Announcing: Improvements to the Udemy Review System

At Udemy, we’re focused on helping students around the world learn from the best instructors in over 15 categories (and growing)!  With hundreds of courses to choose from, peer reviews help students decide which course is right for them.  

Our current 5 star review system was built to be easily understood by and fully transparent to both students and instructors. As the number of courses on Udemy has grown, so has the need to foster a fair marketplace where ratings and reviews accurately reflect the quality of a course. However,the current review system has resulted in:

  1. A low number of reviews per course
  2. A high number of inauthentic reviews  
  3. Limited differentiation between ratings of similar courses

For these reasons, students do not always consider reviews to be accurate and trustworthy, so we embarked on a series of user interviews, experiments, and analyses to ensure a larger volume of trustworthy reviews.  Today we’re excited to announce improvements to the course review system. These changes will be effective beginning in mid-February.   

In-Course Prompts

We understand that getting student reviews is a significant challenge for instructors.  In the improved system, students will be prompted to leave and/or update their rating and review at 3 different points while they are taking the course – the beginning, middle, and end.

This is important for a few reasons. First, prompting students within the course increases the likelihood of leaving reviews.  In our experiment, the in-course prompt response rate was nearly 7x the response rate of the current review system! 

4740f9c3-23e4-41c0-bb41-40a3ccfa5d36

Additionally, capturing student sentiment at multiple points within the course helps students and the instructor understand what parts of the course are great and which parts could be improved.  

More Robust Spam Review Filter

Because reviews impact instructors’ marketing efforts, some instructors have taken steps to solicit fake reviews that are not representative of the course material.  Inauthentic reviews jeopardize the trust built between the student community, instructors, and Udemy.  To address this, our data science team developed a new framework to better detect fraudulent reviews as early as possible.  Our intention is that this early detection system will minimize the periodic large-scale removal of spam reviews that we’ve engaged in over the past several months.  Instead, these inauthentic reviews either will not appear or will be quickly removed from the course landing page.

Increased differentiation among courses   

Despite similar categories, topics, and content, no two courses are the same in the marketplace. However, with only 5 rating options, students don’t always have the full picture.  That’s why we’re introducing the ability to leave half-stars in reviews.  Including 10 choices for students in the review system(1-5 stars in half-star increments) allows them to leave a more granular rating which results in increased distinction between courses.

13cb4ec0-f2a2-49f0-b8b0-d66e59739f63

 

This new review system will help students trust the reviews for courses by

  • Increasing the volume of reviews
  • Decreasing fraudulent reviews
  • Increasing the differentiation between courses

What this means for you:

To ensure that instructors are completely equipped to manage the increased volume, we’re also introducing a brand new dashboard where instructors can view and publicly respond to all reviews.

The ability to respond to reviews will allow instructors to acknowledge positive remarks, while addressing student concerns. This builds goodwill not only with existing students but also with potential students.

The dashboard also makes it easier for instructors to stay on top of reviews by aggregating reviews across courses into one dashboard. On this dashboard instructors have the ability to:

  • Filter reviews by course
  • Filter reviews by rating
  • Filter reviews that have already been responded to
  • Respond publicly to reviews

As we focus on improving the trustworthiness of reviews, we will no longer collect the student feedback displayed on the Student Satisfaction Analytics dashboard. Instead, our review system will serve as the single metric students, instructors and Udemy use to assess student sentiment of a course. Student satisfaction continues to be a top priority at Udemy, and this data will continue to inform courses’ standing across the site, including our promotions and our search algorithms. Accordingly, we will continue to monitor this data closely to ensure it remains an accurate indicator of student sentiment.

We’re incredibly excited to make it easier to collect more reviews, minimize inauthentic reviews, and help excellent courses stand out!  And in the coming months, we’ll examine ways to make reviews even more beneficial to students and instructors.  Let us know what you think of the improved review system by emailing us at instructorsupport@udemy.com.

Frequently Asked Questions 

What happens to my existing reviews?

Reviews collected through the new and improved system will be added to the course landing page along with your existing reviews.  As of mid-February, the new ratings will be factored into the overall course rating.  The course rating is an average (mean) of all ratings.  This is consistent with how the rating is currently calculated. 

What happens when a student leaves a rating and no review?

Ratings without reviews will not appear on the course landing page, as they are not helpful for students yet remain valuable pieces of feedback.  These ratings will appear on the Instructor Reviews dashboard (visible only to you) and are factored into the overall course rating that appears on the course landing page.  As always, if you would like to reach out to the student who left a rating with no text, you can do so via the Messaging tool.  

Why are you prompting students to leave a review at multiple times during my course?

We’re prompting students at the beginning, middle, and end of the course.  Multiple prompts capture the sentiment of students throughout the course, and that data has value for instructors and potential students.  Many students consume only a limited amount of the course, so we also aim to capture feedback from students who choose not to continue with the course.  The timing of the prompt at the beginning is optimized for volume of responses and consumption to ensure we collect as much data as possible.   As a result, we always recommend ensuring that the first few lectures of your courses are engaging and helpful to students to encourage their continued engagement with your content.  

My course is 1 hour (or less).  Are you still asking my students to leave a review 3 times per course?

For courses that are less than 1 hour, students will only see a review prompt 2 times (at the beginning and upon completion).

Are students required to leave a review? If a student doesn’t want to leave a review, what are their options?

Students are not required to leave a review.  When they are prompted to leave a review, they have the option of clicking “Ask Me Later”.  If a student chooses “Ask Me later”, they will be prompted to leave a review at a point later in the course.  

If I publicly respond to a review, can the student reply to my response?

Public responses to reviews are intended to help address a reviewer’s concerns and positive feedback. This is helpful for the reviewer and for other potential students.  You can only leave one public response per review on the course landing page.  The student is not able to reply to your response.  If you would like to engage with a dialogue with the student, please use the Messaging tool.  

If a student leaves multiple reviews, which one appears on the course landing page?

Only the student’s most recent review appears on the course landing page.  Although, if a student leaves multiple reviews, you can view all the student’s reviews of your course on the Instructor Reviews dashboard.   

With the improved review system, will my rating change?

We anticipate an increased volume of reviews with this new system and, as a result, it’s likely that your courses’ ratings will change.  Students often have a diversity of feedback regarding a course – some positive and some negative.  Since all authentic student feedback is valuable, the new rating reflects these variations in opinions.

How can you tell if a review is fake?

Our new, more robust spam review framework includes a wide array of algorithmic and human components parameters designed to detect and remove fake reviews. Some reviews are clearly inauthentic, and will be automatically removed.  There are parameters that we use to ensure that students who leave reviews are authentic, including the amount of content watched and source of enrollment.  We don’t disclose the exact methodology because doing so would make it easier to circumvent our filters.   However, for reviews that are more nuanced, our team will investigate and take appropriate next steps on the review and, if applicable, the instructor. As always, if you suspect reviews are inauthentic, please contact policy@udemy.com.

Where is my Student Satisfaction Analytics dashboard?

Since we will now be prompting students to leave a review during the course, we are removing the early impression survey that previously generated the student feedback section of the “Student Satisfaction Analytics” dashboard. The Early Impressions Survey will be replaced by the new review prompts at the beginning, middle and end of the course. These student reviews will be fully transparent to instructors and students on the course landing page.   

The Student Satisfaction Analytics page will be renamed to “Engagement” and still contains the overall engagement and lecture engagement metrics previously found on this dashboard.  It will no longer show the student feedback by category. Instead, instructors can get a more detailed picture of student opinion of their courses in the new reviews dashboard.

Is Udemy planning any additional improvements to the review system in the future? What’s next?

Adding multiple in-course prompts, building a more robust spam review filter, and introducing of half stars to the rating scale represent the first phase of planned improvements to the Review System. Over the coming weeks and months, we’ll be experimenting with and gathering feedback on additional features and functionalities that will be helpful to students and instructors.  Stay tuned for more improvements soon!

 

3 thoughts on “Announcing: Improvements to the Udemy Review System”

  1. Leslie Lello says:

    This sounds great! Super-excited about the changes! Thank you.

  2. Roger Humbke says:

    It all sounds very good and very fair for all concerned. I hope it works out well.

  3. len smith says:

    This is EXCELLENT news. Thanks.

Got Something To Say:

Your email address will not be published.

*

sixteen + seventeen =