Ratings and reviews are a universally understood signal to help students find the course that’s right for them, help instructors get feedback from students about their courses, and help Udemy match students with courses that are most relevant to them.
As an instructor, you’re working hard to create amazing courses, and star ratings as well as written reviews do affect the success of your course. While the current review system does provide a helpful signal to students today, there is work to be done to make it fair and consistent for instructors, as well as get instructors more relevant and valuable feedback from students.
Over the last year, we’ve been working to get as many reviews for your courses as possible. And this is still important to make the signal strong enough to be useful to students, to Udemy, and to you. However, we need to get smarter about how we collect and use all this information so that we’re taking the context of these ratings into account and representing courses more fairly in the marketplace.
What are we doing?
Over the next two months, you’re going to see the following upgrades being made to the review system.
- Treating ratings differently based on context: We know it’s frustrating when a rating left after 12 minutes is treated the same as a rating left by a student who completed the course. It’s also frustrating when you have a course with a few ratings, and a single low rating significantly affects the average rating. To address these issues, we’ll upgrade how we calculate average rating in January, 2018. We’re going to stop treating all ratings the same, and start weighting them differently based on the level of usefulness of the rating to students choosing courses. Some examples of factors we’ll take into account are the timing of when a student leaves a rating, how much of the course has been consumed by that student, and the length of the written review. By doing this, we’ll be able to help students make better decisions about courses and make the system more fair to instructors.
- Improving the spam review filter: We’re launching an improved spam filter in December 2017 that will leverage machine learning to better identify spam reviews. The gaming of the review system unfairly impacts instructors and creates inaccurate signals for students. Over the next year, we’ll continue to make improvements to the spam review filter to become smarter about how we identify spam reviews.
- Testing “featured reviews” on course landing pages: We’ll also start testing “featured reviews” in December 2017, where we’ll test displaying one highly rated written review more prominently on the course landing page. This review will be one where the student took the time to explain why she/he had a positive experience with the course. If successful, “featured reviews” can help prospective students get the context they need about a course and encourage them to enroll. Please go here to learn more.
In addition, we’re going to invest in collecting and surfacing more relevant, useful feedback from students in 2018.
- Our goal is to ask students more contextually relevant questions based on where students are in a course, so we can get more valuable feedback. For example, it may be more helpful to ask a student just starting out on a course about the video quality of the course, and ask students who are close to completing the course about whether the content met their expectations.
- We’re exploring different features to collect useful feedback from students. One example of this is the “review tags” feature that we’ve been testing for a short while. When a student starts leaving a review, they can choose tags to describe their course experience. We know it’s frustrating when instructors receive low-star ratings without any other feedback. If successful, “review tags” can help instructors get more actionable feedback from these students. While this is one example, we’re exploring a number of other ways to collect more relevant, actionable feedback for instructors.
Of course, we’ll continue to follow our policies on removing reviews that are flagged when they are about Udemy product issues and not about the course, otherwise unrelated to the course, fraudulent, contain offensive or inappropriate language, etc.
We’re committed to building a review system that is helpful and fair to instructors. As we invest more deeply in the review system, the Udemy product team would love to hear more from you and have an opportunity to learn more about your needs. We’ll be hosting a Product Ask me Anything session about reviews in the Udemy Instructor Club in January, 2018. Stay tuned for more details! And in the meantime, if you have any feedback on specific reviews, please continue to flag them to us by using the “Report Abuse” button or writing to firstname.lastname@example.org.