Investments in the Review System – August 22nd Update

Introducing Course Ratings Guidance

 

Today, we’re excited to share our plans for course ratings guidance on Udemy. Our aim with this update is to help students rate courses from a common understanding, by assigning a description for each rating on the five-point scale.

 

Why are we doing this?

Our primary goal with ratings is to help prospective students decide if a course is a match for their learning goals and needs. But without a common understanding of what each individual rating means, a course’s overall rating can’t effectively summarize the response of existing students.

Today, two students leaving the the same rating may have had a very different experience. Two students that leave different ratings may actually be expressing the same thing.

We’ve also been listening to you, our instructor community. You’ve made it clear that you’d appreciate greater clarity on what each rating means.

 

What will it look like?

How did we come up with this?

We wanted to make sure our guidance reflects what most students are already expressing through their reviews. We conducted an extensive analysis to pinpoint how students felt at each rating. We also studied other comparable marketplaces (Amazon, Yelp, etc.) to see how they describe their rating scales.

 

How will this impact instructors?

We’re committed to having marketplace policies that align with what we’ve learned about the student perspective on ratings. So we’re making a few adjustments around star rating thresholds:

• We’re removing the search filter for free courses that hides courses rated 4.1 and below
• We’re adjusting the ratings thresholds for our badges like Bestseller, Hot, and New
• Udemy for Business (UFB) will update its admission criteria, while continuing to focus on curating the freshest, most relevant content for corporate learners

In the short term, we expect a small, gradual dip (between 0.1 and 0.2) in average ratings left on Udemy as whole. But we see this as a good long-term investment. While there may be some change over the next few months, reliable and uniform ratings are vital for a healthy marketplace.

 

When will this happen?

Course ratings guidance will roll out within the next week. It’s an important step forward. It will improve the reliability and usefulness of star ratings by helping students map their experience to a specific rating. It will also give instructors a more reliable understanding of how their students respond to courses.

 

Continuing improvements

We’ll be closely monitoring the effects of this change as ratings guidance rolls out. We’re also going to continue improving our feedback systems in a few other ways:

• Displaying some of the qualitative course attributes we’ve collected over the last several months to differentiate courses and provide more useful feedback
• Improving the spam filter and weighted rating system to ensure course ratings are aligned with other signs of student satisfaction like consumption, re-enrollment, etc.
• Reducing the amount of time between a student rating a course and when that rating appears on the course landing page, instructor dashboard, etc.

Feedback and ratings systems are the backbone of any marketplace, requiring careful and thoughtful maintenance. We will continue our work to support our mission of improving lives through learning and our promise to help both instructors and students “Be able.”

If you have any questions, we’re happy to help. Reach out to us at [email protected].