A Look at Course Quality Standards from Udemy’s Learning Science Team

At Udemy, our mission is to help anyone learn anything online. To make that happen, we’re always looking to empower our instructors by turning the art of teaching online into a science. So, we took a deep dive into the data pool created by 4 million students learning from thousands of amazing instructors worldwide.

Here’s an overview of how we arrived at and continue to improve our course quality standards.

We Observed:
We asked a control group of students to take high (5 star) and low (1-2 star) rated courses on Udemy in different categories and watched them take these courses. We heard about what thrilled them in a course, and what made them feel frustrated, and then we compared and contrasted the results.

We Inferred:
We read thousands of student reviews to identify course traits that were frequently mentioned. The patterns and frequency of these attributes in good and bad reviews provided a gateway to understanding relative importance to students.

We Asked:
We surveyed a large sample of students to understand what they thought was important in a course, and the order of importance of certain traits. These surveys offered up absolute and relative preferences, which (excitingly) lined up cleanly with feedback from the other qualitative pieces of our research.

We Analyzed:
Based on what we observed, inferred, and heard, we broke down the relevant components of a course into dozens of observable attributes and mapped them across the entire marketplace of 20,000 courses and 12 million enrollments. We then used live marketplace data to distinguish between those attributes that were highly statistically correlated with student engagement, and those which played a marginal role or (in some cases) didn’t contribute at all.

We Concluded:
The most important drivers of student engagement broke into two camps: (1) utility factors that were key for delivering on students’ base expectations, and (2) delight factors that held the ability to drive a student’s learning experience from good to great and turn them into an evangelist of your course.

We Implemented
These new insights have made their way into Udemy’s Course Quality Checklist, our Quality Review Process, and even into our post-publishing tips and tricks. You’ll notice that those factors that drive utility show up as “Required” items in the checklist, whereas those that drive delight show up as “Recommended” items. Of course, we always recommend aiming for a perfect course — why settle for satisfaction when you can deliver delight!

Screen Shot 2014-11-14 at 2.33.26 PM

(click on the image to enlarge it)

Below you can find links to all of the places you can receive feedback and guidance while creating your course: