What are the Ingredients of an Awesome Online Learning Experience?

What are the Ingredients of an Awesome Online Learning Experience?Ruben Kogel, Head of Learning Science

190+50% Domestic & 50% International20THOUSANDpublished courses10THOUSANDpublished instructors4MILLIONstudentsSource: Udemycountries

Lifelong Learning: just in case and just in time learningUdemy is the leading marketplace for online learning and teaching, offering people everywhere the opportunity to advance their careers, change professions, develop their passions, or simply learn something new. Udemy has more than 10,000 instructors, and more than 4 million students taking courses in everything from programming to yoga to photographyand much, much more.

awesome learning experiencerepurchase rateWhy care about delivering an awesome learning experience?because we are an education company and we care about education!because we believe students will come back and spend more (and tell friends)!find a course[discover]spend morecome back [re-visit]enroll [conversion]awesome learning experiencelifetime value?YES!care about our mission statement + LTV and referrals (= top-down revenue)

all things equal, does an awesome learning experience translate into higher revenue per student?

both awesome and $$$ difficult to measure

to control for the effect of past experience with other Udemy courses.. we looked at 1st time buyersto control for the fact that different categories have different LTVs.. we looked at 1 category with high traffic, where in rough approximation all courses are substitutesto control for time lag effects.. we looked at the 3-mo LTVCan we measure quality of learning experience reliably?course qualitywhat we measurequality of learning experiencestudent satisfactionplatforminstructoretc.student expectationpriceetc.selection biasdifferent scalesdifferent motivesskewednoisyinaccurateSolutionspam filterselect students with a similar profileaverage out over many responsesuse regression model to see the trendIdeally..In reality..Ideally, wed be able to measure course quality directly from student feedback but in reality..

nebulous concept:many drivers to student satisfactionwhat we measure is itself differentstudent heterogeneity (scale heterogeneity, motives)

spam filter - remove untrusted feedbackselect students with a similar profile - control for motive / prior experienceaverage out over many responses (set the threshold based on standard error)use regression model to see the trend

What are the ingredients to a great learning experience?First lets make sure we are measuring the things that matter..

{ known knowns, known unknowns, unknown unknowns } quality- has outline Y/N- audio quality- etc.- quality of explanations- content depth- etc.list of attributes from:- analysis of reviews- open-ended survey- closed-ended survey- learning scienceStep 1We inferred buckets of things that were important to students through reading reviews for 60 representative coursesWe found very similar buckets of things through independent user testing with 7 ODesk freelance studentsStep 2We distilled important attributes from the reviews and user testing, and asked students with ~300 responses each for 2 separate surveysWe re-reviewed and measured some of these important attributes against student happiness

Seeing the forest for the treesInitial attempts using linear regression failed.... effects are non-linear (e.g. audio quality).. the model of learning experience has lots of interactions (e.g. audio + tone of voice).. endogeneity due to hidden variables (e.g. price)video qualityaverage ratingaudio qualitystructured teachingtone of voicesupp materialInstead, use non-linear model (Random Forest) to rank factors in order of explanatory power + marginal regression to measure the size and direction of the effect

Some factors had wrong effects.. (e.g. supplementary material cause worse experience??).. remember that correlation is not causation!

non-linear effects: audio qualitylearning experience = our perception lots of interactionsinteractions: if audio is awful, tone of voice is uselessendogeneity: supp material is worse b/c it really speaks for a hidden variable instructor is lazy and posts lots of material instead of videosTop Predictors of Course Quality RatingsDeliver the lesson clearly, confidently, and in an engaging mannerReinforce learning with quizzes, exercises, and supplementary materialRespond to students promptlyProduce lectures with high audio and video qualityStructure and organize the content well

Cannot be reliably measured but important:Set the right expectations with realistic course objectivesProvides real world examples throughout the courseDive into the specifics and provide detailed explanationsinstructorinstructorcontentcontentcontentcontentcontentproductionquant analysis can only include attributes that are reliably measuredranking is done by importance in the Random Forest model = change in MRSE when removing this one factor

attributes can be segmented along different axes:- buckets- stage of course creation- utility / delight- measured / inferred (from student feedback)- measured before / after

goal: measure everything before to make the model perfectly predictivethe different layers of a great online course

Absolutely necessaryclear, confident, and engaginghigh audio and video qualityVery important to havedetailed explanationsset the right expectationsreinforced learningFrom good to greatwell structured contentuse real world examplesrespond to students promptlyNow blending the quantitative and qualitative analyses..Now what?Communicate learnings to the instructor communityremember initial goal to improve quality!

Iterate on the model until we can pin down the exact ingredientslots of room for improvement and making it more useful

Think outside the boxlots of science outside, lots of new factors to incorporateCommunicate learnings - this is the primary lever to achieve initial goal of improving quality

Iterate - it is a predictive model and we can make it more accurate (and, therefore, useful)

Think outside the box - this is just the basics, lots of science outside of Udemy, lots of new factors to include

TakeawaysWe are pushing the boundaries of Learning Science

Take a careful approachunderstand what is being measureduse qualitative analysis to inform the modelcorrelation is not causation

Data scientists job does not stop at the analysiscommunicate learnings; focus on actionablessetup processes to maintain and improve the predictive power of the modelincorporate learning from outsideOur findingsfind out what matters, in which orderdifference utility / delight

Take a careful approach = methodologyask the right questions: what do we need to predict, whyunderstand what is being measured (e.g. student satisfaction vs. learning)use qualitative analysis to inform the model and collect the right dataimportant to collect data right (garbage in, garbage out)THANK YOU!

What are the necessary ingredients for a good course?From the qualitative analysis..

Set the right expectations with realistic course objectivesProduces lectures with high quality audio and video Deliver the lesson clearly and confidently in an engaging mannerStructure and organize the content wellProvides real world examples throughout the courseDive into the specifics and provide detailed explanationsUtility factorsWhat are the ingredients to go from good from great?From the qualitative analysis..

Dive into the specifics and provide detailed explanationsSet the right expectations with realistic course objectivesDeliver the lesson clearly and confidently in an engaging mannerStructure and organize the content wellProvides real world examples throughout the courseReinforces learning through quizzes, exercises and supplementary materialDelight factors