RigoRous online CouRses:Ontario Reach Ahead

A Crowdmark white paper / December 11, 2013

1

This white paper by Crowdmark:

• identifiestheproblemobstructingmassiveonlinecourses

fromachievingtheirpotential;

• advancesthesolutionthroughanewonlinecollaborative

gradingplatform;

• describesanonlinecoursescenarioincludingrigorous

assessmentwithaneffectivebusinessmodel;

• outlineshowgovernmentscanbooststudentachievement

whilesavingmillionsofdollars.

Thescenariodescribedhereisjustoneofmanywaysthatonline

collaborativegradingcanimprovetheglobaleducationsystem.

RigoRous online CouRses:Ontario Reach Ahead

2

ReachAheadopportunitiesinOntariopublic

secondaryschoolshavemanybenefits.Stu-

dentswhoearnauniversitycreditduringhigh

schoolaremore likely to finish university, and

can complete their degree sooner, reducing the

demand on the higher education system and

reducing student indebtedness.However,only

35%ofOntariosecondaryschoolsofferReach

Aheadopportunities,denyingstudentsinthe

other65%ofschoolsthebenefitsofearning

universitycredit.ThisviolatestheProvince’s

EquityandInclusiveEducationStrategy. Inthis

whitepaper,Crowdmarkpresentsanonline

coursescenariowithrigorousassessmentthat

enablestheMinistryofEducationtoprovide

ReachAheadopportunitiestoall secondary

schoolsinOntario.

Weuseasanexampleafirst-yearcalculus

class.CrowdmarkwillworkwithanOntariouni-

versityandsecondaryschoolstakeholdersto

createarichonlinecalculuscoursethatcovers

thematerialfromboththeOntarioGrade12

MCV4UCalculusexpectationsandafirst-year

universitycalculuscourse.Thecoursewillbe

broadcasttoeveryOntariosecondaryschool

twiceaweek,whileontheremainingthree

summary: BenefitstotheProvinceofOntario

Only 35% of Ontario secondary schools offer Reach Ahead

opportunities. In this white paper, Crowdmark presents an online course scenario with rigorous assessment that enables the

Ministry of Education to provide Reach Ahead opportunities to all

secondary schools in Ontario.

days,teachersintheparticipatingsecondary

schoolsworkwiththeirstudentsonhomework

andprovideadditionalacademicsupport. Stu-

dentsareassessedusingexamsdistributed

electronicallytoeachschool,printedand

writtenlocally,thenuploadedtoCrowdmarkfor

anonymization,randomization,anddistribution

totheparticipatingteachers,whogradethepa-

personlineusingCrowdmark,inhalftheusual

time.Attheendoftheyear,thetopstudents

(basedontheirassessments)areinvitedtoap-

plyforuniversitycreditatasignificantlyreduced

fee.Theuniversityhasaccesstothestudents’

portfoliosstoredinCrowdmarkandisableto

rapidlydetermineeligibilityforuniversitycredit.

3

Crowdmarkhasdevelopedaweb-basedplat-

formtosupporttherigorousassessmentofstu-

dentsrequiredforthesuccessofthisscenario.

Itcontainstheseessentialelements:

• Distribution Infrastructure: systems to

distributeexamstoandcollectexamsfrom

theparticipatingsecondaryschools;

• Team Dashboard:toolsforassemblingand

managingthemarkingbyteachersfrom

theparticipatingsecondaryschools;

• Crowdmarking Workflow:astreamlined

webprocessformarkingteamstocollabo-

rativelyevaluatestudentwork;

• Data Archive:anorganizedrepositoryfor

storingstudentworkandtheassociated

assessmentsofthatwork.

Thisisjustoneexampleofonlinecollaborative

grading.Onceitsvaluehasbeendemonstrat-

ed,onlineReachAheadopportunitiescouldbe

developedandmadeavailableinthesciences

andmanyotheracademicdisciplines,allowing

Ontariosecondaryschoolstudentstoenteruni-

versitywithuptoafullyearofvalidateduniver-

sitycredit,andgraduatewithafour-yeardegree

inthreeyears.Thisscenarioillustrateshow

theGovernmentofOntariocanbooststudent

achievementwhilereducingtimetodegreeand

savingtheProvincemillionsofdollars.

4

ReachAheadopportunitiesinOntariopub-

licsecondaryschoolshavemanybenefitsto

students.Studentswhoearnauniversitycredit

duringhighschoolhaveexpectationsandstudy

habitsthataremorealignedwithuniversity

requirements.Theyaremorelikelytofinishuni-

versityandcancompletetheirdegreesooner,

reducingthedemandonthehighereducation

systemandreducingstudentindebtedness.

Inthiswhitepaper,wedemonstratethatReach

Aheadopportunitiesprovide significant benefits

to the universityaswell.

Crowdmark presents a scenario with an effec-

tivebusinessmodelthatcreates a new revenue

streamforuniversities,andhasthepotential

to reduce the time to degree.ImagineaReach

Aheadopportunityofferedtoallsecondary

schoolsinOntario:forexample,afirst-yearcal-

culusclass.Crowdmarkwillworkwithuniversity

andsecondaryschoolstakeholderstocreatea

richonlinecalculuscoursethatcoversthema-

terialfromboththeOntarioGrade12MCV4U

Calculusexpectationsaswellasafirst-year

universitycalculuscourse.Thecoursewillbe

broadcasttoOntariosecondaryschoolstwice

aweek,whileontheremainingthreedays,

summary: BenefitstotheUniversity

Students who earn a university credit during high school

have expectations and study habits that are more aligned with university requirements.

Crowdmark presents a scenario ... that creates a new revenue

stream for universities, and has the potential to reduce the

time to degree.

teachersworkwithstudentsinclassonhome-

workandofferadditionalacademicsupport.

Studentsareassessedusingexamsthatmeet

universitystandardsforachievement.The

examsaredistributedelectronicallytoeach

school,printedandwrittenlocally,thenupload-

ed to Crowdmark for anonymization, random-

ization,anddistributiontoparticipatingteach-

ers,whogradethepapersonline.Attheendof

theyear,thetopstudents(basedontheiras-

sessments)willbeinvitedtoapplyforuniversity

credit,forafee.Theuniversitywillhaveaccess

toanarchivedcollectionofstudentworktouse

asthebasisforawardingcredit.

5

Crowdmarkhasdevelopedaweb-basedplat-

formtosupporttheefficient,rigorousassess-

mentofsecondaryschoolstudentsrequiredfor

thesuccessofthisscenario.Itiscomprisedof

theseessentialelements:

• Distribution Infrastructure: systems to

distributeexamstoandcollectexamsfrom

theparticipatingsecondaryschools;

• Team Dashboard:toolsforassemblingand

managingtheteamsofbothteachersfrom

theparticipatingsecondaryschoolsfor

markingexams,anduniversityassessors

foradjudicatingcreditawards;

• Crowdmarking Workflow:astreamlined

webprocessformarkingteamstocollabo-

rativelyevaluatestudentwork;

• Data Archive:anorganizedrepositoryfor

storingstudentworkandtheassociated

assessmentsofthatwork.

Thisisjustoneexampleofthedeploymentof

onlinecollaborativegrading.Oncefeasibility

has been demonstrated, Reach Ahead oppor-

tunitiescouldbemadeavailableinthesciences

andmanyotheracademicdisciplines,providing

asignificantrevenuestreamtotheuniversity.

6

Theemergenceofmassiveonlinecourse

platformshasexpandedtheabilityofeduca-

torstobroadcastrichmultimediainstruction

tohugeclassesofstudentsallovertheworld.

Participantsinthesecoursesdemonstrate

theirlearningbyansweringmultiplechoice

questionsinsidevideos,fillinginwebforms,

andparticipatingingiantonlineforumswith

otherstudents.

Comparedtowhathappensinasmallseminar,

studentsincurrentmassiveonlinecoursesdo

notreceiverichhumanresponsestotheirwork.

Thefeedbackchannelfromstudenttoinstructor

hasmuchlowerbandwidththanthebroadcast

channelfrominstructortostudent.Thecommu-

nicationsmismatchprobleminmassiveonline

coursesobstructstherigorouscertificationof

learningachievementandaccountsforthefail-

ureofMOOCbusinesseffortstodate.

TheInstructionalCommunications MismatchProblem

Collaborative online grading, ‘crowdmarking’, amplifies human

capacity to respond to and evaluate student work to

arbitrary scale. Crowdmark solves the instructional

bandwidth mismatch problem.

7

scenario: Ontario Reach Ahead

TheProvinceofOntariohas913public

secondaryschools.ByofferingInternational

Baccalaureate and AdvancedPlacement

programs,317oftheseschoolsofferaReach

Aheadopportunityallowinggrade12students

toearnfirst-yearuniversitycreditsthrough

studiesattheirhighschool.Studentsatthe

remaining596schoolsaredisadvantagedand

don’thaveReachAheadopportunities.This

situationisunfairandviolatesOntario’sEquity

andInclusiveEducationStrategy.

Theinequitygrowsmoresignificantwhenyou

considerthatstudentswhoearnauniversity

creditduringhighschoolaremorelikelytofinish

university.Thisresulthasbeenestablished

throughresearchandmakeslogicalsense:stu-

dentswhoperformuniversity-levelworkwhilein

highschoolhavetheirexpectationsandstudy

habitsalignedwithuniversitybeforetheyarrive

there.InOntario,thesituationwasexacerbated

bytheeliminationofGrade13OntarioAcadem-

ic Creditwhichproducedagapbetweenthe

K-12andpostsecondarycurriculathatcontin-

uestowiden.

Ontario’sinvestmentinhighereducationis

morelikelytoachievedesiredoutcomeswith

anexpansionofopportunitiesforstudentsto

earnuniversitycreditswhileinhighschool.

Students at 596 schools are disadvantaged and don’t have Reach Ahead opportunities.

317

596schoolswithReach Ahead

schoolswithoutReach Ahead

Reach Ahead opportunities in ontario secondary schools

8

Forsimplicity,let’sassumethat:

• Ontariohasexactly1,000publicsecond-

aryschools(insteadof931);

• Eachpublicsecondaryschoolhasamulti-

functionprinter/scannerandbasicinternet

connectivity;

• Eachpublicsecondaryschoolhasexactly

onegrade12calculusclass;

• Eachgrade12calculusclassistaughtby

exactlyoneteacher;

• Eachgrade12calculusclasshasexactly

25students;

• Eachclasswritestwomidtermsandone

finalexampersemester;

• Eachexamis10pageslongsoeachstu-

dentproduces60pagesofexamwork.

Proposal: ACalculusCourseto1,000Classrooms

We propose a deployment of an online calculus course that equalizes the Reach Ahead opportunity across the entire Province of ontario.

ThroughpartnershipwithanOntarioschool

board(forexample,theTorontoDistrictSchool

Board)oranOntarioindependentschool(for

example,theUniversityofTorontoSchools)

andanOntariouniversity(forexample,theUni-

versityofToronto),webuildateam-taughton-

linecourseandbroadcastrichmultimediacal-

culusinstructionfromtheuniversityinto1,000

Ontarioclassroomsusingtheweb.Thecourse

content,enrichedwithvideos,simulationsand

synchronousquizzesanddifferentiatedin-

struction,canbedesignedtocovertheOntario

Grade12MCV4UCalculusexpectationsaswell

asafirst-yearuniversitycourselikeMAT135 at

UofT.ThecourseisstreamedliveonTuesdays

andThursdays.OnMondays,Wednesdaysand

Fridays,the1,000teachersfliptheclassroom,

supplementingthecoursematerialwithgroup

problemsolvingsessionsandindividualized

instruction.

9

Proposal: ACalculusCourseto1,000Classrooms

exchange Between ontario High schools and the Host university

HostUniversity

HighSchool

10

ArchivedRigorousAssessmentof25,000StudentsArigorousfirst-yearuniversitycalculuscourse

willoftenhavetwomidtermexamsandafinal

exampersemester.Theseexamsinclude

questionsaskingstudentstoexpresstheirideas

throughwritingandcalculation.Eachofthese

examsisabout10pageslong,sothroughout

theacademicyeareachstudentcreates

approximately60exampagestobeassessed.

Theevaluationofresponsesfromstudents

requiresworkbyskilledgraders.Whengraders

giveformativefeedbacktostudents,themid-

termexamscreateopportunitiesforstudentsto

solidifytheirlearning.Thepioneeringresearch

byBlackandWilliamandastreamofsubse-

quentstudieshaveshownthatformativeas-

sessmentvastlyimproveslearningoutcomes.

Bytakingadvantageofcutandpaste,embed-

dinglinkspointingtolearningresourcesonthe

web,andallowingfortheexpressionofelegant

mathematicsincomments,thecollaborativeon-

linegradingplatformempowersteamsofmark-

erstogiverichformativefeedbacktostudents.

Itisadauntingtasktoimaginereading,evalu-

ating,scoringandannotating1,500,000exam

pages(60pages×25,000students).Thisisthe

equivalentofarowofbanker’sboxesaslongas

afootballfieldwitheachboxcontaining5,000

pages.Butwecanaccomplishthistaskby

leveragingtheteamof1,000remoteteachers

involvedusingonlinecollaborativegrading.

When graders give formative feedback to students, the

midterm exams create opportunities for students to

solidify their learning.

11

AssessmentDistributionInfrastructure

UsingCrowdmark,wecanrunandevaluatean

examinthereachaheadcalculusclass.With

LaTeX,MicrosoftWord,AdobeCreativeSuite

orwhatevertoolstheywishtouse,thelead

instructorsinthecourseoverseethecreationof

aPDFfilecontainingtheexamquestionsand

spacesfortheanswers.UsingtheCrowdmark

distributionsystem,25copiesofthisblank

ten-pageexamcanbesentelectronicallyin

theformofa250pagePDFfiletoeachofthe

1,000teachers.Eachpagedistributedthisway

byCrowdmarkhasauniquelyaddressable

graphicalcode.Theteachersprintandstaple

theexamsintobookletslocallyattheirschool.

Theexamsarethenwrittenbystudentsand

collectedbytheteachers.Staffattheschools

(orwithhelpfromoutsourcepartners)scan

the250pagesintodigitalimagesusingthe

school’sphotocopier/scanner.Onstandard

photocopiers,ascanjobof250pageswilltake

approximately20minutes.Thedigitalimages

fromeachschoolareuploadedintoCrowdmark

androutedintoadatabaseformarkingusing

thegraphicalcodesappearingoneachpage.

Rigorous online Course Assessment Distribution

Universityprofessorsendsdigitalassignmentstohighschools

Highschoolteachersprintassignmentsanddistributepaperstostudents

Studentscompleteassignmentsandhand them in to teachers

P

s s s s s s sss

T T T

s

T

P

HighschoolteachersscananduploadassignmentstoCrowdmark.

T

12

Crowdmarking

Eachofthe1,000teachersareinvitedtocollab-

orativelygradetheexam.Thistaskisorganized

usingtheteammanagementdashboard.Each

teachergradesthesamenumberofexamsas

s/hehasstudents,sothereisnoincreasein

workload.Butinsteadofmarkingtheexams

writtenbytheirownstudents,teachersgrade

arandomsampleofexamsfromacrossOn-

tario.Comparedtothetraditionalred-pen-on-

papermethod,thecrowdmarkingworkflowis

streamlined.Inexperimentstodate,Crowdmark

hascutgradingtimeby50%ormore.Graders

can“write”onthestudentexamimagesusinga

mouseorstylus.Graderscanleaverichforma-

tiveassessmentfeedbackcommentswithweb

linksandmathematicalexpressionsparsedby

TeXandMathjax.

Afterthegradingoftheexamisfinished,

theleadinstructorselectronicallyreturnthe

scannedimageswiththeassociatedscores,

commentsandannotationstoeachstudent.

Datafortheirownstudentsisalsomadeavail-

abletoindividualteachers.

Thisstreamlinedprocessallowstheteamto

reviewscoredisputes.Communicationschan-

nelsbetweenstudents,teachers,andparents

(ifappropriate)promoteinformationexchanges

anchoredontheassessmentandbuildnew

opportunitiesforlearningandengagement.

13

ArchivedPortfolio

Theexamimagesandassociatedfeedback

fromthemarkingteamarestoredinthecloud.

Theformativeassessmentfromallpreviousex-

amsremaininstantlyaccessibletostudentsas

theystudy.Theteachershaveaccesstoeach

student’sportfolioofwork.Thefeedbackgiven

bythemarkingteamsisvisibletothecentralin-

structionalteam.Byusingthecommunications

channelsandreferencingtheportfolioofstu-

dentworkandassessment,exchangesamong

thecentralinstructionalteamandamongthe

1,000remoteteacherscanimprovethequality

ofinstructionandpromotebetterGrade12to

universitycurriculumalignment.

Theportfoliobelongingtoeachstudentin-

cludesscannedimagesofeverypageoftheir

workinthecourseandthescores,comments,

andannotationsgivenbygradersoneach

page.Thisrecordenablestheuniversityto

makeadetailed,evidence-basedevaluationof

studentperformanceaspartoftheprocessfor

decidingwhethertoissuecredit.Theuniversity

couldchoosetohidethescoresof1,000teach-

ersandcarryoutacompleteinternalevaluation

ofthestudent’swork.Attheotherextreme,

theuniversitycouldchoosetocompletelytrust

theevaluationskillsofthe1,000teachers.By

buildingsystemstoevaluateandtraincrowd-

markers,Crowdmarkempowerstheuniversity

totailortheirreviewofthestudentportfoliopro-

ducingmorerigorousdecisionsaboutissuing

credit.Theportfolioofstudentworkalsocon-

tributestoamoreequitable,evidence-based

decisionprocessfordeterminingwhethercredit

canbetransferedtoanotheruniversity.

Using the communications channels and referencing the portfolio of student work and assessment ... can improve

the quality of instruction and promote better Grade 12 to

university curriculum alignment.

14

BusinessModelforReachAheadCourses

As a point of reference, and based on FFTE and

OntarioBasicIncomeUnits,traditionalyear-long

creditsattheUniversityofTorontocostabout

$2,800.Wedescribeabusinessmodelinwhich

Grade12studentsinthemassiveonlineReach

Aheadcalculuscourseearncreditsfor$100,

amassivedecreaseof96.5%ofthetraditional

cost.Studentswhoperformwell,forexamplein

thetop20%oftheclass,asmeasuredbytheir

scoresontheexamsaregiventheoptionto

submittheportfoliooftheirworktotheuniversi-

tyforconsiderationforcredit.Thecosttomake

therequestis$100andthe$100iskeptbythe

universityifitdeterminesthatacreditshould

beissued.Ifthecreditisnotissued,the$100

paymentcanbereturnedtothestudent.Ifthe

top-performing20%oftheclassexercisesthis

option,theuniversityreceives$100from5,000

students,generatingrevenueof$500,000from

onecalculuscourse.

Wecanestimatethecostsofrunningthe

course(seethefollowingpage).Notethatthe

salaryofthe1,000teachers,thesalaryofthe

centralinstructionalteam,thepastinvestment

inhardwareandsoftwareinfrastructure,the

pastinvestmentinphysicalinfrastructurelike

buildingsandclassroomsareleveragedinthis

scenarioanddonotrepresentnewcosts.The

mainadjustmentistochangefromanessential-

lyisolatedsetof1,000highschoolclassrooms

generallyfollowingacurriculumdefinedbythe

Provincetoanewcollaborativeworkflowof

1,000classroomsworkingtogetherinpartner-

shipwithauniversityandtheProvince.

Students earn credits for $100, a massive decrease of 96.5% of the

traditional cost.

15

Cost Estimates• Theonlinecoursecontentneedstobedesignedandprepared.

• Eachstudentwrites6examsperyear,producing60pagesofworktobeevaluated.

• 25,000students×60pages=1,500,000totalpages

• Thosepagesneedtobeprintedandscanned(wehaveaquantifiedsolution).

• Thepageswrittenbystudentsinthetop20%needsubsequentreviewbyuniversitystaff.

Quantifications

1. Coursepreparationcost:$100,000(one-time-only)*

2. Papercost:$12,900†

• Acaseofpapercontaining10reamshas5,000pages.

• Eachboxcost$43atOfficedepot.300boxesarerequired.

3. Paperprintingcost:1,500,000×$0.03=$45,000†

4. Outsourcescanning:1,500,000×$0.08=$120,000†

5. Crowdmarkcosts:$5/student×25,000students=$125,000

6. ReviewofTop20%:5,000×60=300,000pages=$80,000‡

• ExperimentsatUofTindicatecrowdmarkerscanreviewcalculusexampagesat150pages

perperson-hour.

• 300,000pagesreviewedat150pagesperhourwilltake2,000hours.

• UofTgraduatestudentTAscost$40/hour.

Total Cost, Year 1 (estimate):$482,900(Subsequentyearswillcostsignificantlyless.)

Representstheonetimeonlycostsofproducinganonlinecourseincludingvideoproductionandtechnicalsupport.Afterthecontentassetsareproduced,laterversionsofthecoursewillbelessexpensivetoproduce.

Canpotentiallybeviewedaspartofthenormalworkflowofthe1,000schoolsandthereforemaynotbeactualcosts.

Estimatedundertheassumptionthateverypagewrittenbystudentsinthetop20%receivesadetailedreviewbyUofTgraduatestudentteachingassistants.Thislevelofreviewmaynotbenecessaryandwillbecomeunnecessaryasthetrainingandevaluationsystemsofthecrowdmarkersdevelopinsophistication.

*

†

‡

16

CascadingBenefitsTherigorousonlinecoursestrategyoutlined

aboveisnotrestrictedtomathematicscourses.

Thesameapproach—leveragingonlinecollab-

orativegradingtoscalerigorousassessment—

canbeusedinothersubjects.Imagineifthis

strategyweresuccessfullyimplementedacross

allmajordisciplines.Wecanenvisionafuturein

whichamajorityofperseveringfirst-yearstu-

dentsarriveatuniversitywithayearofcourse-

workalreadycompleted.Thisshortensthetime

todegreeandopensupcapacityinthehigher

educationsystem.

We can envision a future in which a majority of persevering

first-year students arrive at university with a year of

coursework already completed.

Universitiescanbegintodevotemoreoftheir

limitedresourcestowardadvancedcoursesand

researchefforts.Theportfolioofstudentwork

andalltheassociatedfeedbackfrommarkers

allowsentitieslikeONCAT and ECTStofacilitate

credittransferdecision-makingbasedonmore

thoroughevidenceofstudentachievement.

ConclusionWehavedemonstratedthatOntariocancost-

effectivelyprovidereachaheadcreditsto

studentsProvince-wide.Theriskisminimal,as

theapproachleveragesexistinginvestmentsin

educationalinfrastructureandusesaproven

methodforscalingtheassessmentofstudent

learning.ItprovidestheopportunityforOntario

toimprovestudentachievementandstudent

successacrossourentireeducationalsystem.

17

AboutCrowdmarkCrowdmarkisaneducationtechnologycom-

panybasedinToronto.Crowdmark’sultra-scal-

ableassessmentplatformandbadgedlabor

marketenablesevaluatingandcertifyinglearn-

ingonamassivescale.Theglobaleducation

explosionlacksscalablemethodstocertify

achievement.Evaluatingstudentworkcurrently

involvespagingthroughpilesofpaperorusing

ineffectivemultiple-choiceexams.Crowdmark

streamlinesdocumentassessmentandmakes

iteasyforinstructorstorapidlymarktheworkof

theirownstudents,ortoassembleandmanage

teamsofqualifiedgradersforlargeclasseswith

hundredsorthousandsofstudents.

Contact Crowdmark:

JamesColliander

+1(416)8874833

james@crowdmark.com