Learnersourcing of ComplexAssessments

Piotr MitrosedX141 Portland StCambridge, MA 02139pmitros@edx.org

Permission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bearthis notice and the full citation on the first page. Copyright is held by theowner/author(s). Additionally licensed under a Creative CommonsAttribution-ShareAlike 4.0 International Public License. For all other uses,contact the Owner/Author. L@S 2015, March 14–18, 2015, Vancouver, BC,Canada.ACM 978-1-4503-3411-2/15/03.http://dx.doi.org/10.1145/2724660.2728683

AbstractWe present results from a pilot study where studentssuccessfully created complex assessments for a MOOC inintroductory electronics – an area with a very largeexpert-novice gap. Previous work in learnersourcing foundthat learners can productively contribute through simpletasks. However, many course resources require a highlevel of expertise to create, and prior work fell short ontasks with a large expert-novice gap, such as textbookcreation or concept tagging. Since these constitute asubstantial portion of course creation costs, addressingthis issue is prerequisite to substantially shifting MOOCeconomics through learnersourcing. This represents one ofthe first successes in learnersourcing with a largeexpert-novice gap. In the pilot, we reached out to 206students (out of thousands who met eligibility criteria)who contributed 14 complex high-quality design problems.This results suggests a full cohort could contributehundreds of problems. We achieved this through afour-pronged approach: (1) pre-selecting top learners (2)community feedback process (3) student mini-course inpedagogy (4) instructor review and involvement.

ACM Classification KeywordsK.3.1 [Computer Uses in Education]: CollaborativeLearning.

IntroductionThere are 4,706 degree-granting institutions in the UnitedStates. A specialized introductory course such as circuitsand electronics is taken by 10k-100k students everyyear [8]. This gives 3-4 orders of magnitude inefficiencyon tasks such as creation of lectures and assessments, and4-5 orders of magnitude on per-student tasks such asgrading. Open educational resources (OER) and at-scalelearning organizations (such as MOOC providers) attemptto improve the quality of education by leveraging thoseinefficiencies to provide higher-quality resources at lowercost. The OER approach has met limited adoption due tolack of coherence – finding and adapting relevant andquality OERs is often more time-intensive than creatingnew resources [4].

In contrast, institutional Massive Open On-line Courses(xMOOCs) use a centralized approach where a singlecourse team creates a complete, coherent course.xMOOCs are typically taught by top instructors, andmany employ research-based pedagogies such as activelearning, constructive learning, and mastery-learning [5].They are further enhanced with data-driven techniques.Initial evidence suggests that well-designed xMOOCs canlead to high levels of student learning and satisfaction inboth on-line and blended settings [3][5][1]. In contrast toOERs, xMOOCs are traditionally not open, and do notsubstantially leverage the creative input of externalcontributors. xMOOCs have been limited by to the highcost of creating evidence-based resources, combined witha lack of well-developed economic models.

Mitros [6] introduced a hybrid model of course creationwhere a central staff provides a common course skeleton,and a community of learners, educators, and researchersbuilds and improves on that skeleton. This approach has

been shown successful across a range of areas such asstudent remediation, translation, subtitling, and peergrading of simple assessments, but as predicted inexpert-novice literature [7], had lackluster results for taskssuch as concept tagging or text creation. Cormier [2]piloted how this model may work in courses with a largecommunity of instructors, such as an introductory physicscourse. More esoteric courses, as found at a universitylevel, often do not have such communities.

In this pilot, we explore ways to source content with alarge expert-novice gap. To do this, we first pre-screenstudents for ones who showed a high level of mastery inthe MOOC. Second, we train those students in bestpractices for creating high-quality assessments. Third, wehave a process by which those resources are improvedbased on both community and instructor feedback.

In order to maximize the expert-novice gap, we choosecontribution of assessments to 6.002x, the MITx course incircuits and electronics. 6.002x has a small number ofcomplex assessments involving design and analysis. Togive an idea of complexity of those problems, most weeklygraded assignments consist of three problems and onelaboratory. Students self-report spending an average of 11hours per week on the course (including ungraded work).

The creation of assessments is a complex process:

• Students are taught basic concepts, such as linearityand nodal analysis. A large part of the goal of thecourse is to begin to develop design intuition forhow circuits work and physical intuition for realisticcomponent values. Course creators must be familiarwith a range of circuits in order to be able to findones which illustrate concepts taught in the course,

and to be able to create simplified, realistic, variantsof those circuits. Finding or creating interestingcircuits is the bulk of the assessment creation time.

• Students can continue to work on problems untilthey reach a correct answer, and receive immediatefeedback if they do not. Problems must lendthemselves to automated grading, mastery learning,and infinite attempts (e.g. numeric and equations,rather than multiple choice or freeform pictures).

• Quality problem creation requires extensivepedagogical content knowledge.

• Problems should be fun and interesting.

An example problem from 6.002x is shown in Fig. 1. Thecomplexity of this process is such that the assessmentcreation for the first run of 6.002x required 1.5 semestersof full-time effort from MIT instructors. Creating newexam problems is the major limiting factor on how oftenthe course can be re-offered.

Figure 1: A representativemid-semester problem from theedX offering of 6.002x. Studentsare asked to analyze an amplifierbased on a device they have notseen in the course.

Process and ResultsWe e-mailed two groups of students:

• 106 students from the fall 2014 run of 6.002x(contacted one week after course completion) whoachieved >97.5% on the final exam.

• 100 students from the original spring 2012 run of6.002x who had achieved an ’A’ grade in the course,and had volunteered to participate in such projectsin the end-of-course survey (out of 2833 eligible).

Students were invited to participate in a short courseabout how to create good autograded assessments inintroductory electronics, including content about

pedagogy, capabilities of the platform, as well assubject-specific advice, such as finding good analogcircuits to use as the basis of problems. Students weregiven a 10 day window within which to submit first drafts.Registered students were sent a reminder three days beforethis deadline. Assessments had three possible formats:

• Simple practice problems

• Mid-term exam problems (complex multiconcept)

• Final exam problems (complex multiconcept)

Students could submit problems either through the edXpeer assessment tool, or the discussion forums. Studentsalmost exclusively favored the discussion forum. Coursestaff and peers were active on forums, reviewing problemsand providing feedback. Problems, on average, received3-4 comments. This process was critical – few of thesubmitted problems were directly usable in the firstiteration. Students and instructors formed a community inwhich they reviewed problems, learned from each other,and through the process, many of the problems improveddramatically. Several students indicated a very high levelof value from participation in those forums, and found ithelped for learning about both electronics and pedagogy.

At the time of the deadline, 57 students had registered forthe course, and submitted a total of 17 problems, ofwhich 2 were excellent as evaluated by the course staff1,12 were usable or excellent with relatively small tweaks orchanges, and 3 were incomplete or unusable.Contributions and community continued beyond thedeadline. The level of participation is typical of previousexperiments in sourcing from MOOC students[6], but at amuch higher task complexity. If we maintained similar

1Neither independent nor blind evaluation. The author of thispaper is a co-creator of the course.

levels of participation as we scaled out to moreparticipants, we’d expect 28±6% to enroll in the course,and to receive problems corresponding to 30±12% of theregistrants (25±11% usable problems). For all eligiblestudents, the number of contributed problems would bemany times greater than required for the overall course.

Figure 2: A representativeproblem contributed by astudent, followed by thediscussion around that problem.The problem walks through thedesign of an H-bridge driver. Theauthor rewrote the first draft as aresult of feedback.

The problems ranged from simple single-concept practiceproblems to complex multiconcept problems based onreal-world circuits, such as edge detector, a motor driver,an envelope detector, and a range of amplifiers, but mosttended towards the more complex end. An exampleproblem is shown in Fig. 2. In most cases, the very highquality was the result of the community process. Most ofthe initial submissions had good ideas, but suffered fromany of a range of issues such as ambiguity, inappropriatelevel of difficulty, technical errors, and pedagogicalshort-comings. The community process allowedcontributors to discover and resolve those issues.

ConclusionWe successfully demonstrated the ability to sourcecomplex assessments from MOOC learners. Since this wasa pilot study, we used a small sample of 206 eligiblelearners. The results suggest that using the full cohort ofeligible contributors would have allowed us to substantiallyaccelerate the creation of complex assessments – the mostresource-intensive portion of the course, assuming we wereable to maintain similar contribution levels. A rough guess(with no data) is that the time spent per assessment isroughly an order of magnitude lower than fully manualcreation, but also not zero, due to the need for review andediting. Based on the pilot results, learnersourcing appearsto be a viable strategy for improving MOOC costs ofcreation, including those with a large expert-novice gap.

AcknowledgmentsWe thank all of the 6.002x students for their activeengagement in both the course, and in community effortsto improve the course.

References[1] Cima, M. A mastery-based learning and assessment

model applied to 3.091r (introduction to solid-statechemistry). MIT Interim Report (2014).

[2] Cormier, D., Mitros, P., and Pritchard, D. Teachingphysics at scale. Physics Education ResearchConference (2014).

[3] Ghadiri, K., Qayoumi, M. H., Junn, E., Hsu, P., andSujitparapitaya, S. The transformative potential ofblended learning using MIT edX’s 6.002x onlineMOOC content combined with student team-basedlearning in class. environment 8 (2013), 14.

[4] Harley, D., Head, A., Nasatir, D., Paletz, S., Henke,J., Miller, I., Sheng, X., and Guo, J. Understandingthe use of digital collections in undergraduatehumanities and social science education: Statusreport, preliminary analyses, and next steps. CSHEDigital Resource Study, Center for Studies in HigherEducation, University of California at Berkeley (2004).

[5] Mitros, P., Affidi, K., Sussman, G., Terman, C.,White, J., Fischer, L., and Agarwal, A. Teachingelectronic circuits online: Lessons from MITx’s 6.002xon edX. In ISCAS, IEEE (2013), 2763–2766.

[6] Mitros, P., and Sun, F. Creating educational resourcesat scale. IEEE Conf. on Adv. Learning Technologies(2014).

[7] National Research Council. How People Learn.National Academy Press, 2000, 31–50.

[8] Snyder, T., and Dillow, S. Digest of EducationStatistics 2012. NCES 2012-001. NCES, IES, U.S.DOE, Washington, DC, 2012.