MOOCs: So Many Learners, So Much Potential

2 1541-1672/13/$31.00 © 2013 IEEE Ieee InTeLLIGenT SySTemSPublished by the IEEE Computer Society

Editor: judy Kay, University of Sydney, [email protected]

MOOCs: So Many Learners, So Much Potential ...

they’re free, removing any fi nancial barrier for even the poorest student. Being online means peo-ple can access them on the Internet. In provid-ing courses, MOOCs represent a major shift in scale beyond open learning objects (see the side-bar, “Learning Objects and Open Educational Resources”). They operate at the level of a whole course (or subject), meaning that they provide a coherent learning sequence, with integrated learn-ing materials and formative assessment, all cre-ated and managed by outstanding teachers from the world’s top institutions. If a course is of high quality, free (open), and readily accessible (online), it follows that massive numbers of students will grab the chance to get a fi rst-rate education for free. This creates scalability challenges. We un-derstand how to engineer websites that gracefully handle huge numbers of users; however, we’re still trying to learn how to handle the scalability issues for the teaching, learning, and assessment mod-els. Researchers acknowledge the importance of social interaction among learners,1 which is one benefi t to MOOCs. A MOOC can call upon its large community of learners to support learning via discussions and to assess work based on peer review.

There is a delightful idealism and altruism in the words of many of those driving the MOOC movement. We all value a quality education, we all acknowledge the huge gap between the edu-cational opportunities of the most privileged and the most disadvantaged learners, and we present MOOCs as a means to help close this gap. We can conjure up images of students from the develop-ing world, the most disadvantaged groups in the fi rst world, and lifelong learners with changing needs, all quenching their thirst for knowledge by

learning at the feet of the intellectual giants of the world’s leading research institutions. This is an ex-ample of Friedman’s fl at world.2 We can cater to this large, unmet need because of the wide avail-ability of inexpensive networked computers.

Beyond the excitement of the learning oppor-tunities of the actual MOOC courses, a different dimension of promise is in MOOCs as open plat-forms, built by a new and energetic open source community. Perhaps this will be a revolution in software for authoring and delivering high-quality learning opportunities.

What Is a MOOC?Despite the name, we really have no clear defi ni-tion of exactly what is and isn’t a MOOC. Con-tributing to the confusion around the term, we’ve started defi ning MOOCs differently. Although in 2012 we saw highly publicized MOOCs, such as Coursera and Udacity, we’ve actually used the term since 2008. In the past, we used it to de-scribe a very different kind of online course run by people like George Siemens. Siemens sug-gested that there are in fact two entirely sepa-rate types of online courses sharing the name “MOOC” and he offered new terms to distin-guish them: edX MOOCs (xMOOCs) and con-nectivist MOOCs (cMOOCs) (www.elearnspace.org/blog/2012/07/25/moocs-are-really-a-platform). The xMOOCs are the new and well-publicized type that moves a traditional university learn-ing paradigm into the online learning space. The cMOOCs are the older type, as developed by Sie-mens. Administrators base these around learning socially, inspiring creativity, and believing that knowledge exists in the collective minds of the stu-dents. Rather than assessable content, teachers provide students with content designed to encour-age discussion and debate. Although these terms have gained some traction, the recent media atten-tion has popularized the term MOOC to refer to xMOOCs.

Massive open online courses, (MOOCs) if

true to their name, possess three defi ning

elements. They’re open, meaning that anyone can

use them to learn. This also logically implies that

Judy Kay, Peter Reimann, Elliot Diebold, and Bob Kummerfeld, University of Sydney

A I A N D E D U C A T I O N

IS-28-03-Education.indd 2 15/07/13 2:44 PM

may/june 2013 www.computer.org/intelligent 3

Exemplar MOOCsTo better understand this new and fast-changing phenomenon, let’s be-gin with some exemplars. Table 1 explains some key features for arti-ficial intelligence in education (AIED) across a sample of major MOOC platforms, such as Coursera (www. coursera.org), edX (www.edX.org), Course Builder (www.powersearching withgoogle.com), Class2Go (http:// class2go.stanford.edu), udemy (www.udemy.com), and Lernanta (http://p2pu.org/en). We chose these plat-forms because they represent a di-verse set of some of the better-known MOOC platforms.

Both edX and Coursera repre-sent the new, widely publicized MOOCs—funded and provided by large universities. These focus on video lectures, with an impor-tant innovation being their use of integrated quizzes. These break up the traditional lecture into much shorter parts, ending each lecture snippet with a self-test short an-swer question that is automatically graded so there can be immediate

feedback to the learner. These do the main content delivery, and the platforms tend to be light on other quizzing capabilities.

Google made their Course Builder open source after the company used it to teach their “Power Searching with Google” course. It requires significant technical work and skill to setup and run a course.

A Stanford team created the Class2Go open source platform. The team’s design is similar to edX and they’re working together on a com-bined and open platform. Because it promises to be open and high quality, the AIED community finds it particu-larly interesting.

Compared to the other platforms, udemy stands out because it truly provides a platform for anyone to teach. Notably, it also lets instructors charge students for access to their courses. However, it has no ability to assess or quiz students.

Finally, we included Lernanta as an example of a cMOOC. It pow-ers P2PU, a site for social learning. It has no quiz or video functionality,

but it focuses on peer assessment and discussion.

A Diversity of Features and ApproachesComparing the attributes of each MOOC platform in Table 1, we imme-diately see how different these MOOC platforms are. Even platforms that support video differ in terms of host-ing sites and discussions or quiz inte-gration. Despite their quite minimal-ist quiz capabilities, MOOCs already face fracturing in terms of question types supported. In particular, while many offer a short answer-style ques-tion, they drastically differ in the ways they mark these answers. This ranges from peer assessment to automated assessment, where MOOCs do this in fairly simple ways, such as matching regular expression or simply compar-ing student’s input for an exact match against a list of “correct” answers. All of the systems currently have ru-dimentary facilities to capture learner activity data for analysis. The student can see rather simple information about their marks and progress. The

The IEEE Working Group 12 (WG12) describes learning objects as “any entity, digital or non-digital, which can be used, reused, or referenced during technology-supported

learning” (see http://ltsc.ieee.org/wg12). The working group aims “to enable learners or instructors to search, evaluate, acquire, and utilize learning objects.” Wayne Hodgins1 champi-oned this work on learning objects when the Web was emerg-ing and growing. He visualized making greater use and reuse of high-quality learning resources by making them open and available to all. For example, WG12 describes one of their goals as “to enable computer agents to automatically and dynami-cally compose personalized lessons for an individual learner.” By choosing the word object in their description, they reflect the programming notion, with its associated possibilities of re-use. They worked to define standards around learning objects to enable their description and subsequent reuse, and they created many learning object repositories. Critics of the whole notion of learning objects and their reuse2,3 highlight the problems of reuse, particularly when a teacher needs to adapt a learning object to fit the context and needs of their students.

Another landmark initiative that aimed to make high- quality learning resources publicly available was Open Educa-tional Resources, or OER. A review of this movement described it aiming to ”Sponsor high-quality open content . . . Remove

barriers ... Understand and stimulate use.”4 Its flagship out-come was the MIT OpenCourseWare Project, described as “a very successful, compelling, living existence proof of the power of high-quality open educational resources ... a pio-neering project that has now become a catalyst for a nascent open courseware movement in service of both teachers and learners.” There is a strong similarity between these senti-ments and the current visions presented for massive open on-line courses (MOOCs).

References 1. H. Hodgins, “The Future of Learning Objects,” e-Technolo-

gies in Engineering Education: Learning Outcomes Providing Future Possibilities, Eng. Conf. Int’l (ECI) Symp. Series, paper 11, 2004; http://services.bepress.com/eci/etechnologies/11.

2. P. Parrish, “The Trouble with Learning Objects,” Educational Technology Research and Development, vol. 52, no. 1, 2004, pp. 49–67.

3. N. Friesen, “Three Objections to Learning Objects,” Online Education Using Learning Objects, 2004, pp. 59–70.

4. D. Atkins, J. Brown, and A. Hammond, A Review of the Open Educational Resources (OER) Movement: Achievements, Chal-lenges, and New Opportunities, tech report, William and Flora Hewlitt Foundation, 2007; www.hewlett.org/uploads/files/ ReviewoftheOERMovement.pdf.

Learning Objects and Open Educational Resources


4 www.computer.org/intelligent Ieee InTeLLIGenT SySTemS

teacher facilities are pretty limited, too. Perhaps in the spirit of making use of existing tools, they do not ex-tend beyond exporting comma-sep-arated value (CSV) files of raw data, or using general website analytics, like Google Analytics. Here is a place where there is exciting potential to in-troduce AIED tools and techniques into MOOCs.

From our small but carefully se-lected sample of MOOC platforms, we see the diversity in the features for learning and assessment, even at this early stage. This highlights our dif-ficulties in determining what is and isn’t a MOOC platform. We also have difficulty telling the depth of the differences between MOOCs and the widespread Learner Management Systems (LMS; see the sidebar, “Is an LMS Really a MOOC Platform?”).

Now let’s consider the more gen-eral picture that seems to be emerg-ing. Figure 1 shows a high-level view of the teacher and student views of MOOCs. The teacher needs to de-sign the curriculum, perhaps taking into account formal learning goals for accreditation and certification or broader goals that learners have for lifelong learning on demand, at the times that they recognize the need for new knowledge and competencies. With this foundation, the teacher needs to design and create the learn-ing materials that the students will actually see and interact with. For the current breed of MOOCs, this in-volves making video snippets, short pieces with the lecturer’s face vis-ible, and other supporting material, such as their annotations of materi-als. The lecturer needs to create the

self-test formative quiz questions as well as the larger assessment tasks. The teacher must also make design decisions for the discussion forums and grades or assessments. For ex-ample, the lecturer might create ru-brics or questions for use in self- and peer-assessment. The remaining part of the big picture is the lecturer’s use of data about the learning. Some of this is classic management of marks. Another dimension is to reflect on student’s progress and use that to re-fine the materials and course. AIED has much to offer for all of these stages and elements and we will dis-cuss some examples after we have ex-plored some of the challenges.

So Many LearnersAs exciting as MOOCs are, if we want to exploit their potential, there’s

Table 1. Key features for artificial intelligence in education (AIED), across a sample of major massive open online course (MOOC) platforms.

Features edX Coursera Google Course Builder Class2Go udemy Lernanta

Video lectures

Where are they stored?

YouTube Coursera YouTube YouTube and Amazon S3

udemy or YouTube

N/A

Quizzes integrated with video?

No Yes No Yes No N/A

Discussion on video page?

Yes No No No Yes N/A

Additional files and features

Subtitles Subtitles files Subtitles files Subtitles Subtitles video and slide mashup

N/A

Quizzes

Are there quizzes outside of videos?

Yes Yes Yes Yes No N/A

Question types

Multiple choice ✔ ✔ ✔ ✔

Short answer ✔ ✔ ✔ ✔

Numeric ✔

No. of attempts allowed

Limited Limited Unlimited Limited N/A N/A

Discussion forums

Can posts be rated? Positive Positive/negative N/A None None None

Grading and analytics

Student’s view of progress

Raw marks with graph

Raw marks None Raw Marks Progress percentage

Progress percentage

Teacher’s view of progress

Unknown Unknown CSV* export Google analytics (CSV)

Multiple types of detailed CSV reports

N/A Can see and edit progress

*CSV = comma-separated value.



much to learn from AIED and broader educational research. We do know a good deal about the real challenges of dis-tance learning and what a teacher must do to create effective learning contexts. We now con-sider these and how AIED work can provide foun-dations for MOOCs to help more teachers enable more learners to learn more things, more deeply.

The StudentWhat does it take to be a successful MOOC stu-dent? Current evidence suggests that the success-ful MOOC student isn’t your average student who has de-cided they need to learn. Indeed, that sort of student might be a rare phe-nomenon. Although we can’t pro-vide more reliable numbers this early, the current first-generation MOOCs typically have a 5–10 percent pass rate. For instance, in a truly massive

course run by MIT, only 7,157 of more than 150,000 students who signed up passed (www.i-programmer. i n f o / n e w s / 1 5 0 - t r a i n i n g - a -education/4372-mitx-the-fallout-rate.html). Of course, 7,157 is a very large group of students, but a 95 percent dropout rate is quite high. We clearly

can’t treat this fallout rate of 95 percent as directly comparable with fallout rates in university courses or training courses, be-cause even the sma l l amount of available re-search on student demo-graphics and motivation indicates that a high per-centage of those signing-up are just curious (www.i n s i d e h i g h e r e d . c o m /news/2012/06m/0e5n /eta[r1ly3-].demographic-data-h ints-what-type-student-takes-mooc). But even students who go be-yond the initial “snooping” phase have difficulties finishing the course. Es-sentially, the MOOC stu-

dent faces the same challenges as a distance-learning student, such as time management. Researchers have published extensive literature on these challenges and on the kind of support that teaching organizations should provide.3 These challenges have stayed stubbornly consistent

Since Sebastian Thrun’s artificial intelligence course in 2011, there has been a boom in the number of platforms from which a MOOC can be taught. Many of these plat-

forms share common characteristics, including a focus on short video lectures and questions integrated within those videos. We can identify MOOC platforms by these features.

However, these new platforms have a lot in common with their conceptual ancestors, Learning Management Systems (LMS). Like MOOCs, developers designed LMSs to move learn-ing online. However, engineers designed LMSs to assist a classroom, whereas they designed a MOOC to replace one. In general, learning institutions have already incorporated LMSs, because they’re older. They’re more advanced, espe-cially when it comes to quiz/assignment design. They have many features that most MOOC platforms lack. Indeed, the only feature MOOC platforms possess that LMSs don’t is the integration of questions within videos, something that LMS platforms could easily add and improve, given their superior assessment capabilities.

The recent emergence of MOOCs is already affecting the LMS space. For example, Blackboard—the creators of Blackboard Learn, a leading LMS—recently launched CourseSites, a

hosted version of their popular Blackboard Learn software. Interestingly, despite users describing it as a MOOC plat-form, CourseSites appears to make no attempt to emulate features of other popular MOOC platforms. For instance, they still have poorly implemented videos. They seem to have confused the purpose of MOOCs. This different, per-haps confused, interpretation results in a product that was not designed to entirely replace face-to-face learning. In attempting to do so, they further blur the line between the LMS and a MOOC.

Similarly, the traditional use of LMS platforms is shaping the implementation of MOOC technologies, with some uni-versities having begun adapting MOOC platforms to run in-house courses as a flipped classroom. In this model, educators deliver content such as lectures outside the classroom, so that more class time may be spent on activities and discussions. For example, in January 2013, the Cultural Complexity and Digital Humanities Department at The University of Western Ontario offered a course that used a modified version of the open source openMOOC platform. The teacher delivered all of the lectures and prescribed readings via the platform, ded-icating actual class time to discussions and tutorials.

Is an LMS Really a MOOC Platform?

Figure 1. Overview of the teacher and student views of MOOCs. The platform must have the flexibility to meet each side’s needs.

Design curriculum: accreditation,certification, lifelong learning

Design and create materials- “Lecture” snippets- Associated formative quiz- Major assessments

Discussion, collaboration

Manage grading

Teacher view

Reflect on student progress

Revise materials

Interact with material- “Attend” lectures- Complete and reflect on formative quiz- Complete major assessments

Group formation

Student view

Discussion about material

Self assess, peer assess

Summative assessment



over time. The new generation’s life-long access to technology doesn’t ad-dress the problem. Researchers have even proven that the widely claimed generation-related increases in tech-nical competence of digital natives are largely a myth.4

How might AIED make important improvements to students’ staying power and success in MOOCs? Stu-dents must possess certain competen-cies, such as self-guided learning and time-management skills. Despite the strict regimen that a typical MOOC imposes on students for handing in assignments, students must self-manage what happens between as-signment dates. As we know from psychology, self-regulated learning involves a complex interplay of cogni-tive, metacognitive, and motivational regulatory components.5

According to recent theoretical ap-proaches, ideal regulatory activities during learning include orientation, in order to get an overview of the task and resources, planning the course of action, evaluating the learning prod-uct, and monitoring and controlling all activities. According to research, students who participate in these reg-ulatory activities more successfully learn the content.6 However, only a few students display such learning strategies. Most of us need the educa-tor to prompt or reward us to engage in them.7

Educators might find it hard to change dispositions; however, stu-dents can master learning strate-gies fairly quickly. Keeping a learn-ing journal, if not a more formal learning portfolio, and going beyond the requested assignments by re-flecting on one’s learning is another element that could be likely con-tribute to success in MOOCs.8 In ad-dition, students could organize small learning groups in the context of a MOOC. Engineers built MOOCs

on an important theory of mass collaboration, but it’s different from the cognitive and motivational sup-port we get from communicating with a small group of people we trust and whose interests we share.9 In addition to social networking sites, research-ers recommend using e-portfolios as a tool for planning and documenting learning, reflecting on learning, and sharing learning and reflections with others.10 Related to this, researchers have revealed competence-oriented open learner models that support self-guided, lifelong learning.11

The TeacherResearchers built MOOCs on the as-sumption that there are great teach-ers out there, and that getting them on video (plus a couple of assign-ments and quizzes) is largely suffi-cient for establishing pedagogical quality. No doubt there are great teachers out there, but as MOOCs proliferate (assuming for the sake of argument that they do proliferate), this approach won’t suffice. Not ev-ery MOOC lecturer will be rhetor-ically gifted or trained, and “cus-tomers” will become more critical about quality. Great researchers aren’t necessarily also great teach-ers, and great teachers don’t neces-sarily produce great online learning experiences. We’ll need to support MOOC lecturers and authors in sev-eral of the elements in our table to ensure that they can be effective in instructional design, course develop-ment, and student monitoring.

To support technical course devel-opment (for example, video captur-ing and editing, assignment specifica-tion), one can build on a good many years of research on, and experience with, developing courseware tools and e-learning software.12 Supporting the instructional design process on the pedagogical-conceptual level is no less

challenging, as it has proven difficult to capture the pedagogical rationale and to formalize it in a manner that is conducive to sharing and continu-ous improvement.13 Researchers have discovered that students build impor-tant conceptual foundations by work-ing on learning design patterns.14 On the practical side, engineers have de-veloped numerous tools to support educators in designing for (online) in-struction, and they have made some progress in providing shared reposi-tories for learning designs (see www.lamsfoundation.org). In short, the subject specialist creating a MOOC can gain a lot from the body of knowledge on how to create effective online materials and ways to nurture effective learning experiences for stu-dents. And, happily, there are already some MOOCs for that!

To monitor the learning activities of potentially thousands of students, e-learning platforms are providing increasingly valuable reporting tools. To some extent, they’re also appear-ing in MOOC platforms, and we can anticipate refinement and deve-lopment of these. Engineers are also developing learning analytics for when an educator needs to monitor learning activities across platforms and tools.

Assessment issues are notoriously complex, not only in a technical sense. First-generation MOOCs are fairly simple-minded about assess-ment, following the United States’ undergraduate model of quizzes and assignments. Although MOOCs can automatically assess quizzes, peer reviews must deal with more com-prehensive assignments. However, assessment should be formative, as well. And that goes beyond simple self-test questions at the end of each lecture segment. That is, formative feedback should provide information that can guide learning. In this



respect , neither quizzes nor peer- assessment suffice. Eventually, MOOC course and platform developers must engage with the substantial chal-lenges of online formative and sum-mative assessment.

So Much PotentialMOOCs’ engineers can exploit diverse results and techniques from AIED to improve MOOC platforms and create new opportunities for AIED research. Computer scientists working in fields such as educational data mining15 and learning analytics16 find MOOCs particularly interesting. Not only can they create truly “big” learning- related data from MOOC courses (provided that the dropout rate is suitably managed), they can also pro-vide a very heterogeneous student body, because of the open nature of MOOCs. These students can interact in ways that aren’t further structured by established social contracts and roles; therefore, researchers may ex-plore social network analysis methods on MOOCs.

Methods from educational data mining and learning analytics can in general be applied for knowledge cre-ation (learning more about learning and interaction, and relevant technolo-gies). They can also serve applied pur-poses: supporting students, teachers, educational institutions, and systems. In light of the mentioned attrition rate, computer scientists can use another rather obvious applied challenge—automatically identifying students at risk of failing. They can use similar techniques to “nudge” students who need it, and provide course- or cohort-based monitoring.17 We can expect the growth of large collections of learning data, similar to the Pittsburgh Science of Learning Center (PSLC) Datashop (https: / /ps lcdatashop.web.cmu.edu). This can provide a new scale in testbeds for EDM researchers. We can

then expect to see more innovative uses of learning data to improve teach-ing, such as an elegant system to gener-ate hints for students,18 by drawing on historic data from the paths taken by successful and un successful students.

Pedagogic interface agents are one of the current hot topics in AIED. Researchers have proven that these anthropomorphic conversational char acters provide real benefits for learning.19 Although we might expect this effect to be short-lived once the novelty has worn off, recent results indicate that interface agents actu-ally help people stay the course.20 They seem to promise a valuable role in MOOCs.

In addition to providing general re-search opportunities on how to sup-port online learning with technical means, MOOCs might also provide a particularly fruitful arena for re-search on e-portfolio systems, com-petence management (including as-sessment), and technical support for lifelong learning (including open learner models).

The quality, timing, and form of feedback is critical to effective learn-ing. MOOCs currently rely heavily on self- and peer-review. Higher edu-cation already recognizes the place for such forms.21 However, they are more effective if students are explic-itly taught how to do such self- and peer-assessment, a valuable role for AIED systems.

Another key form of valuable feedback can be provided for learn-ing contexts where high-quality as-sessment can be automated. This is the case for domains like pro-gramming, mathematics, and phys-ics, where there are already many systems that provide high-quality feedback. AIED research has pro-duced many systems that give high-quality feedback in those classes of well-defined learning domains.

These classes of MOOCs can also be part of a hybrid model. For exam-ple, many developing countries have a large unmet need for skilled IT pro-fessionals, where the learning need in-volves well-defined technical skills. The most recent MOOCs already of-fer several attractive options for this situation. Employers can use MOOCs to deliver content and a basic forma-tive assessment to potential employ-ees. The employer can complement this by nurturing learning communi-ties. They can conduct summative as-sessment that determines employment options, which is a significant motiva-tor for students.

More recently, researchers are moving AIED toward ill-structured domains22—most notably, lifelong ge-neric (particularly the meta-cognitive) skills that are a key to success in MOOCs. Educators have successfully taught these skills using AIED.

The rhetoric about MOOCs refers to personalized learning, with reference to Benjamin Bloom’s classic 2-Sigma arti-cle about one-to-one tutoring.23 How-ever, current MOOCs come nowhere near trying to achieve that level of per-sonalization. One key to the success of AIED systems is in the nature of the per-sonalization, which is based on a learner model. Indeed, some have argued that the very core of AIED is the role of the learner model.24 This core notion of creating an explicit learner model could be readily integrated into MOOCs. Open learner models25 have been dem-onstrated to improve learning, and they could be a fundamental means for learners to monitor their prog-ress and plan all aspects of their learn-ing, including setting goals and how to achieve them.

It is hard to conceive of MOOCs as having any lasting impact on (higher) education without concern for how



the single MOOC event (course) gets integrated into individual ca-reer planning and personal devel-opment, as well as into an compre-hensive certification framework.26 Hence, research on how to support the integration of learning events on the individual as well as the societal level will be crucial. The excitement around MOOCs is justified, both in terms of the potential value they offer and the quality of the play-ers who have launched them. What a great opportunity to integrate the lessons, techniques, methods, and tools of AIED!

AcknowledgmentsThe faculty of Engineering and Information Technologies, University of Sydney, Austra-lia partly funded this work.

References 1. R. Light, Making the Most of College:

Students Speak Their Minds, Harvard

University Press, 2001.

2. T. Friedman, The World Is Flat: A Brief

History of the Twenty-First Century,

Farrar, Straus and Giroux, 2006.

3. O. Simpson, Supporting Students in

Online, Open and Distance Learning,

Routledge, 2002.

4. M. Bullen, T. Morgan, and A. Qayyum,

“Digital Learners in Higher Education:

Generation Is Not the Issue,” Canadian

J. Learning and Technology, vol. 37,

no. 1, 2011; http://cjlt.csj.ualberta.ca/

index.php/cjlt/article/view/550/298.

5. M. Boekaerts, “Self-Regulated Learning:

A New Concept Embraced by Research-

ers, Policy Makers, Educators, Teachers,

and Students,” Learning and Instruc-

tion, vol. 7, no. 2, 1997, pp. 161–186.

6. R. Azevedo, J. Guthrie, and D. Seibert,

“The Role of Self-Regulated Learning

in Fostering Students’ Conceptual Un-

derstanding of Complex Systems with

Hypermedia,” J. Educational Comput-

ing Research, vol. 30, no. 1, 2004,

pp. 87–111.

7. M. Bannert and P. Reimann, “Sup-

porting Self-Regulated Hypermedia

Learning through Prompts,” Instruc-

tional Science, vol. 40, no. 1, 2012,

pp. 193–211.

8. D. Cambridge, E-Portfolios for Life-

long Learning and Assessment, Jossey-

Bass, 2010.

9. G. Stahl, Group Cognition: Computer

Support for Building Collaborative

Knowledge, MIT Press, 2006.

10. K. Wall et al., “Developing Digital

Portfolios: Investigating How Digital

Portfolios Can Facilitate Pupil Talk

about Learning,” Technology, Peda-

gogy and Education, vol. 15, no. 3,

2006, pp. 261–273.

11. J. Kay and B. Kummerfeld, “Lifelong

Learner Modeling,” Adaptive Technol-

ogies for Training and Education, Cam-

bridge Univ. Press, 2012, pp. 140–163.

12. W. Horton and K. Horton, E-Learning

Tools and Technologies: A Consumer’s

Guide for Trainers, Teachers, Educa-

tors, and Instructional Designers,

Wiley, 2003.

13. I. Falconer et al., Mod4l Final Report:

Representing Learning Designs,

tech. report, 2007; www.academia.

edu/516561/Mod4L_final_report_

Representing_learning_designs.

14. P. Goodyear and S. Retalis, Technology-

Enhanced Learning: Design Patterns

and Pattern Languages, Sense Publish-

ers, 2010.

15. R. Baker and K. Yacef, “The State of

Educational Data Mining in 2009: A

Review and Future Visions,” J. Educa-

tional Data Mining, vol. 1, no. 1, 2009,

pp. 3–17.

16. R. Ferguson, “Learning Analytics:

Drivers, Developments and Chal-

lenges,” Int’l J. Technology Enhanced

Learning (IJTEL), vol. 4, nos. 5–6,

pp. 304–317.

17. K. Arnold, “Signals: Applying Academic

Analytics,” Educause Quarterly, vol. 33,

no. 1, 2010; www.educause.edu/ero/

article/signals-applying-academic-

analytics.

18. J. Stamper et al., “Experimental Evalu-

ation of Automatic Hint Generation

for a Logic Tutor,” Artificial Intel-

ligence in Education, Springer, 2011,

pp. 345–352.

19. B. Woolf et al., “Affect-Aware Tutors:

Recognising and Responding to Student

Affect,” Int’l J. Learning Technology,

vol. 4, no. 3, 2009, pp. 129–164.

20. L. Pfeifer and T. Bickmore, “Is the

Media Equation a Flash in the Pan?:

The Durability and Longevity of Social

Responses to Computers,” Proc. 2011

Annual Conf. Human Factors in

Computing Systems, ACM, 2011,

pp. 777–780.

21. F. Dochy, M. Segers, and D. Sluijsmans,

“The Use of Self-, Peer and Co-Assessment

in Higher Education: A Review,” Stud-

ies in Higher Education, vol. 24, no. 3,

1999, pp. 331–350.

22. C. Lynch et al., “Concepts, Structures,

and Goals: Redefining Ill-Definedness,”

Int’l. J. Artificial Intelligence in Educa-

tion, vol. 19, no. 3, 2009, pp. 253–266.

23. B. Bloom, “The 2 Sigma Problem: The

Search for Methods of Group Instruc-

tion as Effective as One-to-One Tutor-

ing,” Educational Researcher, vol. 13,

no. 6, 1984, pp. 4–16.

24. J. Self, “The Defining Characteristics of

Intelligent Tutoring Systems Research:

Itss Care, Precisely,” Int’l. J. Artificial

Intelligence in Education (IJAIED),

vol. 10, 1998, pp. 350–364.

25. S. Bull and J. Kay, “Student Mod-

els That Invite the Learner In: The

SMILI:() Open Learner Modelling

Framework,” Int’l. J. Artificial Intel-

ligence in Education, vol. 17, no. 2,

2007, pp. 89–120.

26. R. Gluga, T. Lever, and J. Kay,

“Foundations for Modelling University

Curricula in Terms of Multiple Learn-

ing Goal Set,” IEEE Trans. Learning

Technologies, vol. 6, no. 1, 2013,

pp. 25–37.

judy Kay is a professor of computer science

in the School of Information Technologies



at the University of Sydney, and she is a

principal in the CHAI: Computer Human

Adapted Interaction Research Group. Her

research interests include personalization,

advanced technologies for learning, and sur-

face computing. Kay has a PhD in computer

science from the University of Sydney. She is

the immediate past president for the Artifi-

cial Intelligence in Education (AIED) Soci-

ety. Contact her at [email protected].

Peter Reimann is a professor in the Educa-

tion Faculty of the University of Sydney. His

research interests include cognitive learning

research with a focus on educational com-

puting, multimedia- and knowledge-based

learning environments, e-learning, and the

development of evaluation and assessment.

Reimann has a PhD in psychology from

the University of Freiburg, Germany. He is

currently directing the Next-Tell project,

funded by the European Commission. Contact

him at [email protected].

elliot Diebold is a student at the Univer-

sity of Sydney, studying a combined Bache-

lor of Information Technology and Bachelor

of Medical Science. He is currently complet-

ing a summer research scholarship at the

university, exploring innovative potential

uses of MOOC technology. Contact him at

[email protected].

Bob Kummerfeld is an associate profes-

sor of computer science in the School of In-

formation Technologies at the University of

Sydney, and he’s a member of CHAI: Com-

puter Human Adapted Interaction Research

Group. His current research focuses on

pervasive computing systems, middleware

for long-term user modeling, and innova-

tive user interfaces. Kummerfeld has a PhD

in computer science from the University of

Sydney. Contact him at bob.kummerfeld@

sydney.edu.au.

Selected CS articles and columns are also available for free at

http://ComputingNow.computer.org.

stay connected.Keep up with the latest IEEE Computer Society

publications and activities wherever you are.

| IEEE Computer Society| Computing Now

| facebook.com/IEEEComputerSociety| facebook.com/ComputingNow

| @ComputerSociety | @ComputingNow

| youtube.com/ieeecomputersociety


Documents

MOOCs: So Many Learners, So Much Potential