9-3-2015

Challenge the future

DelftUniversity ofTechnology

An ‘open research’ approach

Thieme Hennis – thieme@hennis.nl

Pieter de Vries - Pieter.deVries@tudelft.nl

TU Delft Online Learning

A bit of context

Big ambition…

• And questions on different levels..

Level (Example) topics of interest and stakeholders

Strategic University role into the future, open en online education, business models, ROIDeans, DelftX, MT, marketing

Organizational Support structure, documentation and guidelines, collaboration protocols, data use, teacher satisfactionO&S

Course Course design, use of media, pedagogy, methods and tools, student support, experiments, interaction & collaborationCourse teams, Media Center

Technology/edX Design of tools for learning, teaching, and data analysis/visualizationedX consortium, ‘TUD tech’

Students Learner demographics, background, performance, engagement, satisfaction, etc.Students

Many opportunities

• … to research MOOCs

• … and use MOOCs for research

But..

• Limited capacity

• No/little history in educational research at university

• Potential to connect with researchers worldwide

Open Research approach

• What is open research?

Open research…

Facilitating research activities

1. …with relevant parties globally and locally

2. …on a broad range of relevant topics

3. …by giving them access to our data and research

instruments

4. …and managing the collaboration workflow

5. …and publishing the results together

1a. Who are these ‘relevant

parties’?

• MOOC Teachers / course teams

• University students / PhD’s

• Researchers from partner institutes

• Researchers globally with expertise on a specific,

relevant topic

1b. Our current ‘MOOC research

network’

• Stanford University - Psychology• UC St. Barbara - Psychology• University of South Australia - Pedagogy, coll. learning• Edinburgh University - Communities of Inquiry• Harvey Mudd College - Gender, diversity• TU München - Gender, diversity• Memphis university - Linguistics• UVA - Methodologies

• TU Delft - Privacy/ethics- Course-relevant topics (researching course teams)

2a. The ‘broad range’ of relevant

topics

• Who is the DelftX MOOC learner? - UNISA

• Does self-affirmation of personal values increase engagement of at-

risk students (motivation research) – Stanford, UC St. Barbara

• (How) does communication in MOOC forums influence social

centrality and performance? – Memphis Univ.

• How well does the Community of Practice framework fit learning in a

MOOC context? What effects of social and teaching presence on social

networks and collaboration? – Edinburgh univ.

• Why do female students have lower performance in our MOOCs? How to

design socially inclusive courses? – Harvey Mudd College & TU

München

• Etc..

2b. Choice of topics

Research questions come/emerge from

• Evaluation outcomes (DelftX MOOCs)

• Literature, current developments in educational research

• Teachers, developers, and other support staff

• External parties interested in research collaboration

3. Giving access to data and

instrumentsData sources Research & evaluation

edX subscription data Number of participants, dropouts, location, age, gender, schooling

edX student data, learning analytics Progress, tests results, quizzes, exams, etc.Forum participationVideo clicks, navigation

External data, other media (i.e. Facebook groups, discussions)

Social networks, content and discourse analysis

Surveys (pre, mid, post) Information about demographics, intention, expectations, satisfaction, media use, etc.Interventions embedded in surveys

Interviews / self-assessment: teachers, NMC, DelftX

Experiences with workflow andorganizationQuestions and expectations for evaluation

4a. Facilitating collaboration

workflow

4b. The Open Research Toolbox

Collaborating• Google Drive folder with

• Documentation for partners• Data files

Documentation• Memorandum of Understanding• Ethical clearance documentation• Paper outline• All available data sources, instruments, and courses

Communication• Monthly Skype conversations (depending on stage in process)• Email

5. Selection of publications

• 6 Working papers

• 5 course reports

• 1 comparative analysis paper

• Understanding Social Learning Behaviors (SEFI’14 - Skrypnyk, Hennis, De Vries, 2014)• Learners’ Social Centrality and Performance (EDM’15 – Dowell, Joksimovic, Skrypnyk,

Hennis, De Vries, 2015)• An exploratory study in the concerns for information privacy (MSc thesis - Hassing,

2015)• Scalability and Flexibility through Open Research (LAK’15) Hennis, De Vries, 2015)

• Who is the Learner in the DelftX Engineering MOOCs? (SEFI’15 - Hennis, Skrypnyk, De

Vries, 2015)

• Diversity in Engineering MOOCs, a first Appraisal (SEFI’15 – Ihsen, Yves, Hennis, De

Vries 2015)

• Reconsidering Retention in MOOCs: the Relevance of Formal Assessment and Pedagogy

(EMOOCS’15 - Skrypnyk, De Vries, Hennis, 2015)

Potential of ‘open research’

• Being highly flexible in what you research

• Being able to scale the research efforts

• Research network and learn from their expertise

• Interdisciplinary research

Next steps

• Optimize the work flow & instruments

• Research across courses (other edX partners/MOOC providers)

• More data cleaning capacity + open data workflow

• Extend the research HUB function (also: organizations)

• Better integration and involvement course teams to achieve

design based research → feedback and forward loops between

research and teaching (online)

Additional ideas:

• Tender for MOOC research

• Promote TU Delft research areas

• Students as researchers (crowdsource research models)

Some interesting outcomes

Gender gap

• Low percentage female students at start

• Even lower percentage female students finishing

• Stereotype threat?→ collaboration Univ. Munich

& Harvey Mudd College

Ihsen, S, Yves, J, Hennis, T. A. & Vries, P. de. (2015). Diversity in Engineering MOOCs, a first Appraisal. In SEFI 43th Annual Conference. Orleans, FR.

Performance attitude

•Among young, male

students

• ‘Spike’ around grades 6

and 10

• Reason to join MOOC:

“To get a certificate..”

Young vs Old

Male vs female

Hennis, T. A., Skrypnyk, O., & Vries, P. de. (2015). Who is the Learner in the DelftX Engineering MOOCs. In SEFI 43th Annual Conference. Orleans, FR.

Dropout rate

• Non-starters are biggest dropout group (up to ~ 80%!)

• Then: after first Homework Assignment (HWA.01)

• GRADE HWA.01 IS IMPORTANT PREDICTOR

• Avg. grade HWA.01 of group that does not continue: to HWA.02: 3.58

• Avg. grade of group that does continue: 8.89

• Also: pass rate is > 80% of group who performs better than 20%

• Including the ‘explorers’

• “No Time” most important factor

Pedagogy and Assessment design

→ Collaboration with University of South Australia

Skrypnyk, O., Vries, P. de. & Hennis, T. A.(2015). Reconsidering Retention in MOOCs: the Relevance of Formal Assessment and Pedagogy. In EMOOCs’15, Mons, BE.

(Professional) background students

• Many professionals: 50-70%

• “No experience” predictor of early

dropout / not starting• Not during course!

• Relevance for course team: • Recommended level course ~

student level?

• Who are my students?

• etc

Hennis, T. A., Skrypnyk, O., & Vries, P. de. (2015). Who is the Learner in the DelftX Engineering MOOCs. In SEFI 43th Annual Conference. Orleans, FR.

Forum (1): performance en structuur

• Forum activity ~ grade

• Facebook groups vs forum• More ‘social’ behavior and peer-

support• Not anonymous• Better performance

• edX forum social network analysis• Few ‘superposters’ responsible for

most posts• Few ‘communities’

Skrypnyk, O., Hennis, T. A., & Vries, P. de. (2014). Understanding social learning behaviours of

xMOOC completers Conference Topic : Technology in Learning. In SEFI 42th Annual Conference.

Birmingham, UK.

Forum (2): Language ~ social capital

+ Community of Inquiry

• Community of Inquiry• Measuring social and teaching

presence• Effects on engagement, performance

and forum activity

• Linguistic analysis• Aumated textual analysis: how

conversations influence development of social capital

→ with Memphis Univ & Univ of Southern Australia

Dowell, N, Skrypnyk, O., Joksimovic ́, S, Graesser, A, Dawson, S, Gasevic, D, Hennis, T. A.,, Vries, P.

de., Kovanovic, V. (2015). Modeling Learners’ Social Centrality and Performance through Language

and Discourse. In EDM’15. Madrid, Spain.

Forum (3): cultural group &

preference to collaborate

• EUR / Anglo > alone• Asia / African /Latin >

together

→ edX forum: does not show this cultural preference!

(vooral EUR/Anglo)