John Whitmer, Ed.D. & Kathy FernandesAcademic Technology Services

California State University, Office of the Chancellor

CATS 2013, Sonoma State UniversityFebruary 19, 2013

Using Learner Analytics to Understand Student Achievement in

a Large Enrollment Hybrid Course

Outline1. Context

2. Learning Analytics, Methods & Tools

3. Findings

4. Conclusions & Next Steps

5. Becoming a Data Scientist (pending time & interest)

Why are you here? 1. To increase my knowledge of “Learning

Analytics”

2. To get ideas about how I can become a “Data Scientist” (or practice analytics)

3. To learn about the results of this study

4. Because this was the closest room

1. CONTEXT

Case Study: Intro to Religious Studies• Undergraduate, introductory, high demand

• Redesigned to hybrid delivery format through “academy eLearning program”

• Enrollment: 373 students (54% increase on largest section)

• Highest LMS (Vista) usage entire campus Fall 2010 (>250k hits)

• Bimodal outcomes:• 10% increase on final exam• 7% & 11% increase in DWF

• Why? Can’t tell with aggregated data

54 F’s

Guiding Questions1. How is student LMS use related to academic

achievement in a single course section?

2. How does that finding compare to the relationship of achievement with traditional student characteristic variables?

3. How are these relationships different for “at-risk” students (URM & Pell-eligible)?

4. What data sources, variables and methods are most useful to answer these questions?

It takes a village … Chico “Graduation Initiative” endorsement Former CIO (Bill Post included in project portfolio) Registrar approval

Active participation: – Kathy Fernandes, Director Academic Technology– William A. Allen, Director of Institutional Research– Scott Kodai, Manager Distributed Learning

Technologies and Classroom Services

Predict the trend: Grade w/Mean LMS Hits

What will the trend look like?

Strong Trend: Grade w/Mean LMS Hits

2. LEARNER ANALYTICS, METHODS & TOOLS

Learner Analytics

“ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)

How? Knowledge & Skills Involved

http://www.drewconway.com/zia/?p=2378

Data Extracts & Sources LMS Data

– SQL queries on WebCT Vista web server log file

Student Data– Query against Peoplesoft data– ERSS data elements (common to CSU)– Required Institutional Research

Course Data– Query against Peoplesoft data

Tools Used

App Function

Excel Early data exploration; simple sorting; tables for print/publication

Tableau Complex data summaries and explorations; complex charts; presentation chartsFinal/formal descriptive data; statistical analysis; some charts (scatterplots)Statistical analysis (factor analysis)

Methods at a Glance1. Clean/filter/transform/reduce data (70% effort)2. Descriptive / exploratory analysis (20% effort)3. Statistical analysis (10% effort)

– Factor analysis– Correlation single variables– Regression multiple variables; partial & complete

Final data set: 72,000 records (-73%)

Variables

3. FINDINGS

Correlation: Student Char. w/Final Grade

Scatterplot of HS GPA vs. Course

Grade

LMS Use Variables

Administrative Activities (calendar, announcements)

Assessment Activities (quiz, homework, assignments, grade center)

Content Activities (web hits, PDF, content pages)

Engagement Activities (discussion, mail)

Student Characteristic Variables Enrollment Status First in Family to Attend

College Gender HS GPA Major-College Pell Eligible URM and Pell-Eligibility

Interaction Under-Represented

Minority URM and Gender

Interaction

Predict the trend LMS use and final grade is _______ compared to

student characteristics and final grade:

a) 50% smallerb) 25% smallerc) the samed) 200% largere) 400% larger

Predict the trend LMS use and final grade is _______ compared to

student characteristics and final grade:

a) 50% smallerb) 25% smallerc) the samed) 200% largere) 400% larger

Correlation LMS Use w/Final Grade

Scatterplot of Assessment Activity

Hits vs. Course Grade

Conclusion: LMS Use Variables better Predictors than Student Characteristics

LMS Use

Variables

18% Average(r = 0.35–0.48)

Explanation of change in final grade

Student Characteristic

Variables

4% Average(r = -0.11–0.31)

Explanation of change in final grade

>

Smallest LMS Use Variable

(Administrative Activities)

r = 0.35

Largest Student

Characteristic

(HS GPA)

r = 0.31

>

Combined Variables Regression Final Grade by LMS Use & Student Characteristic Variables

LMS Use

Variables

25% (r2=0.25)

Explanation of change in final grade

Student Characteristic

Variables

+10%(r2=0.35)

Explanation of change in final grade

>

Predict the trend LMS use and final grade is ______ for “at-risk”*

students?a) 50% smaller than not at-risk studentsb) 20% smaller than not at-risk studentsc) No difference than not at-risk studentsd) 20% larger than not at-risk studentse) 100% larger than not at-risk students

*at-risk = BOTH under-represented minority and Pell-eligible

Predict the trend LMS use and final grade is ______ for “at-risk”*

students?a) 50% smaller than not at-risk studentsb) 20% smaller than not at-risk studentsc) No difference than not at-risk studentsd) 20% larger than not at-risk studentse) 100% larger than not at-risk students

*at-risk = BOTH under-represented minority and Pell-eligible

At-Risk Students: “Over-Working Gap”

33

Question 3 Results:Regression by “At Risk” Population Subsamples

WHAT CAUSED THAT DIFFERENCE?

Activities by Pell and Gradegrade / pelleligible

A B+ C C-

Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible0K

5K

10K

15K

20K

25K

30K

35K

Value

Content

Content

Engage

Engage

Assess

Assess

Admin

Admin

Content

Content

Engage

Engage

Assess

Assess

Admin

Content

Content

Engage

Engage

Assess

Assess

Content

Content Engage

Engage

Assess

Assess

Admin

Admin

Measure NamesAdmin

Assess

Engage

Content

Extra effort in content-related activities

Previous Studies Relating LMS Use to Course Grade

4. CONCLUSIONS & NEXT STEPS

Conclusions1. At the course level, LMS use better predictor of

academic achievement than student demographics (what do, not who are).

2. Small strength magnitude of complete model demonstrates relevance of data, but suggests that better methods could produce stronger results.

3. LMS data requires extensive filtering to be useful; student variables need pre-screening for missing data.

More Conclusions4. LMS use frequency is a proxy for effort. Not a

very complex indicator.

5. Student demographic measures need revision for utility in Postmodern era (importance to student, more frequent sampling, etc.).

6. LMS effectiveness for at-risk students may be caused by non-technical barriers. Need additional research!

CSU Learner Analytics Collaborative Projects

Moodle: mCURL (Moodle Common Use Reporting &

Learning Analytics)

8 CSU & 2 UC Campuses

Building common SQL queries for accurate & comparative use metrics

2 rounds of data collection already completed and discussed

Blackboard: Analytics for Learn Pilot

3 CSU Campuses

Bb Learn Analytics product available “off the shelf”; defined and integrated with Peoplesoft

Building common queries & campus-specific reporting

5. SO YOU WANNA BE A DATA SCIENTIST?

Choose your domain(s)

http://www.drewconway.com/zia/?p=2378

Potential Resources Math/Stats: campus classes

MOOC resources (search “Data”, “Statistics”, or “Analytics”)

Society for Learning Analytics Research (http://www.solaresearch.org/)

Learning Analytics Resources (http://johnwhitmer.net/resources/)

Feedback? Questions?

Kathy Fernandeskfernandes@csuchico.edu

John Whitmer jwhitmer@calstate.edu

Complete monographhttp://bit.ly/15ijySP

Twitter: johncwhitmer

BONUS SLIDES

Missing Data On Critical Indicators

LMS Use Consistent across Categories

Factor Analysis of LMS Use Categories

Ideas & FeedbackPotential for improved LMS analysis methods: time series analysis social learning activity patterns discourse content analysis

Group students by broader identity, with unique variables: Continuing student (Current college GPA, URM, etc. First-time freshman (HS GPA, SAT/Act, etc)