Enhancing Excellence in Assessment: Institutional Effectiveness and Learning

Analytics

SUNY Council on Assessment Learning Analytics Task Group

October 15, 2013

FACT2 and task groups

FACT2:

• “ a well-established venue to foster collaboration and consensus within a highly diverse university community.

• “FACT2governance includes representatives from faculty, librarians, and IT across individual campuses from all Carnegie sectors.”

http://fact.suny.edu

FACT2 and task groups

Task groups• Initiatives developed collaboratively with the SUNY

Provost and the FACT Advisory Council• 2010-2012 projects:– Learning analytics

– E-publishing

– Innovative learning space inventory

• 2013 projects:– Learning analytics

– Online Accessibility

– Experiential Education

2013 Task Group will focus on…

Identifying and sharing best practices and uses of Learning Analytics for

assessment student readiness course placement

& remedial education

Institutional Level Use

Advising Placement Learning outcomes

Course outcomes

Instructional effectiveness

Student Feedback

Intervention

Support Persistence Degree completion

LATG Survey:course success data is used to identify disciplines that have lower than expected success rates; Data are also correlated with student characteristics and other factors like date of registration, data of application.

Institutional Levela) identify students at risk of leaving the college

without a degree from the college [54%]b) identify students who are in need of developmental

courses [46%]c) place students in appropriate credit courses [54%]

d) evaluate student progress on course objectives [38%]

e) predict student performance in courses [15%]f) advise students on course selection [54%]

FACT2 Learning Analytics Task Group, CIT May 2013

Interventions & Feedback

FACT2 Learning Analytics Task Group,

Learning Analytics - Working Definition

Learning analytics can be used to… –diagnose student needs, –provide feedback to the student, faculty,

instructional developer, and advisor, – combine with data from other learning

systems to generate new insights about learning and instruction.

Learning Analytics - Working Definition

• Learning analytics uses software that collects and analyzes multiple data sets related to the process of learning to PREDICT and IMPACT student success.

• This includes data collected in– blended and online learning environments, – online portals, – enrollment data, – and other emergent resources connected to the

teaching and learning experience.

New approachmining the

“pile of big data” generated by technology

longstanding approach “asking research

questions” and gathering data

Where is the data?

DiscreteAnalytics

Tools

LMS Platform Analytics

Stand-alonePlatform Analytics

FACT2 Learning Analytics Task Group

Collected from explicit student actions-completing assignments and exams,

From tacit actions-online social interactions,extracurricular activities, posts on discussion forums, and other activities that are not directly assessed as part of the student’s educational progress.

Data in online activities….

Learning Analytics Tools

SPSS

Approaches• Institutional• Faculty and

advising• Student

assessment and feedback

Traditional analysis tools

Learning Analytics & Online Learning

Examples of uses…• Persistence and

retention (APUS)• Intelligent/adaptive

tutoring (Carnegie Mellon)

• Research on conditions that facilitate learning (CSU Chico)

FACT2 Learning Analytics Task Group

Use Learning analytics to ….

Provide automated feedback to studentsCustomize course delivery to student learning styles.

Quizzes or Learning Sequences in Blackboard or Angel

Use Learning analytics to ….

Provide individualized learning paths to students based on pre-entry conditions.

Provide adaptive learning paths to students based on performance in course.

Learning analytics enables tailoring of responses, such as through adapting

instructional content, intervening with at-risk students, and providing feedback.

Use Learning analytics to ….

Revise course content, activities, assessments and/or course structure.

In adaptive learning, the path of each student is highly personalized

ASSESSMENT AND LEARNING ANALYTICS

Typically use End of semester or mid-term data…Focus on course outcomes, but no real-time data for interventions

PLACEMENT & COURSE SELECTIONLearning analytics in

Predictive Analytics: Building Models

Placement for success and completion..– which students should be steered

toward which courses? Which programs?

Can advising process leverage data on student performance? – If so, what are the best predictors

of performance?

Predictive Analytics: Building Models

• Can we identify characteristics of a successful outcome?

• an unsuccessful outcome?

DATA SOURCES

Grade in course

Can it be predicted by other data?• Major• High school GPA• English placement exam

score• Math placement exam

score• HS Regent scores….• SAT Verbal, SAT math• SAT Writing“every student with a HS

average of 83 or less, did not successfully complete the

course…”

PILOTING APPROACHES

FACT2 Learning Analytics Task Group,

Assessment: study habits

Explore the efficacy of student study habits

Assessment: study habits

Share information about what practices lead to success with students

Automated feedback with quizzes.

What can we learn from the data?

04/11/2023 Teaching, Learning, and Analytics at Michigan

How does E2Coach work?

MTS

Detailed information 

about thousands of students and their current status 

Expertise of hundreds of students, dozens of instructors and behavior change experts Individually 

personalized messages: 

what we all agree we 

would say to each 

student, if only we could…

The Michigan Tailoring System: a mature open-source software system for

creating content designed specifically for an individual based on data about that

individual

Where the real effort lies

Learning Analytics Opportunities

Leverage…• institutional practices and tools in place• interest in using tools for interventions and

student feedback– More systemic and consistent approach to placing

students in appropriate courses and developmental courses.

– Especially for online learning

Explore further…FACT2 Learning Analytics Task Group,

Retention Initiatives

LATG 2013-14 GOALS1. Identify and share known best practices and exemplary uses of Learning Analytics 

for assessment, and early intervention strategies. • Identify the common questions for these areas, and share best practices through web resources and

other communication channels.

2. Develop and conduct professional development activities for use of learning analytics for:

• Assessment, student feedback, and early intervention activities in a course.• Leveraging existing campus data sources to inform strategies for student readiness, course placement,

and remedial education; and to identify what data is readily available and policy guidance in the use of data.

• Identify ways where learning analytics may help to eliminate some administrative burdens while improving academic achievement.

3. Provide opportunities for SUNY faculty to explore Learning Analytics in pilot projects.

• Develop “Proof of concept” projects through IITG using learning analytics approaches/tools. • Develop a Pilot program that would recruit a small group of faculty and courses to implement

assessment strategies based on analytics, and evaluate. (though Open SUNY initiative?)

4. Use the finding from best practices research and pilot projects to identify a course of action for further expansion of Learning Analytics across SUNY.

Adaptive Systems to Pilot or Explore

SUNY Pilot of Learning Analytics Tools

SPSS

QUESTIONS?Visit SUNY Learning CommonsLearning Analytics Task Group

greg.ketcham@oswego.edu