View
100
Download
0
Category
Tags:
Preview:
Citation preview
Tales from
the Frontier
of Higher
Education
Analytics
Vince Kellen, Ph.D.Senior Vice Provost
Analytics and Technologies
University of Kentucky
vkellen@uky.edu
March 4th, 2015
Challenges to higher education
Questions
• What is the purpose of higher education?
– Job training and economic development? An informed citizenry and an antidote to tyranny? Create ‘good’
people? Increase knowledge in civilization? Provide a space for young adults to grow?
• What is the role of government in higher education?
– Is it a private good with access being managed by the market? Is it a public good with access provided by the
people for the people?
Facts
• Pay gaps are widening, skills are hollowing out
– The gap in economic prospects between those without college, those with an undergraduate degree and those
with advanced degrees are diverging.
– Growth in technology is placing new demands on job skills needed in the future
• The price of education, as felt by the public, is increasing
– Increased regulation and student support requirements, diminishing funds from government for education and
research, stagnant wage growth for middle-income families
– Some public colleges are considering ‘going private’ or are nearly there
2
Higher education has a ‘last mile’ problem
Education in any form is
struggling to address families
and communities with
economic and other readiness
problems
Free or low-cost educational
content does not easily solve
readiness problems which have
a multitude of causal factors
For profit models rightfully
struggle with ‘last-mile’
problems. Public policy
matters!
3
What would Abraham Lincoln think of eLearning?
Abraham Lincoln
• Autodidactic
• Books, books, books
• Became a skilled military strategist
• Penchant for poetry, Shakespeare,
politics and history
My nephew
• Not an autodidact
• Good worker, smart kid, but…
• It takes a village
• After a few low-security colleges
and much money borrowed
• He has found an intellectual home
• And a good occupation
4
Volume operations versus complex systems
Excluding the late 20th
century, universities
have been largely
complex systems,
delivering niche and
customizable
interactions F2F
settings. Large
lectures were added
to increase output
while reducing costs
Personalization
technology and e-
Learning approaches
can begin to handle
both high-volume and
specialty classes
5
Geoffrey Moore (2005). Dealing With Darwin
Given all the challenges in higher
education, nearly all roads to
improvement lead through
information technology
6
How technology can help in student success
7
High
effectiveness
Low
effectiveness
Low volume High volume
Small F2F class
Broadcast class
Current eLearning approach
PT + F2F
F2F = Face-to-face
PT = Personalization technology, adaptive learning technology
Some of our student analytic models
8
Model Description
Enrollment Enrollment in a class, midterm and final grades, credit hours attempted and earned, instructor teaching the class,
class level, room building, time and meeting pattern
Student retention and
graduation
Student demographics and cohort identification (e.g., John Doe is in the 2009 entering first-year student cohort),
fall-to-spring, 1, 2, 3, 4, 5 year retention rates; 4, 5, 6-year graduation rates, incremental retention rates
Student demographics Demographics, such as age, high school GPA, entrance test scores (SAT, ACT) and subcomponent scores. Also,
in a secure location, additional personally identifiable demographic details such as name, address, email, financial
aid and unmet need
Student performance Present the enrollment data in such a way as to easily show the student’s performance for each term, including
credit hours earned, term GPA, cumulative GPA for that term, etc., midterm and final grades, academic progress
Student academic career Keep a list of the majors and minors for each student and degrees awarded. Also, include details on students who
transfer in and out, including transfer institution, credit hours transferred in, etc.
Productivity The room utilization model contains every building, every potential classroom and lets users analyze the room
capacity and enrollments for the class or event in the room at five minute intervals, how far students walk to/from
classes. The faculty stats per term model pulls together the number of students and sections taught per term and
will contain other important data such as research expenditures per term and grant proposals submitted and won
Micro-surveys Capture questions and answers from the My UK Mobile micro-survey feature
Student involvement Interaction history with various student software including the learning management system, clickers, course
capture and playback, academic alerts, student organizations, advising interactions, mobile usage
Analytics at the University of Kentucky
Our goal was to utilize high speed analytics to help us improve student retention and
graduation rates
We had to alter our own thoughts about analytics, data warehouse construction and in-
memory computing. We spent one year in DENIAL about the capabilities of the tool
Our approach utilized ‘democratizing’ the data, engaging and fostering a community of
analysts, outside of central IT and institutional research
We developed a ‘single source of truth’
We focused on high-speed analytics to make the analyst experience enjoyable, allowing
aggregation at the highest level with drill-down into the lowest level quickly (within a
second)
After two years, we have collected nearly all data from our current systems and we expect
to gather the rest in the next year9
What does if feel like having high-
speed analytics for nearly all data
available that is easy-to-analyze,
easily sharable, quick to alter or
adjust and of sufficient quality?
10
Mobile & micro-surveys
We connected analytics to our
mobile app, envisioning
personalized messaging
We deliver simple 1-question
surveys with the ability to
trigger message responses
and interactions
In 17 months, we have gotten
162,891 surveys, averaging
about 45% response rates
12
17
Iteratively query any/all fields of your
choosing, linking in an AND or OR
fashion
Combine different lists using SET
manipulations
Refresh lists regularly (nightly or
otherwise)
Apply the set name as a filter on ALL
models
This provides advanced filtering and
combining that works regardless of
the user interface
Our AA team can build and maintain
Lists easily. So can some users
Since lists are refreshed nightly, we
can keep track of each time a student
(or other entity) is added or removed
from a list
We can develop workflow apps using
this. Backend, front-end agnostic
List builder
List Builder
18
Found all students who take a lot of
classes at one point in their career
and then took less classes at
another point in their career.
Interpretation: These students start
with a bang but many fade at the
finish
How long did this analysis take?
Start to finish with this visualization:
25 minutes
Targeted student interactions
In addition to our work on
difficult student cases, we
needed to find a way to reach a
‘murky middle’ group of
students
We identified students who are
just as likely to come back as
they are not
The predicted reenrollment was
about 50%
After interventions, the actual
enrollment was about 65%
Analytics in higher
education: Déjà vu?
The “long tail” concept applies
We have classes than enroll large
numbers of student, usually with
dozens of sections and frequently
with very large lectures
We also have many smaller
classes representing interesting
knowledge niches
These niches are one of the
reasons students globally come to
U.S. higher education
How can technology help students
and instructors in these classes?
22
http://www.thelongtail.com/conceptual.jpg
MOOCs
Large lectures
PHI 698
???
Taxonomy? Automatic metadata? Automatic atomic metadata?
Let learners more easily navigate an audio/visual
stream based on keywords
Let the system learn what are top terms. Let the
system map terms to concepts. Let instructional
designers lightly ‘bump’ the taxonomy, post
production
Record student engagement with and mastery over
specific terms / concepts
Deliver personalized messages to students,
detecting frustration, interest
Guide students with recommendations and
prompting, promoting active learning
23
See http://p.uky.edu
Can we help learners build proper mental models?
24
Natural language processing
(NLP) and neural-networks. Can
these classes of technology:
Extract concepts from audio
and text?
Find highly relevant articles,
scientific papers and
discussions regarding these
concepts?
Help learners construct and
validate their knowledge?
What will instructors do?
University of Kentucky
Personalized messages
in learning systems
Many classes integrate different
content within one experience
The following features are
becoming more common
• Use of video to replace or
augment lectures
• Backchannel ‘tweeting’ for live
student questions
• Use of mobile apps to answer
simple questions in class
• Students sharing notes, rating
content effectiveness
With high-speed analytics, we now
have a window into the mind of the
student, real-time
Analyze this clickstream to deliver
personalized messages
Confusion detector
In the middle of working online, a student searches for
“dorsolateral prefrontal cortex” in the online course content,
replays the associated segment in the video lecture and has
gotten 4 out of 5 related quiz questions incorrect.
1. Through content analysis, real-time analytics detect the
confusion regardless of the type of content and notifies the
peer tutor.
2. The real-time analytics service lets the student know that a
peer tutor is available and will be calling
3. The peer-tutor and the student discuss the concept
“dorsolateral prefrontal cortex” and arrange to meet later to
explain
4. The real-time analytics service places a entry in the
students advising log and notifies the faculty
RAMCaliper
“Uber Tutor”
Person-to-person
Confused student
Real-time analytics Peer tutor
How can we personalize? Let me count the ways…
General cognitive attributes, such as the composite ACT score or subcomponent math and verbal scores, high
school GPA, performance in prior classes
Economic attributes such as level of unmet financial need, level of tuition, prior tuition payment timeliness
Engagement levels such as the learner’s interactivity in the class relative to their peers
Non-cognitive factors, such as their level of conscientiousness, effort-regulation, planning, which affect
performance in classes
Class-level engagement and mastery including concept-level engagement and mastery, mastery over early
tests (including midterms) and participation in discussions, blogs, etc.
Co-curricular engagement including level of activity in student clubs, events on campus, advising services
Physical behavior, including how far they have to walk between classes, what dorm do they live in, etc.
Social network, including who else participates in clubs, discussions and classes with the student
General survey data, including mobile surveys
27
RAMCaliper: Universities defining real-time clickstream events
Purpose
• To allow the university to deliver real-time personalized messages in response to specific
learning system events to the learning system(s) environment(s) via a personalized messaging
region and to any other interaction channel the university utilizes including but not limited to
mobile applications, SMS messaging, email and delivery of messages in the university portal.
• To allow the university to capture data regarding student engagement with the learning
system(s) and add this data to a real-time analytic infrastructure.
• Partners: IMS Global, APLU PLC, about 15 universities
Specifics• API will need to send messages, ideally through web‐based modern streaming approaches. Specifically, messages should be available to
subscribers within 100‐1500 milliseconds, in what is commonly referred to as a publish/subscribe system. A web‐based RESTful "pull"
API, should be used to control the publish/subscribe system. The API need not make available the entire clickstream. It only needs to
make available the "terminal" action coinciding with completion of the important student and faculty events. As described, this results in a
few events per minute per user. This will allow institutional back‐end system to submit timely messages either to the learning system
environment or any other student interaction interface available in response to events originating in the learning platform. The events do
not need to persist indefinitely. Ideally, events will persist for 48 hours and will then be removed from originating streaming event system.
Our intent is to persist the messages within the institutional analytic infrastructure.
28
Is there an opportunity for a real-time event stream broker?
Technical value: Do we need a value-added network (a provider) to make it easy for everyone to connect?
Contractual value: Can this provider enforce contractual obligations regarding data privacy and sharing?
29
RAMCaliper
Hub
Vendor
B
Vendor
A
Vendor
D
Vendor
C
Institutions
The whole enchilada
Personalize learning, learning analytics and IPAS analytics
into one real-time architecture
• Real-time personalized interactions• Target on-demand peer tutoring based on student’s profile
• Deliver micro-surveys and assessments to capture additional information needed to improve personalization
• Give students academic health indicators that tell students where they can improve in study, engagement,
support, etc.
• Let students opt their parents in to this information so the family can support the student
• Tailor and target reminder services, avoid over messaging, enable timing of message delivery based on user
temporal proclivities, mix and match messages across learning, support and progression areas
• Allow for open personalized learning• How content gets matched to students is psychologically complex
• Several theories of how humans learn give many insights
• Students differ in the following abilities and attributes: visual-object, visual-spatial, reasoning, cognitive
reflection, need for sensation, need for cognition, various verbal abilities, confidence, persistence, prospective
memory, etc.
• We need an open architecture to promote rapid experimentation, testing and sharing of what works and what
doesn’tUniversity of Kentucky
What we will be working on this year: Five ‘Easy’ Use Cases
1. Simple, event-driven, real-time personalized messages in classes
• We give recommendations to learners in response to real-time comments, searches, and navigation through educational
content and online discussions, to their system of origination or any other channel to which we have access (e.g., mobile)
2. Guiding learners as they “map” out concepts in order to master them
• In the middle of students actively ‘drawing’ relationships between terms, we can reduce their search time for relevant
information while giving them feedback on how close or far they are from understanding the relationships between key
concepts in the class
3. Peer-to-peer ‘Uber’ tutoring service
• We can let the system detect student difficulties with specific skills in class and automatically pair them up with more
advanced students for just-in-time, mobile-phone assistance 7X24
4. Real-time performance and engagement analysis
• We can move beyond ‘point-in-time’ learning analytics and instead provide analysis of student engagement with and
mastery over critical class concepts across multiple interaction channels
5. Enhanced face-to-face advising interactions
• We can provide human experts with richer information about the learner to help the expert in their F2F advice
31
What will faculty do?
First, we can stop giving passive lectures, which
is educational malpractice
We can do more of the following:
• Monitor student engagement with the material
• Analyze student mastery in real-time
• Provide for more small group interactions
• Give academic career advice
• Connect students to other faculty and disciplines
• Engage students in research opportunities
• Instead of “sage on the stage” become “guide on
the side”
• Meet 1:1 to explain difficult material, probe the
student’s ability, give face-to-face feedback
• Motivate and inspire students
Return to our roots!
32
Organizing the IT unit
Our organizational model makes a
big difference. Other organizations
fail to take advantage of new tools
like this for mostly political reasons
Making key data transparent to all
does not help those who made their
living being the data ‘go to’ person
We had to merge two units, losing 1/3
of the staff. This let us hire three data
scientists with different analytic
backgrounds
The tool let the staff transition their
skills easily
33
Herding cats
We shared with everyone that we are
building the bridge as we walked on it
We established a community of practice and
rules of analysis etiquette
We built tailored objects for colleges, let
users choose their own front end tool
We relied on word-of-mouth adoption and
some teasing-revealing
Guess what happened?
34
Analytics Community of Practice (CoP) Principles
1) Be safe and secure. Respect the acceptable use of information policies and guidelines the university has in place. Please
have good passwords and secure your laptop, desktop and other devices appropriately. Treat private student and UK
information appropriately.
2) Be collegial. University data is a community asset and a community of people steward the data. Use and share the data with
the best interests of the university community in mind. Since parts of our data analysis environment is designed to allow for
greater transparency, analysis will potentially be able to see other unit data. While we will make private to a unit what
absolutely needs to be private, the way the university runs its business often involves multiple colleges and units at the same
time requiring broad data access. Don't use your access to take unfair advantage of another unit.
3) Help improve data quality. If you see data that doesn't appear to be correct, let someone know. We have a team of staff
dedicated to helping improve data quality. This team can work with colleges and units on any data entry and data
management processes that might need to be changed to improve data quality.
4) Be open-minded and inquisitive. Data can be represented in multiple ways at the same time. While the teams are taking
great care to enable multiple views of the data to support the community, you might have a valid and unique perspective. In
time, we can accommodate more ways of looking at the same data while not interfering with other views or taxonomies.
5) Share. The main benefit from open analytics is the power of a community of analysts learning from each other rather than a
few select individuals hoarding knowledge or access. As the community improves its knowledge and skill with the data, the
university can improve accordingly.
35
Top-down versus bottom-up
Doing this top down is like pushing water
uphill. Its harder than pushing a rock
uphill
The great leader is one who the people
say “We did this ourselves”
Consider analytics to be a process of
self-discovery. Each person has to go
through the stages of analytic maturity
Paradoxically, this also requires strong
top-down commitment and action!
Difficult organizational changes are
[often] required37
Recommended