Upload
vince-kellen-phd
View
301
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Deriving value from analytics requires much more than purchasing technology. University of Kentucky's analytics journey utilized fostering a bottom-up emergent community of practice as well as top-down organizational maneuvers. This presentation shares different aspects of the University of Kentucky score.
Citation preview
Organizing to Ret Analytics RightVince Kellen, Ph.D.Senior Vice Provost, Analytics and Technologies [email protected] is a living document subject to substantial revision! September, 2014
SilosAre recursive
Get reproduced across time and space reliably, without effort
Arise naturally due to human sociological/biological tendencies
It takes constant effort to mitigate their adverse effects
Sharing data and analysis widely requires a reconceptualization of silo structures
2
Organization dysfunction
Information as power
Defensiveness
Data hoarding
Process separation
Empire building
Excessive control
Fear of scrutiny
Loss of power
3
We are competitive animalsInformation becomes a [tool, weapon]
We instinctually manage information to enhance our competitiveness
Competition relies on information hiding
IT tools become part of our body
How we personally utilize information is part of our biological heritage. This is hard to change, if at all
4
Shift from production concerns to consumption ones Production
• Collecting, integrating, cataloging, categorizing, transforming, abstracting, analyzing, model-building, visualization, dashboarding, distributing, publishing. If you build it they will come (hopefully)
Consumption• Motivating, collaborating, expressing, integrating, improving action, increasing ambition, desire, recognition.
If they build it everyone will come
5
A Proposed Analytic Maturity Scale
6
Our process
7
A. Merging of mobile and BI strategy
B. Merging of IR and BI units
C. Super high-speed infrastructure
D. Single analytic value chain
E. Analytics community of practice
F. Data transparency
G. Community sourcing and norming
H. Community rules of etiquette8
Our Community of Practice Rules of EtiquetteBe safe and secure. Respect the acceptable use of information policies and guidelines the university has in place. Please have good passwords and secure your laptop, desktop and other devices appropriately. Treat private student and university information appropriately.
Be collegial. University data is a community asset and a community of people steward the data. Use and share the data with the best interests of the university community in mind. Since parts of our data analysis environment is designed to allow for greater transparency, analysis will potentially be able to see other unit data. While we will make private to a unit what absolutely needs to be private, the way the university runs it's business often involves multiple colleges and units at the same time. Don't use your access to take unfair advantage of another unit.
Help improve data quality. If you see data that doesn't appear to be correct, let someone know. We have a team of staff dedicated to helping improve data quality. This team can work with colleges and units on any data entry and data management processes that might need to be changed to improve data quality.
Be open-minded and inquisitive. Data can be represented in multiple ways at the same time. While the teams are taking great care to enable multiple views of the data to support the community, you might have a valid and unique perspective. In time, we can accommodate more ways of looking at the same data while not interfering with other views or taxonomies.
Share. The main benefit from open analytics is the power of a community of analysts learning from each other rather than a few select individuals hoarding knowledge or access. As the community improves its knowledge and skill with the data, the universitycan improve accordingly.
9
Organizing IT
Our organizational model makes a big difference. Other universities fail to take advantage of a tool like this for purely political reasons
Making key data transparent to all does not help those who made their living being the data ‘go to’ person
We had to merge two units (Institutional Research and Business Intelligence), losing 1/3 of the staff. This let us hire three data scientists with different analytic backgrounds
The tool let the staff transition their skills
10
What we have done and what we would like to do First steps over the past year
• Mobile micro-surveys: Learning from the learner. In one year, 134,458 surveys harvested. Survey response rates are holding at about 40%. We can instantly analyze all responses for retention and progression issues
• Student enrollment, retention, demographics, performance, K-Score, facilities utilization, instructor workload, student revenue and financial aid, student progression and more
• High speed, in-memory analytics architecture. Lowest level of detail, maximum semantic expressiveness, one-second per click for analyst are key design philosophies
• Open data and organizational considerations
Coming down the road?• Micro-segmentation tool to enhance user and IT productivity, develop personalized mobile student
interaction/intervention• Models for learner technographics, psychographics, in addition to behaviors, performance, background• Advanced way-finding for streaming content like lecture capture• Content metadata extraction and learner knowledge discovery• Real-time measures of concept engagement and mastery• Real-time learner recommendations and support engine • Use graphing algorithms to perform more sophisticated degree audit what ifs
11
Our analytics technology infrastructure roadmap 2014-15
12
List builder Iteratively query any/all fields of your choosing, linking in an AND or OR fashion
Combine different lists using SET manipulations
Refresh lists regularly (nightly or otherwise)
Apply the set name as a filter on ALL models
This provides advanced filtering and combining that works regardless of the user interface
Our AA team can build and maintain Lists easily. So can some users
Since lists are refreshed nightly, we can keep track of each time a student (or other entity) as it added or removed from a list
We can develop workflow apps using this. Backend, front-end agnostic
13
List builder example
14
List builder example
15
2009 2010 2011 2012 2013 2014 2015Academic year
8
10
12
14
16
18
20
22
24
26
28
30
32
34
36
Avg
19
109
107
8
107
160
161
118
36
154
55
155
165
47
162
14
19
51
109
162
165
56
Fast/Slow ProgressionStudent headcount
8
50
100
165
Cohort YearFall 2008
Fall 2009
Fall 2010
Fall 2011
Fall 2012
List builder visualization exampleFound all students who take a lot of classes at one point in their career and then took less classes at another point in their career.
Interpretation: These students start with a bang but fade at the finish
How long did this analysis take? Start to finish with this visualization:
25 minutes
16
K-Feed:
Intelligent, personalized alerts, news, reminders
17
Identifying smaller segments of students
In addition to our work on difficult student cases, we needed to find a way to reach a ‘murky middle’ group of students
Identify students who are just as likely to come back as they are not
The predicted reenrollment was about 50%
After interventions, the actual enrollment was about 65%
The whole enchiladaPersonalize learning, learning analytics and IPAS analytics
into one real-time architecture• Real-time personalized interactions
• Target on-demand peer tutoring based on student’s profile• Deliver micro-surveys and assessments to capture additional information needed to improve personalization• Give students academic health indicators that tell students where they can improve in study, engagement,
support, etc.• Let students opt their parents in to this information so the family can support the student • Tailor and target reminder services, avoid over messaging, enable timing of message delivery based on user
temporal proclivities, mix and match messages across learning, support and progression areas
• Allow for open personalized learning• How content gets matched to students is psychologically complex • Several theories of how humans learn give many insights• Students differ in the following abilities and attributes: visual-object, visual-spatial, reasoning, cognitive
reflection, need for sensation, need for cognition, various verbal abilities, confidence, persistence, prospective memory, etc.
• We need an open architecture to promote rapid experimentation, testing and sharing of what works and what doesn’t
University of Kentucky
Herding cats We shared with everyone that we are
building the bridge as we walked on it
We established a community of practice and rules of analysis etiquette
We built tailored objects for colleges, let users choose their own front end tool
We relied on word-of-mouth adoption and some teasing-revealing
Guess what happened?
20
Top-down versus bottom-up Doing this top down is like pushing water
uphill. Its harder than pushing a rock uphill
The great leader is one who the people say “We did this ourselves”
Consider analytics to be a process of self discovery. Each person has to go through the stages of maturity
Paradoxically, this also requires strong top-down commitment and action! Organizational maneuvers like reorganizations are [normally] required
21
Questions?
22