View
703
Download
0
Embed Size (px)
DESCRIPTION
Analytics: as if learning mattered Presentation from 'In Focus: Learner analytics and big data', a CDE technology symposium held at Senate House on 10 December 2013. Conducted by Adam Cooper (Co-Director, Cetis) Audio of the session and more details can be found at www.cde.london.ac.uk.
Citation preview
Analytics(as if learning mattered)
Adam Cooper,In Focus: Learner analytics and big data symposium,University of London, December 10th 2013
Structure of the Talk
• My Perspective
• Threats
• Avoiding the threats
Image: cb
a Jaskirat Singh Baw
a
PERSPECTIVE
Perspective of this TalkWhere am I coming from?
UK
Higher Ed
InstitutionCetis is based in the Institute for Educational Cybernetics at the University of Bolton.
“Our mission is to develop a better understanding of how information and communications technologies affect the organisation of education from individual learning to the global system.”
Defining “Analytics”
Analytics is the process of developing
actionable insights
through
problem definition
and the application of statistical models and
analysis against existing and/or simulated future
data.
Learning vs Business Venn
Insights to support
educational aims and
objectives
Insights to support
operational and
strategic activity.
Learning vs Business Venn
Retention
Student Satisfaction
Employability
Achievement
Applications
Easiest Case forInvestment
Structure to Manage ComplexityThe SMT – “what makes a viable organisation?”
Subject/Discipline Member - “what makes a good chemist?” (knowledge, approach to problems, creativity, attitudes, values...)
Course Team – “what curriculum?” (topics, suitable LOs and assessment methods to match our typical student career etc)
Class Teacher – “where are my students collectively and individually?” (intellectually, emotionally, meta-cognitively...)
Student/Learner – time, place and method of studySelf-organising and self-regulating
The Intersection Doppelgänger
Retention
Student Satisfaction
Employability
Achievement
Applications
Values
Epistemological
Cause & Effect Models
Moral Imperatives
The “Accommodation”“The introduction of these techniques [Learning Analytics] cannot be
understood in isolation from the methods of educational management as
they have grown up over the past two centuries. These methods are
conditioned by the fact that educational managers are limited in their
capability to monitor and act upon the range of states which are
taken up by teachers and learners in their learning activities. ... Over
the years, an accommodation has developed between regulatory authorities,
management and teaching professionals: educational managers indicate the
goals which teachers and learners should work towards, provide a
framework for them to act within, and ensure that the results of their activity
meet some minimum standards. The rest is left up to the professional skills
of teachers and the ethical integrity of both teachers and learners.”Prof. Dai Griffiths, “Implications of Analytics for Teaching Practice in Higher Education”
TWO THREATS(+1 AND A BIT)
Threat #1: Interventionism
Image: via Wikimedia Commons p
Threat #2: BI Tradition
• Centralised projects• IT + senior leader• Management reports• KPIs
Image: Paul H
odge
Image: via W
ikimedia Com
mons p
Hhultgren
So, you want to optimise student success?
• Are the following well-specified?• “optimise student success”• “maximise the probability of success”
• What does success look like?• Really?• Says who?• Is a “good education” indicated by student success?
Does everyone understand?Attainment: an academic standard, typically
shown by test and examination results
Progress: how far a learner has moved between assessment events
Achievement: takes into account the standards of attainment reached by learners and the progress they have made to reach those standards
Enrichment: the extent to which a learner gains professional attitudes, meta-cognition, technical or artistic creativity, delight, intellectual flexibility, ...
+1: Technological Solutionism‘Recasting all complex social situations either as neatly defined problems with
definite, computable solutions or as transparent and self-evident processes that
can be easily optimized—if only the right algorithms are in place!—this quest is
likely to have unexpected consequences that could eventually cause more
damage than the problems they seek to address.
I call the ideology that legitimizes and sanctions such aspirations “solutionism.” I borrow this
unabashedly pejorative term from the world of architecture and urban planning, where it
has come to refer to an unhealthy preoccupation with sexy, monumental, and narrow-
minded solutions—the kind of stuff that wows audiences at TED Conferences—to problems
that are extremely complex, fluid, and contentious … solutionism presumes rather
than investigates the problems that it is trying to solve, reaching “for the answer
before the questions have been fully asked.” How problems are composed matters
every bit as much as how problems are resolved.’
Evgeny Morozov, “To Save Everything Click Here: The Folly of Technological Solutionism”, 2013
3M and Six-Sigma“McNerney axed 8,000 workers (about 11% of the workforce), intensified the
performance-review process, and tightened the purse strings at a company that had
become a profligate spender. He also imported GE's vaunted Six Sigma program—a
series of management techniques designed to decrease production defects and
increase efficiency. Thousands of staffers became trained as Six Sigma ‘black belts’.
...
Now his successors face a challenging question: whether the relentless emphasis on
efficiency had made 3M a less creative company. ... Those results are not
coincidental. Efficiency programs such as Six Sigma are designed to identify
problems in work processes—and then use rigorous measurement to reduce
variation and eliminate defects.
...
Indeed, the very factors that make Six Sigma effective in one context can
make it ineffective in another. Traditionally, it uses rigorous statistical
analysis to produce unambiguous data that help produce better quality,
lower costs, and more efficiency. That all sounds great when you know
what outcomes you'd like to control. But what about when there are few
facts to go on—or you don't even know the nature of the problem you're
trying to define?”Source : Bloomberg Business Week, June 2007.
The Bit
Data:
it may be big
but its not clever
The Bit: Net Scale Fascination (delusion?)
Data:
it may be big
but its not clever
• Hendrik’s photo
Source unknown – thanks to Hendrik Drachsler for passing it on.
Threat Rating=UBI*(I+D)UTS
Where:UBI = Uncritical BI tradition factor
I = Interventionist tendenciesD = Big data fascinationUTS = Uncritical technological solutionism
This is not serious...
Why am I Worried?
From
: “An
alyti
cs in
Hig
her E
duca
tion:
Ben
efits
, Bar
riers
, Pr
ogre
ss, a
nd R
ecom
men
datio
ns”
(ECA
R Re
port
)
NEUTRALISING THREATS
Data Is Useless Without the Skills to Analyze It
‘...employees need to become:
Ready and willing to experiment: Managers and business
analysts must be able to apply the principles of scientific
experimentation to their business. They must know how to construct
intelligent hypotheses. They also need to understand the principles
of experimental testing and design, including population selection
and sampling, in order to evaluate the validity of data analyses.
...
Adept at mathematical reasoning: How many of your managers
today are really “numerate” — competent in the interpretation and
use of numeric data?’Jeanne Harris, HBR Blog, Sept 2012
Is This a Real Problem?
From
: “An
alyti
cs in
Hig
her E
duca
tion:
Ben
efits
, Bar
riers
, Pr
ogre
ss, a
nd R
ecom
men
datio
ns”
(ECA
R Re
port
)
Make Some Progress By:
• Directly investing in specialists• E.g. OU, statisticians and data wranglers• ECAR Survey Report 2012: “invest in people over tools...
hiring and/or training... analysis”
• Operating a “shared service”• Higher Education Statistics Agency (HESA)• Universities and Colleges Admissions Service (UCAS)• Jisc
Data Is Useless Without the Skills to Analyze It
‘...employees need to become:
Ready and willing to experiment: Managers and business
analysts must be able to apply the principles of scientific
experimentation to their business. They must know how to construct
intelligent hypotheses. They also need to understand the principles
of experimental testing and design, including population selection
and sampling, in order to evaluate the validity of data analyses.
...
Adept at mathematical reasoning: How many of your managers
today are really “numerate” — competent in the interpretation and
use of numeric data?’Jeanne Harris, HBR Blog, Sept 2012
We Get Further with “Soft Capability”
• Appreciation of the limitations of analytics technologies
• Expertise to comprehend, critique, and converse(as opposed to originate or implement)
Visualisations are the Opiumof the People
(c)2008, Ryan Block/Engadget
Sparklines: Tufte vs ...
Significance and Sample Size“Flagged for action because 74% of students felt
they were well supported by their supervisor compared to 80% in similar institutions (difference >5%)”
“We need to talk to Mr. Benn; the drop-out rate doubled in his class this year.”
Correlation is not Causation“The predictive model indicates that students with
attributes {X, Y, Z} are only 50% likely to complete.”
What Does This Give Us?(aside from people less likely
to cut themselves on sharp tools)
Absorptive Capacity
“...the ability of a firm to recognize the value of new,
external information, assimilate it, and apply it to
commercial ends is critical to its innovative capabilities. We
label this capability a firm’s absorptive capacity and suggest that
it is largely a function of the firm’s level of prior related
knowledge. [...] We formulate a model of firm investment in
research and development (R&D), in which R&D contributes to
a firm’s absorptive capacity, and test predictions relating a
firm’s investment in R&D to the knowledge underlying technical
change within an industry.” Cohen and Levinthal, Admin. Science Quarterly 35(1), 1990
How to do LA as if learning mattered
1.Build soft capability.
2.Engage in participatory (collaborative) design.
3.Develop & prototype technology (& models)
organically.
4.Reflect on the effect LA innovation has on practice.
5.Design and develop with the educator IN the
system.
6.Optimise the environment, not only the outcome.
7.Be realistic about scale/quality of data and its
meaning.
Personal Learning Analytics?• Changed dyamic• Less “big brother”
Image: cbn Jesse Millan
ParticipatoryDesign
LearningAnalytics
Thank-you for your attention
These slides are at: http://is.gd/AaiLM
Reading and ReferencesJacqueline Bichsel, “Analytics in Higher Education: Benefits, Barriers, Progress, and
Recommendations”Available from http://www.educause.edu/ecar
Cohen and Levinthal (1990), “Absorptive Capacity: A New Perspective on Learning and Innovation” Admin. Science Quarterly 35(1)http://faculty.fuqua.duke.edu/%7Echarlesw/s591/Bocconi-Duke/Papers/C10/CohenLevinthalASQ.pdf
Adam Cooper, “What is Analytics? Definition and Essential Characteristics” http://publications.cetis.ac.uk/2012/521
Jeanne Harris, “Data is Useless Without the Skills to Analyze it”http://blogs.hbr.org/2012/09/data-is-useless-without-the-skills/
Bloomberg Business Week, “At 3M, A Struggle Between Efficiency And Creativity”http://www.businessweek.com/stories/2007-06-10/at-3m-a-struggle-between-efficiency-and-creativity
Dai Griffiths, “CETIS Analytics Series: The impact of analytics in Higher Education on academic practice”http://publications.cetis.ac.uk/2012/532
Evgeny Morozov, “To Save Everything Click Here: The Folly of Technological Solutionism”ISBN-10: 9781610391382
Phil Winne, “Improving Education” (lecture)http://www.youtube.com/watch?v=dBxMZoMTIU4
Licence
“Analytics (as if learning mattered)” by Adam Cooper, Cetis,
is licensed under the Creative Commons Attribution 3.0 Unported Licence
http://creativecommons.org/licenses/by/3.0/
Case studies and white papers:
Cetis Analytics Series,
http://publications.cetis.ac.uk/c/analytics
Me:
http://blogs.cetis.ac.uk/adam
Cetis:
http://www.cetis.ac.uk
Learning Analytics Community Exchangehttp://www.laceproject.info