Embed Size (px)
Citation preview
Developing evidence-based strategies to support improved
organisational performance in teaching and learning
Building Futures: Hindsight – Insight – Foresight
National Learning and Teaching Conference, Nelson October 2012
Peter Coolbear
October 2012
The challenge
Achieving synergies
between
individual improvement of practice
and
organisational change
for
the benefit of tertiary learners
October 2012
Plan of presentation
• The context – an inefficient tertiary system which works
well for many learners but very poorly for others
• The need for evidence-based improvement
• What constitutes good evidence?
• Ensuring our projects contribute to sustainable change
• Changing expectations of 21st century vocational
educators
October 2012
The context …
• Significant advances in course and qualification
completion rates, but still wide variation:o Course completions <50% - 100%
o Qualifications 12% - 100%
• Lack of parity of success for Māori and Pacific
learners – what’s the gap in your organisation?
• Low progression to success at higher levels of study
• In creasing lack of confidence about consistent
academic standards
• The inevitability of continued change
• Projects designed to support evidence-based
improvement of practice
October 2012
Since its establishment in 2007, Ako Aotearoa has
provided $5m + funding for 190 projects to
improve tertiary teaching and learning
We’ve learnt a lot!
October 2012
What we’ve learnt …
• We work within a system that allows great teaching and
learning to happen
• We are not very strategic about improving tertiary teaching
• We do not do enough to share good practice
• We work within a system that allows the mediocre (or worse)
• We work in a system that is highly fragmented
• The NZ research base in tertiary education is still quite weak
and has limited impact on practice
Variability of learner outcomes is the immediate problem,
but is not the only fix
October 2012
Excellence
Threshold of
acceptability
Led by
government
Led by the profession and
organisations, enabled by
government
What the Minister sees
as the present state
What also needs to be done
• Tertiary education offers opportunities for empowerment
• for individuals
• for whanau
• for communities
• for New Zealand as a whole
October 2012
Why is it so important?
It’s about moral purpose ......
It’s also about New Zealand’s international reputation
October 2012
Why is evidence of learner benefit fundamental?
October 2012
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
Qu
laif
icati
on
Co
mp
leti
on
Rate
Providers
Variation in qualification completion rates for Level 1-3
students within public tertiary providers
Data for 2009 (TEC)
October 2012
Practice
Policy
The policy - research - practice disconnect
• High performing practitioners and tertiary education
organisations continually test their assumptions
about how and how effectively their learners are
learning
October 2012
Evidence-based enhancement of teaching and
learning
Partnership and continuing dialogue between
professional practitioners and their organisations
• For 70% + traditional practice will not change
without evidence that something else is better
[aid theory]
• Organisational decisions to redirect resources will
(hopefully!) not be made without evidence
[no theory]
October 2012
Testing assumptions and effective dialogue relies on
robust, easily understood evidence
• Focussed on student achievement and outcomes o For the individual learner
o For the employer
o For the community
• Uses robust, transparent, purposeful and well
validated data collection techniques
• Uses mixed methods (qualitative + quantitative)
• Is action focussed
October 2012
Characteristics of sound evidence
October 2012
Meaningful critical self-reflection
Experienced practitioners often do this intuitively and
informally….
“ Well that went well: there was some really great interaction in
that class and the formative assessment demonstrated that
the group really understood those key concepts”
“ There’s some feedback here from their employers that Joe and
Lisa had some difficulties setting the levels on that formwork
correctly, yet both breezed through the class exercise. Clearly
I need to do something different to help them translate theory
into practice.”
October 2012
Evidence-based Enhancement of Teaching and
Learning
A Tertiary Practitioner’s Guide
to Collecting Evidence of
Learner Benefit
Individuals
Organisations
October 2012
Organisations need to do this more purposefully and
formally
What works?
What works most effectively?
Can we make it business as usual and sustainable?• What’s the RoI?
• Is there anything we can stop doing if we do this
instead?
• What are my levers to ensure this happens in the way I
intend?
October 2012
So what do we do?
• Systematise the informal evidence
– Students’ stories have validity
– Student contributions to class discussion
– Student preferences for different activities
– Employer feedback
– Alumni feedback
October 2012
So what do we do?
• Develop new sources of evidence
– Student engagement – e.g. AUSSE
– Critical incident questionnaires
– Research based in cognitive psychology
– Graduate surveys
– Employer surveys
October 2012
An example of robust and rich evidence collection:
• Collects data from students about how students are learning
• Has a very strong research base
• Used by all Australian Universities and now all New Zealand
Universities + pilots in NZ ITPs and PEPs
• International benchmarking against the NSSE used in North
America
• Most highly validated, data-rich student survey instrument
available to tertiary education
• Student satisfaction is just one dimension of thirteen others
• Specifically designed to start organisational conversations.
October 2012
October 2012
So what do we do?
• Locate evidence and projects firmly in an
organisational context.
Guide to change projects
October 2012
• Creating sustainable
change to improve
outcomes for tertiary
learners
(Alkema, 2012)
Changes to our project funding model
October 2012
• Ako Aotearoa funds change projects
• i.e. we fund evidence-based change projects with a
high potential to benefit learners.
• Co-funding model - 50% contribution (or more) by the
organisation
• i.e. we require organisational commitment to change
practice
• We will measure the impact of projects
• We expect to create PD resources from high impact
projects
Logic sequence for sustainable improvement
October 2012
Identify
need /
opportunity
Create
intervention
Investigate options
Practice change
Student
response: do
they like it?
Student success
on course(s)
Better
student
outcomes
Change
becomes
BAU
SHARE
EXTERNALLY
October 2012
So do Ako Aotearoa projects benefit learners?
Impact evaluation of our own projects in providing
learner benefit:
October 2012
• 63 successfully completed small projects
• Interest generated
• Impact on practice
• Evidence of learner benefit associated with the work
• Impact on project team
• Outcomes hierarchy.... direction of travel
October 2012
Pragmatic choices
Evidence that identifies the most effective
interventions when providers need to make
choices
Exemplar project: Dedicated education unit
October 2012
• Project: Piloted 2 DEUs at
Middlemore hospital using action
research
• http://akoaotearoa.ac.nz/nursing-deu
Vision was to change
the way nurses
were supported
as they learnt
Exemplar project: Dedicated education unit
October 2012
• What’s changed? They are now business as usual
• 9 DEUs established, 2 more in 2013 (covers
between 80-90% of nursing students at MIT)
• exploring possibility of multi-disciplinary unit
• Key features
• Robust design, strong organisational buy-
in, focused on change for learners, broad
influence
The key output from the dedicated education
unit project is not the research report
October 2012
It’s the practice guide
Exemplar project: ITO assessment systems
October 2012
• Project: examined assessment systems for industry
training
• http://akoaotearoa.ac.nz/projects/ito-workplace-
assessment-structures
Vision was to create
system level
change in
assessment
practice
Exemplar projects: ITO assessment systems
October 2012
• What’s changed?
• Principles have contributed to organisational wide
practice in 5 ITOs (impacts ~23,000 learners)
• Workshops and new project with BCITO illustrate
how change can be achieved
• Key features
• Issue of critical importance, research based
principles, structure created to embed within
organisational practice
Professional development workshops and
resources
October 2012
October 2012
Organisation
Explicit aspirations for their learners
Practitioners
Individual (and team) aspirations for their
own practice and for their learners
Evidence
about what
works
Mutual
preparedness to
innovate and
change practice
Mutually
congruent
expectations
October 2012
Are we supporting change
to improve what we do now
or
are we fundamentally changing
the way we do things?
The last big question:
October 2012
The changing nature of 21st century tertiary education
October 2012
So what are the characteristics of a high
performing professional tertiary teacher?
These will stay the same:
• Empathy
• Reflection-in-action
• Critical self reflection-on-action
• Emancipation
October 2012
Plus what for the future?
What are the critical shifts?
October 2012
• A focus beyond the boundaries of courses and their
programmes – tighter integration between programmes of
study and work
• Changing expectations about Return on Investment
• Changing expectations about inclusivity
• New technologies allow greater learner autonomy and much
greater expectations around meaningful student centricity
• Open source materials will change the dynamics of delivery
Where is this thinking coming from?
October 2012
Wide range of different conversations about 21st
education and educators, but more recently:
• DEANZ Conference
• Metro ITPs Graduate Diploma in Tertiary Education
• HERDSA Graduate Attributes
• Te Ara Whakamana
• Tertiary Teaching Excellence Awards portfolios
• Ako Aotearoa’s work on its own business model
What are the norms of 21st century education?
October 2012
• Focus on successful outcomes for all learners in their
classes
• Highly culturally competent
• Capability focus rather than content / competence
focus:
“ I teach less content so that my students learn more”
Zoe Jordens
October 2012
More specialist (e.g. learning design / assessment /
curriculum development / open source retrieval)
The 21st Century tertiary educator will spend much less
time developing resources and much more time facilitating
learning and providing facilitative feedback to learners
The 21st Century tertiary educator’s practice will be
increasingly framed by how their students use on-line
technologies
The 21st century educator’s practice will be informed by the
latest understandings in cognitive psychology
What are the changes in skill set for the faculty?
October 2012
It will host, support and develop 21st century educators
who are dual professionals
It will own (and, in turn, expect employee ownership of)
successful learner outcomes
It will seek to disestablish the resource management –
educational performance divide
What are the changes in skill set for the tertiary
education organisation?
October 2012
It will become a learning organisation
What are the changes in skill set for the tertiary
education organisation?
October 2012
Organisation
Explicit aspirations for their learners
Practitioners
Individual (and team) aspirations for their
own practice and for their learners
Evidence
about what
works
Mutual
preparedness to
innovate and
change practice
Mutually
congruent
expectations
October 2012
www.akoaotearoa.ac.nz
Thank you
