Black, Adam Dr - Efficacy and how to improve learner outcomes

Efficacy and how to improve learner outcomes

EAQUALS Conference

April 25, 2014, Budapest

Dr. Adam Black

Efficacy and Research, Pearson (Professional)

Overview

• Defining Efficacy

• Building a path to Efficacy

• Efficacy Reviews: a framework and lessons learned

• Efficacy Studies: holistic measures of impact

• Efficacy Analytics: global trends

• Do an Efficacy Review yourself

• Q&A

Efficacy and improving learner outcomes, EAQUALS April 2014 2

Context: what are we trying to improve

and why now?

Context

―No program can be evaluated properly without a common

understanding of what it’s supposed to achieve.

An unfortunate consequence of treating purposes casually is

a tendency to accept goals that seem important in theory

without pausing to consider whether it is possible to achieve

them within the time available.‖

Our Underachieving Colleges

Derek Bok (former President, Harvard University)


―I have been struck by how important measurement is to improving the human condition. You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal—in a feedback loop.‖

Bill Gates, Jan 2013

Context

Why now?

• There is a shared understanding that high-quality education drives personal, economic and societal growth

• Governments, individuals, employers and institutions recognise the need to deliver high-quality learning

• New technology makes it increasingly possible to see what works and what doesn’t in helping learners to achieve their goals


What do we mean by efficacy?

A measurable impact on learner outcomes


efficacy (dictionary definition)

• ability to produce the intended result

Efficacy (Pearson’s definition)

• make a measureable impact on learner outcomes

efficiency (dictionary definition)

• achieve maximum productivity with minimum wasted effort

Improving Efficacy: how are we going about it?

Pearson’s path to efficacy: three integrated activities for improving learner outcomes

research design and

development piloting

full deployment

customer use

Efficacy Analytics: mine data from products to gain insights into iterative improvements, learner behaviours and future innovation

Efficacy Reviews: predict likelihood of impacting learner outcomes and plan improvements

Efficacy Studies: learn from long-term holistic studies of outcomes

An Efficacy Framework: how likely is it that your project will successfully

improve learner outcomes?

Criteria area Rating Rationale summary

• Action plan

• Governance

• Monitoring and reporting

• Internal capacity and culture

• User capacity and culture

• Stakeholder relationships

Outcomes

• Intended outcomes

• Overall design

• Value for money

• Comprehensiveness of evidence

• Quality of evidence

• Application of evidence

Evidence

Planning and implementation

Capacity to deliver

Efficacy

An Efficacy Framework: likelihood of impact

An Efficacy Framework: an explanation of ratings

Good – requires slight refinement, but on track

Mixed – some aspects require attention, some solid

Problematic – requires substantial attention, some require urgent rectification

Off-track – requires urgent action and problem solving

Ratings are not grades on performance

Ratings prompt discussions that lead to actions

Ratings prioritise and suggest timeline


Criteria area Rating Rationale summary

• Action plan

• Governance

• Monitoring and reporting

• Internal capacity and culture

• User capacity and culture

• Stakeholder relationships

Outcomes

• Intended outcomes

• Overall design

• Value for money

• Comprehensiveness of evidence

• Quality of evidence

• Application of evidence

Evidence


Capacity to deliver

Efficacy

An Efficacy Framework: likelihood of impact

An Efficacy Framework: a deep-dive on outcomes

Overall design

• Is the product designed in a way that will most effectively help your target group reach their goals?

• Does the design allow you to automatically collect evidence of your progress?

• Have you adapted the design based on feedback from users?

• Could the design by used by others?

Value for money

• Do you understand the benefits of your product/service to your target group? Relative to other options?

• Is the cost of the product/service competitive, considering the benefits it would deliver?

Intended outcomes

• Have you identified specific outcomes for your target group?

• Do you have a way to measure the intended outcomes?

• Do you have ambitious and measurable targets in place, and deadlines for achieving them?

• Are your intended outcomes clearly documented and understood by your team and customers?

Example of green rating Example of red rating

• All outcomes are specific and clearly documented.

• People within and outside my organisation understand the intended outcomes and can communicate them clearly.

• Future targets are ambitious and achievable.

• Outcomes can be regularly measured against set targets.

• Design is superior to other options/competitors with features focused on delivering outcomes.

• Real-time evidence is generated.

• The design can be adapted and developed.

• Others could use this design, and it has been shared with them.

• Feedback/research has identified what benefits the product/service needs to deliver to users.

• Feedback and return-on-investment research shows that the cost of the product/service reflects the benefits.

• Outcomes are not documented or specific.

• People within and outside my organisation do not understand the intended outcomes or communicate them in the same way.

• Targets do not exist to measure outcomes against.

• Outcomes are only defined at a high level.

• No feedback from users (formal or informal), and benefits of using the product/service are unclear to our team and users.

• Perceptions of value for money and user experience are poor.

• The design does not meet target group expectations and is difficult to use.

• The design does not reflect intended outcomes.

• The design does not allow for the collection of feedback.

• The design is specific to a local situation and cannot be replicated.

An Efficacy Framework: in action

Review of evidence

• Strategy papers • Customer feedback • Audits • Progression research • Policy briefs

Internal interviews

• Sales • Strategy • Marketing • Planning • Executive leadership

Customer and stakeholder interviews

• Government bodies • Universities • Potential employers • Associations

Efficacy workshop

Outputs

Assessment of current efficacy

Actions needed to enhance efficacy

Highly collaborative and focused on improvement opportunities

Framework area Initial review

3- month estimate

6-month estimate

Comment

Outcomes

Intended outcomes After 6 months, outcomes and metrics will be clear and will influence design. Value for money will be tested from pilots.

Overall design

Value for money

Evidence

Comprehensiveness of evidence After 6 months, the plan to develop the forward evidence base will be finalised and initiated. Quality of evidence

Application of evidence


Action plan After 6 months, long-term plans and reporting structures will be in place and governance agreed. Reporting will be at an early stage.

Governance

Monitoring and reporting

Capacity to deliver

Pearson capacity and culture After 6 months, Capacity issues will be clear, pilots delivered and lessons learned and applied. Stakeholder relationships plans will be launched and gathering feedback.

Customer capacity and culture

Stakeholder relationships

An Efficacy Framework: driving improvement


An Efficacy framework: lessons learned

1. You won’t improve a learner outcome you can’t define clearly!

2. You can’t demonstrate you’re improving a learner outcome if you’re not measuring it!

3. Appropriate learner outcomes vary by age, stage, and situation (school, college, private language school, corporation)

4. To improve learner outcomes, stakeholders must be aligned to the same goals (tutors, administrators, education authorities, etc.)


Efficacy Studies: holistic studies of outcomes

Efficacy Studies: holistic, long-term studies with specific learners, teachers, and institutions


Efficacy (Learning) Analytics:

learning from Big and Small data

Identify common learner difficulties Personalise learning Optimise learning by L1

Research learner behaviours that lead to

success (machine learning)

Improve learner engagement

(activity design)

Predict learners who will fail for early intervention

(predictive algorithms)

Efficacy Analytics: insights into learning behaviours


both students have the same net score

0 100 200 300 400 500 600

-25

-20

-15

-10

-5

0

5

10

Ne

t S

core

Responses Submitted by Student

0 100 200 300 400 500 600

0

50

100

150

200

250

300

Student 57

Fractal D = 1.60

Ne

t S

co

re

Responses Submitted by Student

fractal alert: alert teacher and learner to intervene

responses submitted over course

student who will succeed - smooth Fractal = 1.60

student who will fail or not complete – noisey Fractal = 1.94

Efficacy Analytics: identifying learners at risk

Patent awarded 2013


Want to do an Efficacy Review yourself?

http://efficacy.pearson.com

Efficacy framework: try it yourself




Want more?


Questions and answers…

Contact: [email protected]

Education

Black, Adam Dr - Efficacy and how to improve learner outcomes