Conducting Rigorous Evaluations of Interventions That (May) Improve Student Learning: A...

Conducting Rigorous Evaluations of Interventions That (May) Improve Student

Learning:A Researcher/Teacher Reflects

Jon R. StarMichigan State University

Harvard University (as of July 2007)

2

About me

• Former middle and high school mathematics teacher

• PhD in Educational Psychology, with emphasis on middle/high school students’ learning of mathematics

• Two grants from US Dept. of Ed (IES) exploring interventions designed to improve students’ learning of mathematics

3

Why I was asked to speak today

• Perhaps useful for me to share my experiences as a researcher who designs and conducts rigorous (experimental) evaluations of interventions designed to improve students’ learning of mathematics

• What are the challenges that I face in my work that I feel are endemic to the evaluation work that many of you are trying to do?

4

Research examples (1)

• The benefits of comparison for learning mathematics

• Better to solve problems by viewing worked examples sequentially or side by side?

Example 1

Example 2

p.1

p.2

Example 1 Example 2p.1

5

Research examples (2)

• Learning strategies for solving ratio and proportion problems using schema

• Is using schema is better than not using schema?

?

?= ?

Ratio Problem Schemata

? ?

? ?

If Then

Proportion Problem Schemata

6

Two types of challenges

• Tension between: my role as a researcher, and my heart and soul as a teacher

• Balancing between: the need to design the intervention to be very

‘clean’ to enable an experimental study, and the need to provide experiences for students

that are appreciative of good (and sometimes ‘messy’) instructional practices

7

Researcher vs. teacher

8

Flow of intervention

• Let intervention run its course Give help in standardized way Don’t deviate from the instructional

protocol or script

• If students don’t ‘get it’, spend extra time with them Focus on mastery of the material Spend extra time with struggling

students

9

Scorecard

Flow of intervention

=

10

Assessing learning

• Materials need to discriminate and have room for learning OK with pretest scores of 10% OK with posttest scores of 50%

• Materials should be challenging but students should be able to succeed ‘Failing’ tests is disheartening Average posttest grade of a B?

11

Scorecard

Assessing learning

Flow of intervention

=

12

Relations with schools

• Keep professional distance from schools and teachers Don’t want to ‘spill the beans’ so that

we can use the school in future work Don’t tell if study didn’t work

• Establish deep and long-lasting relationships with schools, teachers, and students Researchers shouldn’t zip in/out Go back and visit; share results

13

Scorecard

Relations with schools

Flow of intervention

Assessing learning

=

14

Ethics of study design

• Experimental design tells us the most about the intervention Control group is necessary There may be “winner” and “loser”

conditions

• All conditions should lead to student learning Design should allow non-treatment

students to experience treatment eventually (if successful)

15

Scorecard

Ethics of study design

Flow of intervention

Assessing learning

Relations with schools

=

16

Two types of challenges

• Tension between: my role as a researcher, and my heart and soul as a teacher

• Balancing between: the need to design the intervention to be very

‘clean’ to enable an experimental study, and the need to provide experiences for students

that are appreciative of good (and sometimes ‘messy’) instructional practices

17

Clean vs. messy

18

Instructional format

• Students working alone is an easier design Logistically Statistically

• Students working with partners is often better for learning Also can help teachers get more

comfortable with this instructional format

19

Scorecard

Instructional format

=

20

Instructional delivery

• Researchers providing instruction is easier: no PD consistency across schools

• Letting students’ regular teachers do the teaching is better: we are unfamiliar with norms we don’t know students’ names

21

Scorecard

Instructional delivery

Instructional format

=

22

Relations with teachers

• Better if teachers don’t know too much about the intervention might bias their interactions with

students

• The best way to have a long term impact on students is by working with teachers on instructional improvement

23

Scorecard

Relations with teachers

Instructional format

Instructional delivery

=

Thanks!

Jon Starjonstar@msu.edu

www.msu.edu/~jonstar