How do you know your product

“works”?And what does it mean for a product to “work”?

www.SEGMeasurement.com

SEG Measurement WWW.SEGMeasurement.com

Purposes and Uses of Evidence

Common Uses of Evaluation Evidence In Process Product Design and Modification Gathering “proof” that the product works Understanding how teachers and students use the product Complying with grant or foundation giving requirements Supporting product adoption decisions from piloting

Understand your purpose for collecting information at the beginning. Design your evidence collection in a way that is consistent with those

goals.

SEG Measurement WWW.SEGMeasurement.com

Sources of Evidence and Ways to Collect It Sources of Evidence

Students Teachers Administrators Parents Experts or other third parties

How to collect evidence Pre/Post Testing (Cognitive and

Affective Anecdotes Surveys Interviews Observation Focus Groups Product Usage Data

Amount its used How its used

SEG Measurement WWW.SEGMeasurement.com

Are you looking for “proof” that use of your product is the “cause” of improved learning outcomes?

“Proof” and “Causality” (The scientific model) The “Medical model”

If we want to know if a drug works, we know you can’t just ask a bunch of people.

Random Assignment to treatment and control groups or matched groups designs

Pre and post testing Sound statistical models to eliminate any potential initial differences or

“interfering” variables

SEG Measurement WWW.SEGMeasurement.com

Challenges

Getting results quickly Desire for rapid cycle, iterative improvement Costs (One good adoption won pays the bill) Lack of Research Literacy (We offer pro bono assistance)

How to prove your product is effective Treatment and Control Group Sufficient Sample Size

Statistical Power Generalizability

Pre post assessment of desired outcomes (external, reliable and valid) Fidelity of implementation Qualitative and quantitative data Third party conducted Peer reviewed Multiple studiesSEG Measurement WWW.SEGMeasurement.com