Strategies and Tools to Study and Evaluate STEM Education-Oriented

Projects

PKAL Transforming STEM Education

November 8, 2019

Chicago, IL

Overview

▪ LLCAL Project Description

▪ Theories of Change & Logic Models

▪ Project Research vs. Evaluation

▪ Activity 1: Sort E, R characteristics

▪ Distinguish between affective, behavioral, & cognitive questions

▪ Activity 2: Sort A, B, C, E, R questions

▪ Assessment tools

▪ Discussion/Questions/Sharing

LLCAL Project Model

▪ Expand Science Residential Learning Communities

▪ Focus on retention

▪ Bridge program

▪ First Year Seminar (Fall), Research Seminar (Spring)

▪ Cohort Scheduling

▪ Summer research opportunity

▪ STEM Leadership program (sophomores)

▪ Active Learning Pedagogy Transformation

▪ Target gateway science courses

▪ Select sophomore+ courses

▪ Mostly Biology & Chemistry plus Calculus 1

Program Models

Theory of Change

Program Logic Model

Knowlton & Phillips, (2013). The Logic Model Guidebook

Theory of Change

St. Ed’s Theory of Change

Program Logic Model

Patton (2014). Evaluation Flash Cards

St. Ed’s Resources

St. Ed’s Activities & Outputs

St. Ed’s Outcomes

St. Edward’s Logic Model

Evaluation ≠ Research

Grantors look for

▪Educational Research

▪External Evaluation

What is research?

• What is purpose?

• Who is audience?

• What are methodologies?

• How are results used?

• How to judge quality?

What is evaluation?

the systematic determination of

the quality, value, or significance of

a program, policy or product

How to evaluate?

Consumer Reports

“Find the machine that’s

right for you”

Evaluation

• Makes judgments from data gathered

• Results from particular context

• Stakeholders & evaluator set agenda

• Intended use of results is important

• Standards of accuracy, utility, feasibility, propriety, accountability

Sort cards sort into 2 piles:

• Research

• Evaluation

Activity 1: Research or Evaluation

Studying Students

• Many projects focus on students

• How well are project interventions working?

• What do the students think about the programmatic

components?

• How do students react to the programs?

• Students can be studied in three different perspectives:

affective, behavioral, and cognitive.

Affective Measures

How do students perceive the programming?

• Do they feel the programs are worth attending?

• Do they see benefit in the programming?

• Do they enjoy the activities?

• Would they recommend the programs to peers?

Behavioral Measures

Do students change the way they do things?

• Do they prioritize class attendance?

• Do they meet deadlines?

• Do they seek assistance when needed?

• Do students attend help sessions regularly?

Cognitive Measures

Does student performance improve?

• Are exam grades improving?

• Are students able to develop research projects?

• Can students recognize authoritative sources?

• Can students extract knowledge from data?

Activity 2:Distinguish A,B,C,E,R

Sort cards into 5 piles

• Affective Questions

• Behavioral Questions

• Cognitive Questions

• Project Evaluation Questions

• Project Research Questions

Common Guidelines for Education Research &

Development

Institute of Education Sciences

&

NSF

https://www.nsf.gov/pubs/2013/nsf13126/nsf13126.pdf

2013

Continuum of ResearchFoundational

Exploratory

Design & Development

Efficacy

Effectiveness

Scale-up

Assessment Tools

Sense of Belonging

Sense of Belonging

I feel a strong sense of belonging to X.

I feel that I am a member of the X community.

I see myself as part of the X community.

Bollen & Hoyle (1990)

Sense of Community

I can recognize most of the people who live on/in my X.

Most of my X know me.

My X and I want the same thing from our X.

If there is a problem on this X, people who live here can

get it solved.

Long & Perkins (2003)

Group Cohesion of LLCs

I feel connected to my math & science peers in the LLC.

I am happy I was selected to be a member of the math & science LLC.

I can recognize most of the people who live in my LLC.

Most of the students in my LLC know me.

I have influence over what my LLC is like.

If there is a problem in this LLC, students who live here can get it solved.

I feel a strong sense of community with the students in my LLC.

Group Cohesion by LLC

Contact Us

Michelle Burd, PhD

michelle@burdseyeview.net

512-797-9532Photos from:

✓ Alpha Stock Images

✓ Wikimedia

✓ National Park Service

Attributions

Richard L. Kopec, PhD

http://kopec.create.stedwards.edu

kopec@stedwards.edu

512-448-8464