W W W . C E S . C L E M S O N . E D U / G E /

Planning Engineering Education Research

Facilitator: Matthew W. Ohland

W W W . C E S . C L E M S O N . E D U / G E /

Workshop Objective

Describe steps in the planning and assessment of engineering education research

Give examples of each step

Small-group discussion to implement the step using a common example

W W W . C E S . C L E M S O N . E D U / G E /

Workshop Agenda

4:00-4:05 Introduction4:05-4:20 Defining the Purpose of the Evaluation

(What questions are we asking?)4:20-4:40 Clarify Project Objectives (What do we expect?)4:40-5:00 Create a Model of Change

(How will our efforts lead to the objectives?)5:00-5:15 Select Criteria and Indicators

(What data do we need to measure our progress?)5:15-5:25 Identify Data Sources (Where will we get that data?)5:25-5:45 Design Evaluation Research

(How will we analyze that data?)5:45-5:50 Monitor and Evaluate (What actually happened?)

Use and Report Results (Who needs to know?)5:50-6:00 Wrap-up and evaluations

W W W . C E S . C L E M S O N . E D U / G E /

Step 1: Purpose of the Evaluation

FormativeSummative

EfficacyEffectivenessCost

Examples• Is there a benefit to doing

this?• Can we improve it?• Does that benefit lead to

better retention / grades?• Do the benefits justify the

program cost?• Can we achieve the same

benefits on a larger scale?

W W W . C E S . C L E M S O N . E D U / G E /

Step 1 Action Steps

Decide purpose(s)

Primary questions? Order?

Primary audience(s)?

How will the results be used?

W W W . C E S . C L E M S O N . E D U / G E /

Step 1: Purpose of the Evaluation

Do we have a learning community?

Is it leading to better course retention / grades?

Can we achieve the same benefits cost-effectively on a larger scale?

W W W . C E S . C L E M S O N . E D U / G E /

Step 2: Project Objectives

Impact

Outcome

Process

Example• Improve graduation rate

by some %

• Increase course passing rate by some %

• Student or faculty opinions / behaviors change

• Reserve any facilities needed

W W W . C E S . C L E M S O N . E D U / G E /

Step 2 Action Steps

Write down project objectives• Impact• Outcome• Process

W W W . C E S . C L E M S O N . E D U / G E /

Step 2: Project Objectives

Increase graduation rate from 50% to 75%

Increase first-time Calculus passing rate from 50% to 70%

Students have positive opinion of group work

Secure dormitory and classroom space

W W W . C E S . C L E M S O N . E D U / G E /

Step 3: Create a Model of Change

Identify assumptions you can assessChoose relationships to test based on• Resources• Where you anticipate

problems• Where you have

control / can make improvements

Link what you do to what you expect to happenExample• Common residence /

classes • in between, we need a

sound theory of why this will happen

more will graduate

W W W . C E S . C L E M S O N . E D U / G E /

Step 3 Action Steps

Model of change• as specific / complete as needed

Review model assumptions

Use criteria to prioritize• Resources• Relevance• Control

W W W . C E S . C L E M S O N . E D U / G E /

Step 3: Create a Model of Change

Common residence / classes affiliation

Affiliation support

Support performance

Performance future performance

Future performance more will graduate

W W W . C E S . C L E M S O N . E D U / G E /

Step 4: Criteria and Indicators

Validity

Reliability

Sensitivity

Ease of interpretation

Usefulness

Example: define• Graduation rate

• Course pass rate

• Affiliation

• Life-long learning

• Enrollment

W W W . C E S . C L E M S O N . E D U / G E /

Step 4 Action Steps

Define a set of indicators and criteria • Impact• Outcome • Process

W W W . C E S . C L E M S O N . E D U / G E /

Step 4: Criteria and Indicators

Graduation rate• # graduated / # in original cohort

Calculus pass rate• # A, B, C / # in original cohort• D’s are no good – must be retaken

Affiliation • # in study group at end of sophomore year /

# students

Number of students enrolled in program

W W W . C E S . C L E M S O N . E D U / G E /

Step 5: Data Sources

ExamsSurveysObservationsStudent recordsSAT scores

Frequency• Resources• Guidance• Change

Example• Institutional Research

office (annual)• Course records (each

semester)• Registrar• Admissions• Assessment office• Process - monitor until

achieved

W W W . C E S . C L E M S O N . E D U / G E /

Step 5 Action Steps

Define data sources

Define frequency of measurement

W W W . C E S . C L E M S O N . E D U / G E /

Step 5: Data Sources

Grad / retention rate – Inst. Res. – AnnualCalculus performance – Math dept. – includes course grades and common exam results –end of semesterDorm space allocated – Housing – monitor until achievedSurvey of program participantsObserver evaluations of class interactionSAT scores—Admissions

W W W . C E S . C L E M S O N . E D U / G E /

Step 6: Design Evaluation Research

Selection

Mortality

Placebo is not an issue

Qualitative• Interviews

• Focus groups

• Systematic observation

Quantitative• Non-experimental

- Posttest only

- Pretest-Posttest

• Quasi-experimental- Time Series

- Nonequivalent control

• Experimental- Pretest-Posttest Control

- Multiple Intervention

W W W . C E S . C L E M S O N . E D U / G E /

Step 6 Action Steps

Design evaluation research studies for key questions

W W W . C E S . C L E M S O N . E D U / G E /

Step 6: Example Designs

Example-Interviews• Student expectations—

invitation process modified

• Resource utilization—approaches to motivate attendance

Example-Efficacy• Non-equivalent control

group

• Test for selection bias using baseline measures

• Matched pairs / groups

• Calculus grades and overall GPR

W W W . C E S . C L E M S O N . E D U / G E /

Step 7: Monitor and Evaluate

Establish project information system

Budget for evaluation

Evaluation meetings

Review and revise evaluation plan

Carry out studies

W W W . C E S . C L E M S O N . E D U / G E /

Step 8: Use and Report Results

Report results – to everyone

Use results to make improvements

W W W . C E S . C L E M S O N . E D U / G E /

Conclusions

Planning and assessment is essential

Start small if resources are limited

Develop a plan before starting

NSF and other agencies support well-designed educational research• CCLI—EMD / A&I, ASA, and other programs

Seek appropriate partners from education, psychology, sociology, statistics, etc.