“What's measured improves”

Peter Drucker

Yvonne Pelham, Manager, Educational Outreach, EAD

IEEE Symposium on Pre-University Teacher TrainingHilton Tampa Airport Westshore19 – 20 May 2012

Topics of Discussion

Outcome, measures of success, and impactMetrics on workshops and in-service presentationsImpact assessmentTISP champion opportunities

04/10/232

Evaluating Outcomes, Measuring Success and Determining Impact

To evaluate outcome, define the vision/aspiration, and determine if the program activities fulfill that visionTo measure success, define the goals/objectives and measure efforts using a common set of metricsTo determine impact, define the value of the program and assess the effort using key performance indicators

04/10/233

Vision

Through a sustained program lead by IEEE volunteers in every section, IEEE will be

recognized by the pre-university community as a leading provider of professional development for science, technology,

engineering and mathematics teachers.

4

How to Get There:Long-Term Goals

IEEE Teacher In-Service Champions in all sections– Empower IEEE “champions” to develop collaborations

with local pre-university education community– Increase the level of understanding of the needs of

educators – Identify ways that engineers can assist schools and

school systems (i.e. teacher training, curriculum enhancements, etc.)

Technological Literacy– Enhance the level of technological literacy of pre-

university educators and students

5

Measures of Success

6

Empower IEEE “champions”Technological literacy of pre-university educators Technological literacy of pre-university students

Understanding of the needs of educatorsSchool systems assisted by IEEE

2,203 trained attendees (81% volunteers)93% agreed that program enhanced technological literacy94% believe that student’s technological literacy would increase (anecdotal)Sustained programs in several sections

168 presentations reported

Current MetricsWorkshop Evaluations to measure effectiveness of the training experience– Did the training inspire the attendees to get involved

in outreach activities with teachers and schools?– Did the training increase understanding and provide

resources to enable collaboration?Teacher Questionnaires to measure usefulness of the in-service presentation – Will the teachers use the resources provided in the

classroom?– Did the in-service affect level of technological literacy?

04/10/237

Training Workshops – Empowering Champions

04/10/238

Training Workshops – Empowering Champions

04/10/239

Effectiveness of Training Workshops

10

38% prior68% after

More stats on TISP Workshop Feedback

72% said the workshop definitely provided valuable information/resources to participate in pre-university outreach

69% said the workshop contributed to their understanding of the impact engineers can make through pre-university outreach

11

Teacher Feedback – Questionnaire Cumulative Results

12

93% of attendees who attended a TISP Presentation and completed a feedback form are teachers.

68% of educators teach secondary level and 25% teach primary level.

The subject areas most educators teach are science, technology and math.

93% felt that the program added to their technical knowledge base.

92% stated that they will use the concepts presented in their classroom instruction in the next 6 months.

94% agreed that the “hands-on” portions of the presentation were helpful in understanding the concepts discussed.

Impact on Technological Literacy

04/10/2313

Perceived impact on student

94% agreed that the hands-on activities will increase their student’s level technological literacy

Impact on teacher

91% agreed that the presentations increased their level of technological literacy.

Program Evaluation

04/10/2314

Key Research Questions

How does TISP affect teacher’s pedagogy for science, technology and mathematics?How do the in-service sessions affect teachers attitude towards teaching technology subjects in their classroom?How do IEEE hands-on activities affect what students know about engineering, technology, and related science and mathematic topics?How do the activities affect student’s attitudes towards science, technology, engineering, mathematics activities, and careers?How do the activities impact understanding and attitudes of female students and under-represented group in engineering and technology fields?

04/10/2315

2009 Pilot Teacher Assessment: Survey Questions

Number of IEEE lesson plans or activities that have been used, in full or in part, in the classroom – Reasons(s) that influenced your inability to use

the lesson plansThe lesson plan(s) supports classroom instruction for the subject areas that I teachThe students expressed greater interest in science, technology, engineering or math after completing these hands-on activities (anecdotal)

04/10/2316

2009 Pilot Teacher Assessment: Results

Of the respondents, – 48% taught in the secondary school– 57% were science teachers– 48% used 2 to 5 lesson plans in the classroom– Of the 26% that did not use the lesson plans, 6

out of 11 stated that they ran out of time or was unable to fit into schedule

– 74% agreed that the lesson plans supported their classroom instruction

– 88% agreed that the students expressed more interest after completing the activities

04/10/2317

Assessment Effort

Engage a research firm to examine TISP impact on teacher attitudes, classroom practice and knowledgeIdentify section(s) to participate in the study

04/10/2318

Opportunities to contribute to the enhancement of TISP

Make recommendations on enhancements to the train-the-trainer workshopsRemind teams to report TISP activities and distribute feedback questionnaires to update global statisticsMake recommendations on follow-on activities with teachersParticipate in assessment research

04/10/2319

04/10/2320

Let’s have dialogue