Embed Size (px)
Citation preview
Project Evaluation
Don Millard John Yu [email protected] [email protected]
March 27, 2012
Guy-Alain Amoussou Lou Everett [email protected] [email protected]
March 28, 2012
Handout 1
Before you leave the session please complete the assessment survey:
http://www.nsflsu.com
Note: please correct this URL in your handouts
BEFORE you leave today!
Most of the information presented in this workshop represents the presenters’ opinions and not an official NSF position
Local facilitators will provide the link to the workshop slides at the completion of the webinar.
Participants may ask questions by “raising their virtual hand” during a question session. We will call on selected sites and enable their microphone so that the question can be asked.
Responses will be collected from a few sites at the end of each Exercise. At the start of the Exercise, we will identify these sites in the Chat Box and then call on them one at a time to provide their responses.
3
Important Notes
Learning must build on prior knowledge ◦ Some knowledge correct ◦ Some knowledge incorrect – Misconceptions
Learning is ◦ Connecting new knowledge to prior knowledge◦ Correcting misconceptions
Learning requires engagement◦ Actively recalling prior knowledge◦ Sharing new knowledge◦ Forming a new understanding
Framework for the Session
4
Effective learning activities ◦ Recall prior knowledge -- actively, explicitly◦ Connect new concepts to existing ones◦ Challenge and alter misconceptions
Active & collaborative processes◦ Think individually◦ Share with partner◦ Report to local and virtual groups ◦ Learn from program directors’ responses
5
Preliminary Comments
Active & Collaborative Learning
Coordinate the local activities
Watch the time◦Allow for think, share, and report phases◦Reconvene on time -- 1 min warning ◦ With one minute warning, check Chat Box to see if you will be
asked for a response
Ensure the individual think phase is devoted to thinking and not talking
Coordinate the asking of questions by local participants and reporting local responses to exercises
6
Facilitator’s Duties
The session will enable you to collaborate more effectively with evaluation experts in preparing credible and comprehensive project evaluation plans…. it will not make you an evaluation expert.
Goal for Project Evaluation Session
7
After the session, participants should be able to: Discuss the importance of goals, outcomes, and
questions in the evaluation process◦ Cognitive and affective outcomes
Describe several types of evaluation tools◦ Advantages, limitations, and appropriateness
Discuss data interpretation issues◦ Variability, alternative explanations
Develop an evaluation plan in collaboration with an evaluator◦ Outline a first draft of an evaluation plan
Session Outcomes
8
The terms evaluation and assessment have many meanings◦ One definition
Assessment is gathering evidence Evaluation is interpreting data and making value judgments
Examples of evaluation and assessment◦ Individual’s performance (grading)◦ Program’s effectiveness (ABET and regional accreditation)◦ Project’s progress and success (monitoring and validating)
Session addresses project evaluation◦ May involve evaluating individual and group performance – but in the
context of the project Project evaluation
◦ Formative – monitoring progress to improve approach◦ Summative – characterizing and documenting final accomplishments
Evaluation and Assessment
9
Think about your favorite course. What types of in-class activities could be called:◦ Assessment versus Evaluation?◦ Formative versus Summative Evaluation?
Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min
Watch time and reconvene after 6 min Use THINK time to think – no discussion, Selected local
facilitators report to virtual group
With one minute warning, check Chat Box to see if you will be asked for a response
Activity
Evaluation vs AssessmentFormative vs Summative
10
An in-class quiz could be called assessment. Using the in-class quiz results to realize the students are not
achieving a learning outcome is an evaluation. Using your evaluation results to change your approach as the
course progresses is a formative evaluation. Integrating your evaluation over the whole semester to
determine if you achieved your objective is summative evaluation.
11
PD Response
Evaluation vs AssessmentFormative vs Summative
Handout 2
Project Goals, Expected Outcomes, and Evaluation Questions
Effective evaluation starts with carefully defined project goals and expected outcomes
Goals and expected outcomes related to:◦Project management
Initiating or completing an activity Finishing a “product”
◦Student behavior Modifying a learning outcome Modifying an attitude or a perception
Evaluation and Project Goals/Outcomes
13
Goals provide overarching statements of project intention
What is your overall ambition? What do you hope to achieve?
Expected outcomes identify specific observable or measureable results for each goal
How will achieving your “intention” be reflected by changes in student behavior?How will it change their learning and their attitudes?
Learning Goals and Outcomes
14
Goals → Expected outcomes
Expected outcomes → Evaluation questions
Questions form the basis of the evaluation process
The evaluation process consists of the collection and interpretation of data to answer evaluation questions
Goals, Expected Outcomes, and Evaluation Questions
15
Read the abstract -- Goal statement removed Suggest two plausible goals
◦ One on student learning Cognitive behavior
◦ One on some other aspect of student behavior Affective behavior
Focus on what will happen to the students ◦ Do not focus on what the instructor will do
Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min
Watch time and reconvene after 6 min Use THINK time to think – no discussion, Selected local facilitators
report to virtual group
Activity
Identification of Goals/Outcomes
16
The goal of the project is …… The project is developing computer-based instructional modules for statics and mechanics of materials. The project uses 3D rendering and animation software, in which the user manipulates virtual 3D objects in much the same manner as they would physical objects. Tools being developed enable instructors to realistically include external forces and internal reactions on 3D objects as topics are being explained during lectures. Exercises are being developed for students to be able to communicate with peers and instructors through real-time voice and text interactions. The project is being evaluated by … The project is being disseminated through … The broader impacts of the project are …
Two goals: one for student learning and one for student behavior
Non engineers should substitute: “Organic chemistry” for “statics and mechanics of materials” “Interactions” for “external forces and internal reactions”
Abstract
17
GOAL: To improve conceptual understanding and processing skills
In the context of course◦ Draw free-body diagrams for textbook problems◦ Solve 3-D textbook problems ◦ Describe the effect(s) of external forces on a solid object orally
In a broader context◦ Solve out-of-context problems ◦ Visualize 3-D problems◦ Communicate technical problems orally◦ Improve critical thinking skills◦ Enhance intellectual development
PDs’ Response: Goals on Cognitive Behavior
18Handout 3
GOAL: To improve◦ Self- confidence◦ Attitude about engineering as a career
PDs’ Response: Goals on Affective Behavior
19
Write SMART outcomes for your goals• Specific• Measurable• Attainable• Realistic• Timely
Achieving outcomes brings you closer to your goal, e.g. ◦Goal: My students will be life long learners◦Outcomes (some of these lack SMART components):
reading an unassigned technical article, attending a professional society meeting attending a non required seminar or talk
Outcomes provide observable effects that goals are being achieved
Transforming Goals into Outcomes
20
Write expected measurable outcomes for each of the following goals:
◦ Improve the students’ understanding of the fundamental concepts in statics (cognitive)
◦ Improve the students’ self confidence (affective)
Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min
Watch time and reconvene after 6 min Use THINK time to think – no discussion, Selected local facilitators
report to virtual group
Activity
Transforming Goals into Outcomes
Non engineers may substitute: “Organic chemistry” for “statics”
21
PDs’ Response: Expected Outcomes
Understanding of the fundamentals◦ By the end of the class, 70% of the students will be able to:
Correctly draw freebody diagrams of 2D truss structures Correctly write Newton’s laws when given a FBD Describe the effects on member force when one angle in a
2D truss is changed
Self-Confidence◦ By the end of the semester:
30% of the class volunteers to show the solution to any homework problem on the board
Self reported test anxiety reduces to 50% of the initial amount 80% will say the class was easier than they expected it would be 50% report they are excited about taking the follow-on course
22Handout 4
Evaluation Questions
Understanding of the fundamentals◦ Are the students better able to describe the effects of
changing some variable in a simple problem◦ Are the students better able to describe the effects of
changing some variable in a simple problem as a result of the intervention
Self-Confidence◦ Do the students express more confidence in their solutions◦ Do the students express more confidence in their solutions as
a result of the intervention
23
Questions
“Hold-up your virtual hand” and you will be called upon after we unmute your mike.
BREAK15 min
BREAK1 min warning
Tools for Evaluating Learning Outcomes
Surveys ◦ Forced choice or open-ended responses
Concept Inventories◦ Multiple-choice questions to measure conceptual understanding
Rubrics for analyzing student products◦ Guides for scoring student reports, tests, etc.
Interviews◦ Structured (fixed questions) or in-depth (free flowing)
Focus groups◦ Like interviews but with group interaction
Observations◦ Actually monitor and evaluate behavior
Olds et al, JEE 94:13, 2005
NSF’s Evaluation Handbook
Examples of Tools for Evaluating Learning Outcomes
28
Comparing Surveys and Observations
Surveys Efficient Accuracy depends on
subject’s honesty Difficult to develop reliable
and valid survey Low response rate
threatens reliability, validity & interpretation
Observations Time & labor intensive Inter-rater reliability must
be established Captures behavior that
subjects are unlikely to report
Useful for observable behavior
Olds et al, JEE 94:13, 2005
29
Example – Appropriateness of Interviews
Use interviews to answer these questions:◦What does program look and feel like?◦What do stakeholders know about the project?◦What are stakeholders’ and participants’ expectations?◦What features are most salient?◦What changes do participants perceive in themselves?
The 2002 User Friendly Handbook for Project Evaluation, NSF publication REC 99-
12175
30
Originated in physics -- Force Concept Inventory (FCI) Several are being developed in engineering fields Series of multiple choice questions◦ Questions involve single concept
Formulas, calculations or problem solving skills not required
◦ Possible answers include distractors Common errors -- misconceptions
Developing CI is involved◦ Identify misconceptions and detractors◦ Develop, test, and refine questions◦ Establish validity and reliability of tool ◦ Language is a major issue
Tool for Measuring Conceptual Understanding – Concept Inventory
31
Tool for Assessing Attitude
Pittsburgh Freshman Engineering Survey◦ Questions about perception
Confidence in their skills in chemistry, communications, engineering, etc.
Impressions about engineering as a precise science, as a lucrative profession, etc.
Validated using alternate approaches: ◦ Item analysis◦ Verbal protocol elicitation◦ Factor analysis
Compared results for students who stayed in engineering to those who left
Besterfield-Sacre et al , JEE 86:37, 1997 32
Tools for Characterizing Intellectual Development
Levels of Intellectual Development◦Students see knowledge, beliefs, and authority in
different ways “ Knowledge is absolute” versus “Knowledge is
contextual” Tools ◦Measure of Intellectual Development (MID)◦Measure of Epistemological Reflection (MER)◦ Learning Environment Preferences (LEP)
Felder et al, JEE 94:57, 2005
33
Suppose you were considering an existing tool (e. g., a concept inventory) for use in your project’s evaluation of learning outcomes
What questions would you consider in deciding if the tool is appropriate?
Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min
Watch time and reconvene after 6 min Use THINK time to think – no discussion Selected local facilitators report to virtual group
Activity
Considering an Existing Tool
34
Nature of the tool◦ Is the tool relevant to what was taught? ◦ Is the tool competency based? ◦ Is the tool conceptual or procedural?
Prior validation of the tool◦ Has the tool been tested? ◦ Is there information concerning its reliability and validity? ◦ Has it been compared to other tools? ◦ Is it sensitive? Does it discriminate between a novice and an expert?
Experience of others with the tool◦ Has the tool been used by others besides the developer? At other
sites? With other populations? ◦ Is there normative data?
PDs’ Response
Evaluating an Existing Tool
35Handout 5
Questions
Hold up your “virtual hand” to ask a question.
Interpreting Evaluation Data
Interpreting Evaluation Data
Comparision Group
Experimental Group
Comparision Group
Experimental Group
1 25 30 29% 23%2 24 32 34% 65%3 25 31 74% 85%
- - - - -
Question or
Concept
Percent w ith Correct AnswerNo. of Students
Data suggest that the understanding of Concept #2 increased
38
Comparision Group
Experimental Group
Comparision Group
Experimental Group
1 25 30 29% 23%2 24 32 34% 65%3 25 31 74% 85%
- - - - -
Question or
Concept
Percent with Correct AnswerNo. of Students
Data suggest that the understanding of Concept #2 increased
One interpretation is that the intervention caused the change
List some alternative explanations ◦ Confounding factors◦ Other factors that could explain the change
Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min
Watch time and reconvene after 6 min Use THINK time to think – no discussion Selected local facilitators report
to virtual group
Activity
Alternative Explanations For Change
Students learned the concept out of class (e. g., in another course or in study groups with students not in the course)
Students answered with what they thought the instructor wanted rather than what they believed or “knew”
An external event distorted the pretest data The instrument was unreliable Other changes in the course and not the intervention was
responsible for the improvement The characteristics of groups were not similar
PDs’ Response
Alternative Explanations For Change
40Handout 6
Data suggest that the understanding of the concept tested by Q1 did not improve
One interpretation is that the intervention did cause a change that was masked by other factors
Think about alternative explanations How would these alternative explanations
(confounding factors) differ from the previous list?
Alternative Explanations for Lack of Change
Comparision Group
Experimental Group
Comparision Group
Experimental Group
1 25 30 29% 23%2 24 32 34% 65%3 25 31 74% 85%
- - - - -
Question or
Concept
Percent w ith Correct AnswerNo. of Students
Evaluation Plan
List the topics that need to be addressed in the evaluation plan (a.k.a. summarize)
Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min
Watch time and reconvene after 6 min Use THINK time to think – no discussion Selected local facilitators report to virtual group
Activity
Evaluation Plan
43
Name & qualifications of the evaluation expert◦ Get the evaluator involved early in the proposal development phase
Goals, outcomes, and evaluation questions Instruments for evaluating each outcome Protocols defining when and how data will be collected Analysis & interpretation procedures Confounding factors & approaches for minimizing their
impact Formative evaluation techniques for monitoring and
improving the project as it evolves Summative evaluation techniques for characterizing
the accomplishments of the completed project.
PDs’ Response
Evaluation Plan
44Handout 7
Workshop on Evaluation of Educational Development Projects◦ http://www.nsf.gov/events/event_summ.jsp?cntn_id=108142&org=NSF
NSF’s User Friendly Handbook for Project Evaluation ◦ http://www.westat.com/westat/pdf/projects/2010ufhb.pdf
Online Evaluation Resource Library (OERL)◦ http://oerl.sri.com/
Field-Tested Learning Assessment Guide (FLAG)◦ http://www.wcer.wisc.edu/archive/cl1/flag/default.asp
Student Assessment of Their Learning Gains (SALG)◦ http://www.salgsite.org/
Science education literature
Other Sources
45
Identify the most interesting, important, or surprising ideas you encountered in the workshop on dealing with project evaluation
Take ---- 4 min◦ Think individually -------- ~2 min◦ Report in local group ---- ~2 min
Watch time and reconvene after 4 min
Use THINK time to think – no discussion, Selected local facilitators report to virtual group
Activity
Final Reflection
46
Questions
Hold up your “virtual hand” to ask a question.
Proposal & Award Policies & Procedures Guidehttp://www.nsf.gov/pubs/policydocs/pappguide/nsf11001/
To download a copy of the presentation- go to:
http://www.nsflsu.com
Please complete the assessment survey-go to:
http://www.nsflsu.com
Thanks for your participation!
