Evaluation Basics Evaluation Basics for Youth Programsfor Youth Programs

Ben Silliman, Ph.D.Ben Silliman, Ph.D.

Dept. of 4-H Youth DevelopmentDept. of 4-H Youth Development

Reasons to LoveReasons to LoveProgram EvaluationProgram Evaluation

Personal: feedback on Personal: feedback on personal effectivenesspersonal effectiveness

Practical: information Practical: information for improving deliveryfor improving delivery

Professional: process Professional: process and outcomes data to and outcomes data to show accountability, show accountability, impactimpact

Understanding EvaluationUnderstanding Evaluation

Name the best thing(s) Name the best thing(s) you’ve seen/experienced you’ve seen/experienced in a youth-at-risk in a youth-at-risk programprogram

Name the worst thing(s) Name the worst thing(s) you’ve seen/experienced you’ve seen/experienced in a youth-at-risk in a youth-at-risk program program

Dimensions of EvaluationDimensions of Evaluation

Good things come from…Good things come from…

A clear vision of what is A clear vision of what is happeninghappening A positive climate for youthA positive climate for youth Programs that make a Programs that make a differencedifference Organizations that run Organizations that run effective programseffective programs

Dimensions of EvaluationDimensions of Evaluation

Good things come from Good things come from research-based…research-based…

Environmental scanningEnvironmental scanning Program evaluationProgram evaluation Outcome evaluationOutcome evaluation Organizational/Organizational/ professional evaluationprofessional evaluation

Environmental ScanningEnvironmental ScanningWhat do people need?What do people need?

What resources do people have?What resources do people have?

Needs and Asset MappingNeeds and Asset Mapping

Demographic DataDemographic Data

Community Economic and Social SurveysCommunity Economic and Social Surveys

Focus GroupsFocus Groups

Targeted Needs AssessmentsTargeted Needs Assessments

Community Coalition consensusCommunity Coalition consensus

Program EvaluationProgram EvaluationHow good is the program?How good is the program?

Positive Climates/Best PracticesPositive Climates/Best Practices Safe EnvironmentsSafe Environments

Structure Appropriate for Structure Appropriate for

Supportive RelationshipsSupportive Relationships

Social InclusionSocial Inclusion

Standards of ConductStandards of Conduct

Skill Learning FocusedSkill Learning Focused

Contribution-OrientedContribution-Oriented

Connections to Family, CommunityConnections to Family, Community

Program ClimateProgram Climate

Physical and Psychological SafetyPhysical and Psychological Safety

o Safe, healthy facilitiesSafe, healthy facilitieso Conflict resolutionConflict resolutiono Wellness habitsWellness habits

Program ClimateProgram Climate

Appropriate StructureAppropriate Structure

o Clear consistent boundariesClear consistent boundarieso Age-based opportunitiesAge-based opportunitieso Effective monitoringEffective monitoring

Program ClimateProgram Climate

Supportive RelationshipsSupportive Relationships

o Warmth, affirmationWarmth, affirmationo CommunicationCommunicationo Positive guidancePositive guidance

Program ClimateProgram Climate

Opportunities to BelongOpportunities to Belong

o Acceptance, inclusionAcceptance, inclusiono Cultural diversity/ toleranceCultural diversity/ toleranceo TeamworkTeamwork

Program ClimateProgram Climate

Positive Social NormsPositive Social Norms

o High expectations for ethics, High expectations for ethics, behaviorbehavior

o High achievement orientationHigh achievement orientationo Service orientationService orientation

Program ClimateProgram Climate

Efficacy and MatteringEfficacy and Mattering

o Growth in confidence and Growth in confidence and autonomyautonomy

o Ability to make a differenceAbility to make a differenceo Meaningful changes effected Meaningful changes effected

with individuals, organizations, with individuals, organizations, communitiescommunities

Program ClimateProgram Climate

Opportunities for Skill BuildingOpportunities for Skill Building

o Mastery of practical skills in many Mastery of practical skills in many domainsdomains

o Ability to relate and navigate Ability to relate and navigate

varied culturesvaried cultureso Improvements in vocation-Improvements in vocation-

related skillsrelated skills

Program ClimateProgram Climate

Integration with Family, School, Integration with Family, School, and Communityand Community

o Parent involvement and family Parent involvement and family activities at homeactivities at home

o Increased academic performance Increased academic performance and school leadershipand school leadership

o Youth-adult partnershipYouth-adult partnership

Outcome EvaluationOutcome EvaluationWhat’s happening with the kids?What’s happening with the kids?

Logic Model:Logic Model:Building a Game PlanBuilding a Game Plan

The Big Picture: Winning LegacyThe Big Picture: Winning Legacy

Giant Steps: A Habit of Playing WellGiant Steps: A Habit of Playing Well

Small Steps: Mastering the FundamentalsSmall Steps: Mastering the Fundamentals

The Right Stuff: Good PreparationThe Right Stuff: Good Preparation

Setting the Table: Good ResourcesSetting the Table: Good Resources

The Logic of EvaluationThe Logic of Evaluation

InputsInputs

Investments Investments of resources of resources

MoneyMoneyStaffStaffVolunteersVolunteersEquipmentEquipmentSuppliesSupplies

Effects ofEffects of constraintsconstraints

LawsLawsRegulationsRegulationsFunder requirements Funder requirements

Outputs Outputs

Products of programsProducts of programsClasses taughtClasses taughtMaterials distributedMaterials distributedHours of serviceHours of serviceParticipants served Participants served

The Logic of EvaluationThe Logic of Evaluation

ImpactsImpacts

Long-term outcomesLong-term outcomes

Behavioral Behavioral changechange Social-economic Social-economic changechange Cultural changeCultural change

OutcomesOutcomesShort-term outcomesShort-term outcomes

KnowledgeKnowledgeAttitudeAttitude SkillsSkills AspirationsAspirations

Evaluating OutcomesEvaluating Outcomesin Small Stepsin Small Steps

KnowledgeKnowledge

AttitudesAttitudes

SkillsSkills

AspirationsAspirations

Outcome DomainsOutcome Domains

PhysicalPhysical HealthHealth SafetySafety WellnessWellness Risk ReductionRisk Reduction

Outcome DomainsOutcome Domains

Intellectual/CognitiveIntellectual/Cognitive Critical thinking/Critical thinking/

problem solvingproblem solving Academic achievementAcademic achievement Project skills/masteryProject skills/mastery Decision-makingDecision-making Creative thinkingCreative thinking

Outcome DomainsOutcome Domains

EmotionalEmotional CopingCoping Stress managementStress management OptimismOptimism Self-efficacySelf-efficacy CharacterCharacter

Organizational/ProfessionalOrganizational/ProfessionalEvaluationEvaluation

How (well) do leaders make things happen?How (well) do leaders make things happen?

Organizational ProcessesOrganizational Processes

Serving PeopleServing People

Designing and Producing Better Designing and Producing Better Goods and ServicesGoods and Services

Building an Efficient and Effective Building an Efficient and Effective WorkforceWorkforce

Six Sigma ProcessesSix Sigma ProcessesDefineDefine the project goals and customer the project goals and customer (internal and external) requirements(internal and external) requirements

MeasureMeasure and determine customer needs and determine customer needs and specifications; benchmark competitors and specifications; benchmark competitors and industryand industry

AnalyzeAnalyze the process options to meet the the process options to meet the customer needscustomer needs

DesignDesign (detailed) the process (detailed) the process

to meet the customer needsto meet the customer needs

VerifyVerify the design performance the design performance

and ability to meet customer needsand ability to meet customer needs

Professional EvaluationProfessional Evaluation

Professional StandardsProfessional Standards

Novice-to-Expert Rubrics and Novice-to-Expert Rubrics and Certification for General & Certification for General & Specialized CompetenciesSpecialized Competencies

Individual and Team goals, Individual and Team goals, performance standards, and performance standards, and professional developmentprofessional development

Continuous quality improvementContinuous quality improvement

Human SubjectsHuman Subjects

Satisfaction, description is Satisfaction, description is

program feedbackprogram feedback

Individual performance Individual performance change is consent-required change is consent-required feedbackfeedback

Replicating and SustainingReplicating and SustainingProgramsPrograms

Stage 1: Program running as Stage 1: Program running as planned, having intended effect planned, having intended effect (Process Evaluation/Formative (Process Evaluation/Formative Evaluation/Quality Control)Evaluation/Quality Control)

Stage 2: Change in knowledge, Stage 2: Change in knowledge, attitude, skill, or aspiration attitude, skill, or aspiration (Outcome Evaluation/Summative (Outcome Evaluation/Summative Evaluation/Achievements & Impact)Evaluation/Achievements & Impact)