Susan Carter Evaluation Coordinator National Indian Youth Leadership Project

Peering into the black box between pre and post: A theory of change related to interactive/ experiential delivery methods

Susan CarterEvaluation Coordinator

National Indian Youth Leadership Projectsusanleecarter@comcast.net

with J. Fred Springer, EMT Associates

fred@emt.org

A presentation for the 6th Annual Research and Evaluation in Adventure Programming Symposium

Gaithersburg, MDMarch 17-19, 2010

The Multi-Site High Risk Youth Study funded by SAMHSAFred Springer, EMT, Co-PI

48 site quasi-experimental design10,500 youth4 point longitudinal studyCommon instrumentComprehensive process dataHierarchical statistical designAmerican Evaluation Association Study of the

Year award, 2000Largest direct service ATOD prevention

evaluation ever implemented

4

Common Factors Reducing 30-day Drug Use

Behavioral Life-Skills FocusedConnection Building

Program CoherenceIntrospective Learning

Intense Contact

0

0.045

0.09

0.135

0.18

Effe

ct S

ize

.18

.135

.09

.045

Life-skillsFocused

ConnectionBuilding

Coherent IntrospectiveOrientation

IntenseContact

HRY Common Factor Findings

Promoting introspection / connectedness are critical common factors

Out of school more effective Manualized programs less effective Intensity countsPromoting behavioral skills more effective

than focus on knowledge and attitude change

Application: Project Venture

Project Venture identified as one of eight best programs in HRY study◦PV included all of the common factors

Incorporates connectedness building and introspection

Uses interactive experiential program delivery strategy

Essential Elements of Project Venture

Population◦Middle school aged American Indian youth◦High school aged AI service staff◦Mix of risk levels (at risk + high risk)

Program Components◦Experiential education◦Connecting to the natural world◦Physical and social-emotional challenge◦Service learning◦Focus on developing positive relationships◦Positive Youth Development approach

Project Venture: Overview

Experiential, adventure-based positive youth development program for American Indian youth

Recognized by NREPP as evidence-basedThe only American Indian ATOD prevention

program currently in NREPPWidely disseminated throughout the US and

CanadaTwenty year implementation and evaluation

history

Project Venture: Strategies

Outdoor, adventure activities:ropes course, rappelling, canoeing, backpacking, camping, mountain biking, etc.

Service LearningMeaningful projects that are youth centered and designed, address cultural, environmental and other community needs.

In a research-supported, year long sequential program.

Project Venture: Evaluation Findings

Positive substance abuse prevention findings (prevent, reduce, delay onset) especially related to alcohol abuse

Increased internal assetsIncreased external assets, especially

community and peer domains

Purpose of proposed study

We propose to advance knowledge concerning how adventure based strategies and activities can be strengthened, promoted, and replicated.

Hypothesis

Interactive /experiential activities that clearly incorporate reflective learning skills and promote connectedness to positive people and environments will promote intended behavioral change more effectively than interactive/experiential activities that have less focused emphasis on these interim outcomes.

Method notes

Quasi-experimentBaseline, exit, six and twelve month follow

upAssess similar participants in Project

Venture programs:◦6 programs using an enhanced version of PV◦6 “regular” PV programs◦(perhaps 6 groups with no PV)

Measures

Behavioral outcomes (ATOD, etc.)Connectedness (e.g., Michael Karcher’s

Hemmingway scales)Reflective learning (perhaps adapted from

education literature)

Analysis notes

Rigorous development and testing program to develop the measures

Rigorous statistical methodology including propensity scores for improving comparison equivalence

Hierarchical modeling to account for nesting effects, treatment interactions, treatment settings

Strong fidelity/coherence measurements

High Risk Youth Study, SAMHSA mapped onto Project Venture

Characteristics of Most Successful Prevention Programs

How Project Venture Incorporates these Characteristics through Interactive Experiential Delivery Strategies

Engagement Cultural content, leadership, values / experiential learning / connecting to the natural world / physical and emotional challenge

Intensity 150 + hours year (3 + hours per week)68 sessions / In & after school, weekends, summer

Reflective Orientation Experiential learning cycle, debriefing; Outdoor adventure: problem solving; Service learning: student led, problem solving; Camp, treks: journaling, values, norms, natural world, spiritual awareness; Native cultural values: spiritual traditions; In school: Problem solving, skill building, socialization

Connectedness Building Outdoor adventure: team building, trust building; Service learning: youth led, meaningful, relationship emphasis; service ethic; Camp, treks: build caring, cooperative community; Native cultural values; family events, cultural traditions

Life-skills Focused Outdoor adventure: cooperation, leadership, efficacy ; Service learning: problem solving, leadership, cooperation; Camp, treks: leadership, efficacy; In school: problem solving, skill building

Positive Environment Cultural content; adult participation, interaction, trust, support;

Coherence Clear purpose and process; training; collaboration; implementation project

Community Support Guided by Native Elders; Native cultural values; positive youth development approach; collaboration.

Other study ideas

Secondary data analysis of 5 years of pre/post survey data from Project Venture 7th grade participants (600) and matched comparison group (400) collected between 2005-2010

ANDLongitudinal follow-up study of PV program

participants who were in 7th grade in 2006 – including original program survey (adapted for older youth cohort) AND life history interviews