Upload
harriet-lewis
View
215
Download
2
Embed Size (px)
Citation preview
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Measuring Progress: Measuring Progress: Improved Outcomes for Improved Outcomes for
Children and FamiliesChildren and Families
Measuring Progress: Measuring Progress: Improved Outcomes for Improved Outcomes for
Children and FamiliesChildren and Families
Kathy HebbelerThe Early Childhood Outcomes Center
SRI International
Virginia EI Conference Creating Connections: Strengthening Partnerships between
Families and Providers Roanoke, VA, March 2007
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 2
ObjectivesObjectives
Review history and rationale related to the selection of child indicators
Describe what we have learned about implementing measurement systems
Describe why child indicators are important for improved services
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 3
Goal of the ECO CenterGoal of the ECO Center
Promote the development and implementation of child and family outcome measures for infants, toddlers and preschoolers with disabilities that can be used in local, state, and national accountability systems
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 4
Being clear…Being clear…
For VA, the word “indicator” refers to the 3 OSEP outcomes
to distinguish them from IFSP outcomes
ECO Centers materials use the word “outcomes” to refer to both
I will try to use “indicator” for the OSEP outcomes unless I mean all kinds of outcomes
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 5
What are Outcomes?What are Outcomes?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 6
The Concept of OutcomesThe Concept of Outcomes
An outcome is the result of some action or set of actions
Often expressed in the form of a statement
Examples: All passengers will wait 10 minutes or less
in the airport security line. All children will be fully immunized by 2.
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 7
The Concept of OutcomesThe Concept of Outcomes
Two parts: The expected result The action(s) that produces the
result May not be stated but always
present
Outcomes are the “effect” in cause and effect
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 8
Outcomes Closer to HomeOutcomes Closer to Home
EI programs provide high quality services and supports
EI services are family centered Families help their child develop
and learn Children take appropriate action to
meet their needs
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 9
Different Kinds/Levels of Different Kinds/Levels of OutcomesOutcomes
Program or systems outcomes EI programs provide high quality
services and supports EI services are family centered EI services are coordinated
What are the actions/activities that produce these results?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 10
Different Kinds/Levels of Different Kinds/Levels of OutcomesOutcomes
Child or family outcomes Families know their rights and advocate
effectively for their children Children have positive social
relationshipsWhat are the actions/activities that
produce these results?
Supports•Preservice training•Inservice training
Logic Model for Child and Family Logic Model for Child and Family OutcomesOutcomes
Good Federal policies and programs
Good State policies and programs
High quality services and supports for children 0-5 and their families
Good outcomes for children and families
Good Local policies and programs
Adequate funding
Strong Leadership
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 12
Two Critical Outcome/Indicator Two Critical Outcome/Indicator Questions for Building High Quality Questions for Building High Quality ProgramsPrograms
What is the intended result?
What is the action/set of action that will produce the intended result?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 13
How Looking at Child Indicators How Looking at Child Indicators
for EI Became Importantfor EI Became Important
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 14
Focus on AccountabilityFocus on Accountability Followed an era of focusing on and
measuring actions/activities Call for interest in the “ultimate
result”: child and family outcomes/indicators
Cut across both the public and private sectors
Funders demanded data to determine whether a program was doing what it was supposed to do
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 15
Critical Events in EI Critical Events in EI Accountability Accountability 1992 –Osborne and Graebler, Reinventing
Government 1993 – GPRA (Government Performance and
Results Act) passed Intervening years…. 2002 – PART finds there are no data on
outcomes for Part C 2003 – OSEP begins to ask states for EI child
outcome data (and funds the ECO Center!) 2005, 2006 – OSEP releases revisions to the
reporting requirements
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 16
PART Review for Part C and Part PART Review for Part C and Part B PreschoolB Preschool
Results Not Demonstrated
Part C“While the program has met its goal relating to the number of children served, it has not collected information on how well the program is doing to improve the educational and developmental outcomes of infants and toddlers served.”
Part B Preschool“The Department has no performance information on preschool children with disabilities served by this program.”
Read more at Expectmore.gov
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 17
Intervening YearsIntervening Years Special Education –
National study found poor outcomes for special education adolescents Results
Push to include students with disabilities in statewide assessment systems
Early Childhood Debate about whether child outcomes
should be measured at all Discussion of all the problems in trying to
measure outcomes for young children with disabilities
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 18
Measuring Child Indicators for EIMeasuring Child Indicators for EI
The PART findings put an end to the debate about whether or not to do it
Unfortunately, almost no progress had been made in the intervening years as to HOW to do it
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 19
Meanwhile: Re-thinking Meanwhile: Re-thinking Assessment in Early ChildhoodAssessment in Early Childhood
Major changes in last 15 years in how assessment of young children is viewed
Old position: Do not test little kids New position: Ongoing
assessment is part of a high quality early childhood program
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 20
What changedWhat changed
New and different tools became available Curriculum-based assessments were
developed General EC: Tools for 3-5 came first; 0-3
tools are coming now Interesting sidebar: Curriculum-based
assessments for programs serving children 0-5 with disabilities have been around for years
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 21
What changedWhat changed
The purpose of assessment was redefined
Not about: sorting, labeling, using to deny access (“the unready”)
Now about: Getting a rich picture of what children can do and can’t do and using that information to help them acquire new skills “progress monitoring”
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 22
What changedWhat changed
Assessment had always been seen as a process with multiple purposes
Distinctions were made between good and bad uses of assessment with young children
Good uses are now promoted For more information: NAEYC web site
(Position statement on Curriculum, Assessment and Evaluation)
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 23
Interesting IronyInteresting Irony
Even though the disability community had developed many curriculum-based assessment tools, currently **many** programs do not practice ongoing assessment Assess for eligibility only
The push for ongoing assessment to monitor how a child is doing and plan for instruction/intervention is coming from the general education community
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 24
Where We Are NowWhere We Are Now
Push for measuring outcomes for accountability (top down – but not just federal)
Push for ongoing assessment of progress (monitoring outcomes) as best practice in EC programs
Done well this could be a powerful blending of forces to improve outcomes for children
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 25
Why These 3 Indicators?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 26
OSEP Reporting Requirements: OSEP Reporting Requirements: Child Indicators Child Indicators
Positive social emotional skills (including positive social relationships)
Acquisition and use of knowledge and skills (including early language/ communication [and early literacy])
Use of appropriate behaviors to meet their needs
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 27
Origin of the Child IndicatorsOrigin of the Child Indicators
ECO Center stakeholder meetings followed by public comment period (2004)
First, collected themes and ideas Then, drafted and re-drafted
indicator wording Made recommendation to OSEP
(2005)
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 28
Themes from the Stakeholders: Themes from the Stakeholders: Child IndicatorsChild Indicators
Consistent with IDEA and legislative intent
Reflect what EI and ECSE are trying to do
One set for birth to 5 Reflect what is known about
development and learning
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 29
Themes from the StakeholdersThemes from the Stakeholders
Be compatible with best practice (esp. transdisciplinary service models, functional behaviors) Do not base them on domains
Have potential to influence practice in a positive way
Incorporate universal design Be readily understood
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 30
Make Outcomes/Indicators Make Outcomes/Indicators Functional Functional
Functional refers to things that are meaningful to the child in the context of everyday living
Refers to an integrated series of behaviors or skills that allow the child to achieve the outcomes
They are not a single behavior, nor are they the sum of a series of discrete behaviors
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 31
FunctionalFunctional
What does a child typically do?
Actual performance across settings and situations
How child uses his/her skills to accomplish tasks
Not the child’s capacity to function under unusual or ideal circumstances
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 32
RecommendationsRecommendations
Decision: One set of indicators birth to 5
Decision: Functional outcomes Decision: Global, not specific
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 33
The need for an introductory The need for an introductory statementstatement
Family and child outcomes are linked There are overarching goals for children
and family that cut across the outcomes Helpful to frame the outcomes with
these overarching goals
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 34
Concepts in the introduction: Goal for Concepts in the introduction: Goal for childrenchildren
The ultimate goal is for young children to be active and successful participants now and in the future in a variety of settings– in their homes, in their child care, preschool or school programs, and in the community
http://www.fpg.unc.edu/~eco/pdfs/eco_outcomes_4-13-05.pdf
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 35
Concepts in the introduction: Concepts in the introduction: Goal for childrenGoal for children
Active and successful participants Now and in the future In variety of settings
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 36
Concepts in the introduction: Concepts in the introduction: Goal for familiesGoal for families
Enable families to provide appropriate care for their child
Have resources they need to participate in community activities
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 37
Concepts in the introduction:Concepts in the introduction:Effective programsEffective programs
Support families in their quest for a satisfactory quality of life
Provide needed supports and services in a timely fashion
***Remember: Outcomes are the result of actions***
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 38
Concepts in the preface: Concepts in the preface: Outcomes and accountabilityOutcomes and accountability
Outcomes determined by variety of factors
Not all families and children will achieve all outcomes
BUT, system should still strive to achieve them
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 39
ECO Family OutcomesECO Family Outcomes
Understand their child’s strengths, abilities, and special needs
Know their rights and advocate effectively for their children
Help their children develop and learn Have support systems Access desired services, programs,
activities in their community
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 40
The OSEP Reporting Categories
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 41
OSEP Reporting CategoriesOSEP Reporting Categories
Percentage of children who: a. Did not improve functioningb. Improved functioning, but not sufficient to
move nearer to functioning comparable to same-aged peers
c. Improved functioning to a level nearer to same-aged peers but did not reach it
d. Improved functioning to reach a level comparable to same-aged peers
e. Maintained functioning at a level comparable to same-aged peers
3 outcomes x 5 “measures” = 15 numbers
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 4213
Functioning
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 43
Entry
14
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 44
Entry Exit
15
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 45
Entry Exit
16
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 46
CISF Numbers to OSEP CISF Numbers to OSEP Reporting CategoriesReporting Categories
Each possible combination of numbers (and answer to the “b” question at entry and exit generates a line, i.e., a developmental trajectory)
Each trajectory corresponds to an OSEP category
Taking measurements more frequently than entry and exit generates more powerful information
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 4713
Functioning
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 48
OSEP Reporting Categories: OSEP Reporting Categories: Child OutcomesChild Outcomes
Percentage of children who:
a. Did not improve functioningb. Improved functioning, but not sufficient to
move nearer to functioning comparable to same-aged peers
c. Improved functioning to a level nearer to same-aged peers but did not reach it
d. Improved functioning to reach a level comparable to same-aged peers
e. Maintained functioning at a level comparable to same-aged peers
3 outcomes x 5 “measures” = 15 numbers
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 49
Comparing to Same Aged PeersComparing to Same Aged Peers
Deficit model? Not individualized? Goal of EI: Active and successful
participation now and in the future Kindergarten, school readiness, having
friends, community participation Setting high expectations Moving from 1 to 2 = moving from 4 to 5 How much progress is enough?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 50
COSF/CISF Implementation Issues
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 51
OverallOverall
Child indicators have been very well received
Many states have embraced outcomes measurement at some level
State Approaches to Child Outcomes Measurement - Part C ProgramEarly Childhood Outcomes Center - Information current as of 1/3/07
COSF
Publisher’s on-line system
One tool statewide
Other
Don’t know
Legend:
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 53
Assessing functioningAssessing functioning
Process helps teams think about the child functionally across a variety of settings and situations
Requires good assessment practices (and suggests need for improving current practice) Assessment only at entry. What is the
role of assessment in EI? Moving from domains to functioning
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 54
Assessing functioningAssessing functioning
Providers need a way to have a conversation with families about child functioning across settings and situations
Providers need training in explaining assessment results to parents “soft pedal”
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 55
Age ExpectationsAge Expectations
Requires providers understand typical child development and how children of different ages are expected to function in all 3 indicators
Reveals need for training in this area
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 56
Working with FamiliesWorking with Families
Changing the conversation between providers and families?
Changing the nature of IFSPs? “Do we write IFSPs to these
indicators?” Would that be such a bad thing?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 57
Opportunity?Opportunity?
Measurement of indicator data can encourage more frequent collection of assessment data Progress monitoring
Improve practice… And provide outcome indicators
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 58
Validity and ReliabilityValidity and Reliability
Validity is not a characteristic of a tool Validity is a characteristic of a set of
scores produced by a tool CISF is judgment-based Substantial research to support that
with defined criteria, providers and parents can reach reliable judgments
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 59
Validity and ReliabilityValidity and Reliability
ECO working on procedures to help states assess validity of their data
Good training and ongoing supervision will be critical
Quality assurance will be critical
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 60
Some ConcernsSome Concerns
Being implemented very quickly Not all implementers are well
trained Few states have procedures in
place for assuring the process is high quality and reliable
States doing the minimum will not have very useful data
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 61
How Will These Data Be Used?How Will These Data Be Used?
Hopefully not at all -- UNTIL checks are in place to insure they are of high quality
Meaning needs to be attached to the findings by those closest to the program
Craft your own message
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 62
Next steps for ECONext steps for ECO
More support and training materials
More guidance on supervision and quality assurance
More guidance on using data for program improvement
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 63
A VisionA Vision
(What Does This Mean for You)(What Does This Mean for You)
Supports•Preservice training•Inservice training
Logic Model for Child and Family Logic Model for Child and Family OutcomesOutcomes
Good Federal policies and programs
Good State policies and programs
High quality services and supports for children 0-5 and their families
Good outcomes for children and families
Good Local policies and programs
Adequate funding
Strong Leadership
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 65
Building a high quality systemBuilding a high quality system
What if every piece of the system functioned as it was supposed to?
What information is needed to know if this is happening?
What information is needed to know how to identify and address weaknesses in the system?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 66
Your contributionYour contribution
Where do you fit? What is your contribution to good
outcomes? What actions do you take that produce good child and family outcomes?
What information would you need to know if your actions were producing good outcomes?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 67
The ideal outcome system gives persons at each level of the system the information to know if they are producing good outcomes -and if not, to change their practices/policies/procedures so eventually they will.
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 68
Remember the goal for childrenRemember the goal for children
Active and successful participation now and in the future
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 69
And an overarching goal for And an overarching goal for familiesfamilies
To enable families to provide care for their child and have the resources they need to participate in their own desired family and community activities
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 70
The Take Home MessageThe Take Home Message
Measuring outcomes is not primarily about data
It is about doing good things for children and families
And using data as a tool to help programs and providers to know whether what they are doing is making a difference
Early Childhood Outcomes CenterEarly Childhood Outcomes Center 71
Finding out more Finding out more
Additional information about OSEP requirement and state activities
New resources will be coming including training materials, resources on assessment and typical child development, and materials for parents
www.the-eco-center.orgwww.the-eco-center.org