Upload
guy
View
30
Download
0
Embed Size (px)
DESCRIPTION
The Family Outcomes Survey: Revisions, Data, Uses. Measuring Child and Family Outcomes National TA Meeting Baltimore, MD August 26, 2007. Don Bailey, RTI International Robin Nelson, Texas Part C Program Nyle Robinson, Illinois Part C Program Chelsea Guillen, Illinois Part C Program - PowerPoint PPT Presentation
Citation preview
1
The Family Outcomes Survey:Revisions, Data, Uses
Don Bailey, RTI InternationalRobin Nelson, Texas Part C Program
Nyle Robinson, Illinois Part C ProgramChelsea Guillen, Illinois Part C Program
Melissa Raspa, RTI International
Measuring Child and Family Outcomes National TA MeetingBaltimore, MDAugust 26, 2007
Goals for today
Remind everyone of the goals and format of the Family Outcomes Survey
Describe the revisions made to the survey this year and the rationale
Summarize selected data from an initial pilot study using the scale in Illinois and Texas
Describe plans to validate the scale Discuss ways the scale could be used to
help states answer questions of interest
ECO Family Outcomes
Understand their child’s strengths, abilities, and special needs
Know their rights and advocate effectively for their children
Help their children develop and learn Have support systems Access desired services, programs, activities
in their community
ECO Scale:Family Outcomes Survey
Three items for each of the five proposed family outcomes
Each item rated on a scale from 1-7 Descriptive statements for ratings of 1, 3, 5, 7 Blanks left for “in between” ratings of 2, 4, 6 Three additional items for states to use in
responding to APR requirements (two versions, one for Part C, one for Part B)
UNDERSTANDING YOUR CHILD'S STRENGTHS, ABILITIES, AND SPECIAL NEEDS
1.Your child is growing and learning. How much does your family understand about your child’s development?
12 3 4 5 6 7
We are just beginning to understand our child's development
We understand some about our child's development
We understand a good amount about our child's development
We understand a great deal about our child's development
2. Some children have special health needs, a disability, or are delayed in their development. These are often referred to as “special needs.” How familiar is your family with your child's special needs?
12 3 4 5 6 7
We are just beginning to understand our child’s special needs
We understand some about our child’s special needs
We understand a good amount about our child’s special needs
We understand a great deal about our child’s special needs
3. Professionals who work with you and your child want to know if the things they do are working. How often is your family able to tell if your child is making progress?
12 3 4 5 6 7
We seldom can tell if our child is making progress
We sometimes can tell if our child is making progress
We usually can tell if our child is making progress
We almost always can tell if our child is making progress
THE HELPFULNESS OF EARLY INTERVENTIONThe next questions ask how well early intervention has helped your family. When answering, think about the early intervention services you have received.
16. To what extent has early intervention helped your family know and understand your rights?
1 2 3 4 5 6 7
Early intervention has done a poor job of helping us know our rights
Early intervention has done a fair job of helping us know our rights
Early intervention has done a good job of helping us know our rights
Early intervention has done an excellent job of helping us know our rights
17. To what extent has early intervention helped your family effectively communicate your child’s needs?
1 2 3 4 5 6 7
Early intervention has done a poor job of helping us communicate our child’s needs
Early intervention has done a fair job of helping us communicate our child’s needs
Early intervention has done a good job of helping us communicate our child’s needs
Early intervention has done an excellent job of helping us communicate our child’s needs
18. To what extent has early intervention helped your family be able to help your child develop and learn?
1 2 3 4 5 6 7
Early intervention has done a poor job of helping us help our child develop and learn
Early intervention has done a fair job of helping us help our child develop and learn
Early intervention has done a good job of helping us help our child develop and learn
Early intervention has done an excellent job of helping us help our child develop and learn
What have we done in the past year?
States have begun to use the instrument Conducted a pilot study in Illinois and Texas
(summer, 2006) Reviewed and revised the scale (fall, 2006) Submitted application to IES to validate the scale
(summer, 2007) Completed cognitive testing of scale with Spanish
speaking families (summer, 2007) Currently conducting a second, larger pilot study in
Illinois and Texas (summer, 2007)
Questions Addressed in Pilot Study
How should the survey be distributed and what response rates might be expected?
What distribution of responses is seen for each item? Are respondents more likely to pick the odd
numbered items? How did responses on the APR items compare with
the outcome items? Did patterns of responses differ across the states? Did ratings of outcomes vary as a function of
language of the survey (English vs. Spanish?) What is the internal reliability of the instrument?
Methodological Similarities
Identical survey Similar accompanying materials Distribution method almost
identical Identical return mail methods Timeframe was almost the same
Differences in Approach
Texas used sampling Illinois surveyed all programs statewide Illinois found that IFSPs near 3 were
often extended, new meetings not held So, Illinois did not distribute surveys within
3 months of age 3 during the pilot Texas did more follow-up, utilized
replacement, Illinois depended on volume
Sampling, Distribution, Return Rates
Illinois Did not sample Distributed 1,976 surveys at 6-month and/or annual reviews
Texas Stratified programs by region and size Randomly selected 3 programs from 7 regions Randomly selected 1262 families
Return rates Texas – 44%, representative Illinois – 29%, not as representative
Data presented today represent 890 surveys (~100 Spanish)
Additional Construct: Looking Towards the Future
Both states added three additional items to the survey to assess: Current life situation Transition preparation Future life situation
Understanding Your Child’s Strengths, Abilities & Special Needs
0
10
20
30
40
50
60
Perc
enta
ge
1 2 3 4 5 6 7
Response
Item 1: How well do you understand your child's development?
Understanding Your Child’s Strengths, Abilities & Special Needs
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 2: How much do you know about your child's special needs?
Understanding Your Child’s Strengths, Abilities & Special Needs
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 3: Are you able to tell if your child is making progress?
Knowing Your Rights & AdvocatingFor Your Child
0
10
20
30
40
50
60
Perc
enta
ge
1 2 3 4 5 6 7
Response
Item 4: Do you know what programs and services are available for your child and family?
Knowing Your Rights & AdvocatingFor Your Child
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 5: How comfortable are you participating in meetings with professionals to plan services or activities?
Knowing Your Rights & AdvocatingFor Your Child
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 6: How well do you know your rights and what to do if you are not satisfied?
Helping Your Child Develop & Learn
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 7: How would you describe your ability to help your child develop and learn
Helping Your Child Develop & Learn
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 8: How would you describe your ability to help your child learn to behave the way you would like?
Helping Your Child Develop & Learn
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 9: How much are you able to help your child learn or practice new skills at home or in the community?
Having Support Systems
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 10: Does your family have someone you trust to listen and talk with you?
Having Support Systems
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 11: Do you have someone you can call on when you need help with things?
Having Support Systems
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 12: How much is your family able to do the things you enjoy?
Accessing Your Community
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 13: How would you describe the medical care you have for your child right now?
Accessing Your Community
0
10
20
30
40
50
60
Perc
enta
ge
1 2 3 4 5 6 7 NA
Response
Item 14: How would you describe the child care you have for your child right now?
Accessing Your Community
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7 NA
Response
Item 15: How would you describe your child's participation in activities right now?
Findings on Range of Responses
The items generated a range of responses We are not getting a normal distribution at the
item level, but what kind of distribution would you expect?
Ratings vary across items, suggesting that parents respond differentially and specifically to each item
77% of responses used the odd-numbered ratings (the ones with descriptors)
Knowing and Understanding Rights
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 6: How well do you know your rights and what to do if you are not
satisfied?
Knowing and Understanding Rights
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 6: How well do you know your rights and what to do if you are not
satisfied? (Mean = 4.81)
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 16: To what extent has early intervention helped you know and
understand your rights? (Mean = 5.54)
Helping your Child Develop and Learn
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 7: How would you describe your ability to help your child develop and
learn?
Helping your Child Develop and Learn
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 7: How would you describe your ability to help your child develop and
learn? (Mean = 5.11)
0
10
20
30
40
50
60
Per
cent
age
1 2 3 4 5 6 7
Response
Item 18: To what extent has early intervention helped your family be able to help your child develop and learn?
(Mean = 5.98)
Summary Comments and Analyses of APR and Outcome Items
The two sets of items do not generate equivalent response patterns
APR items more likely to have ratings of 6 and 7
Correlations are moderate .58 knowing rights .44 helping child develop and learn .26 communicate needs
Other Findings
IL & TX had almost identical mean scores across items, although patterns across some items differed
Same highest (EI help child dev & learn) and lowest (participation in activities) rated items
High internal consistency for total scale and OSEP items (Cronbach’s alpha > .90)
Factor analysis yields 1-2 factors Means for Spanish-speakers considerably
lower than English-speakers on most items
Specific Question Comparisons
IL slightly higher on “access to medical care” Better, more inclusive Medicaid system
TX slightly higher on “knowledge of rights” Both states have strong emphasis; does service
model make a difference? IL lower on “transition preparation”
Already aware of the problem, IL has put considerable emphasis on improving transition
Example: Effects of Primary Language (English vs. Spanish)
beginning to learn
programs and srvcs
2 3 4 5 6 very aware of program
and srvcs
know available services
0
10
20
30
40
Perc
ent
Lang
Eng
Spa
Example: Effects of Primary Language (English vs. Spanish)
beginning to learn
programs and srvcs
2 3 4 5 6 very aware of program
and srvcs
know available services
0
10
20
30
40
Perc
ent
Lang
Eng
Spa
just beginning 2
34
56
understand very well
understand child's development
0
10
20
30
40
50
Perc
ent
Lang
Eng
Spa
Value of Comparisons
Does service model make a difference? Texas SC has more contact with family
Did Illinois exclusion of kids nearest exit have an impact? Families were receiving services longer
Effect of demographics? Culture, acculturation
Demographic Comparisons
Illinois 55% White 24% Hispanic 17% Black 3% Asian 13% primary
language Spanish 60% Medicaid
Texas 38% White 47% Hispanic 12% Black 2% Asian 19% primary
language Spanish 61% Medicaid
Scale Revision
Systematic review by survey methodologists
Cognitive testing with 12 parents
Results from Scale Analysis
Problems identified with unclear meanings and inconsistencies across items
MUCH back and forth between methodologists and IL/TX folks about specific wording, resulting in significant revision (rewording only, same format and # items
Revised instrument now available on ECO web site
Consistent Set of Descriptors Used at Each Level
7 Great deal, almost always, very,
almost all, excellent 5 Usually, good amount, generally,
many, good 3 Some, sometimes, somewhat, fair 1 Just beginning, seldom, few, poor
Changes in APR Item Modifiers
Old Version1 - EI has not
helped…3 - EI has done a
few things…5 - EI has provided
good help…7 – EI has provided
excellent help…
Current Version1 – EI has done a
poor job…3 – EI has done a
fair job5 – EI has done a
good job7 - EI has done an
excellent job
IES Application: Validating the Family Outcomes Survey
Specific Aim 1: Determine the test-retest reliability of the FOS by examining stability of scores over a short (two-week) period of time.
Specific Aim 2: Establish the criterion validity of scores on the FOS by examining its relationship with six well-established criterion measures of family well being, addressing family empowerment, social support, parenting self-efficacy, quality of life, hope, and parenting stress.
Specific Aim 3: Determine sensitivity to change in scores on the FOS by comparing changes in family outcomes over 18-months to changes in scores on the six criterion measures.
Specific Aim 4: Document the convergent validity of scores on the FOS by examining the relationship between changes in family outcomes over 18 months and the extent to which early intervention programs used recommended practices in providing services to families.
Using the Family Outcomes Survey:What are the possibilities?
Using the scale to describe and classify outcomes attained by families
Using the scale to identify child, family, or program variables associated with outcome attainment
Using the scale to improve programs at the state or local level
Using the scale to work with individual families
Data Uses
How can states use the data from the Family Outcomes Survey to help identify both the “success stories” of early intervention and the families who may need more immediate help?
How do states track the progress of families in early intervention?
How can states examine the relationship between the quality and delivery of services and family outcomes?
Are family outcomes related to child outcomes?
Data Uses
Three areas of analysis Measurement-related analyses
Internal consistency Factor structure
Descriptive analyses Basic descriptive statistics Grouping families
Predictive analyses Describing variation between families Predicting variation between families
Measurement-Related Analyses
Internal consistency Total scale Five subscales
Factor Structure One factor Multiple factors
Descriptive Analyses
Basic descriptive statistics Mean, min, max, standard deviation
Threshold scores 5 or higher = outcome attained 3 or lower = “score of concern”
Score variation Most variable respondent Least variable respondent
Descriptive Analyses
Grouping families with similar response patterns Empirically derived
Success stories (scores of 5 or higher) Borderline (scores consistently 3 to 5) Immediate concern (scores 3 or below)
Statistically derived Clusters of families with similar response
patterns
Predictive Analyses
Two types of analyses: describing variation and predicting variation
Three types of “predictor variables” Child and family variables (e.g., health
status, eligibility, ethnicity, income, mother’s education)
Service quality variables (e.g., family-centered practices)
Service delivery or models (e.g., service coordination model, amount of services, location of services)
Predictive Analyses
Describing variation between families Do outcomes vary by child and family
characteristics? Does the quality of early intervention
services relate to family outcomes? Are service delivery models
associated with family outcomes?
Predictive Analyses
Predicting variation between families Do families of children who just entered
early intervention have lower family outcomes than families whose children have been receiving services for a longer period of time?
Is a dedicated and not independent service coordination model associated with better family outcomes?
Family Outcomes vs. Child Outcomes
Needs
Family
Concerns
Resources Priorities
Goals
ChildDevelopmental Status
Functional Abilities
Disability Type
Learning Style
Early Intervention
Child-Focused Services
Family-Focused Services
· Demonstrate positive social – emotional skills
· Acquire and use knowledge and skills
· Use appropriate behavior to meet needs
· Understand child’s abilities and special needs
· Know rights and advocate effectively
· Help child develop and learn
· Have support systems· Access the community
Child Outcomes
Family Outcomes
Discussion