Getting Ready to Measure Ready Children Ready …uwsemi.3cdn.net/d07fb097a035b31f9f_25m6vgz93.pdf · Produced by SPEC Associates Detroit, Michigan Getting Ready to Measure Ready Children

Produced by

SPEC Associates

Detroit, Michigan

Getting Ready to Measure Ready Children

Ready Communities…

Baseline SIF Evaluation Report

July 2014

“My child is much more social

than when we started P.A.T.”

…Program participant

TABLE OF CONTENTS

Executive Summary ........................................................................................................................................ i

Brief Description of the Ready Children Ready Communities Interventions ................................................ 1

Evaluation Design .......................................................................................................................................... 3

Impact Evaluation Design and Planned Analyses ..................................................................................... 3

Counterfactual Conditions .................................................................................................................. 4

Non-equivalent Comparison Group .................................................................................................... 4

Baseline equivalence analysis: Propensity Score Matching and Counterfactual Items ...................... 5

Planned Statistical Analyses ................................................................................................................ 6

Missing data analysis ....................................................................................................................... 6

Pre-Post HOME, Pre-Post DECA, and RPT outcome analysis .......................................................... 7

Implementation Evaluation Design and Planned Analyses ....................................................................... 7

Quality Control Processes in Place ...................................................................................................... 8

Fidelity to program design .................................................................................................................. 8

Quality of Program Delivery ................................................................................................................ 9

Participant satisfaction ...................................................................................................................... 10

Implementation Data Analysis ................................................................................................................ 11

Data Collection and Measurement ............................................................................................................. 13

Definition of Program Participants ......................................................................................................... 13

Data Collection Procedures..................................................................................................................... 14

Data Collection Instruments ................................................................................................................... 15

Intake (Enrollment) Form .................................................................................................................. 15

Home Visitation Program Survey ...................................................................................................... 15

Play & Learn Program Survey ............................................................................................................ 15

RPT Parent Survey ............................................................................................................................. 16

HOME 0-3 and 3-6 Forms .................................................................................................................. 19

Devereux Early Childhood Assessment (DECA) ................................................................................. 20

Ages and Stages Questionnaire ......................................................................................................... 20

Screening and Referral Tracker ......................................................................................................... 21

Evaluation Findings to Date ........................................................................................................................ 22

Implementation Evaluation Findings ...................................................................................................... 22

Fidelity to Program Design: PAT Home Visits ......................................................................................... 22

Fidelity to Program Design: Play & Learn Groups ................................................................................... 23

Program Delivery .................................................................................................................................... 25

Date program delivery began ........................................................................................................... 25

Number of program units ................................................................................................................. 25

Quality/satisfaction with program delivery ...................................................................................... 25

Impact Evaluation Findings ..................................................................................................................... 29

Validation Results: RPT Parent Survey Item Reliability Analyses ...................................................... 29

Comparison RPT and ECLES Items ..................................................................................................... 30

Lessons Learned .......................................................................................................................................... 32

What helped the evaluation? ................................................................................................................. 32

What hindered the evaluation? .............................................................................................................. 32

Appendices .................................................................................................................................................. 35

Executive Summary This report presents the design, data collection and analysis plan, progress to date, and lessons learned during the first year of SPEC Associates’ evaluation of Ready Children Ready Communities.

Evaluation Design SPEC Associates (SPEC) worked collaboratively with the Macomb Intermediate School District, Leaps & Bounds Family Services and Macomb Family Services to collect moderate evidence via a quasi-experimental Retrospective Pre-Posttest (RPT) research design to test the hypothesis that parents who participate in Parents as Teachers (PAT) home visits will show greater improvement than comparison parents in their:

• Understanding of the principles of early childhood development • Valuing of reading to their children daily • Knowledge of how to access community services

The program group consists of parents who receive at least five PAT home visits. The comparison group includes (a) parents who participate in Play & Learn Group sessions conducted by one of the three participating agencies but choose not to receive PAT home visits, and (b) parents who receive PAT home visits who do not participate in Play & Learn groups. Propensity score matching on seven risk factors will be used to control in the analyses for selection bias. Exploratory evidence will also be collected to test the hypotheses that: (a) children whose family members participate in PAT home visits experience improved general support and literacy stimulation at home, and (b) children with developmental delays in social emotional health show improvement in this growth area after receiving intensive mental health services.

Fidelity and Quality Assessment Fidelity of PAT home visits and Play & Learn Group implementation was assessed by an early childhood education expert who observed one session/visit of each project staff using guidelines developed through review of the PAT curriculum and early childhood best practices. The assessment found that PAT home visits were being implemented with fidelity to the model. In the absence of a standardized curriculum for Play & Learn groups, the expert assessment examined quality of implementation against best practices and found all groups were implemented with high quality. Quarterly Play & Learn Group program surveys also found that in both quarters of data collection so far, parents gave high quality ratings to their experiences:

97% to 100% reported that the facilitators were friendly and respectful toward adults and helped them learn from each other

95% to 100% reported that they learned something about how play encourages their child’s development

91% to 95% reported they received a book or other resources during their session 79% to 86% reported their children were able to use materials they don’t have or use at home

Comments parents wrote to open-ended questions support the quantitative results. Parents praised the Play & Learn Groups and the most frequent comment for change was wanting more or longer sessions and/or activities for their younger children (who are ineligible for the program).

Data Collection and Measurement All data are collected by project staff. All parents enrolled in PAT home visits and/or Play & Learn Groups are administered an Intake Form that collects demographic and risk data and documents the consent of

SPEC Associates for Macomb Intermediate School District Page i Baseline SIF Evaluation Report

parents to be surveyed for the evaluation. The Home Observation for Measurement of the Environment (HOME) versions 0-3 and 3-6 are used as the pre-post assessment of changes in family literacy environment. On a quarterly basis, consenting parents who participated in at least four Play & Learn Group sessions and/or five PAT home visits are asked to complete an RPT survey asking them to rate themselves on the three expected outcomes “now” and “before” participating in the intervention. Test development activities were undertaken this year which yielded RPT scales for each of the three outcomes with more than acceptable Chronbach’s Alpha coefficients of: 0.859 for knowledge of how to access community resources

0.801 for understanding principles of early childhood development 0.690 for valuing reading to child daily The RPT survey questionnaire was translated into Arabic using standard translation-back translation techniques. Counterfactual items were created, pilot tested and selected for inclusion on the final RPT survey to test the extent of social desirability among the survey responses. The standardized and validated Ages and Stages Questionnaire (ASQ) is used to screen children for developmental delays upon enrollment into the PAT home visits. If indicated by ASQ results, the more in-depth ASQ-Social Emotional (ASQ-SE) is used to assess families’ needs for referral to mental health services. Families referred for intensive mental health services are administered the Devereux Early Childhood Assessment on a pre-post basis. Referrals of children on whom one or more screening was performed for developmental delays are tracked using a Screening and Referral Tracker. Parents’ reports of Play & Learn Group quality are measured by a Play & Learn Program Survey.

Program Delivery In Year 1, the program served 177 families. Through June 2014, the program served 261 families. Since the program began: 467 PAT home visits were conducted

301 Play & Learn Group sessions were facilitated 356 Social Emotional (intensive mental health) home visits were made

28 Outreach activities were completed (not including flyer distribution and email blasts)

Lessons Learned The evaluation was successful in implementing the planned research design, creating reliable data collection tools, and instituting processes for assuring data collection is managed appropriately by the three participating agencies. Stakeholder engagement with the evaluation via an Evaluation Advisory Group, joint problem-solving and measure creation, and monthly check-in calls were keys to Year 1 successes. Having an IRB that could rapidly review applications/changes, using a third-party data entry company, and community relationships that the participating agencies had with many schools also contributed to success. Challenges to the evaluation in Year 1 included slower than anticipated SEP approval and parent recruitment, late development of the database partly due to staff turnover at MISD, slow start of data collection because of the time needed to develop the RPT survey, failure to receive anticipated help with fund development for matching funds, unanticipated need to use grant funds to pay for third-party evaluation services, tough winter weather and the general difficulties of working with high risk communities. These challenges will ultimately result in fewer numbers of parents included in the evaluation than originally anticipated.

SPEC Associates for Macomb Intermediate School District Page ii Baseline SIF Evaluation Report

Brief Description of the Ready Children Ready Communities Interventions

Ready Children Ready Communities aims to make Michigan’s children ages 0-5 living in South Warren, Detroit and Macomb County ready for a successful start in Kindergarten by:

• Screening and assessing children for developmental delays and mental health needs • Providing Home Visitation for at risk families using the evidence-based Parents As

Teachers (PAT) curriculum • Providing a project therapist, as well as referrals to other mental health services, for

children and families who need more intensive support • Offering Play & Learn Groups for parents1 and pre-schoolers at seven to eight community

sites, including two within existing Early Learning Centers • Staging parent-child community events focused on early childhood learning for families,

schools and communities and on providing information about resources such as the Imagination Library and relevant community services

The three interventions of Ready Children Ready Communities studied by this evaluation are Parents as Teachers (PAT) home visits, Play & Learn Groups, and intensive mental health services. The PAT home visit model is approved by the federal Health Resources and Services Administration and the Administration for Children and Families as a home visiting model meeting the evidence-based criteria of the Maternal, Infant, Early Childhood program. Research has demonstrated that Kindergarten readiness is sharply impacted by early literacy home visiting programs.2

The Play & Learn Groups model is unique to Ready Children Ready Communities and has not yet been designated as an approved evidence-based model. Play & Learn Groups offer an environment that supports early learning and gives parents the information and skills that they need to provide the appropriate, stimulating, literacy-rich environment for their children. A child's first teacher is his or her parents who care for him during the day. Experts say that the choices families make regarding literacy are more important than the family's income or the caregiver's educational background in predicting future success.3 Research shows that when young children and adults interact through talking, singing, and

1 The term “parent” in this report is used to refer to any adult family member or legal guardian who is acting in the role of the parent vis a vis participation in the intervention. 2 For example, one research study demonstrated that “at age six, children who participated in the NFP home visiting program in Memphis had higher cognitive and vocabulary scores than those in the control group.” (D. Olds, et al., "Effects of Nurse Home-Visiting on Maternal Life Course and Child Development: Age 9 Follow-up Results of a Randomized Trial." Pediatrics 120 (2007): e832-e845). 3 Ballen, J., & Moles, O. (1994, September). Strong Families, Strong Schools: Building Community Partnerships for Learning. Washington, DC: U.S. Department of Education. [ERIC No. ED 371909].

Screen

Community Events

Play & Learn Groups

HOME Visits

Mental Health Services

Increasing Kindergarten

Readiness

SPEC Associates for Macomb Intermediate School District Page 1 Baseline SIF Evaluation Report

rhyming together they stimulate language development which creates the foundation for learning to read.4 These are the types of activities parents and children engage in during Play & Learn Groups.

All children referred to the PAT home visits receive screening for developmental delays in social emotional health. Those who are determined to be at risk for social emotional delays receive intensive mental health services, tailored to the specific cognitive, social and/or emotional needs of the child. Experts agree that early intervention is more effective than later interventions,5 and that “it is essential to treat young children’s mental health problems within the context of their families, homes, and communities.”6

Two community-based agencies are implementing the PAT home visits: Leaps & Bounds Family Services and Macomb Intermediate School District. These two agencies plus the Macomb Family Services are implementing Play & Learn Groups. Macomb Family Services is also implementing intensive mental health services.

4 Hart, B & Risley, T.R. (1995). Meaningful differences in the everyday experiences of young American children. Baltimore, MD: Brookes Publishing. 5 National Symposium on Early Childhood Science and Policy (undated) In Brief: The Impact of Early Adversity on Children’s Development www.developingchild.harvard.edu. 6 Harvard University Center on the Developing Child (undated) In Brief: Early Childhood Mental Health www.developingchild.harvard.edu.


http://www.developingchild.harvard.edu/

http://www.developingchild.harvard.edu/

Evaluation Design SPEC Associates (SPEC) is conducting an impact and implementation evaluation of Ready Children Ready Communities. This section of the report describes the evaluation questions, design, and planned analyses. The Ready Children Ready Communities logic model is contained in Appendix A.

Impact Evaluation Design and Planned Analyses The research hypothesis guiding the impact evaluation is that parents who participate in PAT home visits will improve more on the expected program outcomes than parents who participate only in Play & Learn Groups. The program outcomes for parents are:

• Better understanding of the principles of early childhood development

• Greater valuing of reading to their children daily

• More knowledge of how to access community resources

The first research hypothesis for children is that after parents participate in PAT home visits, preschoolers will experience:

• Improved literacy stimulation at home • Improved general support at home

The second research hypothesis for children is that developmental delays in social emotional health will be reduced for children receiving intensive mental health services.

If parents and children achieve these outcomes, more children in the community are expected to be academically ready for Kindergarten. For many of these children Kindergarten will be years away. Therefore, Kindergarten readiness of children is not being measured as part of the evaluation. More appropriately, the following two immediate outcomes for children are being assessed:

• Early detection of developmental delays • Access to services needed by children with developmental delays

The impact evaluation is addressing the following evaluation questions:

1. Do parents report significantly higher ratings on any of the three outcomes after participating in PAT home visits?

2. Do parents report significantly higher ratings on any of the three outcomes after participating in Play & Learn Groups?

3. Do parents who participate in PAT home visits report greater improvement on the three outcomes than parents who only participate in Play & Learn Groups?

“I learned my child needs more socialization so he can learn to share and get along with other kids.”



4. Do parents who participate in PAT home visits report their children as more ready for Kindergarten than parents who only participate in Play & Learn Groups?

5. Is there significant improvement from the first to the last home visit in the quality and quantity of stimulation and support in the home environment for at least 65% of the children whose parents participate in the PAT home visits?

6. Do at least 65% of children receiving early childhood mental health services improve on at least one protective factor (initiative, self-control, attachment) or decrease on at least one behavioral concern (withdrawal/depression, emotional control problems, attention, aggression)?

Counterfactual Conditions The evaluation of Ready Children Ready Communities will gather moderate evidence of the impact of PAT home visits on creating a literacy-rich home environment for children as reflected in the parent and child outcomes listed above. The impact evaluation uses a quasi-experimental Retrospective Pre-Posttest (RPT) design to examine the degree to which PAT home visits contribute to expected parent outcomes apart from and beyond participating in Play & Learn Groups. This design is appropriate since high-risk families being served by Ready Children Ready Communities cannot be randomly assigned to intervention and control groups.

Non-equivalent Comparison Group The comparison group for PAT home visits are the parents in the Play & Learn Groups who do not get home visits. This is the appropriate comparison group because the Play & Learn Groups are a major source for referral of high-risk families into PAT home visits. Since all Play & Learn Group participants are invited to receive PAT home visits, the major difference between the PAT home visit group and Play & Learn comparison group is self-selection. Demographically, these two groups are expected to be similar.

One risk of this design is that the non-equivalence of the comparison group may yield results contrary to the research hypothesis. Self-selection, for example, may result in parents who already understand principles of early childhood development and who already value reading to their child electing to participate in the PAT home visits as well as the Play & Learn Groups. The “before” scores on the RPT survey will allow for an exploration of self-selection by providing information regarding how much parents claim they already understood principles of child development, valued reading to their child, and knew about community resources before participating in these programs. The evaluation will assess whether the two groups are similar in this regard.


Baseline equivalence analysis: Propensity Score Matching and Counterfactual Items Propensity score matching will be used to control in the analyses for selection bias of parents who elect to participate in the PAT home visits. Propensity scores will be generated for Play & Learn Group and PAT home visit parents using the following variables collected using the Intake Form (described below) that estimate risk of literacy problems:

• Low income • Diagnosed disability or identified developmental delay • Severe or challenging child behavior • Primary home language other than English • Low parental education attainment • Abuse/neglect of child or parent • Environmental risk (e.g., significant loss in family, chronic illness, teen

parent, homelessness, etc.)

These seven variables are the same risk factors used throughout Michigan by the Michigan Department of Education’s Great Start Readiness Program to rank children and determine eligibility for the state-funded preschool program. The risk factor list was initially much longer (including more than 20 risk factors), but changed in the last five years to create an easier system for enrolling children. It is important to note that many of the more specific risk factors from the larger list were combined to be part of the broader risk factors from the new list.

The second method for validating the impact evaluation results is the inclusion of counterfactual items on the Play & Learn Group and PAT home visit parent surveys (parent surveys are described below). Counterfactual items are survey questions that ask about outcomes that are not expected to change as a result of participation in either of these programs. The validity of the results is enhanced if participants change in the expected direction on the items measuring program outcomes, but do not change on the counterfactual items. The counterfactual is an outcome that should not occur as a function of the treatment but is reasonably associated with potential confounds and covariates. In this evaluation counterfactual outcomes were measured such as a child learning to use the toilet (see Table 4 for a list of the counterfactual items) which is reasonably associated with the natural maturation of a child. If there is a change in both the treatment outcome and the counterfactual outcome then the change in both could be due to some third variable (e.g., maturity) because the intervention could not directly lead to a change in the counterfactual outcome. If the counterfactual outcome does not change and it is linked to potential confounds with treatment, then it can be assumed that the confounds were not acting on the sample and the intervention was producing the change in treatment outcome.


Counterfactual items also estimate the extent of threats to internal validity such as social desirability. Since in the RPT design the participant is his/her own comparison, the use of counterfactual items eliminates the problem of non-equivalent comparison groups.7

Planned Statistical Analyses The primary analyses will consist of a repeated measures (pre-post), between-group Analysis of Covariance (ANCOVA) with 2 groups (home visit X play & learn group). Further, there may be as many as 7 covariates depending on baseline differences (see below). G*Power (v. 3.1.9.2) was used to estimate the minimum sample size required to achieve a power of .8 for a moderate effect size (f=.25) at an alpha of .05, which is 98 families (approx 49 families per group).

Missing data analysis Retention of program participants in the evaluation will parallel their retention in the program. To test the self-selection bias resulting from program dropout, PAT home visit parents who drop out before the post-visit HOME observation will be compared with parents who complete the post-visit HOME observation on the same seven risk characteristics noted above which are assessed on the Intake Form. These are:

• Low income • Diagnosed disability or identified developmental delay • Severe or challenging child behavior • Primary home language other than English • Low parental education attainment • Abuse/neglect of child or parent • Environmental risk (e.g., significant loss in family, chronic illness, teen

parent, homelessness, etc.)

Dropout for the Play & Learn Group will not be problematic for the evaluation because the “now” and “before” RPT data are collected at a single point in time, and will be collected at quarterly intervals from any parents who had attended four or more sessions. Intake Form data will be used to estimate and report evaluation dropout rate from the Play & Learn Groups since any parent who completes the Intake Form will have completed at least four Play & Learn Group sessions and will therefore be eligible to complete the RPT parent survey. Differences between parents who complete the RPT survey and those who do not will be examined using the same risk items from the Intake Form listed above.

Missing data on RPT survey items will be assessed using pattern missingness analyses which will determine the reasonableness of the Missing At Random (MAR) assumption. Assuming the data to be MAR then multiple imputation will

7 For a more detailed discussion of this methodology see Trochim, W. (1985) Pattern Matching, Validity, and Conceptualization in Program Evaluation. Evaluation Review Vol. 9 No. 5, October pp. 575-604.


be used in SPSS v. 20 to impute the missing values (see Graham and Schaeffer, 2002)8 on survey items.

Pre-Post HOME, Pre-Post DECA, and RPT outcome analysis Repeated measures ANCOVA will be used to analyze the HOME, DECA and RPT survey data (these instruments and data collection processes are described below). For each of the three outcomes measured by the RPT survey, the dependent variable will be the difference in scale scores between participants’ assessments of themselves “now” and their retrospective assessments of themselves “before” engaging in the intervention. For the HOME and DECA, the dependent variable will be the pre-to-post change on total scores. The independent variables in the RPT survey and HOME analyses will be the level of participation in the PAT home visits (number of visits), and the level of participation in the Play & Learn Groups (number of sessions attended). The independent variable in the DECA analysis will be the level of participation in mental health services (number of sessions attended). Covariates in the equations will be the seven participant risk characteristics listed above. Baseline differences between the Play & Learn Group and PAT home visit participants will be calculated for each of the seven risk factors. At baseline if the two groups differ significantly on any risk factors (using a conservative p<.10) those factors will be included in the statistical analyses in addition to the independent variables. The same selected subset of risk factors will be included as covariates in the pre-post DECA analysis.

Because they may remain in the Play & Learn Groups as long as they like, parents may complete more than one RPT survey. The last survey that parents complete will be used in the impact analyses.

Simple frequencies will be used to address Impact Evaluation Question #5 and #6 listed above. For Question #5, the percent of parents whose HOME score improves from pre to post will be calculated (i.e., the percent for whom the pre-post difference is statistically significant). Similarly, for each DECA subscale, the percent of children who improve from pre to post will be calculated.

Implementation Evaluation Design and Planned Analyses The implementation evaluation is addressing the following exploratory questions:

1. Is the community outreach successful in recruiting the targeted number of children and parents for participation in the PAT home visits, early childhood mental health services and community events?

2. Did the program achieve its goal of screening 80% of PAT home visit children for developmental delay (using the ASQ)?

3. Did 75% of children with developmental delays access appropriate developmental health services (individual or group counseling, consulting with teachers of students with

8 Schafer, J. L. and J. W. Graham (2002). "Missing data: Our view of the state of the art." Psychological Methods 7(2): 147-177.


developmental delays or socio-emotional behavior problems, referrals elsewhere for other risk factors)?

4. Are the PAT home visits implemented with fidelity to the model? 5. What types of problems arise in implementing the PAT home visits and how are they

resolved? 6. What types of problems arise in implementing the Play & Learn Groups and how are they

resolved?

Quality Control Processes in Place The evaluation has instituted a number of control processes to assure that the interventions are implemented with fidelity and that data are collected according to the plan. To assure proper implementation of the evaluation, an Evaluation Advisory Group was established which includes management from Leaps & Bounds Family Services, Macomb Family Services, the Macomb Intermediate School District, United Way of Southeast Michigan and SPEC. The Evaluation Advisory Group meets quarterly to guide the design of instruments and data collection procedures, to receive updates about program implementation, to discuss any problem areas related to the evaluation, and to interpret the implications of any new evaluation findings available during the quarter. In addition to these meetings, on a monthly basis the evaluation manager from SPEC holds telephone check-in calls with the management at each of the three implementing agencies to address any issues related to data collection.

To assure proper program implementation, the SIF Program Director holds meetings with management of the implementing agencies and, in separate meetings and as needed, with parent educators who are implementing the Play & Learn Groups and the PAT home visits. The SIF Program Director along with the MISD Early Childhood Consultant host a parent educator meeting every other month for programs funded through SIF as well as another Early Childhood grant. Time is spent at the beginning of each meeting reviewing any new procedures or protocols as well as addressing any issues or concerns they might have. These meetings also provided additional professional development to meet the needs that the parent educators have in regard to addressing current family issues.

Annually, an early childhood education expert unaffiliated with the program observes a sample of the Play & Learn Groups and PAT home visits using a standard set of criteria (described below). The assessment results are provided, by individual and for the agency as a whole, to each agency. The results are also provided, by agency, to the SIF Program Director; overall results for the three agencies combined are provided to SPEC.

Fidelity to program design Fidelity of PAT home visits and Play & Learn Groups was assessed as part of the baseline evaluation activities. Fidelity was assessed by a third-party, early childhood expert. The fidelity assessor holds an M.A. in speech and language pathology and post-graduate courses in Early Childhood Education and Educational Leadership. She is a Credentialed Reliability Assessor for the High Scope Program Quality Assessment rating instrument. She has been consulting on early childhood education issues with public schools and social service agencies since 2003.


The early childhood expert observed one Play & Learn Group session of each facilitator and one PAT home visit of each parent educator. The assessor met with the home visitors before the visits: (1) to review typical practice, (2) to gain an understanding of what to expect during the visit, and (3) to ask questions about the curriculum that is being used, whether written materials are distributed, risk factors for the family, and other support services initiated for the family.

To assess the fidelity of the home visits to the PAT model, the early childhood expert reviewed the PAT program curriculum and created an observation checklist based on the five essential aspects of PAT home visit implementation:

• Reflections on prior visit • Focus on parent-child interactions • Development-centered parenting around sleep, attachment, discipline,

health, transitions, safety and nutrition • Discussion of family well-being (parental resilience, social emotional

development, social connections, concrete support, knowledge of child development, parenting)

• Inclusion of literacy enrichment activities

• Discussion of next steps at closing

The checklist used to guide the observations is included in Appendix B.

There is no evidence-based curriculum for the Play & Learn Groups. Therefore, assessing fidelity is not applicable to the Play & Learn Groups. Rather, the quality of program delivery was assessed as described below.

Quality of Program Delivery In the absence of fidelity criteria, the quality of the Play & Learn Groups was assessed by the same early childhood education expert who observed the PAT home visits. Using her expertise as an early childhood education expert, the assessor commented on the following characteristic in assessing the quality of Play & Learn Groups:

• Presence of a written plan/focus for the session • Extension of learning beyond the group • Collection of parent feedback at the end of the session • Provision of an appropriate learning environment • Facilitation of appropriate parent-child interactions • Parent-child interactions are rich • Discussion of developmental domains • Inclusion of literacy activities

“Sometimes I can tell how he is getting older, and how his development is changing.”



• Promotion of positive parenting • Appropriateness of materials used • Provision of a rationale for each activity • Respect shown for each family’s uniqueness • Modeling of development-centered parenting • Assistance to parents in tracking their child’s development • Promotion of social connections • Alerting parents to community resources and assistance in accessing

them

Quality of program delivery was also assessed by paper-and-pencil, self-report surveys (referred to as program surveys) that PAT home visit and Play & Learn Group parents were asked to complete every quarter. The two versions of the program survey both focus on quality, but are different. The Play & Learn Group program survey contains open-ended questions asking parents what aspects of the program are important to keep, what could be eliminated, what should be changed, and what should be added. The Play & Learn Group program survey also includes five closed-ended questions that asked parents to indicate whether they agree, disagree or are undecided regarding whether:

• The session facilitator helped parents learn from each other • The children in their care were able to use materials that they don’t

have/use at home • The session facilitator was friendly and respectful towards the adults • They learned something about how play encourages their child’s

development • They received a book or other resources after each week’s Play & Learn

group

The PAT home visit program survey includes the following questions assessing quality of the home visits:

• What did you enjoy most about this program? • What would you have changed about the program? • Is there anything missing or is there something you wish could be added

to this program?

Participant satisfaction Parent ratings of satisfaction with the program were not obtained as part of this evaluation. Because of the high social desirability bias expected in these kinds of measures, the Evaluation Advisory Group decided that more objective measures of program quality – including parent reports of program quality – were more meaningful for the implementation evaluation than the likely high scores that would be achieved if parents were asked about their satisfaction with the program.


Parent satisfaction was inferred from responses to the open-ended questions on the program survey in terms of parents’ assessments of what they thought should be kept, changed, removed or added from the PAT home visits and the Play & Learn Groups.

Implementation Data Analysis Implementation data come from the Intake Form, Screening and Referral Tracker, and the PAT home visit and Play & Learn Group program surveys (described below). Frequency tables from the Intake Form data are computed on the following items to assess the extent to which the program is reaching its screening, enrollment and referral targets:

• Total number of forms completed • Number parents referred to programs within the three Ready Children Ready

Communities agencies • Number of parents referred to programs outside of the three agencies • Type(s) of services parents are referred for • Gender of parents • Ethnic group of parents • Zip codes of parents • Parent involvement in other pre-school programs (Early Head Start, Great Start,

Early Learning Center, Early On, Preschool, other SIF program) • Family risk characteristics (as listed above)

Frequency tables from the Screening and Referral Tracker are computed to count:

• Number and types of referrals made each quarter among the three Ready Children Ready Communities agencies

• Number of and types of referrals made each quarter to other early intervention programs

• Number and types of screenings done each quarter • Number of children already receiving services for their needs

Cross-tabulations of the Intake Form data are calculated to determine if there are any differences in risk characteristics between: (a) those who complete the Play & Learn Group and/or PAT home visit RPT parent surveys and those who do not, and (b) those who complete the post HOME observation and those who do not.

Frequency tables of the Play & Learn Group program survey closed-ended questions (described above) are used to describe program quality and any changes to program quality over time (data are analyzed quarterly).

Descriptive statistics from the community outreach data provided by the SIF Program Director are used to assess the number and types of outreach efforts and to estimate the number of people touched by these efforts.

Narrative data from the fidelity assessor’s report of program fidelity provide information about program strengths and areas needing improvement.


Content analysis of responses to open-ended questions on the PAT home visit and Play & Learn Group program surveys identify areas of strength and areas needing improvement.

Findings related to quality of program implementation are presented below in the Evaluation Findings to Date section of the report. Future evaluation reports will present the other findings about implementation that are listed above.


Data Collection and Measurement Data for the evaluation are gathered about:

• Parents who participate in the Play & Learn Groups and/or the PAT home visits (via Intake Forms and RPT parent surveys)

• Children who are screened for developmental delay (via ASQ) and/or assessed for behaviors related to social and emotional resilience and concerns (via DECA/DECA-C)

• Families who are touched by community outreach efforts (provided in aggregate to the evaluator)

• Families who are referred to other services (via Screening and Referral Tracker and Intake Form) both among the participating agencies and externally

• Families who receive mental health services (enrollment form)

Since outcome-related data are not being collected from families attending community outreach activities or from families who receive only mental health services, there is no need to recruit these parents into the evaluation. Data collection methods and procedures for the PAT home visit and Play & Learn Group parents and children who are recruited for participation in the evaluation are described below.

Definition of Program Participants9 The entire population of PAT home visit and Play & Learn Group parents are invited to participate in the implementation and impact evaluation. Play & Learn Groups are open to all parents who live in the targeted geographic areas of South Warren, Detroit and Macomb County, Michigan. The high concentration of risk factors in the targeted areas makes it highly probable that the preschoolers of any family that elects to participate in Play & Learn Groups and/or PAT home visits will be at-risk for less-than-adequate literacy environments in their homes.

Program participation is defined as attendance at a minimum of four Play & Learn Groups or five PAT home visits. While parents will be eligible to receive a minimum of eight monthly home visits, discussions with the parent educators revealed that not all families participate in all eight visits, and that a minimum of five visits should be sufficient to see improvement on the expected outcomes. These variations of “dosage” in the PAT Home Visit program are expected to have limited negative effects. Sweet and Applebaum (2004)10 found that length in the Home Visiting program did not relate to effect size, despite the fact that families’ intended length spanned from 0-3 months to between 3 and 5 years.

Parents who meet the participation criteria are formally invited to participate in the evaluation. The staff facilitating the Play & Learn Group sessions and the PAT home visits read a script to the parent or guardians at the time they complete the Intake Form. The script explains to the parents what the evaluation involves, its benefits, how data will be collected and how confidentiality will be protected. Parents are told that they have the right to refuse participation

9 IRB approval has been obtained for these consent methods.

10 Sweet, M. A., & Appelbaum, M. I. (2004). Is home visiting an effective strategy? A meta-analytic review of home visiting programs for families with young children. Child Development, 75(5), 1435-1456.


in the evaluation without repercussion. The script instructs the parents to check one of two boxes on the Intake Form to indicate their consent – or refusal – to participate in the evaluation. In some cases it is the guardian and not the parent who attends the Play & Learn Groups. To accommodate this, the Intake Form for Play & Learn Groups requires two different consents to be signed: 1. consent of the primary caregiver bringing the child to the groups, and 2. consent of the parent or legal guardian.

Because Play & Learn Groups are also one source of referral for the PAT home visits, care has been taken to clearly delineate which intervention(s) parents participated in. As an accuracy check for group assignment, in addition to program records defining which parents are in the Play & Learn Group and which are in the PAT home visit group, there is a survey item on the Play & Learn Group RPT parent survey that inquires about parent participation in the PAT home visits. This will enable clean separation of any parent that participates in PAT home visits from membership in the Play & Learn only comparison group.

Data Collection Procedures All evaluation data are collected by project staff. For the PAT home visits, the Intake Form data are collected at enrollment if the parent did not already participate in four or more Play & Learn Groups. For the Play & Learn Groups, Intake Form data are collected either after the parent completes four sessions or at enrollment (and then parents are assigned to the evaluation after they have completed four sessions).

Baseline and end-of-program HOME data are collected by the PAT home visit staff who also administer the PAT home visit RPT parent survey and program survey to parents after at least five home visits or at the end of the home visit cycle (typically but not always eight visits).

Any Play & Learn Group parents who participated in four or more sessions are asked to complete the RPT parent survey at quarterly intervals. Each quarter, one week is designated for surveying Play & Learn Group parents. Play & Learn Group staff collect RPT parent survey data from eligible participants during that week. In order to distribute the right survey to the right parent, each survey is labeled with a unique identification number (without parent name), and is placed inside an envelope with a label that has the matching ID and “Parent/Guardian of (CHILD’S NAME INSERTED).” Only parents eligible to receive the parent survey are included (i.e., they had attended at least four Play & Learn Groups and they consented to participate in the evaluation on the Intake form). This procedure retains the original intent of not giving the evaluator access to identifying information about the parents or their children while simultaneously assures that data are not collected from parents who are not eligible to be surveyed.

Staff read a script (described above) that provides instructions for parents completing the RPT survey. Each copy of the survey is attached to a blank envelope. Staff have a large envelope for collecting the consent forms and completed surveys. The script and written instructions tell

“They like to read at home more because he enjoys reading time.”



parents that their answers are confidential, and instruct them to tear off the consent form after signing it and place it in the large envelope that the staff have. Parents are instructed to place the completed survey into the small blank envelope and seal the envelope before giving it to the staff. The staff place all small envelopes into the large envelope and seal the large envelope in front of the participants to demonstrate that confidentiality of their responses will be maintained. A data entry company enters all survey data with 100% blind key verification, and creates the program databases.

A parallel script is used by PAT home visit staff to administer the RPT survey to program participants. The same RPT data collection procedures are used for the home visit parents as are described above for the RPT parent surveys. Parents are not asked to complete another RPT parent survey if they remain in the PAT home visit program another year. Similarly, only one post HOME is completed for each parent regardless of the number of PAT home visit sessions that they receive.

Data Collection Instruments The following instruments are being used (see Appendix C for a copy of each of the instruments that were developed; copyrighted instruments are not shown in the Appendix):

Intake (Enrollment) Form An Intake Form was created for this evaluation which collects information about:

• The preschool child targeted by the interventions • Legal parent/guardians • Literacy risk factors (listed above) • Consent to participate in the evaluation

Home Visitation Program Survey The Home Visitation Program Survey is a short, open-ended survey that staff ask parents to complete at the end of the series of PAT home visits. The survey is used to assess the quality of the PAT home visits from the perspectives of the parents (see section on Quality of Program Delivery above).

Play & Learn Program Survey The Play & Learn Program Survey is a short survey administered once every three months to all parents attending Play & Learn Groups including those who are not participating in the evaluation (because staff want feedback from all parents for continuous quality improvement purposes). The survey contains both open-ended and closed-ended questions to assess quality of program delivery from the parents’ perspectives (see section on Quality of Program Delivery above). The survey also asks parents to report their perceptions of how they used the information and materials from the Play & Learn Groups.


RPT Parent Survey The RPT parent survey has two versions: Play & Learn Group and PAT home visit. The RPT questions were developed for this evaluation as the key measure of the three parent outcomes:

• Better understanding of the principles of early childhood development

• Greater valuing of reading to their children daily • More knowledge of how to access community resources

The RPT parent survey also contains counterfactual items (see section above on Baseline Equivalence Analysis: Propensity Score Matching and Counterfactual Items).

RPT Methodology

The RPT parent survey asks parents to rate themselves in terms of their understanding of the principles of early childhood development, their valuing of reading to their child daily, and their knowledge of how to access community resources. A second series of questions, included on the same survey, asks respondents to reflect on the same questions but relative to their status prior to participation in the intervention. The difference between the “now” and the “before” rating constitutes the measure of change.

RPT is particularly suited for situations in which respondents have little prior knowledge or experience with the concepts or skills that they will be exposed to during the intervention, such as with the Play & Learn Group literacy activities and PAT home visit discussions. The theory behind the RPT methodology is that only after parents have gained some understanding of the concepts can they adequately assess their level of pre-intervention knowledge or skills. The validity of these “now” versus “before” self-ratings in evaluations of training programs has been well documented in the literature.11

RPT Instrument Design

The RPT items measuring the three outcomes were created during the first year of the evaluation. The items were developed through a review of existing, similar instruments including the Early Childhood Literacy Scale (ECLS) that was created and validated by Dr. Ty Partridge, the evaluator for the SIF grantee – Southeast Michigan United Way.12 The population of identified items were reviewed by Play & Learn Group and PAT home visit program management and staff to assess which best reflected each of the three outcomes (i.e. had the strongest face

11 See, for example, Pratt, C. C., McGuigan, W. M., & Katzev, A. R. (2000) Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3). 341 – 349.; Campbell, D. T. & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Reprinted from Handbook of research on teaching, American Educational Research Association. Chicago: Rand McNally; Howard, G. S. (1980). Response-shift bias. A problem in evaluating interventions with pre/ post self-reports. Evaluation Review 4(1) Feb. 93 – 106.

12 Partridge, T. (2012) Reliability and Validity of the Early Childhood Literacy Scale. Unpublished Manuscript cited with permission of author.


validity). Program staff selected the subsets of items that they felt best reflected the most subdimensions of the outcomes (i.e. has the strongest content validity). In a second meeting with program staff, the evaluator facilitated discussions to generate additional items to add any dimensions of the outcomes that were not already present. During these discussions, staff were asked to explain how they know when a parent achieves each of the three outcomes. For example, how would parents behave, feel or think if they understood early childhood principles? Staff responses were catalogued and used to create additional items for the RPT survey. Draft items were then presented at a third meeting with staff who collectively selected which were the best items for measuring each outcome. During this discussion, staff were also invited to edit the items and to create additional items that might better reflect indicators of each outcome.

A similar process was used to generate the counterfactual items. Program staff were asked to brainstorm a list of things that no parents should report they learned from participating in either the Play & Learn Groups or the PAT home visits. Collectively, the best set of statements was selected as the counterfactual items. During the third meeting with staff, various options for formatting the RPT survey were presented. Program management was also asked for their feedback on survey formatting. For example, was it easier for parents to complete an RPT survey in which both the “now” and “before” questions are presented in tandem; or was it easier for parents if all of the “now” questions are presented first followed by a repeated list on a separate page of the “before” items. The final format separates the “now” and “before” items on separate pages of the survey instrument. This selection aligns with recent research on RPT that suggests that administering the two versions as far apart as possible could help to reduce social desirability factors that may be present when “now” and “before” items are presented close to each other.13

RPT Pilot Testing

The prototype RPT parent survey was pilot tested with a total of 44 parents between August 2013 and March 2014. The purpose of the pilot testing was to refine the questionnaire by selecting the best set of items that, together, measured each of the three parent outcomes. The prototype survey contained:

11 possible items for assessing valuing of reading daily

9 possible items for assessing understanding of early child development principles

7 possible items for assessing knowledge of how to access community resources

13 Nolte S., Elsworth G. R., Sinclair A. J., and Osborne R. H. (2012). The inclusion of 'then-test' questions in post-test questionnaires alters post-test responses: A randomized study of bias in health program evaluation. Quality of Life Research. 21:487-494.


5 possible counterfactual items

Parents were asked to rate each “now” and “before” item on a scale ranging from 1 meaning “not at all true” to 5 meaning “completely true.” For each item, parents were also given an option to check a box indicating that they found the item to be confusing.

Forty pilot test surveys were completed by Play & Learn Groups parents and four were completed by parents receiving PAT home visits. Of the 44 survey respondents a total of 27 (81%) had one or more child over the age of three. Seventeen (39%) had one or more child under the age of three. Of these parents:

38 (82%) were mothers

3 (7%) were fathers

2 (5%) were grandparents

2 (5%) were aunts

1 (3%) was a guardian

An item analysis was conducted using Chronbach’s Alpha to select the best set of items to measure each outcome. The results were presented to the program management. Together, the evaluator and program management selected the five or six items for each scale. Criteria used to select the final items measuring outcomes were:

• As a set, do the selected items achieve an alpha of 0.70 or higher? • Does the particular item improve the overall alpha (i.e., keep item)? • Does the particular item decrease overall alpha by more than 0.02 (i.e.

remove item)? • Did any parent mark the item as confusing (i.e., remove item)?

The criterion used to select the counterfactual items was the smallest difference in the mean scores of the parents’ ratings of the “now” and “before” items.

RPT Arabic Translation

A substantial number of PAT home visit parents speak Arabic as their primary language. Therefore the home visit RPT parent survey questionnaire was translated into Arabic. Standard translation-back translation methods were used to create the Arabic version of the questionnaire. Certified Arabic translators translated the surveys from English into Arabic. Arabic-speaking program staff back-translated the surveys into English. Back-translated items that did not match the original English were revised by the original translator and the process continued until the program staff indicated that all items on the Arabic version of the questionnaire adequately matched the English version.


HOME 0-3 and 3-6 Forms The Home Observation for Measurement of the Environment (HOME) is used as a measure of program outcomes for the PAT home visits. The HOME is used as a pre-post assessment of changes in children’s family literacy environment after the intervention. The Administration for Children and Families offers the following description of the measure: “The Home Observation for Measurement of the Environment (HOME) Inventory is designed to measure the quality and extent of stimulation available to a child in the home environment. The Infant/Toddler HOME Inventory (IT-HOME) comprises 45 items that provide information from the child’s perspective on stimuli found to affect children’s cognitive development. Assessors make observations during home visits when the child is awake and engaged in activities typical for that time of the day and conduct an interview with a parent or guardian. The IT-HOME is organized into six subscales:

(1) Responsivity: the extent of responsiveness of the parent to the child;

(2) Acceptance: parental acceptance of suboptimal behavior and avoidance of restriction and punishment;

(3) Organization: including regularity and predictability of the environment;

(4) Learning Materials: provision of appropriate play and learning materials;

(5) Involvement: extent of parental involvement;

(6) Variety in daily stimulation.

For the IT-HOME, 18 items are based on observation, 15 on interview, and 12 on either observation or interview.”14

There is also an early childhood version of HOME for 3-6 year olds made up of 55 items and eight subscales. The PAT home visit staff use the version of HOME appropriate for the age(s) of child(ren) in the home except in instances where the child ages out of one version of HOME between the pre and post test. In these cases, the earlier version of HOME is used for the post test so that the appropriate change analysis can be performed. The Administration for Children and Family Web site noted above reports that three tests of reliability for HOME (inter-rater and internal consistency) were all 0.65 or higher, and two assessments of test-retest reliability were mostly under 0.65.

Prior to the use of the HOME, the SIF Program director trained all staff who were expected to conduct the pre and post assessments. Then, to assure the accuracy of the post HOME assessments, the HOME was tested for inter-rater reliability. Pairs of home visit staff rated a total of four families. One rater was the “usual” PAT home visitor for that family. The other rater was also a trained PAT home visitor from the same agency who is participating in this evaluation. Inter-rater agreement of 70% or higher on the total subscale scores was the criterion used to indicate adequate reliability and to mean that there is no rater bias in the post HOME rating. In fact, the inter-rater agreement

14 This description of the HOM Observation Tool was taken from: http://www.acf.hhs.gov/programs/opre/ehs/perf_measures/reports/resources_measuring/res_meas_phio.html.


http://www.acf.hhs.gov/programs/opre/ehs/perf_measures/reports/resources_measuring/res_meas_phio.html

results showed that in all four cases, there was 100% agreement between the raters. Staff attributed the very high inter-rater agreement to the ease of use of the HOME, to the training they had, and to the on-the-job discussions they had with each other after each rating session.

Devereux Early Childhood Assessment (DECA) Devereux Early Childhood Assessment (DECA) and the DECA-Clinical Form (DECA-C) are pre-post assessments of resilience in preschoolers ages 2 to 5 with social and emotional problems or significant behavioral concerns. The DECA or DECA-C are administered by staff for children entering or being assessed for intensive early childhood mental health services. Use of the DECA will allow for a pre-post outcome assessment of changes in resilience of children after participating in intensive mental health services. Staff use the DECA in instances where the mental health services are provided to individual children and when staff are working in consultation with teachers of an enrolled child. The creators of the DECA describe its use as follows:

“The Devereux Early Childhood Assessment-Clinical Form (DECA-C) is an assessment of resilience in preschoolers ages 2 to 5 with social and emotional problems or significant behavioral concerns. Whereas the DECA is designed to be used with all children to promote healthy social and emotional growth, the primary purpose of the DECA-C is to support early intervention efforts to reduce or eliminate significant emotional and behavioral concerns in preschool children. The DECA-C may also be used to guide interventions, help identify children needing special services, assess outcomes, and help programs meet Head Start, IDEA and similar standards.”15 “Studies indicate that the DECA-C is a reliable instrument for assessing preschool children's behavioral concerns. The internal reliability estimates for each scale were calculated separately for each rater (parent or teacher). For parents, the alpha coefficients range from a low of .66 on Withdrawal/ Depression to a high of .78 on Emotional Control Problems, with a median of .76. For teachers, the alpha coefficients range from a low of .80 on Withdrawal/Depression to a high of .90 on Attention Problems, with a median of .88. The teacher alpha coefficients all meet or exceed the standard suggested by Bracken.”16

Ages and Stages Questionnaire The Ages and Stages Questionnaire (ASQ) is used to screen children with developmental delays and other risk characteristics at enrollment into the PAT home visits. If the ASQ screen suggests developmental delays, the more in-depth ASQ-SE (Social-Emotional) is used to determine whether the family should be referred for mental health services. The evaluation does not analyze any ASQ data. Rather, the evaluation only records whether children were screened using the ASQ upon enrollment into the PAT home visits (see Implementation Evaluation question #2). The Administration for Children and Families reports the following description of the ASQ: “The Ages & Stages Questionnaires, 3rd Edition (ASQ-3) is a series of 21 parent-

15 Retrieved 10-27-12 from: http://www.kaplanco.com/product/98817/devereux-early-childhood-assessment-deca-c-clinical-kit?c=17%7CEA1000). 16 Retrieved from: http://www.devereux.org/site/DocServer/DECA-C-Booklet.pdf?docID=3262.


http://www.kaplanco.com/product/98817/devereux-early-childhood-assessment-deca-c-clinical-kit?c=17%7CEA1000

http://www.kaplanco.com/product/98817/devereux-early-childhood-assessment-deca-c-clinical-kit?c=17%7CEA1000

http://www.devereux.org/site/DocServer/DECA-C-Booklet.pdf?docID=3262

completed questions to help screen infants and young children for developmental delays during their first 5.5 years. It is completed by parents or caregivers for children 1 to 66 months of age. Each questionnaire includes 30 developmental items and focuses on assessment of five key developmental areas: Communication, Gross Motor, Fine Motor, Problem Solving, and Personal-Social. Communication addresses babbling, vocalizing, listening, and understanding; Gross Motor focuses on body movements, including arm, body, and leg; Fine Motor addresses finger movements; Problem Solving focuses on learning and playing with toys; and Personal-Social addresses social play and play with toys and other children. Parents rate each item as ‘Yes’ the child does the behavior, ‘Sometimes,’ and ‘Not Yet.’ Items about behaviors that are challenging to describe (for example, putting beads on a string) include illustrations to help parents guide their responses. The items include a mix of skill ranges, including activities the child may not have tried before. Usually program or clinical staff work with parents to complete the questionnaires and encourage the parent to assist the child in trying an activity to see if they can do it (for example, if the child has not had the opportunity to draw a line, the parent or staff member might give the child a crayon and paper to see if she makes a line). The ASQ-3 also includes a section for parents to record general concerns and issues not captured by the questionnaire. One reviewer notes that the measure is a valuable screening tool for identifying potential developmental disabilities that are suspected in infants and young children but that it is not a standardized measure to be used for placement decisions for special education services (Hanig 2010).”17

Screening and Referral Tracker A Screening and Referral Tracker was created for staff to log each child on whom one or more screening was performed for developmental delays and to record where each child was referred. The Screening and Referral Tracker captures information on:

• Date of screening/referral • Type(s) of screening performed • Developmental delays identified • If the child is already receiving services for the identified needs and what

type of services s/he is receiving • Which agencies the child is referred and for what type(s) of service(s)

17 Information about the ASQ is taken from: http://www.acf.hhs.gov/programs/opre/ehs/perf_measures/reports/resources_measuring/res_meas_cdib.html.


http://www.acf.hhs.gov/programs/opre/ehs/perf_measures/reports/resources_measuring/res_meas_cdib.html

Evaluation Findings to Date During Year 1, the evaluation focused on defining measures, designing instruments, establishing data collection and processing procedures, and creating an evaluation database. As noted above, an Evaluation Advisory Group was established that meets quarterly to monitor evaluation progress, support evaluation processes, and interpret evaluation results. Agency-level data collection activities are now monitored monthly via check-in calls between SPEC and agency management.

As described above, the evaluation was successful in creating, piloting and revising measures of the three key parent outcomes. Databases have been designed and data are now being submitted regularly to a data processing company that generates analyzable databases from the Intake Form, Play & Learn and PAT home visit program surveys, the Screening and Referral Tracker, DECA, and the two versions of the RPT parent surveys. Program management has been tracking community outreach and provides these data in aggregate to SPEC. This section of the report presents findings from the data available to date from these activities.

Implementation Evaluation Findings This section describes the results from the early childhood expert’s observations of PAT home visits and Play & Learn Groups, the volume of services that were delivered during Year 1, and the parent’s perceptions of program quality.

Fidelity to Program Design: PAT Home Visits Between February 2014 through March 2014, the fidelity assessor accompanied each of the four PAT home visit staff on one home visit. Her role was to assess the alignment of their work to the PAT Home Visit Curriculum which focuses on:

• the parent reflecting on the prior visit and how they have utilized their learning from it • the appropriateness of the content of the current visit • planning the content of the next visit • inclusion of literacy enrichment activities

Reflection on Prior Visit: According to the assessor, each PAT home visit staff took time at the beginning of their visit to reflect on the prior visits. These conversations were rich and reflective and most often lead to additional suggestions from the Home Visitors to the parents. No weaknesses were observed in this aspect of the visits.

Parent-Child Interactions: According to the assessor, each PAT home visit staff was adept at setting the stage for the parent to help their child with the rich array of activities introduced. Each staff always gave a rationale for the activity and described the developmental domains they were targeting. They all also pointed out developmental domains associated with the child’s


independent activities and with reports from the parents of their child’s activities and accomplishments since the last visit. No weaknesses were observed in this aspect of the visits.

Development Centered Parenting: PAT has identified seven developmental topics: sleep, attachment, discipline, health, transitions, safety, and nutrition. According to the assessor, each PAT home visit staff respectfully answered and asked questions regarding at least four of these topics during the monitoring visits. No weaknesses were observed in this aspect of the visits.

Family Well-Being: According to the assessor, each PAT home visit staff discussed protective factors with parents (parental resilience, social emotional development, social connections, concrete support, knowledge of child development and parenting). There was always discussion about the parents’ current well-being and plans for personal growth. Community resources were always presented and challenges to participating in those resources were discussed. All of these discussions were comfortable and respectful. No weaknesses were observed in this aspect of the visits.

Closing: According to the assessor, each PAT home visit staff ended their session with reviewing what the goal, activities and next steps were. Supportive written materials connected to current developmental stages and the focus of the activities were reviewed and given to the parent. A children’s book was reviewed and given to child. The parent was asked if there was anything else they needed from the home visitor and also asked what they would like to focus on at the next visit. No weaknesses were observed in this aspect of the visits.

The assessor found it difficult to identify areas for improvement because each staff was very knowledgeable about the curriculum, child development in general and local resources. The assessor reported that each staff managed, through their own personal style, to be professional and friendly at the same time. The parents were at ease asking them many questions, and were very open with them. The only recommendations the assessor made after observing the home visits was for the staff to stay current with developments in their field and take every opportunity possible to further develop their knowledge and skills. She also encouraged the program administrators to be aware of any signs of “burn-out” among the staff, to be ready and willing to support them through stressful times, and to assure that they are provided ample opportunities for professional development.

Fidelity to Program Design: Play & Learn Groups As noted above, there is no standardized curriculum against which the fidelity assessor could assess alignment of Play & Learn Group facilitation. Rather, she drew on her expertise as an early childhood education expert to assess the quality of program implementation.

Between March 2014 through May 2014, the fidelity assessor observed five Play & Learn Group facilitators, three of whom are also PAT home visit staff. According to the assessor, each facilitator artfully arranged the room, introduced materials and introduced activities in a way that

“I love that every week we get 2 ‘homework’ packets and a book. My child learned the days of the week, so he could remember playgroup was on Friday's.”



facilitated parent-child interaction. Parent to parent interaction was also rich. Developmental domains and activities supporting those domains were discussed; literacy, with story time, was a part of each group. Throughout the group time, positive parenting was promoted through modeling by the provider. Each provider also offered books and many other take home activities and resources. Feedback was gathered from parents informally at each meeting and in written form at regular intervals throughout the year. Several of the same parents were present at different play groups which the assessor considered to be a testament to the importance that parents placed on this experience. The only weakness the assessor reported was that attendance at two of the groups on the day of her visit was low.

Although there is no standardized Play & Learn Group curriculum, the assessor found core aspects that were present in the Play & Learn Groups and also in the PAT home visits. These core aspects could be used to begin to standardize Play & Learn Group implementation:

• Respecting each family’s uniqueness and each parent’s knowledge of their own child’s needs, wants and abilities.

• Involving parents in planning • Facilitating and supporting parent-child interactions • Assisting parents in tracking their child’s development (Milestones, ASQ) • Modeling and supporting development centered parenting • Providing information and learning activities that focus on the domains of school

readiness • Providing rationale for each activity • Literacy – book sharing • Facilitating and supporting social connections • Alerting parents to community resources and assisting them in taking advantage of these

resources

The assessor also found core differences between the PAT home visits and the Play & Learn Groups:

• There is more “free choice” for the parents and the children in the Play & Learn Groups. The PAT home visits are designed to have prescribed activities that are presented to the parent and child together with the parent taking the lead in implementing the activities. In the Play & Learn Groups, there are prescribed activities such as story time or circle but most of the activities are made available for the child/parent to participate in as they choose.

• There is conversation during the home visits regarding family and parental concerns and supports that are not a part of Play & Learn Groups. These conversations would be inappropriate in a group setting.

• There is less conversation directed towards developmental milestones during the Play & Learn Groups, even though parents are given the opportunity to complete Ages and Stages Questionnaires on their own. During PAT home visits, developmental milestones and progress toward them is integral to the visit and parents periodically complete an ASQ or a “Milestones” in conversation with the PAT home visit staff.

• Social connections are a major part of the Play & Learn Groups that are not available during PAT home visit.


• Availability of transportation to Play & Learn Groups is a core difference compared to PAT home visits that are conducted at home or, in some cases, at a designated place already identified as easily accessible to parents.

Program Delivery Date program delivery began

Program delivery started in January 2013.

Number of program units Table 1 shows the number of units of program activities accomplished to date. Across all agencies, the program served:

Year 1: 177 families

Year 2: 261 families to date

Table 1:

Units of Program Activities to Date

Program Unit Type Leaps & Bounds Family Services

Macomb Family Services

MISD18

Play & Learn Groups 180 sessions 64 sessions 57 sessions Outreach Activities 28 activities (not including flyer distribution and email blasts) PAT Home Visits 456 visits 11 visits Social Emotional Home Visits

356 visits

Quality/satisfaction with program delivery Table 2 shows the first two quarter results from the Play & Learn Group program surveys. Data are collected once every three months by program staff for quality improvement purposes. As Table 2 shows, overall parents report high quality of the Play & Learn Groups in terms of facilitator friendliness, respect and helpfulness (97% to 100%). Almost all parents (95% to 100%) reported learning something about how play encourages their children’s development. Almost all (91% to 95%) reported receiving a book or other resources after each week’s Play & Learn session. Fewer, but still the majority of parents (79% to 86%) reported that their children were able to use materials that they don’t have or use at home.

18 MISD started implementing Play & Learn Group sessions and PAT Home Visits in 2014.

“My kids are sharing more and playing better with others.”



Table 2: Play & Learn Group Parents’ Perceptions of Quality of Program Delivery

Question/Response Qtr1

(N=34) Qtr2

(N=38) Facilitator was friendly and respectful towards adults

% Agree 100% 97% % Undecided 0% 0%

% Disagree 0% 3% Facilitator helped us learn from each other


% Disagree 0% 3% I learned something about how play encourages my child’s development


% Disagree 0% 3% I received a book or other resources after each week’s Play & Learn I attended


% Disagree 3% 5% Children were able to use materials we don’t have/use at home


% Disagree 15% 8%


Table 3 presents a summary of the responses to each open-ended question on the Play & Learn program survey. Generally parents and their children engage and enjoy the program. They gave comments suggesting that Play & Learn groups are perceived to be high in quality.

Table 3:

Summary of Results from Open-Ended Questions about Play & Learn Group Quality

Question Most Common Finding

Interesting Findings 1st Data Collection 2nd Data Collection

Q1. What did your child enjoy the most?

Activities Interactions

Of the activities Arts & Crafts and Letters were valued most across all agencies. Songs, story time, and circle time appreciated at at least two agencies. Gym was reported multiple times as appreciated at one agency, giving a sense of space and having the “room to play.” At two agencies themes were recognized and linkages between the themes and the activities noted. Interactive opportunities with the child were appreciated.

Of the activities art projects were valued most across all the agencies. Crafts, music/songs, story time/reading were appreciated at two agencies. Two activities were appreciated by multiple parents but only reported at one agency: parachute and gym.

Q2. What did you learn about your Child’s Development

Ability to recognize child’s needs Awareness of child’s skill or development

Types of things they learned: Social Skills Independence Child’s Interests Confidence in Child’s development Child’s learning disabilities

Types of things they learned: Child’s abilities more advanced than expected Child’s abilities are improving Child’s interests Activities to do with child and help their development

Q3. What would you have changed?

Majority say nothing. Singular suggestions were made.

Suggested changes: More days More time with circle time More songs Timed activities to force children to stay focused for 10-15 minutes

Suggested changes: More age appropriate activities/toys for younger children Longer sessions


Table 3 (Con’t)

Summary of Results from Open-Ended Questions about Play & Learn Group Quality

Question Most Common Finding

Interesting Findings 1st Data Collection 2nd Data Collection Q4. Was something was missing?

Vast majority say nothing. A few comments were made

Types of things reported missing: Parent Involvement in facilitation More time in a particular activity More child friendly activities for younger children

Types of things reported missing: A few desired more music or

crafts More children in the group More activities for children aged

one and two

Q5. What would you add?

A few suggestions were made. More suggestions made here than for Q3 and Q4 above.

More time: Numbers and letters Songs Paint More information on: Positive discipline Teamwork Potty training Language development

More activities: Hands-on Finger play Painting Dancing and singing to learn More information Nutrition Some parents used this space for affirmations, like “everything is great.”

Data were collected at two points in time at the time this report was written. As shown in Table 3, the ‘most common findings’ were the same during both time periods, however there were some differences in the types of things parents reported.

In response to the question, ‘What did your child enjoy the most?’ across the two time periods parents most commonly reported that their children enjoy activities and the interactive nature of the program. The types of activities parents reported differed from one time period to the next. In the first round of data collection more parents across all three agencies discussed the use of themes and learning letters than in the second round of data collection. In the second round of data collection parents discussed more art project activities across the different agencies. The nature of the opportunities to interact slightly changed from the first to second round of data collection. In the first round parents remarked on interactions with their children, with other adults, and child to

“Love story time and welcome song. Love interaction between my daughter and other kids.”



child. In the second data collection period interactions with other adults were not reported.

In response to the question, ‘What did you learn about your child’s development?’ across the two time periods parents most commonly discussed ways to better recognize their child’s needs and being more of aware of their child’s skills or level of development. In the second data collection round more parents reported improvements in their child’s skills or development like being more sociable, able to sit on the floor, and sharing better with other.

Three questions probed for suggestions from parents on ways to improve the program by asking what they would have changed, thought was missing, or would add. In the second round of data collection more suggestions were made than in the first round. In the second round of data collection across these three questions, parents most commonly made three suggestions. Parents asked for there to be an increased number of activities or toys that were made available for younger children, one specified between the ages of one to two. A few parents asked that the length of the sessions be longer. A few parents requested there be more children participating. For the most part, parents used these questions to make positive remarks like, “Nothing that I can think of, everything is perfect.”

Impact Evaluation Findings This section describes the results from the psychometric analyses of the RPT survey and revisions to the original plan to use the Early Childhood Literacy Scale as part of this evaluation. An impact analysis of the RPT survey will be included in the next report, once more data for PAT home visit and Play & Learn comparison group parents are collected.

Validation Results: RPT Parent Survey Item Reliability Analyses The psychometric analyses examined the internal consistency and convergent validity of the three scales measuring the three parent outcomes. Internal consistency reliability was defined as the extent to which the correlation among items measuring the same outcome is stronger than the correlation among items measuring different outcomes. Chronbach’s alpha was used to establish internal consistency. An Alpha score of 0.70 or higher was set as an adequate level of internal consistency. For the three parent outcome scales derived from analysis of the pilot test data, the Alpha coefficients were:

• Understanding the principles of early childhood development: (α=0.801, variance=0.018, mean=4.701)

• Valuing reading to their children daily: (α=0.690, variance=.025, mean=4.496) • Knowledge of how to access resources: (α=0.859, variance = 0.054, mean=4.507)

“He has started to talk more clear(ly). Family members are talking about that they can understand my son now.”



Table 4 shows the final set of items for measuring each of the three parent outcomes as well as the three selected counterfactual items. Management and staff reviewed the final set of items and indicated that the items are appropriate for children aged 0 to 3 as well as for children aged 3 to 5. Therefore, parents of children ages zero to five are being asked to complete the RPT parent surveys.

Table 4

Items Analysis for Three Parent Outcomes Measured by RPT Parent Surveys

OUTCOME: UNDERSTANDING PRINCIPLES OF EARLY CHILDHOOD DEVELOPMENT (Chronbach’s Alpha = 0.801) I am aware of activities I can do to help my child learn and develop. I believe that things I do every day will influence the kind of adult my child will become. I feel confident in my role as my child’s first teacher. I feel confident in my ability to redirect my child’s behavior. I know how to guide my child in making friends. I know what my child should be able to do for his age. OUTCOME: VALUING OF READING TO CHILD DAILY (Chronbach’s Alpha = 0.690) I read to my child every day. I understand the importance of reading the same stories again and again. When I read stories to my child(ren) I ask what they think will happen next and why they think that. My child sees me read. We talk about the pictures when we look at a book. When I’m outside with my child(ren) I point to words on buildings or street signs. OUTCOME: KNOWLEDGE OF HOW TO ACCESS COMMUNITY RESOURCES (Chronbach’s Alpha = 0.859) I know how to access information on community events. I know where to go if someone in my family needs education services (e.g., preschool, speech and language). I know what kinds of help are available in my community. I know where to find information that I need to help my family (e.g., medical care, dental care). I feel comfortable going to talk to someone to make sure my child’s educational needs are met. COUNTERFACTUAL ITEMS: I know that it’s important to teach my child to not talk to strangers. (mean difference = 0.024) I know that it’s important to teach my child to not run into the street. (mean difference=0.071) I know to teach my child to use and flush the toilet. (mean difference = 0.071)

Comparison RPT and ECLES Items The original evaluation plan called for creating an RPT adaptation of the Early Childhood Literacy Scale (ECLS), developed and currently being studied by Ty Partridge at Wayne State University, as a surrogate measure of Kindergarten readiness. The ECLS was designed as a pre-post assessment. The ECLS questions were to be appended to the RPT survey questionnaire. There are three forms of the ECLS; infant, toddler, and preschool.


The ECLS has 8 subscales: Supportive Reading Environment, Adult Reading Quality, Child Interest in Reading, Child Reading Engagement, Child Narrative Skills, Child Language Awareness, Letter & Sound Recognition Skills, and Child Emergent Writing.19

The ECLS has demonstrated reliability and validity. Partridge (2012) reported recent research on a random subset of women participating in United Way of Southeast Michigan’s Women’s Initiative. The sample of 250 was used to conduct a factor analysis on the ECLS. An 8 factor solution accounted for 71% of the variance in the variance-covariance matrix and there were no cross-factor loadings greater than 0.40. The overall reliability of the ECLS was α=0.93. Individual sub-scale reliabilities ranged from α=0.76 - α=0.91. Twelve week test-retest reliabilities ranged from r=0.42 to r=0.61. The ECLS has good convergent and discriminant validity. The total ECLS was correlated with the communication (r=0.53), personal/social (r=0.22), and problem solving (r=0.21) subscales of the ASQ, but not the fine and gross motor subscales. Predictive validity was also demonstrated with correlations for the ECLS and BSRA ranging from r=0.29 to r=0.44.

The ECLS was reviewed as one of several instruments examined in the process of creating the RPT survey items (described above). Many of the concepts from the ECLS were incorporated into the RPT survey items measuring the outcome of valuing reading to the child daily. While the wording is not the same between the ECLS and the RPT, the conceptual basis for the items on the RPT for the subscale of valuing reading are the same. So as not to over burden parents with data collection, and because the evaluation does not include a pretest survey of parents, the ECLES itself was dropped from use in this evaluation.

19 Partridge, T. (2012) Reliability and Validity of the Early Childhood Literacy Scale. Unpublished Manuscript cited with permission of author.


Lessons Learned The first year’s experiences offer many lessons about what helped and what hindered the evaluation. This section summarizes these lessons.

What helped the evaluation? The Evaluation Advisory Group made up of the three implementing agencies plus the United Way of Southeast Michigan was very helpful in terms of:

• Streamlining communications about all aspects of the evaluation • Enabling joint problem-solving as problems that arose • Management understanding what the program staff were being asked to do in support of

the evaluation

Other things that helped this evaluation were:

• Having an Institutional Review Board that was flexible in terms of rapid review of applications and changes to the evaluation design

• Monthly check-in calls to the management of the three implementing agencies to help them recall all of the nuances to the sophisticated data collection that is needed for this evaluation

• Using a third-party data entry company rather than relying on very busy program staff to enter evaluation data into a database

• Good relationships that the agencies established with each other enabled transparent sharing of problems and weaknesses

• Community relationships with many schools in the area have been formed and are thriving which suggests that recruitment of parents for the evaluation will increase in Year 2

What hindered the evaluation? The following were hindrances to this evaluation:

• The slower than anticipated recruitment of parents into PAT home visits may reduce the power of the evaluation analyses

• Many months in the early phases of the evaluation were spent waiting for approval of the SEP. As a result, parents who were participating in the PAT home visits could not be recruited for participation in the evaluation.

• Staff turnover at Macomb Intermediate School District meant that additional time needed to be taken to orient and train the new SIF Program Director

• The late development of the database limited any developmental feedback that could be provided by the evaluators

• The time needed to develop the evaluation data collection tools meant that data were not collected from some parents who were in the program early on

• Reductions in funding available to the implementing agencies, partly due to the unanticipated need to use grant funds for a third-party evaluation and partly because help with fund development was initially promised but did not come to fruition. This will


ultimately result in fewer parents served than expected and therefore fewer parents available for the evaluation

• Tough winter weather resulted in school closures and made it difficult for parents to get to the Play & Learn Groups decreasing the overall participation rate

• The general difficulties working in high risk communities means that it takes time to build trusting relationships where parents are willing to invite program staff into their homes; this may result in fewer numbers of parents included in the evaluation than originally anticipated.


Appendices

Appendix A: Logic Model

Appendix B: Home Visit Observation Synopsis Sheet

Appendix C: Instruments Developed



Appendix B: Home Visit Observations Synopsis Sheet (Parents as Teachers)

Parent Educator Agency

Written Plan / Goal

Selected Activity

Date Observer

Reflections on Prior Visit

Parent/ Child Interaction

Development Centered Parenting

Family Well Being

Across the Visit

Additional Resources Offered

Synopsis


Appendix C: Instruments Developed

Intake (Enrollment) Form

Home Visitation Parent Survey

Play & Learn Parent Survey

Home Visitation Program Survey

Play & Learn Program Survey


Enrollment Form

What other programs are you involved in? Early Head Start/Head Start Great Start Readiness Preschool Early On Early Learning Community Preschool SIF Program ________________________

SIF Agency ___________________________________________ Representative _________________________________________________

Has the child participated in additional SIF programs? ______________________________________________________________________

Last Name ___ First Name Middle Initial_________

Gender _________________________ Birth date __________________________ Birth Order _______________________

Primary Language__ _______________________ Other languages used in the home ____ ___________________________ ___________

Please check appropriate ethnic group: � American Indian � Black � Asian � Hispanic � White � Arabic � Other__________________

Address ________________ _________ Apt _ ____ City/Township _________ State_MI Zip _____ ____

Phone Birth City & State/County

E-Mail _________ Attending Preschool/Daycare

Family/Child Code: 06_ ___ ___ ___ ___ ___ -- ___ ___ ___ ___ ___ ___

Internal Referral from: Agency: � LBFS � MFS � VDPS Type of Service: � HV � PL � MH

For Service: � LBFS � MFS � VDPS Agency � HV � PL � MH Type of Service

District: _____________________ Previous:_____________________ Involved Prior to 1/2013: � Y � N � HV � PL School: ___________________

Date of Enrollment: _______________ ______

Date of Exit: __________ Code:__________________

If additional mailings are needed, please indicate the person and new address on the back of this sheet. Married __________ Single __________ Teen __________ Foster__________

Mother’s Full Name Father’s Full Name

Resides with child _____Y _____ N Resides with child _____ Y _____ N

Birth date Birth date

Employer Employer

Work Phone Work Phone

Cell Cell

Last Grade Completed Last Grade Completed

Guardian/Step-Parent Name Guardian/Step-Parent Name

Last Grade Completed Last Grade Completed

Siblings living in the home (list oldest to youngest)

Last Name _____________________________________ First Name ______________________ Age ________ Date of Birth ______________

School ____________________________________________________________________________ Grade ____________________________


School ____________________________________________________________________________ Grade ____________________________


School ____________________________________________________________________________ Grade ____________________________

Number of other people living in the home ______________________Total number of people living in the home _______________________

Emergency Contact (other than parents)

Last Name ______________________________________ First Name _______________ Relationship ____ _______

Home Phone ______________________________ Work Phone ______________________________ Cell _____________________________

CHILD INFORMATION

PARENT / GUARDIAN / FAMILY INFORMATION

Place an X next to all risk factors that apply:

1. Low family income: Low family income is equal to or less than 250% of federal poverty level (FPL).

2. Diagnosed disability or identified developmental delay: Child is eligible for special education services or child’s developmental

progress is less than that expected for his/her chronological age, or chronic health issues causing development or learning problems.

3. Severe or challenging behavior: Child has been expelled, or is at risk of expulsion from preschool or child care center.

4. Primary home language other than English: English is not spoken in the child’s home; English is not the child’s first language.

5. Parent/s with low educational attainment: Parent has not graduated from high school or is illiterate.

6. Abuse/neglect of child or parent: Domestic, sexual, or physical abuse of child or parent; child neglect issues; parent or caregiver

substance abuse (e.g., alcohol, prescription or non-prescription drugs).

7. Environmental risk: Parental loss due to death, divorce, incarceration, military services, or absence; sibling issues (e.g., chronic illness

or disability); teen parent (not yet age 20 when first child born); family is homeless or without stable housing; residence in a high-risk neighborhood (area of high poverty, high crime, with limited access to critical community services); parental or postnatal exposure to toxic substances known to cause learning or developmental delays.

TOTAL NUMBER OF RISK FACTORS

Qualifies for Free/Reduced Meal Program _____ Y _____ N

A staff person read the Program Feedback Consent Statement to me. I AGREE DO NOT AGREE to take part in the surveys that will be given to me sometime in the future. If I change my mind, I know I can just not complete the surveys. [This statement should be filled out by the primary person taking the child to Play and Learn or participating in Home Visits. If this primary person for Play and Learn is not the parent/guardian, please complete the statement and sign the Caregiver option below.]

I, Parent/Guardian, agree to hold harmless the Macomb Intermediate school district, the local school district, Macomb Family Services and Leaps and Bounds Family Services, their staff and volunteers against any and all claims, liabilities or legal actions that may result from any act or omission. I understand information will be shared with project partners. I further understand that the responsibility for my child's safety and well being during activities and/or home visits is solely my responsibility. I give my permission to these agencies to use my own or my child's picture, data, photograph or video for the project. I understand that these materials may appear in public and I release the Macomb Intermediate school district, the local school district, United Way for Southeastern Michigan, Macomb Family Services, Leaps and Bounds Family Services and the grantor (CNCS) from any liability in connection with the use of these pictures, data, photographs or video materials. The information I have provided is true and factual. Signature of Parent/Guardian:__________________________________________________________ Date: _______________________________

Printed Name of Parent/Guardian:___________________________________________________________________________________________

Signature of Caregiver at Play and Learn Group: ___________________________________________ Date: _______________________________

Health Insurance ___________________________________________________________________________________________________

Name of Medical/Service Provider(s) __________________________________________________________________________________

Does your child have any of the following medical conditions that we should be aware of?

Asthma Y N Vision Problem Y N Orthopedic Handicap Y N

Allergies Y N Glasses/Contacts Y N Kidney Disease Y N

Epilepsy Y N Hearing Problem Y N Insect Bite Reaction Y N

Diabetes Y N Uses hearing Aid Y N Physical Handicap Y N

Heart Condition Y N Other: Y N Other: Y N

Any additional medical conditions? Yes No If yes, please describe _____________________________________________

☐ Above ☐ At or below the following: # of persons in household 2 3 4 5 6 7 8

Amount of income $20,163 $25,389 $30,615 $35,841 $41,067 $46,293 $51,519

MEDICAL INFORMATION

FAMILY INCOME

Home Visitation Parent Survey

Please continue to the next page.

Please tell us about you…

A. Will you help us and complete this survey? Yes (1) No (2)

B. What is today’s date? ______ / _______ / _______ Mo Day Yr

C. What is your relationship to the child(ren):

Mother (1) Father (2) Grandparent (3) Other (see below) (4) If Other, please state your relationship: ________________________

D. What ages (in years) are your children who are involved in the home visits?

Child #1 Child #2 Child #3 Child #4

E. Have you participated in Play & Learn Groups since January 1, 2013? (Check one)

Yes (1) No (2) Don’t know (3)

F. Check this box if someone is helping you complete the rest of this form:

For office use: Staff, please enter ID# and # of Home Visits before administering. ID # __________________________

Home visit #: ______ visits

1. Please tell us how true this is for you now…

Not at all true

Completely true

a. I know how to access information on community events…..................................... 1 2 3 4 5

b. I know that it’s important to teach my child to not talk to strangers……………………. 1 2 3 4 5

c. I am aware of activities I can do to help my child learn and develop....................... 1 2 3 4 5

d. I read to my child every day...................................................................................... 1 2 3 4 5

e.

I know where to go if someone in my family needs educational services [e.g. preschool, speech and language].............................................................................. 1 2 3 4 5

f. I know that it’s important to teach my child to not run into the street………………… 1 2 3 4 5

g.

I believe that things I do every day will influence the kind of adult my child will become..................................................................................................................... 1 2 3 4 5

h. I understand the importance of reading the same stories again and again............. 1 2 3 4 5

i. I know what kinds of help are available in my community….................................... 1 2 3 4 5

j. I know to teach my child to use and flush the toilet……….........……….........………...... 1 2 3 4 5

k. I feel confident in my role as my child's first teacher............……….........……….......... 1 2 3 4 5

l.

When I read stories to my child I ask what s/he thinks will happen next and why s/he thinks that………………………….......................……….........………....................... 1 2 3 4 5

m.

I know where to find information that I need to help my family [e.g. medical care, dental care].....…………………….....................……….........………............................. 1 2 3 4 5

n. I feel confident in my ability to redirect my child’s behavior….........……….........……. 1 2 3 4 5

o. My child sees me read………………………………………….............…….........……….........…… 1 2 3 4 5

p.

I feel comfortable going to talk to someone to make sure my child's educational needs are met........................................................................................................... 1 2 3 4 5

q. I know how to guide my child in making friends……................................................. 1 2 3 4 5

r. We talk about the pictures when we look at a book……........................................... 1 2 3 4 5

s. I know what my child should be able to do for his/her age...................................... 1 2 3 4 5

t. When I’m outside with my child I point to words on buildings or street signs. 1 2 3 4 5

PLEASE continue on the next page >>>>>>>>>

2. Please tell us how true this was for you before you started having Home Visitations…

Not at all true

Completely true

a. I knew how to access information on community events............................................. 1 2 3 4 5

b. I knew that it’s important to teach my child to not talk to strangers……………………...... 1 2 3 4 5

c. I was aware of activities I could do to help my child learn and develop....................... 1 2 3 4 5

d. I read to my child every day........................................................................................... 1 2 3 4 5

e.

I knew where to go if someone in my family needs educational services [e.g. preschool, speech and language].................................................................................. 1 2 3 4 5

f. I knew that it’s important to teach my child to not run into the street……………........... 1 2 3 4 5

g.

I believed that things I do every day will influence the kind of adult my child will become.......................................................................................................................... 1 2 3 4 5

h. I understood the importance of reading the same stories again and again.................. 1 2 3 4 5

i. I knew what kinds of help were available in my community…...................................... 1 2 3 4 5

j. I knew to teach my child to use and flush the toilet...................................................... 1 2 3 4 5

k. I felt confident in my role as my child's first teacher………….......................................... 1 2 3 4 5

l.

When I read stories to my child I asked what s/he thinks will happen next and why s/he thinks that…………………….................................................................................. 1 2 3 4 5

m.

I knew where to find information that I needed to help my family [e.g. medical care, dental care] ......……………………....................................................................................... 1 2 3 4 5

n. I felt confident in my ability to redirect my child’s behavior........................................ 1 2 3 4 5

o. My child saw me read…………………………………................................................................ 1 2 3 4 5

p.

I felt comfortable going to talk to someone to make sure my child's educational needs were met............................................................................................................ 1 2 3 4 5

q. I knew how to guide my child in making friends……………............................................. 1 2 3 4 5

r. We talked about the pictures when we looked at a book.……...................................... 1 2 3 4 5

s. I knew what my child should be able to do for his/her age.......................................... 1 2 3 4 5

t.

When I was outside with my child I pointed to words on buildings or street signs………………………………………...................................................................................... 1 2 3 4 5

Thank you for completing this survey!

Play & Learn Parent Survey

Please continue to the next page.

Please tell us about you…

A. Will you help us and complete this survey? Yes (1) No (2)

B. What is today’s date? ______ / _______ / _______ Mo Day Yr C. What is your relationship to the child(ren):

Mother (1) Father (2) Grandparent (3) Other (see below) (4)

If Other, please state your relationship: ________________________ D. What ages (in years) are your children who are involved in Play & Learn? Child #1 Child #2 Child #3 Child #4 E1. Including this group session, have you participated in at least four Play & Learn sessions?

Yes (1) No (2) Don’t Know (3)

E2. Have you participated in the Home Visitation program since January 1, 2013? Yes (1) No (2) Don’t Know (3)

F. Check this box if someone is helping you complete the rest of this form:

ADD “Survey Label(s)” HERE

Child ID code(s)

1. Please tell us how true this is for you now…

Not at all true

Completely true

a. I know how to access information on community events…..................................... 1 2 3 4 5

b. I know that it’s important to teach my child to not talk to strangers……………………. 1 2 3 4 5

c. I am aware of activities I can do to help my child learn and develop....................... 1 2 3 4 5

d. I read to my child every day...................................................................................... 1 2 3 4 5

e.

I know where to go if someone in my family needs educational services [e.g. preschool, speech and language].............................................................................. 1 2 3 4 5

f. I know that it’s important to teach my child to not run into the street………………… 1 2 3 4 5

g.

I believe that things I do every day will influence the kind of adult my child will become..................................................................................................................... 1 2 3 4 5

h. I understand the importance of reading the same stories again and again............. 1 2 3 4 5

i. I know what kinds of help are available in my community….................................... 1 2 3 4 5

j. I know to teach my child to use and flush the toilet……….........……….........………...... 1 2 3 4 5

k. I feel confident in my role as my child's first teacher............……….........……….......... 1 2 3 4 5

l.

When I read stories to my child I ask what s/he thinks will happen next and why s/he thinks that………………………….......................……….........………....................... 1 2 3 4 5

m.

I know where to find information that I need to help my family [e.g. medical care, dental care].....…………………….....................……….........………............................. 1 2 3 4 5

n. I feel confident in my ability to redirect my child’s behavior….........……….........……. 1 2 3 4 5

o. My child sees me read………………………………………….............…….........……….........…… 1 2 3 4 5

p.

I feel comfortable going to talk to someone to make sure my child's educational needs are met........................................................................................................... 1 2 3 4 5

q. I know how to guide my child in making friends……................................................. 1 2 3 4 5

r. We talk about the pictures when we look at a book……........................................... 1 2 3 4 5

s. I know what my child should be able to do for his/her age...................................... 1 2 3 4 5

t. When I’m outside with my child I point to words on buildings or street signs. 1 2 3 4 5

PLEASE continue on the next page >>>>>>>>>

2. Please tell us how true this was for you before you started Play & Learn…

Not at all true

Completely true

a. I knew how to access information on community events............................................. 1 2 3 4 5

b. I knew that it’s important to teach my child to not talk to strangers……………………...... 1 2 3 4 5

c. I was aware of activities I could do to help my child learn and develop....................... 1 2 3 4 5

d. I read to my child every day........................................................................................... 1 2 3 4 5

e.

I knew where to go if someone in my family needs educational services [e.g. preschool, speech and language].................................................................................. 1 2 3 4 5

f. I knew that it’s important to teach my child to not run into the street……………........... 1 2 3 4 5

g.

I believed that things I do every day will influence the kind of adult my child will become.......................................................................................................................... 1 2 3 4 5

h. I understood the importance of reading the same stories again and again.................. 1 2 3 4 5

i. I knew what kinds of help were available in my community…...................................... 1 2 3 4 5

j. I knew to teach my child to use and flush the toilet...................................................... 1 2 3 4 5

k. I felt confident in my role as my child's first teacher………….......................................... 1 2 3 4 5

l.

When I read stories to my child I asked what s/he thinks will happen next and why s/he thinks that…………………….................................................................................. 1 2 3 4 5

m.

I knew where to find information that I needed to help my family [e.g. medical care, dental care] ......……………………....................................................................................... 1 2 3 4 5

n. I felt confident in my ability to redirect my child’s behavior........................................ 1 2 3 4 5

o. My child saw me read…………………………………................................................................ 1 2 3 4 5

p.

I felt comfortable going to talk to someone to make sure my child's educational needs were met............................................................................................................ 1 2 3 4 5

q. I knew how to guide my child in making friends……………............................................. 1 2 3 4 5

r. We talked about the pictures when we looked at a book.……...................................... 1 2 3 4 5

s. I knew what my child should be able to do for his/her age.......................................... 1 2 3 4 5

t.

When I was outside with my child I pointed to words on buildings or street signs………………………………………...................................................................................... 1 2 3 4 5

Thank you for completing this survey!

Home Visitation Program Survey

What is today’s date? ______ / _______ / _______

Please help us understand how this program can be better. Please write in your responses below the question. Do not put your name on this survey – thank you.

1. What did you enjoy the most about this program? Please be specific.

2. During the home visits, what are a couple of the most important things that you learned (about your child(ren), their development, yourself)?

3. What would you have changed about the program?

4. Is there anything missing or is there something you wish could be added to this program?

Thank you for your feedback!

For office use only: Facilitator’s Name: Entered: _____________________ .

Rev. 07-31-13

Play & Learn Monthly Evaluation

(Program Survey)

Month: ______________________________________ Location: ______________________________________ Number of Play & Learn Groups attended this month: (circle one)

1 2 3 4 5

1) During this month’s Play & Learn groups, what did you enjoy the most? Please be specific. (Ex: themes, activities, interactions, etc...)

2) During this month’s Play & Learn groups, what did you learn about your child(ren) and their development?

3) During this month’s Play & Learn groups, what would you have changed?

4) During this month’s Play & Learn groups, was there anything missing or was there something you wish was added to this program?

5) In the next month’s Play & Learn groups, what would you like to see more of or get more information about?

(Over)

For office use: Facilitator’s Name: Entered: _____________________

Rvsd 06/26/13

6) How have you used the information and materials from this month’s Play & Learn groups? Check ALL that apply.

____ It has improved the way I play/interact with my child(ren). ____ I have used materials in different ways at home. ____ I better understand my child(ren)’s development.

____ Other (Please describe): __________________________________________

7) Thinking about this month’s sessions, please rate them using the 3-point scale from 1= Disagree, 2= Undecided, and 3= Agree.

1 Disagree

2 Undecided

3 Agree

a) The session facilitator helped us learn from each other.

b) The children in my care were able to use materials that we don’t have/use at home.

c) The session facilitator was friendly and respectful towards the adults.

d) I learned something about how play encourages my child’s development.

e) I received a book or other resources after each week’s Play & Learn group I attended.

Thank you for your time and support for the ELC!

Rvsd 06/26/13

Documents

Getting Ready to Measure Ready Children Ready …uwsemi.3cdn.net/d07fb097a035b31f9f_25m6vgz93.pdf · Produced by SPEC Associates Detroit, Michigan Getting Ready to Measure Ready Children