Healthy Start EPIC Center Year 1 Evaluation and …...To answer the evaluation questions, the Healthy Start EPIC Center used a combination of process and outcome monitoring methods

Healthy Start EPIC Center Year 1 Evaluation and Quality Improvement Report

May 2015 Reporting Period July 1, 2014 – April 30, 2015

Table of Contents Introduction ....................................................................................................................................................... 3

Key Findings ........................................................................................................................................................ 3

Methods ............................................................................................................................................................. 5

Summary of Methods ..................................................................................................................................... 5

Identification and Baseline of CBA Needs ...................................................................................................... 6

Results ................................................................................................................................................................ 8

Overview of CBA activities ............................................................................................................................. 8

Reach and Participation ............................................................................................................................... 11

CBA Relevance .............................................................................................................................................. 15

CBA Quality ................................................................................................................................................... 19

CBA Effectiveness ......................................................................................................................................... 23

Conclusion and Plans for Year 2 ....................................................................................................................... 26

Summary of Results ...................................................................................................................................... 26

Evaluation Questions for Year 2 ................................................................................................................... 27

Key Findings of QI Cycles .............................................................................................................................. 28

Applying Evaluation Findings in Year 2 ........................................................................................................ 30

Appendix A: Description of Methods ............................................................................................................... 32

Appendix B: Evaluation Surveys ....................................................................................................................... 33

Page|3

Introduction The Healthy Start EPIC Center is the resource and support center for all Healthy Start grantees funded by the Health Resources and Services Administration’s Maternal and Child Health Bureau (HRSA/MCHB). The Healthy Start EPIC Center website (www.healthystartepic.org) hosts a menu and calendar of training options, a resource inventory of evidence-based practices, and forums for grantees to communicate. Additionally, the Healthy Start EPIC Center offers discussion groups/peer learning networks, and one-on-one and group technical assistance (TA).

The primary goal of the Healthy Start EPIC Center evaluation is to determine the extent to which capacity building assistance (CBA) provided by the Healthy Start EPIC Center supports Healthy Start grantees in implementing the Healthy Start framework as a means to improve maternal and perinatal health outcomes. The Healthy Start EPIC Center identified the following evaluation questions.

1. How many CBA activities were provided by content and delivery mode? 2. Which Healthy Start grantees received CBA by grantee characteristics and type and content of CBA

provided? 3. To what extent did CBA provided meet the identified needs of the Healthy Start grantees? 4. To what extent did JSI effectively communicate the Healthy Start Model to grantees? 5. What elements of CBA were reported to be most effective by Healthy Start grantees? 6. To what extent did CBA recipients report an increase in their capacity to implement the Healthy

Start framework, including increase in knowledge, skills, and intended use? 7. To what extent did Healthy Start grantees report changes in their program practices to align with

the Healthy Start Model in response to CBA provided? 8. To what extent did JSI support HS grantees expressing interest in adding to the evidence base on

perinatal and maternal health practices?

The Healthy Start EPIC Center is committed not only to conducting the evaluation but also to using evaluation data for quality improvement. The Healthy Start EPIC Center applies a Plan, Do, Study, Act approach to quality improvement, using evaluation data to inform and improve future Healthy Start EPIC Center CBA activities. The Quality Improvement and Evaluation teams meet quarterly to share the evaluation data, review progress towards objectives, and address any challenges. The teams use the data to answer questions, such as:

To what extent are we reaching grantees with their identified needs?

Who are we reaching? Who are we missing?

Are we using appropriate training modalities?

Are grantees reporting changes in response to CBA provided?

This Year 1 Evaluation Report summarizes and evaluates Healthy Start EPIC Center activities from July 1, 2014 to April 30, 2015. The report also reviews how the evaluation data has been used during Year 1 for quality improvement, and how the findings of this report will be further used in the coming year.

Key Findings Evaluation results are organized to answer six evaluation questions relevant during this first 10-month reporting period. The section begins with a description of the CBA provided during the reporting period then describes the reach, quality, and effectiveness of those activities. The highlights of these sections are:

http://www.healthystartepic.org/

Page|4

CBA delivered: During the reporting period, the Healthy Start EPIC Center:

Delivered two orientation webinars, one three-day convention, and 16 training activities including 3 informational webinars, 2 Hear From Your Peers, and 11 Ask the Experts

Responded to 13 TA requests

Established the website including a searchable inventory of evidence-based practices

Launched two Discussion Groups

Convened the Healthy Start Collaborative Improvement and Innovation Network (CoIIN)

Participation was assessed by the number of and which grantees participated in each type of activity:

On average, grantees attended 4.0 out of 16 training activities (min 0, max 16) and each training activity had representation from 25 grantees (min 7, max 64). Average attendance for each of the training activities was 44 individual participants (min 14, max 102)

9 grantees requested TA across 13 individual TA requests

99 grantees (197 individuals) attended the convention; 19 attended the QI Institute

Almost all (95%) of respondents to the customer satisfaction survey reported that they had accessed the website and that they receive communications from the Health Start EPIC Center

Relevance was assessed by alignment of topics with CBA worksheet results and post-training surveys:

15 of 16 training activities addressed topics identified as priority areas in the CBA worksheets

100% of respondents participating in Hear From Your Peer events reported that the events were relevant to their needs, as did 94% of respondents for informational webinars, and 82% of respondents for Ask the Expert events

All TA requests (n=13) fell into areas of the CBA worksheet determined to be priorities

84% of convention and 66% of QI institute respondents agreed/strongly agreed it was relevant

Over 70% of satisfaction survey respondents distributed the newsletters/training announcements to staff but just about 1 in 6 said they distributed the information to CAN partners

Quality was assessed through post-training surveys and the customer satisfaction survey results:

The Healthy Start EPIC Center met its goal that at least 80% of respondents would positively rate the training (e.g., agree/strongly agree on a four-point scale) for all quality indicators across all training activities

Since providing training materials in advance in response the participant feedback, the proportion of respondents that indicated that the training was satisfactory with no additional improvements needed increased from 68% to 86%

With respect to timeliness, on average, the Healthy Start EPIC Center responded to TA requests in 2.6 calendar days and completed the TA request in 34 days. This was in line with our goal of responding within 2 days and closing out the request in 45 days.

Although participants rated content during the convention as highly understandable, grantees suggested increasing the complexity of the content, providing more opportunities for interaction, and using a location with more options for activities and meals.

Over 90% of satisfaction survey respondents rated the searchable database of EBPs, training calendar, and information on Healthy Start EPIC Center services on the website as useful/very useful

Page|5

Effectiveness was assessed through post-training surveys and the customer satisfaction survey results:

The Healthy Start EPIC Center met its goal that at least 80% of respondents would positively rate the training (e.g., agree/strongly agree on a four-point scale) for all effectiveness indicators across all training activities

In training activity evaluation surveys, 99% of respondents reported they were either very or somewhat confident they could achieve the learning objectives, and 95% of respondents indicated they either agreed or strongly agreed that the training activity would help job performance.

A large majority of convention attendees said they would share information with program staff (95%) and with CAN stakeholders (80%)

Suggested areas of focus as a result of year 1 findings include:

Increase promotion of training resources by:

Cross-promoting in the newsletter

Highlighting resources on the website

Encouraging live training attendees to access the recordings and slides on the website Increase awareness of and access to TA by:

Highlighting stories of the types of TA that have been requested and provided

Reminding grantees that TA is available through the website

Encouraging Project Officers to continue to inform grantees about opportunities for TA Improve post-training evaluation response rates by:

Adapting evaluations to ask critical evaluation questions first

Engaging Project Officers to encourage grantees to complete the evaluations

Looking into alternative processes to send the evaluation link to attendees

Methods

Summary of Methods

To answer the evaluation questions, the Healthy Start EPIC Center used a combination of process and outcome monitoring methods. In designing the data collection sources, JSI’s goal was to answer the evaluation questions effectively while minimizing burden to grantees. The Healthy Start EPIC Center used five data sources to answer the evaluation questions (see Appendix A for detailed descriptions and the follow-up surveys):

A) CBA tracking system

B) Website analytics

C) Annual grantee CBA worksheet

D) CBA follow-up surveys (i.e., for training activities, TA, discussion groups)

E) Qualitative feedback from grantees

The CBA tracking system and website analytics provide process indicators, while the annual grantee CBA worksheet provides measures of implementation of the Healthy Start framework. CBA follow-up surveys are used to identify effective CBA, an additional outcome measure. Table 1 outlines data sources used to answer the evaluation questions.

Page|6

Table 1. Data Sources to Answer Evaluation Questions

Question Data sources

1. How many CBA activities were provided by content and delivery mode?

A. CBA tracking system

B. Website analytics

2. Which Healthy Start grantees received CBA by grantee characteristics and type and content of CBA provided?


B. Website analytics

3. To what extent did CBA provided meet the identified needs of the Healthy Start grantees?

C. Annual grantee CBA worksheet


E. Qualitative feedback

4. To what extent did JSI effectively communicate the Healthy Start Model to grantees?


D. CBA follow-up surveys

5. What elements of JSI CBA were reported to be most effective by Healthy Start grantees?



6. To what extent did CBA recipients report an increase in their capacity to implement the Healthy Start Model, including increase in knowledge, skills, and intended use?




7. To what extent did Healthy Start grantees report changes in their program practices to align with the Healthy Start Model in response to CBA provided?



8. To what extent did JSI support HS grantees expressing interest in adding to the evidence base on perinatal and maternal health practices?

A. CBA Tracking system

Identification and Baseline of CBA Needs

The Healthy Start EPIC Center is committed to providing data-driven training and technical assistance. To identify areas of training and TA needs, the Healthy Start EPIC Center developed a series of worksheets to assess: 1) organizational capacity and staff core competencies, and 2) practices in line with the Healthy Start framework. The Healthy Start EPIC Center created four individual CBA worksheets (collectively “CBA worksheet”) to align with the EPIC framework (Evaluate need and impact, Partner for Collective Impact, Implement evidence-based practices, and Consider the participant’s context). The items assessed in the CBA worksheet were adapted from the OMB-approved NHSPS Program Survey (Attachment D of the planned National Evaluation of Healthy Start).

Baseline information about grantees’ status on these measures was collected through the CBA worksheet implemented at the November Convention in 2014. To assist grantees with understanding program expectations detailed in the CBA worksheet, general sessions were conducted with an expert speaker and peer grantee for each of the four components of the Healthy Start EPIC framework. Following each general session, conference attendees participated in facilitated breakout discussions and completed the relevant CBA worksheet.

The primary use of the CBA worksheet results was to develop the CBA workplan. The Healthy Start EPIC Center determined topics to be priorities if a large proportion of grantees indicated they had not yet started implementing the activity and expressed interest in receiving support on the topic. The Healthy Start EPIC Center Training and TA team used these priority topics to develop the Hear From Your Peer and Ask the Expert training events.

Page|7

In addition to highlighting areas of need for support, the CBA worksheet results provide baseline measures of grantee competencies. The Healthy Start EPIC Center sent individual reports to Healthy Start grantee Project Directors; Regional Reports to Division of Healthy Start and Perinatal Services (DHSPS) Project Officers; and an aggregate report to the Healthy Start EPIC Center and DHSPS teams. The CBA worksheet is intended to be implemented annually to track progress in program implementation at the grantee level. Furthermore, aggregating at the regional and national level allows for tracking of group changes over time.

Table 2. Average CBA Worksheet Scores

Average Score Possible Points

Percent of Possible Points

EVALUATE NEED AND IMPACT

Community needs assessment 5.2 10.0 Quality improvement and evaluation 4.5 8.0 Subtotal 9.7 18.0 54% PARTNER FOR COLLECTIVE IMPACT

CAN Member 1.4 2.0 Identify and recruit CAN members 3.4 6.0 Set CAN Priorities and Action Plan 3.8 10.0 Communicate/Coordinate Shared Agenda 3.9 10.0 Document CAN Impact 1.3 4.0 Number of partners 3.0 4.0 Partner type 2.4 6.0 Subtotal 16.2 42.0 39% IMPLEMENT EVIDENCE-BASED PRACTICES

Recruit participants 1.6 2.0 Provide screening and case management 5.2 8.0 Assist in insurance enrollment 3.0 6.0 Offer services to promote

wellness/prevention 11.3 18.0

Ensure access to primary health services 2.7 4.0 Encourage partner involvement 2.8 6.0 Deliver evidence-based practices (11

practices) 16.7 22.0

Solicit input 1.7 2.0 Subtotal 45.0 68.0 66% CONSIDER CONTEXT

Assess social context 2.6 4.0 Provide appropriate services 3.6 6.0 Promote cultural competency 3.6 6.0 Assess life conditions 1.7 2.0

Subtotal 11.4 18.0 63% TOTAL SCORE 82.3 146 56%

In addition, JSI developed a score system (Table 2) that assigned points to the activities included in the CBA worksheet. Grantees were assigned points based on their reported progress in implementation, number of partners and partner type, and degree to which they reported implementing evidence-based practices. Based on their scores, grantees received a quintile assignment for each of the EPIC components, as well as

Page|8

an overall quintile assignment. These quintiles were shared with Project Officers in February 2015. The scores are another source of data for the Healthy Start EPIC Center and Project Officers to identify grantees who may need additional support on a topic, as well as grantees with strengths in a given area. JSI also intends to use these scores to track effectiveness of the Healthy Start EPIC Center in helping grantees implement components of the Healthy Start framework.

Results Evaluation results are organized to answer the evaluation questions of focus during this first reporting period. The section begins with a description of the CBA provided during reporting period, then describes the reach, quality, and effectiveness of those activities. The registration, attendance, and evaluation data excludes Healthy Start EPIC Staff, and includes grantees, HRSA/MCHB staff, and other HRSA program grantees.

Overview of CBA activities

The purpose of this first section is to answer the question: How many CBA activities were provided by content and delivery mode? CBA provided by the Healthy Start EPIC Center between July 1, 2014 and April 30, 2015 can be grouped into seven types (Table 3). The Healthy Start EPIC Center held two orientation webinars in September (when grantees were funded) and October, and an orientation Convention in November. Since grantees began ramping up in the fall, the time to deliver CBA to grantees in year 1 was limited to 4 months. The majority of activities took place after January 1, after the needs were identified through the CBA worksheet.

Table 3. Overview of CBA Activities (July 1, 2014 - April 30, 2015)

Type of Activity Description Number

Orientation Webinars

The Healthy Start EPIC Center supported DHSPS in providing two webinars with overviews of program requirements.

2

Convention and Institute

Three-day conference including: 1) 1.5 day Quality Improvement institute for Level 3 grantees and 2) 1.5 day conference for all grantees to introduce the EPIC framework.

1

Training Activities Training activities include: informational webinars, Hear From Your Peer events, and Ask the Expert events.

16

Informational Webinars

Webinars providing general information about the Healthy Start program and Healthy Start EPIC Center using a standard webinar approach.

3

Hear From Your Peers

Webinars hosted by a Healthy Start grantee and another field expert to highlight information and experiences around implementation. The design stemmed from grantee feedback from the Convention, where grantees expressed a desire to take advantage of the knowledge and experience among Healthy Start grantees.

2

Ask the Experts Brief presentations by a field expert, followed by a 30-minute question & answer period during which the expert answers live questions from the audience.

11

Discussion Groups A series of three conference calls with approximately 10 grantees who are interested in in-depth discussion on a topic. Discussion Groups are launched after a Hear From Your Peer event on the same topic.

2

Page|9

Type of Activity Description Number

Technical Assistance Requests

Request for one-on-one assistance/support from a grantee, submitted through the Healthy Start EPIC Center website (after approval from the grantee Project Officer).

13

Healthy Start Collaborative Improvement and Innovation Network (CoIIN) Support

Subset of Healthy Start grantees charged with defining and articulating the Healthy Start Model on behalf of all grantees. The Healthy Start EPIC Center has supported the Quality Improvement Institute, an in-person meeting, and a phone meeting of members.

3

Healthy Start EPIC Center Website

Resource clearinghouse for materials produced by the Healthy Start EPIC Center, as well as other evidence-based resources, designed to support Healthy Start grantees.

Temporary website Oct. 2015; Fully functional

website Jan. 2015

Training Activities

There are three types of training activities: informational webinars, Hear From Your Peer events, and Ask the Expert events. Table 3 summarizes the training activity conducted during the reporting year, including activity topic and type. In addition, data on the number of grantee attendees, individual attendees, and completed evaluations is provided.

Overall, there were 16 activities, averaging four activities per month (January through April). This included three informational webinars, two Hear From Your Peer events (1 event per month March-April) and 11 Ask the Expert events (average of 3.7 events per month February-April, range of 3-5 events).

Table 4. Training Activities by CBA Topic

Training Activity Title Date Training Type Number of

grantee attendees

1

Number of individual attendees

2

Number of evaluations

received3

Using CBA Worksheet Results to Prioritize Support

1/15/15 Informational

Webinar 44 78 36

An Overview of the Healthy Start EPIC Center Website

1/29/15 Informational

Webinar 64 102 35

Healthy Start and Collective Impact 2/12/15 Informational

Webinar 34 62 19

Healthy Start Client Recruitment Strategies

3/12/15 Hear From Your

Peer 19 27 7

Care Coordination and the Healthy Start Community

4/14/15 Hear From Your

Peer 22 39 9

The Value of Integrating Trauma Informed Care for the Healthy Start Community

2/3/15 Ask the Expert 25 42 --4

An Overview of the H.U.G Your Baby Program

2/10/15 Ask the Expert 20 33 --

1 Grantees with representation at the event. A grantee may have had more than one individual attendee.

2 Number of individuals in attendance. This count excludes Healthy Start EPIC Center staff.

3 Number of evaluations received. This count excludes Healthy Start EPIC Center staff.

4 The Healthy Start EPIC Center began evaluating Ask the Expert events after Preconception Care (3/10)

Page|10

Training Activity Title Date Training Type Number of

grantee attendees

1

Number of individual attendees

2

Number of evaluations

received3

Stress, Depression, and Resilience 2/14/15 Ask the Expert 26 48 --

Preconception Care 3/10/15 Ask the Expert 21 34 8

The Happiest Baby 3/19/15 Ask the Expert 18 29 7

Learn the Signs, Act Early 3/24/15 Ask the Expert 16 27 5

An Overview of the Gabby System 3/31/15 Ask the Expert 14 21 4

Oral Health: Why It’s Important and What Healthy Start Communities Can Do to Improve It

4/7/15 Ask the Expert 7 14 3

Using Doulas as a Resource for Case Management


Reproductive Life Planning: Setting Goals for a Healthy Family

4/28/15 Ask the Expert 37 87 10

CenteringPregnancy and CenteringParenting: Innovative Models for Prenatal, Well-Woman and Well-Baby Care


Two Discussion Groups have begun. One focuses on care coordination and the other, a longer-term Peer Learning Network, focuses on achieving collective impact. Since no Discussion Groups were completed at the time of this report, a future report will include process and evaluation results on these activities.

Technical Assistance

The Healthy Start EPIC Center began offering one-on-one technical assistance in January 2015. Between January 25, 2014 (when the Healthy Start EPIC Center website was launched) and April 30, 2015, there have been 13 requests (an average of 4.3 requests per month). This includes seven requests that have been closed by the Healthy Start EPIC Center and six requests that are still in process (Table 5).

Table 5. TA Requests

Topic of TA Request Number of Requests

TA Provision

Community Health Workers 1 Expert Referral

Evaluation 2 Expert Referral (2)

Participant Recruitment 2 Expert Referral (1), None (1)

Reflective Supervision 1 Expert Referral

Evidence-Based Practices 1 TBD

Fatherhood 2 Referral to Discussion Group (1), None (1)

Collective Impact (CI) 2 Referral to Discussion Group (1), Expert Referral (1)

Evidence-Based Practices & CI 1 Expert Referral

Internal Review Board 1 Expert Referral

Total 13

*One TA request was for both collective impact and evidence-based practices

Page|11

The majority of TA requests (n=8, 62%) were referred to expert consultants. Two requests required only a brief check-in call from the Healthy Start EPIC Center (indicated by “None” in Table 5). Two requests were referred to a planned Discussion Group so the grantee could benefit from multiple experiences on the topic. One request is currently under review to determine the follow-up plan.


The Healthy Start QI Institute (1.5 days) and Grantee Convention (1.5 days) took place November 18-20, 2014, at the Hilton Alexandria Mark Center. The convention provided a venue for communicating program expectations about the new Healthy Start model. It also offered opportunities for grantees to network with each other, with MCHB, and with JSI, and to share challenges and areas of expertise for the coming years. The framework of the agenda was built around the EPIC principles. Each EPIC session included a keynote speaker who provided an overview of the topic and a grantee or client speaker that focused on the principle’s implementation in the field. After the general EPIC session, participants broke into regional, smaller facilitated discussions with their Project Officer and Healthy Start EPIC Center facilitators to identify technical assistance needs, reflect on the general session presentation, and complete the CBA worksheet.

Website and Communications

The Healthy Start EPIC Center developed and hosts a branded Healthy Start website to serve as a home for the Healthy Start knowledgebase, a portal for technical assistance, and a workspace to facilitate collaboration and communications. The website includes:

An interactive map of grantee locations with descriptive information to facilitate peer collaboration. Low birth weight, infant death, and preterm birth data are included as filters on the map. Addresses of the NFIMR locations were obtained from the American Congress of Obstetricians and Gynecologists.

A searchable database of evidence-based practices

Information on Healthy Start EPIC staff, services, and the EPIC framework

A training calendar with the capability to register for upcoming trainings

Request form for technical assistance

Message boards

The Healthy Start EPIC Center hosts listservs for all grantees and CoIIN members. There are two emails per month: the first is a training announcement and the second a newsletter.

Reach and Participation

The purpose of this section is to answer the question: Which Healthy Start grantees received CBA by grantee characteristics and type and content of CBA provided? The Healthy Start EPIC Center’s goal is to provide high quality, data-driven CBA. In order to understand whether this CBA is reaching the target audience, the reach and participation of grantees in CBA activities was examined. The evaluation primarily assessed reach and participation by tracking the number of grantees that participated in each type of CBA. While it is not assumed, nor intended, that all grantees would participate in all activities, the Healthy Start EPIC Center seeks to ensure that all grantees have access to the activities provided. Table 6 summarizes reach and participation by each activity type, allowing for a comparison across activities.

Page|12

Table 6. Summary of Reach and Participation by Activity Type

Activity Type Number of grantees 5

attending one or more events

Average number of attendances per grantee

Average number of attendances per grantees with at least one

attendance

Training Activities 87 4.0 4.6

Informational Webinar

75 1.4 1.9

Hear From Your Peer

34 0.4 1.5

Ask the Expert 70 2.2 3.1

Technical Assistance 9 0.13 1.4

Institute 19 (100% of Level 3) 1.0 among Level 3 grantees 1.0

Convention 99 1.0 1.0

Training Activities

The Healthy Start EPIC Center tracks participation in training activities (including informational webinars, Hear From Your Peer events and Ask the Expert events) at two levels: grantee and individual. Overall, 87% of grantees have had a representative in attendance for one or more of the 16 training activities. Nearly 1 in 5 grantees have attended more than seven training events, while 13% have attended none (Figure 1).

Figure 1. Grantee Participation in All Training Events

Each grantee has had representatives in attendance at an average of 2.2 out of 16 training activities (min 0, max 16). Crozer-Keystone Healthy Start has had a representative at all 16 activities. Training activities have had a range of seven to 64 grantees attending, with an average of 25.

In addition to grantee participation, the Healthy Start EPIC Center looks at individual participation statistics. On average, there are 67 individual registrants and 44 individual attendees per training activity. Informational webinars (which address program-related topics, such as the Healthy Start EPIC Center

5 Reach and participation in this table is at the grantee level. Each grantee may have had more than one individual who attended.

13%

42%

25%

15%

4%

0%

10%

20%

30%

40%

50%

0 1 to 3 4 to 6 7 to 10 11 or more

Pe

rce

nt

of

Gra

nte

es

Number of activities attended (total n=16)

Page|13

website and collective impact) had slightly higher average registrations than the other training activities, suggesting that these topics were considered immediately relevant to all grantees. There was very little variation in attendance rate – with approximately 65% of registrants attending all activities.

By topic, the most well attended training event was the Overview of the Website informational webinar with 102 individual attendees across 64 grantees. The Care Coordination and Participant Recruitment Hear From Your Peer events were also well attended with 27 and 39 individual attendees, respectively. Reproductive Life Planning was the most-attended Ask the Expert event, with 87 individual attendees representing 37 grantees.

Table 7. Participation by Training Type

All Training Activities

6

Informational Webinar Hear From Your Peer

Ask the Expert

Total number of activities

16 3 2 11

Average # of trainings attended per grantee

4.0 1.4 0.4 2.2

Number of grantees attending at least 1 training

87 75 34 70

Average # of grantee attendees per training

25 47 21 20

Minimum and maximum grantee participation

Min = 7 (Oral Health)

Max = 64 (Overview of Website)

Min = 34 (Healthy Start and CI)

Max = 64 (Overview of Website)

Min = 19 (Recruitment)

Max = 22 (Care Coordination)

Min = 7 (Oral Health)

Max = 37 (Reproductive Life Planning)

Average # of individual

7 registrants

per training

67 122 51 55

Average # of individual attendees per training

44 81 33 35

Average individual attendance rate

65% 66% 65% 65%

Maximum individual attendance rate

93% (Oral Health) 70% (Healthy Start and CI) 66% (Recruitment) 65% (Care Coordination)

93% (Oral Health)

In addition to offering live events, the Healthy Start EPIC Center posts all trainings to the website for view as a recording. The Healthy Start EPIC Center examined the reach of the recorded trainings by looking at the number of playbacks from non-Healthy Start EPIC Center staff. The average time that the recording was watched was also noted; playbacks of less than one minute were excluded from the count of playbacks and the average time. To date, the most popular recorded trainings are the Overview of Healthy Start and Collective Impact informational webinar (19 views); Stress, Depression, and Resilience Ask the Expert event

6 Includes informational webinars, Hear From Your Peer and Ask the Experts

7 Individual registrant and attendee counts do not include attendance by Healthy Start EPIC Center-affiliated staff.

Page|14

(15 views) and Care Coordination Hear From Your Peer event (13 views). People viewing the trainings stayed for the longest duration for the Reproductive Life Planning Ask the Expert event (55 minutes) and Healthy Start and Collective Impact informational webinar (52 minutes) (Table 8).

Table 8. Reach of Training Activity Recordings (Number of Playbacks and Avg. Time Watched)

Name Number of playbacks

Average playback duration

(minutes)

Total recording duration (minutes)

Healthy Start and Collective Impact 19 52 111

Stress, Depression, and Resilience 15 42 78

Care Coordination and the Healthy Start Community 13 45 83

An Overview of the H.U.G Your Baby Program 10 18 72

The Value of Integrating Trauma Informed Care 8 39 75

The Happiest Baby 7 47 76

CenteringPregnancy and CenteringParenting 6 27 65

Reproductive Life Planning: Setting Goals for a Healthy Family 6 55 77

Healthy Start Client Recruitment Strategies 4 29 91

Using Doulas as a Resource for Case Management 4 36 70

Using CBA Worksheet Results to Prioritize Support 3 10 74

Learn the Signs, Act Early 3 17 61

Preconception Care 3 34 64

An Overview of the Healthy Start EPIC Center Website 1 16 85

Oral Health: Why It’s Important and What Healthy Start Communities Can Do to

1 25 52

An Overview of the Gabby System 0 N/A 68


The 13 TA requests that been responded to represent nine unique grantees. Six grantees have requested TA once; two grantees have requested TA twice; and one grantee has requested TA three times. Grantees who have requested TA have been diverse in level with six Level 1 grantees, one Level 2 grantee, and two Level 3 grantees.


On final count, 240 people attended the convention including 197 participants representing all but one grantee; 13 MCHB staff; 19 JSI staff and 11 guest speakers. At the Quality Improvement Institute, there were 19 grantee attendees including 18 Level 3 and 1 Level 2 grantee (100% of invited grantees).

Page|15


Most grantees appear to be using the Healthy Start EPIC Center website. Almost all (95%) of the Healthy Start Customer Satisfaction Survey respondents (n=608) had accessed the website. The two respondents (5%) who had not yet accessed the website cited a lack of time for the main reason why. Almost all (95%) of respondents had also signed up for communications from the Healthy Start EPIC Center; 98% reported getting training notifications. Slightly fewer (89%) reported getting the monthly newsletter.

Website activity is evaluated based on the number of sessions (a period of time during which a user is actively engaged in the website), users (someone who has had at least one session, including both new and returning individuals), page-views (the total number of pages viewed, including repeated views of a single page), and duration (the average length of one session). In general, website activity, as defined by these indicators, peaked in the first month that the website was live (February). Since this initial surge, the activity dipped in March, but then increased again in April (Table 9). The number of new grantee message boards registrants has increased slightly; actual participation (e.g., posting/responding to messages) has remained low. The Healthy Start EPIC Center expects message board participation to increase with the full implementation of the Discussion Groups (which launched in April).

Table 9. Website Activity and Participation

Indicator February March April

Number of sessions 1332 830 887

Number of users 662 430 471

Average duration 5 m 37 s 5 m 01 s 4 m 35 s

Number of page-views 6895 3869 3512

Total number of grantees registered for message board 29 34 36

Total number of message board registrants 48 54 56

CBA Relevance

The purpose of this section is to answer the question: To what extent did CBA provided meet the identified needs of the Healthy Start grantees? Relevance was assessed in two ways. First, the Healthy Start EPIC Center assessed relevance based on alignment between CBA activity topics with topics identified by grantees in the CBA worksheets. Second, the Healthy Start EPIC Center assessed respondent perception of activity relevance. Table 10 summarizes relevance on these two indicators, by activity type.9

The Healthy Start EPIC Center previously set a quality goal that all events would receive a mean score of at least 3 on a 4-point scale that the activity was relevant; this metric was met for the informational webinars and Hear From Your Peer events. Due to the low response rate for individual Ask the Expert events (range of respondents between 2 and 10), the Healthy Start EPIC Center did not evaluate events at the individual level; however, at the aggregate level, Ask the Expert events also achieved the quality metric. In looking at

8 At the time of report development (5/19/15) there were 60 responses to the Customer Satisfaction Survey. The survey closed on 5/22.

9 Regarding the evaluation results of training activities, there are some cases when informational webinar and Hear From Your Peer

results are combined, but Ask the Expert event results are presented separately. This is because the informational webinar and Hear From Your Peer evaluations are tailored to the specific activity’s learning objectives but are otherwise identical. The Ask the Expert evaluation was conducted across all Ask the Expert events, and was shortened to reduce burden on respondents.

Page|16

the data, the Healthy Start EPIC Center has opted to amend our quality metric to a goal that 80% of respondents agree/strongly agree that the training was relevant at the individual and aggregate levels. This quality metric was met across all training activities.

Table 10. Summary of Relevance by Activity Type

Indicator Percent of activities which are CBA worksheet priority

% of evaluation respondents agree/strongly agree training was relevant

Training Activities 93% 91%

Informational Webinar 100% 94%

Hear From Your Peer 100% 100%

Ask the Expert 91% 82%

Technical Assistance 100% No evaluation results

Convention N/A 86%

Training Activities

Overall, 15 of 16 training activities (94%) addressed topics identified as priorities from CBA worksheet results (Table 11). The only topic not identified in the CBA worksheets was oral health.

Table 11. Training Activities by CBA Topic

Name Training Type Priority CBA Topic

Using CBA Worksheet Results to Prioritize Support Informational Webinar

An Overview of the Healthy Start EPIC Center Website Informational Webinar

Healthy Start and Collective Impact Informational Webinar

Healthy Start Client Recruitment Strategies Hear From Your Peer

Care Coordination and the Healthy Start Community Hear From Your Peer

The Value of Integrating Trauma Informed Care for the Healthy Start Community

Ask the Expert

An Overview of the H.U.G Your Baby Program Ask the Expert

Stress, Depression, and Resilience Ask the Expert

Preconception Care Ask the Expert

The Happiest Baby Ask the Expert

Learn the Signs, Act Early Ask the Expert

An Overview of the Gabby System Ask the Expert

Oral Health: Why It’s Important and What Healthy Start Communities Can Do to Improve It

Ask the Expert

Using Doulas as a Resource for Case Management Ask the Expert

Reproductive Life Planning: Setting Goals for a Healthy Family Ask the Expert

CenteringPregnancy and CenteringParenting Ask the Expert

Page|17

Overall, training activity respondents indicated that the training was relevant to their needs. Almost all of respondents (94%, n=70) evaluating informational webinars and 82% of respondents evaluating Ask the Expert events (n=43), indicated they either agreed or strongly agreed that the training was relevant. Meanwhile, 100% of respondents (n=14) agreed or strongly agreed that Hear From Your Peer events were relevant to their needs.

Figure 2. Evaluation of Relevance of Training Activities


As with training activities, the Healthy Start EPIC Center has responded to requests on a range of topics that align with the priorities identified through the CBA worksheet (Table 12). Slightly less than half of the requests, or 6 of 13 requests (46%), were related to implementing evidence-based practices; two of these related to participant recruitment and two related to fatherhood involvement.

Table 3. TA Responses by CBA Topic

Area of the EPIC Framework Number of TA requests Priority CBA Area

Evaluate need and impact 2 High

Partner for collective impact 3 High

Implement evidence-based practices 6 High

Consider the participant’s context 0 Low

Total 13


Overall, 84% of convention evaluation respondents (n=83) indicated they found the convention to be a useful experience, while 11% reported that they were unsure about the convention’s utility. Just under two-thirds (66%) of the Quality Improvement Institute evaluation respondents (n=19) agreed or strongly agreed that the Institute was relevant. Meanwhile, 86% of all respondents (n=84) agreed or strongly agreed that the all-grantee convention was relevant to their needs.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Webinar

Hear From Your Peer

Ask the Expert

Strongly agree Agree Disagree Strongly disagree

86%

66%

0%

50%

100%

Convention QI Institute

Figure 3. Proportion of Convention and QI Institute Evaluation Respondents Agree/Strongly Agree

That It Was Relevant to Needs

Page|18

With respect to the overall scope of the convention, grantee feedback was mixed, with some grantees indicating that convention tried to do too much, and others not enough, as evidenced by the following quote: “The training was too slow for those with prior expertise, too limited for those new to topics.” While some grantees indicated that the breakout sessions to explore the EPIC framework were useful and relevant (“The second day was very worthwhile and the breakout sessions provided opportunities to network”), others indicated dissatisfaction with the breakout session content (“Needed more [on] practical implementation process”).


The Healthy Start EPIC Center strives to offer resources that align with the priorities identified in the CBA worksheets. In looking at the resources most frequently downloaded in March and April 2015, there was significant interest in the Institute for Health and Recovery’s 5Ps, a screening tool to assess current and past drug and alcohol use. This may be related to the Ask the Expert event on depression, stress, and resilience on February 24. Also in April, there was an increase in the number of downloads of resources related to doulas, which may be correlated to the Ask the Expert event on doulas which took place on April 21. The Growing Great Kids Curriculum was a popular resource in both months.

Table 4. Most Downloaded Resources by Month

March April

Partners for a Healthy Baby (15) Institute for Health and Recovery’s (IHR) 5P's (43)

Growing Great Kids: Prenatal to 36 Mos. Curriculum (12) Community Based Doula Programs (14)

Healthy Steps (12) Growing Great Kids: Prenatal to 36 Mos. Curriculum (11)

AAFP Updated Guidelines for Prenatal Care (9) The Tampa Bay Doula Program (8)

Institute for Health and Recovery’s (IHR) 5P's (8) Depression in Mothers: More Than The Blues (7)

Respondents of the Healthy Start Customer Satisfaction Survey (n=60) were asked the extent to which they distribute the training announcements and newsletters to staff and peers. Figure 4 demonstrates that the majority of grantees are distributing information to staff every time or almost every time (73% for newsletters and 79% for training notifications). However, grantees are largely not distributing this information to partner organizations (15% for newsletters and 16% for training notifications).

Figure 4. Percent of Respondents That Report Distributing Communications Every Time or Almost Every Time

73%

15%

79%

16%

0%

20%

40%

60%

80%

100%

Distribution ofnewsletter to staff

Distribution ofnewsletter to partner

organizations

Distribution of trainingnotifications to staff

Distribution of trainingnotifications to partner

organizations

Page|19

CBA Quality

This section explores the question: What elements of CBA were reported to be most effective by Healthy Start grantees? This section examines qualities expected to reflect the CBA’s ability to produce outcomes. Specifically, this section examines grantee feedback on areas of improvement for training activities; presenter quality; and process indicators regarding the Healthy Start EPIC Center’s responsiveness to TA requests. Respondents’ qualitative feedback in comment boxes provides further insight.

Table 5. Summary of Quality by Activity Type

Indicator Information was presented in understandable way (% agree

or strongly agree)

Level of satisfaction; areas of improvement

Training Activities

Informational Webinar 95% 68% rated training satisfactory

Add pre-training material, poll questions

Hear From Your Peer 100% 86% rated training satisfactory

Ask the Expert 84% No consensus

Technical Assistance No evaluation results Continued effort to maintain goals of 2 calendar days to follow-up email; 30 days to

complete TA request

Convention 94% Less rushed and more complex content; More time for networking; Improved location

Training Activities

Informational Webinars and Hear from Your Peers

Overall, training activity evaluation respondents (total n=85) agreed or strongly agreed that the information was presented in an understandable way (96%) and that the delivery method was appropriate (97%) (Figure 4). All (100%) agreed or strongly agreed that the information was presented in an understandable way and the delivery method was appropriate for the two Hear From Your Peer events (n=14), and for the informational webinar on Healthy Start and Collective Impact (n=16). This was similarly high for the website overview (97%, both indicators; n=25) and the results of the CBA worksheets (92%, both indicators; n=30).

Figure 5. Evaluation Responses Assessing Quality of Informational Webinars and Hear From Your Peer events

For Hear From Your Peer and informational webinar events, evaluation respondents were asked what improvements could be made to the training; one response to this question is the training was satisfactory

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Information was presented in understandable way

Delivery method was appropriate

Strongly Agree Agree Disagree Strongly disagree

Page|20

with no needed improvements (Figure 6). Two thirds (67%, n=72) of evaluation respondents rated informational webinars as satisfactory and 86% of respondents (n=14) rated Hear From Your Peer events as satisfactory. From the list of potential improvements, the most common selected response was to provide materials in advance of the training (14%, or 12 of 86 respondents). It is worth noting that feedback around needing pre-training material came from the first three training activities (informational webinars). The Healthy Start EPIC Center reviewed early evaluation data after the second informational webinar and began sending the slides with the registration link. Since that time (i.e., for the two subsequent Hear From Your Peer events), the percentage of respondents who said the training would have been improved if they received pre-training material has dropped to 0%.

Figure 6. Areas of Improvements for Informational Webinars and HFYP

Among the seven “Other Responses,” there were three comments expressing a desire for an electronic version of the materials to share with staff unable to attend the training. In the early informational webinars, there were requests for more interaction. This led the Healthy Start EPIC Center to include poll questions in all future training activities. The Healthy Start EPIC Center has also made audio recordings and transcripts of training activities available on the website.

Ask the Experts

The format of the Ask the Expert events evaluation surveys differed slightly from the other training activity events. Given the nature of the events, which entail a 20 minute overview on a topic followed by audience Q & A, the Healthy Start EPIC Center opted to eliminate questions about learning objectives, and pare down the evaluation to fewer questions as compared to the longer informational webinar/Hear From Your Peer event evaluation surveys. A question about speaker quality was included in the Ask the Expert evaluation.

Overall, 84% of respondents (n=43) agreed or strongly agreed that the information in the Ask the Expert events was presented in an understandable way, and 84% (n=43) also agreed or strongly agreed that the expert was knowledgeable. One quarter, or 10 of the 43 completed evaluations were specific to one Ask the Expert event, Reproductive Life Planning. When examining the results of this event separately, the results were comparable (80% for both indicators, n=10) to the results overall. Given the small respondent samples for the other individual events (range of two to eight respondents, see Table 3 above), event-specific comparisons were not appropriate.

0%

1%

1%

1%

2%

3%

8%

14%

74%

0% 20% 40% 60% 80%

Too much material

Too slow

Too complex

Technical challenges

Too quick

Too basic

Other

Pre-training materials desired

Training satisfactory

Page|21

Figure 7. Evaluation Responses Assessing Quality of Ask the Experts

In an effort to keep evaluations short, the Healthy Start EPIC Center did not ask specifically for areas of improvement for the Ask the Expert events. However, qualitative responses submitted via comment boxes on the survey were reviewed in this context. With regards to content, there were six comments. One comment indicated the respondent would have preferred a longer presentation, while another said the training was “short but informative.” There was one comment each reminding about being sensitive to cultural references and being sensitive to differences in time zones when scheduling events. The final two comments indicated that the respondents were pleased with the content; said one respondent, “I feel that all the information given today was great and I will apply what I have learned.”


The Healthy Start EPIC Center originally planned to evaluate TA delivery at three months after the close of the TA request. After a sufficient number of TA requests had been addressed and subsequent discussions with the Training and TA CBA Coordinators, the evaluation approach was changed to entail an immediate survey, followed by a 3-month follow-up survey. This change was implemented starting in April 30, 2015; thus, no evaluation survey results are included in this Year 1 Report. That being said, the sample size of evaluations (n=7 closed requests) would have been small.

To assess quality of TA provided, the Healthy Start EPIC Center tracks process measures to evaluate the timeliness and responsiveness of TA. All TA requests are entered through the Healthy Start EPIC Center website. Those who request TA immediately receive an email confirming the Healthy Start EPIC Center has received the TA request. This process is automatic; the Healthy Start EPIC Center reviews TA request records to confirm the functionality of the submission process. The Healthy Start EPIC Center then sends a follow-up email to the Project Officer and person requesting TA to establish communication and to schedule a kick-off phone call to finalize the scope of the request. This process reflects a discussion with MCHB/DHSPS Project Officers about how to best respond to TA request needs. The Healthy Start Center EPIC Center has committed to send this follow-up email within two business days. Records indicate that the average number of days between receiving the TA request and sending the follow up email is 2.6; however, this is skewed by a two requests in early February (11 and 5 days) when the process for responding to TA requests was still being finalized. Without these data, the average time to follow-up email is 1.4 days.

Table 6. TA Response Quality Indicators

Indicator Performance

Average number of days to confirmation email 0

Average number of calendar days to follow-up email 2.6

Average days to complete TA request 34

0% 20% 40% 60% 80% 100%

The expert was knowledgable

The information was presented in an understandable way

Strongly agree Agree Disagree Strongly disagree

Page|22

The Healthy Start EPIC Center has a goal to complete the TA request within 45 days. This goal was defined recognizing that some grantees may want to postpone the start of the TA request, and that experts may not always be available to respond in that window. The Healthy Start EPIC Center has achieved its goal, despite the recognized limitations. Overall, the average time a TA request was open (if closed) or has been open (if still in process) is 34 days (min 2, max 90). The outlier, a 90-day request, was delayed due to challenges securing and scheduling the TA expert consultant in collective impact, in line with the expected challenge of variations in expert availability.

Overall, eight of the 13 requests (62%) were referred to a content expert. Two requests (15%) were referred to a Discussion Group. Two requests (15%) required no more engagement after the initial kick-off phone call, and the plan for one TA request (received 4/29) is yet to be determined.


Among the respondents of the convention evaluation survey (n=85), 90% agreed or strongly agreed that the in-person training delivery method was appropriate and 94% agreed or strongly agreed the material was presented in an understandable way (n=85). Three quarters of respondents (74%) agreed or strongly agreed that there were adequate opportunities to connect and share resources, while 25% disagreed/strongly disagreed. This suggests that future in-person trainings should have more opportunities for grantee networking.

Figure 8. Proportion of Convention and QI Institute Evaluation Respondents (n=85) Agree/Strongly Agree

In terms of areas of improvement, a third (31%) of respondents indicated a desire for pre-training materials (n=85). A quarter (25%) indicated the content was too basic, while 4% indicated it was too complex and 18% indicated there was too much material allotted for the amount of time. Meanwhile, 15% indicated the pace was too fast and 8% indicated the pace was too slow. A quarter (25%) reported that the convention was satisfactory with no needed improvements.

The few comments on the location of the convention indicated that grantees would have preferred a different location, closer to transportation and to food and activity options.

90% 94% 74%

0%

20%

40%

60%

80%

100%

In-person trainingdelivery method was

appropriate

Material presented inunderstandable way

Adequateopportunities to shareresources and connect

Page|23


Respondents of the Healthy Start Customer Satisfaction Survey (n=60) offered insight into the quality of the website and communications. With regards to the website, almost all respondents rated very useful or somewhat useful the following elements of the website: training calendar (93%), searchable database of EBP (91%), and information on HS EPIC services (91%). Just about a third (32%) rated the message boards as very or somewhat useful, indicating that this is an area of improvement. With regards to the interactive map, 65% of respondents rated it useful/very useful; this suggests there may be an opportunity for training on how to use the interactive map to help conduct a needs assessment or facilitate a strategic planning session.

Figure 9. Percent of Respondents Rating Website Component Useful or Very Useful

Overall, respondents did not have any qualitative comments for how the website could be improved. Eight of the 9 open ended responses indicated that respondents felt the website was satisfactory as is. One respondent suggested providing a glossary of definitions of activities, services, and interventions, as defined by HRSA for Healthy Start grantees.

Respondents were asked for how the newsletter could be improved. The majority of respondents (71%) said the newsletter was satisfactory as is. Almost a quarter (23%) felt that the newsletter could be improved but did not offer specificity on areas of improvement.

CBA Effectiveness

The purpose of this section is to answer the question: To what extent did CBA recipients report an increase in their capacity to implement the Healthy Start Model, including increase in knowledge, skills, and intended use? Effectiveness is assessed by the extent to which evaluation respondents report confidence to implement learning objectives identified a priori; to what extent they indicate the training will help job performance; and how they intend to apply what they have learned. The Healthy Start EPIC Center asks how respondents intend to use materials with options that range from sharing information (with staff and CAN members/stakeholders) to making a change in program practices, and ultimately to making a change in procedure or policy. The goal of the Healthy Start EPIC Center is to encourage grantees to make concrete changes (in practice, procedure, or policy) since these changes are ingrained in the infrastructure and are therefore more sustainable. That being said, the Healthy Start EPIC Center also recognizes the importance of spreading awareness to increase knowledge and change attitudes, which are precursors to these more intensive behavior changes. Thus, intention to apply information from CBA activities across the spectrum is considered an important measure of effectiveness.

93% 91% 91% 79%

65%

32%

0%

20%

40%

60%

80%

100%

Training calendar Searchabledatabase of EBP

Information on HSEPIC Services

Information on HSEPIC framework

Interactive map Message boards

Page|24

Table 7. Summary of Effectiveness by Activity

% of respondents that report they will… Hear From Your Peers

Ask the Expert Convention

share information with staff 88% 87% 95%

share information with CAN/stakeholders 51% 38% 80%

make a change in program practices 13% 22% 47%

make a change in procedure or policy 10% 11% 24%

Training Activities

Webinar/Hear From Your Peers

The evaluation for the three informational webinars and two Hear From Your Peer events was tailored to assess confidence to achieve or implement the learning objectives, determined a priori by the training presenter. Across these five events, 99% of respondents (n=86) indicated that they were very or somewhat confident to achieve the learning objective. This suggests that the Healthy Start EPIC Center has clearly defined objectives, and for the most part, has ensured that the trainings fulfill the predetermined goals. In terms of the Hear From Your Peer events, 93% and 86% of evaluation respondents felt very confident to achieve the learning objectives for the Participant Recruitment (n=5) and the Care Coordination (n=9) events, respectively.

Figure 10. Average Confidence of Evaluation Respondents to Achieve Learning Objectives of Informational Webinar/HFYP

Overall, 95% of respondents (n=79) either agreed or strongly agreed that the training activity would help job performance. There was some variation by activity, with 100% of respondents for the Recruitment and Retention Hear From Your Peer event (n=5) and the Care Coordination Hear From Your Peer event (n=7) reporting that they agreed or strongly agreed that the training would help job performance (among those who selected a response).

Figure 11. Proportion of Evaluation Respondents Reporting Informational Webinar/HFYP Would Help Job Performance

In terms of using information from the training events, a majority of respondents of the informational webinar and Hear From Your Peer events (n=86) reported that they intend to share the information with

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Very Somewhat Not too Not at all

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Strongly Agree Agree Disagree Strongly disagree

Page|25

staff (88%) and CAN partners or other community stakeholders (51%) when aggregating across trainings. One in 10 (10%) respondents reported that they would make a procedure or policy change and 13% reporting they would make a change in program practices. Among the “other” responses, grantees said they would: take advantage of more EPIC support (n=2) and explore the training topic further.

Figure 12. Proportion of Evaluation Respondents Who Intend to Use Material from Informational Webinar/HFYP

Ask the Expert

The breakdown of intention to use material for the Ask the Expert evaluations was quite similar to the breakdown for informational webinar and Hear From Your Peer event evaluations, with a large majority of respondents (total n=45) reporting that they would share information with Healthy Start program staff (87%). A third (38%) indicated they would share the information with CAN members or other stakeholders. Almost a quarter indicated they would make a change in program practices (22%) and 11% indicated they would make a change in policies. Among the “other” responses, grantees said they would: share information with participants (n=3), explore additional funding, and propose sustainable ideas.

Figure 13. Proportion of Evaluation Respondents Who Intend to Use Material from Ask the Expert

5%

10%

13%

13%

51%

88%

0% 20% 40% 60% 80% 100%

None

Change program procedures

Other

Change program practices

Share info with partners/community

Share info with program staff

2%

11%

13%

22%

38%

87%

0% 20% 40% 60% 80% 100%

None


Other




Page|26


The Healthy Start EPIC Center had originally intended to implement all TA evaluations at 3 months after the close of the TA. Because the Healthy Start EPIC Center began responding to TA requests in late January/early February, no evaluation surveys had been sent to TA recipients during the reporting timeframe. The Healthy Start EPIC Center has since changed its evaluation procedure, recognizing that real-time evaluation feedback is needed. Moving forward, after a TA close-out, the recipient will receive an evaluation survey immediately following the TA, and another three months after the TA is provided.


Like the training activities, a large majority of respondents (total n=85) reported that they would share information from the Convention with Healthy Start program staff (95%) and with CAN members or other stakeholders (80%). Almost half (47%) indicated intentions to make a change in program practices, and almost a quarter (24%) indicated intentions to make a procedure or policy change.

Figure 14. Proportion of Evaluation Respondents Who Intend to Use Material from the Convention/Institute

Conclusion and Plans for Year 2

Summary of Results

The table below summarizes the results across activity type by the indicators of interest: reach, relevance, quality, and effectiveness.

Table 8. Summary of Evaluation Results by Activity

Measure Hear From Your Peer

Ask the Expert

TA Convention

Reach

Number of grantees with at least 1 participation

34 70 9 99

1%

9%

24%

47%

80%

95%

0% 20% 40% 60% 80% 100%

None

Other





Page|27

Measure Hear From Your Peer

Ask the Expert

TA Convention

Relevance

% agree/strongly agree training was relevant

100% 82% No results 86%

Relevance

% of activities in a priority area (determined by CBA worksheets)

100% 91% 100% NA

Quality

% agree/strongly agree information was presented in understandable way

100% 84% No results 94%

Quality

Areas of improvement

86% rated training

satisfactory

No clear consensus

among respondents

Continued achievement of timeliness goals (2 days for follow up email, 45 days to

close out request)

Less rushed; More complex content; More

time to network; Improved location

Effectiveness

% that report they will share information with staff

88% 87% No results 95%

Effectiveness

% that report they will share information with CAN/stakeholders


Effectiveness

% that report they will make a change in program practices


Effectiveness

% that report they will make a change in policy or procedure


Evaluation Questions for Year 2

This evaluation report has focused on evaluation questions 1-3 and 6. The Healthy Start EPIC Center will explore Questions 4, 7 and 8 in future evaluation reports.

Question 4, To what extent did JSI effectively communicate the Healthy Start Model to grantees?, requires that there be a Healthy Start Model to communicate to grantees. The process of articulating the Model began at the November Convention, when qualitative feedback from attendees indicated that grantees did not feel there was a defined Healthy Start Model. The Healthy Start CoIIN, a subset of grantees selected to reflect the larger Healthy Start community, has been tasked with further articulating the Healthy Start Model. The Healthy Start EPIC Center provides logistical support to the CoIIN on regular phone and in-person meetings. At the first in-person meeting in March 2015, CoIIN members agreed on the goals of standardizing the Healthy Start Model, developed a common understanding of what could and could not be standardized, and established priorities for the Healthy Start CoIIN over the first six months. The members determined that their first task would be to define the assessment process so that all women who enter a Healthy Start program are consistently evaluated.

The Healthy Start EPIC Center sees our role in communicating the Healthy Start Model as twofold: first, to provide logistical and facilitation support to the Healthy Start CoIIN meetings and confirm points of agreement; and second, to relay those findings back to the broader Healthy Start community. The first task is ongoing, while the second task is expected to take place after the CoIIN completes its task and expresses

Page|28

readiness to communicate its decisions more broadly. Question 7, To what extent did Healthy Start grantees report changes in their program practices to align with the Healthy Start Model in response to CBA provided?, requires a second implementation of the CBA worksheets (planned for late 2015) to track progress in implementing components of the Healthy Start framework. For Question 8, To what extent did JSI support HS grantees expressing interest in adding to the evidence base on perinatal and maternal health practices?, the Healthy Start EPIC Center originally planned to look at TA requests related to adding to the evidence base, but few TA requests have been logged to date (and none on the topic), due to the short window between the launch of the Healthy Start EPIC Center and the evaluation report.

Key Findings of QI Cycles

To date, the Healthy Start EPIC Center has implemented the following changes to improve the quality of CBA services, based on evaluation data:

Actively engage grantees in all aspects of CBA to leverage the benefit of grantee expertise: At the November 2014 Convention, grantees expressed that there is a significant amount of knowledge and experience in the grantee field. There was a desire among grantees to leverage that expertise for other grantees. Based on the feedback, the following changes were made:

The Healthy Start EPIC Center shifted its plan to conduct action learning collaboratives around key topic areas, and established a CBA delivery calendar that emphasized peer involvement. The Hear From Your Peer webinars are co-facilitated by a grantee and other expert and designed to highlight specific grantee evidence-based practice implementation experiences and insights. Feedback from Hear From Your Peer evaluation surveys suggests that this was a successful change. Discussion Groups also provide a forum for shared learning across grantees.

Six grantees participated in collective impact trainings and will lead the Collective Impact Peer Learning Networks.

The membership of the CoIIN was expanded to include a Level 1 and Level 2 grantee, recognizing the expertise of all levels of grantees.

The Healthy Start EPIC Center has positioned itself as a supporting infrastructure to facilitate or support grantees identifying and meeting their needs, and the needs of the Healthy Start Program. Grantees are actively engaged in all CBA planning processes.

Foster leadership development among Healthy Start CoIIN members as they evolve into their advisory role: During the initial Healthy Start CoIIN meetings, members expressed their desire to facilitate and lead the effort to define and articulate the Healthy Start Model. Based on this feedback, the Healthy Start EPIC Center has explicitly filled a support role. The Healthy Start EPIC Center facilitated establishing the first round of co-chairs, who are fully engaged in planning and leading each virtual and face-to-face CoIIN meeting. To date, meetings facilitated by grantee co-chairs have resulted in identifying case management as a central Healthy Start component through group consensus. The co-chairs are currently engaged in leading the CoIIN through a process of prioritizing risk factors to include in a standardized Healthy Start screening tool. CoIIN members have actively engaged in respectful discussion.

Promote broader participation in Healthy Start EPIC CBA activities: The Healthy Start EPIC Center heard anecdotal feedback from grantees that the planned training calendar (two Hear From Your Peer events per month) was too intensive, and that the volume of communications from the Healthy Start EPIC Center

Page|29

related to the training calendar was overwhelming. Grantees also indicated that they needed more notice of training opportunities. Based on the feedback, the following changes were made:

Because the Healthy Start EPIC Center encouraged grantees to sign up for post-webinar Discussion Groups, the T/TA Team shifted the planned calendar to one event per month. In the May Healthy Start Customer Satisfaction survey, respondents were asked how they felt about the training calendar. Over half (59%) of respondents said the number of trainings was about right, while 9% said too little and 31% said too much.

Figure 15. Evaluation of Frequency of Trainings

The Healthy Start EPIC Center developed a communications schedule to reduce the volume of communication. The schedule now includes twice-monthly emails. The first email is a training announcement and the second is a newsletter.

The Healthy Start EPIC Center communicated that TA is free of charge to grantees.

Another change that the Healthy Start EPIC Center made was to encourage Project Directors to forward and disseminate training announcements to other staff members. DHSPS Project Officers were also encouraged to spread this message during regular monthly calls.

After the first three informational webinars, the evaluation data indicated that 16% of respondents (n=72) suggested that providing pre-training materials would have improved the training experience, prompting the Healthy Start EPIC Center to begin sending slides in advance of the training. Since then, the percentage of evaluation respondents asking for pre-training materials dropped to 0%. Of note, this question has not previously been asked of Ask the Expert event evaluation respondents; however, the question has been added for future evaluation reports.

Figure 16. Proportion of Respondents Requesting Pre-Training Materials, by Training Activity Type

9%

59%

31% Too little

About right

Too much

16%

0% 0%

10%

20%

Informational Webinars(January-February)

HFYPs (March-April)

Page|30

Applying Evaluation Findings in Year 2

The Healthy Start EPIC Center plans to implement the following changes in year 2 based on the results of the evaluation data presented in this report.

Increase promotion of available training resources: The evaluation results suggest that respondents are largely content with the material’s relevance and quality, but attendance may not be reaching maximum capacity. While the Healthy Start EPIC Center does not expect all grantees to attend all trainings, we do want to ensure that all grantees are aware of all training opportunities and other services. Specific action items to increase the awareness of available resources include:

Continue advertising and marketing of Hear From Your Peer events as a way to hear how peer grantees are implementing changes on the ground

Use the website’s home page to highlight training opportunities

Promote the availability of recorded trainings on the website

Cross-promote training opportunities in the newsletter, including adding quotes from evaluation respondents of the Hear From Your Peer events to the newsletter and adding a standard message that includes a link to recorded webinars in the newsletter

Include a standardized message in trainings about where to find the recording (for other staff members who may want to watch but were unable to attend the live training) and slides on the website

Engage Project Officers by providing messaging around available resources so they can convey that information to grantees during their scheduled interactions

Increase awareness of and access to TA: The number of TA requests has been relatively low (n=13) in the three months since TA has been made available to grantees. Early results from the Healthy Start grantee satisfaction survey (conducted in May 2015) suggest that the main reason grantees are not requesting TA is that they have no perceived TA needs; when asked why they had not requested TA, 75% of grantees said they had no TA needs, but 11% indicated they were unsure of how to request TA. Specific changes that the Healthy Start EPIC Center will make to ensure that grantees are aware of what TA is available and how to request it include:

Continue advertising the availability of free TA in the newsletter including examples of the type of TA that has been provided to grantees to date

Highlight quotes from evaluations of TA in the newsletter and on the website

Engage Project Officers to encourage grantees to submit a TA request for any questions or concerns they may have

Engage Project Officers to share stories of other grantees who asked for TA, including a description of the TA request and provision

Success Story

The Healthy Start EPIC Center documented that 126 individuals registered and 87 attended (across 37 grantees) the Reproductive Life Planning Ask the Expert event. This was noticeably higher than previous Ask the Expert events, leading the Healthy Start EPIC Center to consider factors contributing to the high participation. For example, the Ask the Expert event was promoted three times (one more than usual) in the newsletter and mentioned in the e-News. The Healthy Start EPIC Center will continue to examine the impact of this enhanced promotion strategy.

Page|31

Increase post-training evaluation response rates. There has been a steady decline in the number of completed evaluations (from 20 for the first informational webinar to 9 for the latest Hear From Your Peer) and the evaluation response rate (from 46% for the first informational webinar to the 23% for the latest Hear From Your Peer event). This trend is concerning since the Healthy Start EPIC Center relies on grantee feedback to understand what is working well and identify areas of improvement. The Healthy Start EPIC Center will make the following changes to address this trend:

Modify the question order of post-training evaluations to put demographic questions (e.g., Healthy Start Project Name and City, State) at the end. There is concern that grantees are turned off by having to answer questions multiple times and stop the survey before getting past this point.

Delete all unnecessary questions including grantee level and zip code, which are available on file

Add a question around identifying areas of improvement to the Ask the Expert evaluation to understand the somewhat lower relevance and quality ratings, compared to Hear From Your Peer events

Examine options in Ready Talk to provide webinar links in an alternative way. Anecdotal evidence suggests that some people may close out of the trainings before they receive the evaluation link

Engage Project Officers to encourage grantees to complete evaluations as it is the best way for grantees to tell the Healthy Start EPIC Center what we can be doing better to serve them

Additionally, the Healthy Start EPIC Center intends to implement the following components of the evaluation:

Evaluation of the Discussion Groups, which began in April 2015. Evaluations of the Discussion Groups will take place after the third group call

Launch of immediate follow-up and 3-month follow-up surveys for recipients of technical assistance

The Healthy Start EPIC Center will also consider the development of a rubric to evaluate the CoIIN’s effectiveness after the objectives of the CoIIN have been finalized.

Page|32

Appendix A: Description of Methods This section further details the methods used to evaluate the Healthy Start EPIC Center.

A. CBA tracking system: The CBA tracking system is made up of two components. First, there is the TA tracking system, which collects and manages TA requests in Salesforce. The Salesforce system was preprogrammed with Healthy Start grantee information including: grantee level, ZIP code, target population, and contact information. Grantees input TA request information into the website, including content and timeline, and then EPIC Center staff responds to the TA request. Second, the CBA tracking system includes registration and attendance information, by grantee, for trainings, including Hear from Your Peers and Ask the Expert events.

B. Website analytics: EPIC Center staff update a monthly tracking sheet of process indicators for the website, including number of website visits, number of clicks on resources, number of downloads, geographic reach of website, and use of website for training registration.

C. Annual grantee CBA worksheet: The primary purpose of the annual grantee CBA worksheet is to identify areas of training and TA needs. A secondary purpose is to track grantee stage of Healthy Start Model implementation. The annual grantee CBA worksheet includes measures of 1) organizational capacity and staff core competencies, and 2) practices in line with the Healthy Start framework. The annual CBA worksheet items were adapted from the OMB-approved NHSPS Program Survey (Attachment D of the planned National Evaluation of Healthy Start).

D. CBA follow-up surveys (immediate and up to 3-months post): Feedback on all capacity building activities are collected through evaluation surveys. Standard evaluations are used for each training activity (Informational Webinar, Hear From You Peer, Ask the Expert), immediately post and 3-months post TA delivery, and at the conclusion of a Discussion Group. Similar measures are used on all evaluations to enable cross-activity comparisons. These forms collect outcomes of interest including the extent to which attendees/recipients of CBA are confident they can complete the intended learning objectives, the degree to which the training was relevant and will help recipients do their job, and intention/actual use of the information delivered. The training evaluation is designed to be administered immediately following the training, while the TA evaluation will be administered up to 3-months post-delivery, depending on the complexity of the TA provided. In addition to these recurring evaluations, JSI implemented a convention evaluation survey; the results were included with the December 2014 Progress Report.

E. Qualitative feedback from grantees. The Healthy Start EPIC Center had planned to conduct formal qualitative interviews with grantees to enhance the findings of the Year 1 Evaluation Report. Due to the relatively short amount of time between the launch of the Healthy Start EPIC Center for grantees and the timing of the evaluation report, the Healthy Start EPIC Center opted instead to incorporate feedback into this report through two channels: 1) a series of questions provided to Project Officers (PO) to collect information during regular PO/Grantee calls and 2) open-ended questions to CBA follow up surveys.

Page|33

Appendix B: Evaluation Surveys This section provides the evaluation surveys for trainings, TA, the convention, and discussion groups.

Webinar Evaluation Form [To be completed by each training participant immediately after training] [Instructions for Participants] Thank you for participating in the Healthy Start training. Please take a moment to provide us with feedback. Your feedback is important to us, and will help us provide high quality training/technical assistance (TA) in the future. All of your responses will be kept confidential and will only be presented in summary form. Thank you in advance for your feedback!

1. Which of the following best describes you? (*required) Healthy Start grantee HRSA/MCHB staff Other HRSA program grantee Other (please specify name of your organization): _________________

If Healthy Start Grantee in Q1 2. What is the name of your Healthy Start Project? ________________________(*required)

3. What level of Healthy Start grantee do you represent?

Level 1 Level 2 Level 3

4. What is your organization’s state? _____ (*required)

5. What is your organization’s ZIP Code: _______ (*required) This question relates to the learning objectives of this training. Please indicate your CONFIDENCE in your ability to do the following:* (select one response per row, required)

Not at all confident

(1)

Not too confident

(2)

Somewhat confident

(3)

Very confident

(4) Not

applicable

6. Learning Objective 1




10. How do you intend to apply what you learned in this training? (check all that apply, required)

Share information with Healthy Start program staff Share information with CAN partners and/or other community stakeholders Make a change in program practices Make a procedure or policy change None of the above Other, specify: _________

Page|34

Please indicate the degree to which you agree with the following.* (select one response per row, required)

Strongly disagree Disagree Agree

Strongly Agree

Not applicable

11. The training was relevant to my needs

12. The training will help me perform better in my job

13. The information was presented in an understandable way

14. The training delivery method was appropriate

15. We are always looking for ways to improve our trainings. If this training did not meet your needs or expectations,

please tell us why: (check all that apply, required) N/A - I found the training to be satisfactory and have no feedback at this time The pace of the training was too quick The pace of the training was too slow The content was too complex The content was too basic There was too much material for the time allotted Pre-training supplemental materials would have been helpful Technical challenges made it hard for me to follow the training (e.g., it was difficult to hear) Other, please specify: _____________________________

16. Please provide comments, proposed training topics, and/or suggestions for improving future trainings (please be specific) (Optional)

17. What is your role in your organization? (required)

Front Line Staff (such as case manager, health educator, outreach worker) Program Management (such as Project Director, administrator) Evaluator Other (Please specify): _______________________

Page|35

Immediate TA Evaluation Form [To be completed by recipients of TA immediately after TA provision]

[Instructions for Participants] You recently received technical assistance (TA) from the Healthy Start (HS) EPIC Center. Please take a moment to provide us with feedback. Your feedback will help us ensure high quality training/technical assistance. All of your responses will be kept confidential and will only be presented in summary form. The HS EPIC Center may use anonymous comments provided in the feedback surveys in promotional materials. Thank you in advance for your feedback!


Not too confident

Somewhat confident

Very confident

Not applicable

1. How confident are you in your ability to apply the information and/or skills presented during the TA provision?* (required)

2. How have you used the information or materials provided during the technical assistance? (check all that apply,

*required

o Develop a plan for implementation o Share information with Healthy Start program staff o Share information with CAN partners and/or other community stakeholders o Make a change in program practices o Make a procedure or policy change o Evaluate changes made as a result of the TA o None of the above o Other, specify: _________

3. Please indicate the degree to which you agree with the following.* (select one response per row, required)


Strongly Agree

Not applicable

The technical assistance was relevant to the needs of my Healthy Start project

The technical assistance will help me and/or my Healthy Start project staff perform better in their roles

4. Please rate the following attributes of the primary technical assistance provider.* (select one response per row,

required)

Poor Fair Good Excellent Not

applicable

Knowledge level

Preparation

Understanding of the context of my Healthy Start project

Responsive to my needs

Communication style

Page|36

5. This question relates to the process of developing and implementing the TA. Please indicate the degree to which you agree that:


Strongly Agree

Not applicable

The HS EPIC Center developed a clear scope of work describing the AIM of the TA

The HS EPIC Center implemented the TA scope of work identified in the plan.

The HS EPIC Center checked in with you about the usefulness of the TA provided.

6. What is the name of your Healthy Start Project? ________________________(*required) 7. If you would like someone from the HS EPIC Center to follow-up on your feedback, please share your name and e-mail

address. (optional) a. Name b. Email address

8. Please provide any comments and/or suggestions for improving future technical assistance (please be specific). (Optional)

Page|37

3-Month TA Evaluation Form [To be completed by recipients of TA 3-months after TA provision]

[Instructions for Participants] A few months ago, you received technical assistance (TA) from the Healthy Start (HS) EPIC Center. Please take a moment to provide us with feedback. Your feedback will help us ensure high quality training/technical assistance. All of your responses will be kept confidential and will only be presented in summary form. The HS EPIC Center may use anonymous comments provided in the feedback surveys in promotional materials. Thank you in advance for your feedback!


Not too confident

Somewhat confident

Very confident

Not applicable

1. How confident are you in your ability to apply the information and/or skills presented during the TA provision?* (required)

2. How have you used the information or materials provided during the technical assistance? (check all that apply,

*required) Develop a plan for implementation Share information with Healthy Start program staff Share information with CAN partners and/or other community stakeholders Make a change in program practices Make a procedure or policy change Evaluate changes made as a result of the TA None of the above Other, specify: _________

3. Please indicate the degree to which you agree with the following.* (select one response per row, required)


Strongly Agree

Not applicable

The technical assistance was relevant to the needs of my Healthy Start project

The technical assistance will help me and/or my Healthy Start project staff perform better in their roles

4. Please tell us two specific ways the TA has impacted your Healthy Start project. You may also include how the TA

impacted the CAN or the community at large. (required)

5. What, if anything, could the HS EPIC Center have done differently to better meet your needs identified in this TA request? (optional)

6. What is the name of your Healthy Start Project? ________________________(*required) 7. If you would like someone from the HS EPIC Center to follow-up on your feedback, please share your name and e-mail

address. (optional) c. Name d. Email address

8. Please provide any comments and/or suggestions for improving future technical assistance (please be specific). (Optional)

Page|38

Conference Evaluation Form [To be completed by each training participant] [Instructions for Participants] Thank you for participating in the Healthy Start Conference, including the Healthy Start grantee conference in Washington, DC this past week. Please take a moment to provide us with feedback. Your feedback is important to us, and will help us provide high quality training/technical assistance (TA) in the future. All of your responses will be kept confidential and will only be presented in summary form. Thank you in advance for your feedback!

1. What is the name of your Healthy Start Project ____________________________ (*required) 2. What is your ZIP Code: _______ (*required)

This question relates to the learning objectives of the general grantee conference (November 19-20), “A New Era of Healthy Start.” Please indicate your CONFIDENCE in your ability to do the following:* (select one response per row, required)


(1)

Not too confident

(2)

Somewhat confident

(3)

Very confident

(4) Not

applicable

3. Describe the components of the Healthy Start EPIC model

4. Ensure your Healthy Start project includes activities in each of the four components of EPIC

5. Ensure your Healthy Start project includes activities in each of the four phases of the perinatal period (4 Ps)

6. Identify next steps for strengthening quality improvement processes in your Healthy Start project

7. Identify next steps for strengthening documentation of your Healthy Start project outcomes


Share information with Healthy Start program staff Share information with CAN partners and/or other community stakeholders Make a change in program practices Make a procedure or policy change Other, specify: _________ None of the above



Strongly Agree

Not applicable




12. The in-person training delivery method was appropriate

13. The breakout discussions on the EPIC topics enhanced my understanding of the expectations for my Healthy Start project

14. There were adequate opportunities for me to connect and share resources with other Healthy Start grantees

Page|39

15. We are always looking for ways to improve our trainings. If this training did not meet your needs or expectations, please tell us why: (check all that apply, required)

I found the training to be satisfactory and have no feedback at this time The pace of the training was too quick The pace of the training was too slow The content was too complex The content was too basic There was too much material for the time allotted Pre-training supplemental materials would have been helpful Technical challenges made it hard for me to follow the training (e.g., it was difficult to hear) Other, please specify: _____________________________

16. Did you participate in the QI institute? o Yes o No (skip to Question 27)

This question relates to the learning objectives of the quality improvement training (November 18-19), “Engaging and Expanding Healthy Start Grantee Leadership and QI Institute.” Please indicate your CONFIDENCE in your ability to do the following:* (select one response per row, required)


(1)

Not too confident

(2)

Somewhat confident

(3)

Very confident

(4) Not

applicable

17. Describe the role of Leadership Grantees as part of the complex Healthy Start system of care and services.

18. Describe a model for systems change and quality improvement as a foundation for further development throughout the CoIIN.

19. Identify strengths of Leadership Grantees that align with Healthy Start Program benchmarks

20. Describe communication guidelines and next steps for learning collaboratives.

21. Identify specific educational and training needs among Leadership Grantees.


Share information with Healthy Start program staff Share information with CAN partners and/or other community stakeholders Make a change in program practices Make a procedure or policy change Other, specify: _________ None of the above



Strongly Agree

Not applicable




26. The in-person training delivery method was

Page|40

appropriate

27. The use of the group case study enhanced my understanding of the QI process

28. The use of formulized reflections enhanced my understanding of the QI process

29. We are always looking for ways to improve our trainings. If this training did not meet your needs or expectations,

please tell us why: (check all that apply, required) I found the training to be satisfactory and have no feedback at this time The pace of the training was too quick The pace of the training was too slow The content was too complex The content was too basic There was too much material for the time allotted Pre-training supplemental materials would have been helpful Technical challenges made it hard for me to follow the training (e.g., it was difficult to hear) Other, please specify: _____________________________

30. Overall, did you find the conference useful?

o Yes o No

31. Please provide comments and/or suggestions for improving future trainings (please be specific) (Optional)

32. What is your role in your organization? (required)


33. What Health Start Grantee Level do you represent? (select one, required)

Level One Level Two Level Three

34. Which of the following best describes your Healthy Start Project setting? (select one, *required)

Community health center Community-based organization Faith-based organization Hospital-based clinic Indian tribe or tribal organization Local health department Other (please specify)_____

35. How many years have you worked in the field of maternal and child health? (select one, *required)

Less than 2 years 2 to 5 years 6 to 10 years 11 to 20 years Greater than 20 years

Page|41

Discussion Group Evaluation Form [To be completed by participants in discussion groups]

[Instructions for Participants] You recently participated in a series of discussion groups. Please take a moment to provide us with feedback. Your feedback will help us ensure high quality training/technical assistance. All of your responses will be kept confidential and will only be presented in summary form. Thank you in advance for your participation!

1. What is the name of your Healthy Start Project? ________________________(*required) 2. What level of Healthy Start grantee do you represent? (*required)

Level 1 Level 2 Level 3

3. What is your organization’s state? _____ (*required) 4. What is your role in your organization? (required)


5. How many discussion group sessions did you attend?

None Less than half (for example, 1 out of 3 sessions) More than half but not all (for example, 2 out of 3 sessions) All

6. If 5 = None or Less than half, We want the discussion groups to meet participants’ needs. Please describe why you

participated in only some (or none) of the discussion group sessions. (Then skip to end) This question relates to objectives of the discussion group. Please indicate the degree to which you agree or disagree that these objectives were met:


Strongly agree

Not applicable

7. The discussion group identified effective strategies related to the topic.

8. The discussion group identified implementation challenges related to the topic.

9. The discussion group identified recommendations and lessons learned related to the topic.

10. The discussion group findings were summarized and distributed to participants.

11. How have you used the information (for example, lessons learned/recommended practices) or materials provided

during the discussion group? (check all that apply, *required) Shared information with Healthy Start program staff (for example, staff training) Shared information with CAN partners and/or other community stakeholders Made a change in program practices Made a procedure or policy change None of the above Other, specify: _________

Page|42

12. Overall, did you find participation in the discussion groups to be useful? Yes No



Strongly Agree

Not applicable

13. The discussion group was relevant to the needs of my Healthy Start project

14. Participating in the discussion group sessions will help me and/or my Healthy Start project staff perform better in their roles

15. The final product/resources developed during the course of the discussion group will be useful to the Healthy Start community

16. We are always looking for ways to improve our support. If this discussion group did not meet your needs or

expectations, please tell us why: (check all that apply, required) N/A - I found the discussion group to be satisfactory and have no feedback at this time There were too many discussion sessions There were not enough discussion sessions The discussion sessions were too long The discussion sessions were too short The co-facilitators did not keep the discussions focused The co-facilitators constrained the discussion too much The content of the discussions was too complex The content of the discussions was too basic Additional materials would have been helpful Technical challenges made it hard for me to participate (e.g., it was difficult to hear) Other, please specify: _____________________________

17. Please provide any comments and/or suggestions for improving future discussion groups (please be specific). (Optional)

Documents

Healthy Start EPIC Center Year 1 Evaluation and …...To answer the evaluation questions, the Healthy Start EPIC Center used a combination of process and outcome monitoring methods