Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
FFY 15
Iowa’s SSIP Phase III Year 1
Progress Report – Part C
Submitted April 3, 2017
Iowa Department of Education
[Intentionally left blank.]
i
Iowa Part C State Systemic Improvement Plan Phase III Year 1
Table of Contents
INTRODUCTION
Acronyms .................................................................................................................................................... 1
Structure of Early ACCESS, Iowa's IDEA Part C Early Intervention System ................................................. 2
Stakeholder Involvement ....................................................................................................................................... 2
SECTION A: Summary of Phase III .......................................................................................................................... 5
SECTION B: Progress in Implementing the SSIP .................................................................................................... 8
Table 1 Progress on Evidence-Based Practices Activity 1 of 2 from Logic Model .......................... 9
Table 2 Progress on Evidence-Based Practices Activity 2 of 2 from Logic Model ........................ 13
Table 3 Progress on Implementation Strategies Activity 1 of 4 from Logic Model ...................... 14
Table 4 Progress on Implementation Strategies Activity 2 of 4 from Logic Model ...................... 15
Table 5 Progress on Implementation Strategies Activity 3 of 4 from Logic Model ...................... 16
Table 6 Progress on Implementation Strategies Activity 4 of 4 from Logic Model ...................... 17
Table 7 Progress on High Quality System Activity 1 of 3 from Logic Model ................................. 18
Table 8 Progress on High Quality System Activity 2 of 3 from Logic Model ................................. 18
Table 9 Progress on High Quality System Activity 3 of 3 from Logic Model ................................. 19
Figure 1 Cohort 4 Training Cycle ................................................................................................... 10
Figure 2 Cohort 4 Example of Expert Coaching Timeline for Quads of Service Providers............ 10
Figure 3 Example of Cohort 4 Expert Coaching Session Cycle Responsibilities of
Quad Members ................................................................................................................. 11
Figure 4 Cohort 4 Roles and Responsibilities of the Service Provider Participants ...................... 11
Figure 5 Community of practice Schedule of Webinars to Support Shift to Evidence-Based
Practices ............................................................................................................................ 12
Figure 6 Dedicated Website for Iowa Distance Mentoring Model for Early ACCESS ................... 12
Figure 7 Core Principles of Adult Learning.................................................................................... 13
Figure 8 List of Implementation Team Recorded Webinars from IA DMM Website ................... 15
Figure 9 Example of Adjusting Written Agreements Based on Feedback
and Changing Needs ......................................................................................................... 16
Figure 10 Iowa Family Support Network Early ACCESS Webpage with Sections Identified for
Different Stakeholders ...................................................................................................... 17
SECTION C: Data on Implementation and Outcomes ......................................................................................... 20
Table 10 Selected Elements from Direct Service Provider Level
Evaluation Plan L2Q5, L2Q5.1 ........................................................................................... 21
Table 11 IA DMM Cohort 4 First Session, Day 1 ........................................................................... 24
Table 12 IA DMM Cohort 4 First Session, Day 2 ........................................................................... 24
Table 13 IA DMM Cohort 4 Second Session, Day 1 ...................................................................... 24
IOWA Part C SSIP Phase III Year 1
ii
Table 14 IA DMM Cohort 4 Second Session, Day 2 ...................................................................... 24
Table 15 Selected Elements from Direct Service Provider Level
Evaluation Plan L2Q6a-d ................................................................................................... 25
Table 16 Selected Elements from Direct Service Provider Level
Evaluation Plan L2Q6.1, L2Q6.2 ........................................................................................ 27
Table 17 Selected Elements from Direct Service Provider Level Evaluation Plan L2Q6.3 ............ 28
Table 18 Selected Elements from Direct Service Provider Level Evaluation Plan L2Q7 ............... 29
Table 19 ROLE Areas, Subareas, and Number of Items ................................................................ 29
Table 20 Individual AEA/LEA Mean Scores for ROLE Culture and Leadership.............................. 31
Table 21 Individual AEA/LEA Mean Scores for ROLE Systems & Structures, Communication,
Teams and Evaluation ....................................................................................................... 32
Table 22 Selected Elements from Direct Service Provider Level Evaluation Plan L2Q8 ............... 33
Table 23 Selected Elements from Family Participant Level Evaluation Plan L1Q1....................... 36
Table 24 Early Intervention Parenting Self-Efficacy Scale for Cohorts 1 through 3, Fall 2015 ..... 37
Table 25 Selected Elements from Family Participant Level Evaluation Plan L1Q2a-b, L1Q3 ....... 38
Table 26 Selected Elements from Family Participant Level Evaluation Plan L1Q4....................... 38
Table 27 Selected Elements from Implementing Agency Level Evaluation Plan L3Q9-Q11 ........ 41
Table 28 Levels of Collaboration Definitions and Relationship Characteristics ........................... 42
Table 29 Selected Elements from Implementing Agency Level
Evaluation Plan L3Q12, L3Q12.1 ....................................................................................... 48
Table 30 Regional Implementation Team Report Questions & Examples of Data Collected ....... 49
Table 31 Selected Elements from Implementing Agency Level
Evaluation Plan L3Q13, L3Q14 .......................................................................................... 50
Table 32 Selected Elements from State Level Evaluation Plan L4Q15, L4Q16 ............................. 51
Table 33 Selected Elements from State Level Evaluation Plan L4Q17, L4Q17.1 .......................... 52
Table 34 Selected Elements from State Level Evaluation Plan L4Q18a-b .................................... 53
Table 35 Selected Elements from State Level Evaluation Plan L4Q19 ......................................... 55
Table 36 Selected Elements from State Level Evaluation Plan L4Q20 ......................................... 55
Table 37 Selected Elements from State Level Evaluation Plan L4Q21 ......................................... 56
Figure 11 Learning Based Assessment of Training—IA DMM Cohort 4 First Session Day 1 ........ 22
Figure 12 Learning Based Assessment of Training—IA DMM Cohort 4 First Session Day 2 ........ 22
Figure 13 IA DMM Cohort 4 Second Session
(6 months after 1st Face-to-Face Training), Day 1 ............................................................ 23
Figure 14 IA DMM Cohort 4 Second Session
(6 months after 1st Face-to-Face Training), Day 2 ............................................................ 23
Figure 15 Cohort 1-4 Observation Results, FGRBI SS-OO-PP-RR Key Indicators, Coaching
Strategies, Routines Used ....................................................................................................... 26
Figure 16 Number of Participants by Cohort and Percentage of Disciplines
Trained Cohort 1-4 ............................................................................................................ 27
Figure 17 Examples of Interview Quotes from Cohorts 1 and 2 ................................................... 34
Figure 18 Examples of Interview Quotes from Cohorts 1 and 2 ................................................... 35
IOWA Part C SSIP Phase III Year 1
iii
Figure 19 Caregiver Key Indicators Form ...................................................................................... 40
Figure 20 Caregiver Key Indicators Observations Cohort 4 Video 1 and Video 2 ........................ 40
Figure 21 AEA "X" Network Maps for 2014 and 2015 .................................................................. 43
Figure 22 AEA "X" 2014 and 2015 Ties Valued as Coalition or Higher ......................................... 43
Figure 23 AEA "Y" Network Maps for 2014 and 2015 .................................................................. 44
Figure 24 AEA "Y" 2014 and 2015 Ties Valued as Coalition or Higher ......................................... 44
Figure 25 AEA "Z" Network Maps for 2014 and 2015 .................................................................. 45
Figure 26 AEA "Z" 2014 and 2015 Ties Valued as Coalition or Higher.......................................... 45
Figure 27 All AEAs Network Maps, 2014 ...................................................................................... 47
Figure 28 All AEAs Network Maps, 2015 ...................................................................................... 47
Figure 29 ECTA System Framework Self-Assessment Governance Results .................................. 54
Figure 30 ECTA System Framework Self-Assessment Personnel/Workforce Results .................. 54
SECTION D: Data Quality Issues ........................................................................................................................... 57
SECTION E: Progress Toward Achieving Intended Improvements ..................................................................... 58
Table 38 Evidence-Based Practices Short, Intermediate, Long-term Outcomes.......................... 58
Table 39 Implementation Strategies Short, Intermediate, Long-term Outcomes ....................... 61
Table 40. High Quality Infrastructure Short, Intermediate, Long-term Outcomes
Governance ....................................................................................................................... 65
Table 41 High Quality Infrastructure Short, Intermediate, Long-term Outcomes
Personnel/Workforce ....................................................................................................... 66
Table 42 Licensed Area Education Agency and District Staff that Could Retire in Next Five Years
(Based on Rule of 88 or Retirement at Age 65. ................................................................ 67
Table 43 High Quality Infrastructure Short, Intermediate, Long-term Outcomes Accountability &
Quality, Quality Standards, Data System, Finance ........................................................... 68
SECTION F: Plans for Next Year ............................................................................................................................ 68
References ............................................................................................................................................................ 69
Appendix A
Logic Models: Instructional Practices, Implementation Strategies, High Quality Part C System ............. 70
Appendix B
System Level Evaluation Plan for Early ACCESS ........................................................................................ 73
1
Acronyms
AEA ..................... Area Education Agency
AI ........................ Active Implementation
CPT ..................... Core Planning Team for personnel/workforce development (interagency team of consultants for birth to age 5 services)
DaSy ................... Center for IDEA Early Childhood Data Systems
EA ....................... Early ACCESS
EAGL ................... Early ACCESS Leadership Group (Liaisons from all Area Education Agencies, Departments of Education, Public Health, and Human Services, Child Health Specialty Clinics, Special Education Directors, Iowa School for the Deaf, Iowa Educational Services for the Blind and Visually Impaired, Autism)
EC CSPD .............. Early Childhood Comprehensive System for Personnel Development
ECSE .................... Early Childhood Special Education
ECTA ................... Early Childhood Technical Assistance Center
FGRBI .................. Family-Guided Routines-Based Intervention
FSU ..................... Florida State University
GV ....................... Governance (component of the ECTA System Framework)
ICEA .................... Iowa Council for Early ACCESS (Iowa's interagency coordinating council)
IA DMM .............. Iowa Distance Mentoring Model of Professional Development for Early ACCESS
MOA ................... Memorandum of Agreement between Departments of Education, Public Health, and Human Services, and Child Health Specialty Clinics, University of Iowa
OT ....................... Occupational Therapist
PD ....................... Professional Development
PDSA ................... Plan Do Study Act
PN ....................... Personnel/Workforce (component of the ECTA System Framework)
PSP ...................... Primary Service Provider
PT........................ Physical Therapist
RIT ...................... Regional Implementation Teams (at each AEA and Des Moines Public Schools)
ROLE ................... Readiness of Organizational Learning and Evaluation
SC ........................ Service Coordinator
SiMR ................... State-identified Measurable Result
SIT ....................... State Implementation Team (early childhood interagency team)
SLP ...................... Speech Language Pathologist
SS-OO-PP-RR ...... Setting the Stage, Observation and Opportunities to practice, Problem solving and Planning, Reflection and Review
SSIP ..................... State Systemic Improvement Plan
SWT .................... State Work Team (Departments of Education, Public Health, and Human Services, and Child Health Specialty Clinics)
IOWA Part C SSIP Phase III Year 1
2
Structure of Early ACCESS, Iowa's IDEA Part C Early Intervention System
Iowa Administrative Rules for Early ACCESS establish that four signatory agencies commit to implementation
and evaluation of the IDEA Part C early intervention system. “Signatory agency” means the:
Iowa Department of Education (lead agency),
Iowa Department of Public Health,
Iowa Department of Human Services, and
Child Health Specialty Clinics of the University of Iowa.
There are 9 regional grantees, called Area Education Agencies (AEA), which guarantee services are delivered to
all parts of the state.
STAKEHOLDER INVOLVEMENT IN EARLY ACCESS Who's Who in Early ACCESS: Stakeholder Groups, SSIP Phase I, II and III The following groups routinely engage in Early ACCESS activities. Long-standing relationships between and within these groups allows for smooth transitions when membership changes due to retirements, new hires, changing roles or jobs, or expiring terms for membership. Each group has a role in supporting the successful implementation of Early ACCESS in Iowa. This includes engaging with implementation strategies, continuous improvement and evaluation. In addition to permanent long-standing groups, task teams form for the purpose of dealing with specific, time-bound activities. During the current reporting period, 13 different task teams (e.g. service coordinator competency training revision, eligibility criteria, Every Student Succeeds Act, Autism Navigator Implementation Plan) formed and included diverse groups of stakeholders that volunteered to work on different aspects of the early intervention system. There is always an electronic format for joining so that anyone from across Iowa can participate. It is not unusual that a service provider "attend" a task team meeting from a car between home visits or that a family member be at home with the sounds of children and pets in the background.
IOWA Part C SSIP Phase III Year 1
3
There is a prevailing belief that having the voice of people who are impacted most by the work must be included in helping to define problems, develop and carry out solutions, and define success. This belief translate into action through the task team and stakeholder group work. The success of the Early ACCESS system depends on the knowledge and skills the following permanent, stakeholder groups: Early ACCESS State Work Team (SWT) (6 members)
Staff from Iowa Department of Education, Iowa Department of Public Health, Department of Human Services,
and Child Health Specialty Clinic.
Two full-day meetings per month; two-day summer retreat; contact between meetings to get work done. This
is the core team that is responsible for the day-to-day operations of Early ACCESS and supporting the
work of the SSIP.
Early ACCESS Leadership Group (EAGL) (30-34 members)
Area Education Agencies (liaisons/providers/service coordinators, special education director as liaison to all
directors, mid-level management)
Des Moines Public Schools Iowa School for the Deaf
Department of Education Department of Public Health
Department of Human Services Child Health Specialty Clinics
Autism Consultant Iowa Educational Services for the Blind and Visually Impaired
Iowa Council for Early ACCESS (ICEA) (18-25 members)
Parent Leadership and Members
Multi-Agency Stakeholders
Department of Education Department of Public Health
Department of Human Services Child Health Specialty Clinics
Signatory Agency Leadership Team (4 members)
Administrators from Iowa Department of Education, Iowa Department of Public Health, Department of Human
Services, and Child Health Specialty Clinic who are bound by a written Memorandum of Agreement (MOA) to
ensure the Early ACCESS system functions successfully. MOA is executed every 5 years.
Regional Implementation Teams (RIT) (10 teams, 5-13 members per team, 80 individuals statewide)
Personnel Preparation/Higher Education
AEA Administrators, Supervisors, Coordinators, Liaisons
Service Providers (SLP, ECSE Teachers, OT, PT, Deaf/Hard of Hearing Teachers, Social Workers, Autism
Consultant)
Media Specialists
Service Coordinators
Out-of-State PD Trainer
IOWA Part C SSIP Phase III Year 1
4
State Implementation Team (SIT) (10-15 members)
State Work Team members (Staff from Iowa Department of Education, Iowa Department of Public Health,
Department of Human Services, and Child Health Specialty Clinic)
Area Education Agencies (Special Education Directors, EA Liaisons)
Early Childhood Iowa (Early Head Start, Head Start; Child Care; Maternal Infant Early Childhood Home
Visitation; Iowa Association for the Education of Young Children)
Child Care Resource and Referral
Personnel Preparation/Higher Education
Iowa Distance Mentoring Model for Early ACCESS Project Administration Team (5-7 members)
Florida State University, College of Communication and Information, School of Communication Science and
Disorders
Iowa Department of Education
Evaluation Team (5-7 members)
Department of Education
Florida State University IA DMM project staff
External evaluator
IFSP/IEP Data System Core Team (3-5 members)
Department of Education
Area Education Agency
IFSP Data System Team (3-5 members)
Department of Education
Department of Public Health
Private contractors
Early Childhood Comprehensive System for Personnel Development (EC CSPD) (6-8 members)
Department of Education, Early ACCESS (IDEA Part C)
Department of Education, ECSE, (IDEA Part B 619)
Department of Human Services
Iowa Association for the Education of Young Children
National Center for Systemic Improvement Cross State Learning Collaborative (Family Outcomes)
Multiple, various state and US Territory membership
Division of Early Childhood, Early Intervention Special Interest Group
Multiple, various state, organization and individual membership
Infant Toddler Coordinator Association
IDEA Part C coordinators from states and US Territories
IOWA Part C SSIP Phase III Year 1
5
SECTION A: Summary of Phase III
Theory of Action, including the State-identified Measurable Result (SiMR)
The Early ACCESS Theory of Action was originally developed in 2012 and updated in 2014 when submitted to OSEP. Iowa is focusing on increasing the percentage of families that report early intervention has helped them help their child develop and learn (the Part C SiMR) and corresponding information in the Theory of Action is highlighted by the light blue box.
Improvement Strategies, Logic Models including the SiMR
Iowa's Theory of Action has three action strands: practice, professional development, and infrastructure that lead to three coherent improvement strategies.
Improvement Strategy 1: New Instructional Practices
Shifting instructional practice away from teaching the child to using family-guided routines-based interventions(FGRBI) where the focus is on coaching the caregiver to support the child.
IOWA Part C SSIP Phase III Year 1
6
Improvement Strategy 2: New Implementation Strategies
Incorporate implementation science frameworks in order to develop the capacity to make effective, statewide, and sustained use of evidence‐based practices.
Improvement Strategy 3: New High Quality System
Use the ECTA System Framework self-assessment processes in order to develop a high-quality Early ACCESS system that encourages, supports, and requires implementation of evidence-based practices.
A logic model was created for each of the 3 improvement strategies. The logic models (see Appendix A) were
used to help build the Systems Level Evaluation Plan for Early ACCESS (see Appendix B).
Specific Evidence-Based Practices Implemented To Date
Family-Guided Routines-Based Interventions (FGRBI), caregiver coaching and Active Implementation (AI)
Frameworks to sustain evidence-based practices are being implemented as part of the Iowa Distance
Mentoring Model (IA DMM) of Professional Development for Early ACCESS. Through coaching caregivers in
family-guided routines-based interventions, there will be an increase in the competence and confidence of
caregivers which will lead to an increase in the percentage of families reporting that early intervention helped
them to help their children develop and learn (Iowa's SiMR). Using AI Frameworks will ensure implementing
with high fidelity, scaling up and sustainability. As of the current reporting period, 4 cohorts of early
intervention service providers have been trained in FGRBI and caregiver coaching.
IOWA Part C SSIP Phase III Year 1
7
Brief Overview of the Year’s Evaluation Activities, Measures, and Outcomes
The majority of evaluation measures were completed as scheduled. Family level pre-interview surveys and
interviews did not occur as scheduled. Solicitation for participants did take place during the current reporting
period and took longer than expected. Regional and state level measures changed from Active
Implementation (AI) Hub quizzes to more contextual measures and data was collected. The Levels of
Collaboration Scale for state level evaluation was dropped. This is a very time intensive and expensive survey
to analyze. Results from the scale are used to support 3 short-term outcomes that have other, better
measures already being used to show progress. Evidence from multiple evaluation measures shows progress
is being made on short-term outcomes.
Highlights of Changes to Implementation and Improvement Strategies
Improvement strategies remain the same as planned and submitted in the SSIP Phase II. Implementation of IA
DMM involves a continuous improvement process driven by stakeholder input and data. Shifts in the makeup
of cohort groups happen based on feedback from the proceeding cohort.
Original plans included training pairs of service providers from each agency per cohort. Stakeholders asked for
more providers to be included per cohort. Training was redesigned to accommodate groups of 4 providers or
quads for cohort 4. In June 2016 at a Joint Implementation Team Meeting (regional and state teams), in
preparation for cohort 5, data related to cohort 4 structure changes was reviewed. Based on the review,
cohort 5 structure would be changed to groups of 3 service providers or trios. In addition, cohort 5 is to
include a veteran IA DMM trained provider to begin training as an internal coach to build agency's capacity to
have expert coaches that will focus on fidelity.
One of the biggest changes was giving up, not just the toy bag, but my ability to bring the latest most greatest interesting and exciting toys into the home to entice children to get involved and to give that up and realize that the toys I need are in the home and that those are the things that will be there every day for children to use to learn. So that's been a challenge but it's been a positive change. Also through that process learning that my role really is to empower parents to teach their children every day during their routines and activities. What I do in the few minutes I am there really doesn't make as much difference as what the parents are doing everyday with their child.
IOWA Part C SSIP Phase III Year 1
8
SECTION B: Progress in Implementing the SSIP
The focus of Section B is on progress in implementing activities in each of the 3 improvement strategies
chosen to improve outcomes for children and families served in Early ACCESS:
Improvement Strategy 1: New Instructional Practices
Shifting instructional practice away from teaching the child to using family-guided routines-based
interventions (FGRBI) where the focus is on coaching the caregiver to support the child.
Improvement Strategy 2: New Implementation Strategies
Incorporate implementation science frameworks in order to develop the capacity to make effective,
statewide, and sustained use of evidence‐based practices.
Improvement Strategy 3: New High Quality System
Use the ECTA System Framework self-assessment processes in order to develop a high-quality Early
ACCESS system that encourages, supports, and requires implementation of evidence-based practices.
Tables are used to share progress in each of the activities identified in the logic models (Appendix A) for Early
ACCESS (submitted in Phase II) and include:
activities taken directly from the logic model,
evidence or outputs of the activity,
if implemented as intended including description of changes in an activity based on data collected, and
successes and challenges to implementation.
NOTE: Data generated from evidence listed for an activity are shared in Section C.
IOWA Part C SSIP Phase III Year 1
9
Progress on Instructional Practices Table 1. Progress on Evidence-Based Practices Activity 1 of 2 from Logic Model
Act
ivit
y
Design and implement professional development plan to build and expand capacity of providers in coaching caregivers and family-guided routines-based intervention (FGRBI)
Evid
en
ce
Timelines and training cycles (link?)
Training attendance.
Pre-then-Post retrospective surveys collected at
end of each face-to-face training
Journey reflection documents
IA DMM website visits and materials used
# of Weekly emails read and used consistently
# of online modules
Facebook usage
# videos submitted
External coaching sessions completed
# of self assessments completed
# of webinars and attendance
Online Community of Practice usage
Survey results for importance of training
features
Participant feedback video recordings
Observational data on routines used, SSOOPPRR
home visit components, key indicators to FGRBI
including coaching strategies.
Survey results for each cohort on next steps,
what to keep and what to adjust.
Imp
lem
en
ted
as
inte
nd
ed?
[ye
s/n
o]
Tim
elin
e
Yes; As of 6/30/16, completed 4 cohorts and prepared for cohort 5 as scheduled. However, structures of the cohorts changed over time. Cohorts 1-3 (trained in FFY 2013-2014) had pairs of providers from each region. This changed to having quads or 4 providers or Quads from each region. Changes were made based on feedback from stakeholders asking for more people to be trained per cohort. After that change occurred, cohort 4 Quad data was reviewed to inform cohort 5 plans and included: (1) participation and completion rates of Quads; (2) journey reflections; (3) Quad survey results; (4) frequency counts of participant contacts; (5) Google analytics; (6) FSU feedback and (7) input from regional and state implementation teams at Joint Implementation Team Meeting. Based on all these pieces of information, the makeup of the training groups changed to trios. In addition, cohort 5 is to included a veteran IA DMM trained provider to begin training as an internal coach to build agency's capacity to have expert coaches that will focus on fidelity. Implementation of this change takes place during the next reporting period and will be address in SSIP Phase III Year 2 report.
Succ
ess
es
&
Ch
alle
nge
s
Contracting with experts in FGRBI and caregiver coaching who are both flexible and organized.
Well planned timelines and training cycles.
Written roles and responsibilities.
Multiple learning opportunities designed to support change (e.g. face-to-face training, webinars, distance expert coaching sessions, community of practice, online modules).
Using core principles of adult learning in the training while teaching integration of adult learning principles in family centered practices.
Having a core team of people responsible for the overall plan is both a success and challenge so that the work is bigger than any one person should they leave.
Making professional development part of service providers' everyday work is a challenge that is strong even into year 4 of the work.
Less technology challenges over time.
IOWA Part C SSIP Phase III Year 1
10
Figure 1-7 are illustrations of the successful elements of the Iowa Distance Mentoring Model of Professional Development for Early ACCESS. Figure 1. Cohort 4 Training Cycle Timeline
Figure 2. Cohort 4 Example of Expert Coaching Timeline for Quads of Service Providers
IA DMM Cohort 4 Training Cycle Timeline
IOWA Part C SSIP Phase III Year 1
11
Figure 3. Example of Cohort 4 Expert Coaching Session Cycle Responsibilities of Quad Members
Figure 4. Cohort 4 Roles and Responsibilities of the Service Provider Participants
IA DMM Cohort 4 Session Cycle –Post Follow Up Workshop Example
Jane – March 8 Suzi – April 12
Carol – May 10 Amy – June 7
IOWA Part C SSIP Phase III Year 1
12
Figure 5. Community of Practice Schedule of Webinars to Support Shift to Evidence-Based Practices Figure 6. Dedicated Website for Iowa Distance Mentoring Model for Early ACCESS (http://dmm.cci.fsu.edu/IADMM/index.html)
IOWA Part C SSIP Phase III Year 1
13
Figure 7. Core Principles of Adult Learning
Table 2. Progress on Evidence-Based Practices Activity 2 of 2 from Logic Model
Act
ivit
y
Develop and implement plan for tracking other related professional development (PD) that occurs within individual Area Education Agencies and signatory agencies that supports coaching caregivers and FGRBI
Evid
en
ce List of PD funded and offered through regional
Area Education Agencies
Notes from statewide stakeholder meetings (EAGL)
Lists of data collection tools and methods for measuring FGRBI and coaching caregivers.
Imp
lem
ente
d a
s
inte
nd
ed
? [y
es/n
o]
Tim
elin
e
Yes. As of 6/30/16 have met regularly with the Early ACCESS Leadership Group where information is gathered on the following: writing functional, participatory child and family outcomes; routines-based interviewing; Primary Service Provider (PSP) teaming practices; and, data collection tools and methods used at the agency level to track progress in FGRBI and coaching caregivers.
Succ
ess
es
&
Ch
alle
nge
s
Using already existing stakeholder groups and meetings.
Using Google Docs to collect and share information.
Discussing in large groups supports cross-agency learning.
Nuances from different experts in coaching strategies can confuse some service providers.
Each agency is independently governed and each is at different stages of practice shift which is
challenging.
IOWA Part C SSIP Phase III Year 1
14
Progress on Implementation Strategies
Table 3. Progress on Implementation Strategies Activity 1 of 4 from Logic Model
Act
ivit
y
Establish collaborative planning & implementation teams & networks
Evid
en
ce Regional Implementation Teams (RIT) rosters
RIT reports and updates
State Implementation Team (SIT) rosters
SIT meeting agendas and notes
Imp
lem
ente
d
as in
ten
ded
? [y
es/n
o]
Tim
elin
e
Yes, teams created as scheduled.
Succ
ess
es
&
Ch
alle
nge
s
In the beginning (2013) teams were established quickly in order to meet timelines.
Now, membership based on roles, responsibilities and needs.
Some regional teams include early childhood partners outside of the Area Education Agency system.
SIT serves more as an advisory and high-level problem-solving and reflection team. The State Work
Team functions as what Active Implementation Frameworks describe as a state implementation team,
meeting often and doing the daily work of supporting the system change.
State Implementation Team meeting, small group discussion
IOWA Part C SSIP Phase III Year 1
15
Table 4. Progress on Implementation Strategies Activity 2 of 4 from Logic Model A
ctiv
ity
Design and implement plan to build capacity of state and regional teams in Active Implementation Frameworks
Evid
en
ce
Evidence-Based Practices and Professional Development (PD) Implementation Plan
Written PD vision
Written agreements between administrators, providers, and training organization
Meeting agenda items, notes, and completed modules related to active implementation frameworks by State Work Team, EA Leadership Group, Iowa Council for EA, SIT, RIT
Recorded webinars
Imp
lem
ente
d
as in
ten
ded
? [y
es/n
o]
Tim
elin
e
Yes, implemented as planned and is ongoing.
Succ
ess
es
&
Ch
alle
nge
s
Webinars specifically for implementation teams.
Written action plans for regional teams.
Use Google Docs to routinely update progress on RIT activities.
Face-to-face EAGL meetings used as RIT events where team members across the state who are not part of EAGL join electronically (via Zoom) to participate while other EAGL members who typically are uninvolved with RIT work participate as well.
Annual joint face-to-face meeting between RIT and SIT facilitated by Florida State University where data are shared and team plans are reviewed.
State Work Team functioning as AI Hub definition of state implementation team.
Even though implementation team work is better, it has been a struggle to maintain well functioning teams. Teams continue to need regular support which is addressed routinely at meetings and through webinars.
Figure 8. List of Implementation Team Recorded Webinars from IA DMM Website
IOWA Part C SSIP Phase III Year 1
16
Table 5. Progress on Implementation Strategies Activity 3 of 4 from Logic Model A
ctiv
ity
Assess readiness in the Area Education Agencies
Needs assessment
Assess current EI policies, practices, etc.
Assess and acquire resources needed to do the work
Evid
en
ce
Needs assessment results from practices survey
Needs assessment results from technology
survey
Administrator agreements include commitment
of resources to do the work
Notes from review of current procedures,
policies, practices
Feedback surveys showing PD is meeting the
needs of the participants
Imp
lem
ente
d
as in
ten
ded
? [y
es/n
o]
Tim
elin
e
Yes, completed as scheduled.
Succ
ess
es
&
Ch
alle
nge
s
Adjusted agreements based on feedback from previous year and needs of the upcoming year (see
Figure 9).
June Joint Implementation Team meetings, share data and what to keep and what to change for next
cohort. Each cohort contributing to the continuous improvement of the PD
Getting timely consents from families and providers has always been a challenge. Providers get the
needed consents after several reminders from Florida State University.
Figure 9. Example of Adjusting Written Agreements Based on Feedback and Changing Needs
IOWA Part C SSIP Phase III Year 1
17
Table 6. Progress on Implementation Strategies Activity 4 of 4 from Logic Model A
ctiv
ity
Maintain communication with stakeholders, PD participants, EA leadership team, and council
Evid
en
ce
Written communication plans for internal and
external communications
List of individualized one-day trainings at each
AEA including FGRBI 101 complete by Florida
State University
Written documents explaining FGRBI for various
stakeholders
Video recordings posted on public website
Lists of presentations and PowerPoints used
Task team rosters of volunteers helping with
various aspects of the work
Imp
lem
ente
d
as in
ten
ded
? [y
es/n
o]
Tim
elin
e
Yes, implemented as planned and is ongoing.
Succ
ess
es
&
Ch
alle
nge
s
History of long-standing, regular meetings of statewide stakeholders.
Use of Iowa Family Support Network website for public access to information
(http://www.iafamilysupportnetwork.org/) (see Figure 10).
Good participation in task teams when work needs to be done.
Unsure of level of communication and engagement with families so a task team has been created to
select a statewide self-assessment and develop plan around engaging families in all levels of the early
intervention system (next reporting year activity).
Figure 10. Iowa Family Support Network Early ACCESS Webpage with Sections Identified for Different Stakeholders
IOWA Part C SSIP Phase III Year 1
18
Progress on High Quality System Table 7. Progress on High Quality System Activity 1 of 3 from Logic Model
Act
ivit
y
Complete ECTA self-assessment for Governance (GV) component; engage stakeholders to develop and implement GV improvement plan
Evid
en
ce
Completed self-assessment with priorities identified and work plans that address priorities
Updated self-assessment document
Early Childhood Comprehensive System of Personnel Development plan (EC CSPD)
Memorandum of Agreement (MOA) Action Plan between Iowa Department of Education, Iowa Department of Public Health, Department of Human Services and Child Health Specialty Clinics of the University of Iowa
Imp
lem
ente
d
as in
ten
ded
? [y
es/n
o]
Tim
elin
e
Yes, implemented as planned.
Succ
ess
es
&
Ch
alle
nge
s
Self-assessment reviewed at Early ACCESS state work team meetings and updated annually using the Early Childhood Technical Assistance (ECTA) Center resources.
Multi-state agency shared responsibility to complete MOA Action Plan.
Revisions to Endorsement 100 (Birth-3rd Grade general & special education) approved.
Challenge to balance infrastructure development and shift to evidence-based practices which must happen simultaneously.
Table 8. Progress on High Quality System Activity 2 of 3 from Logic Model
Act
ivit
y
Complete ECTA self-assessment for Personnel/Workforce (PN) component; engage stakeholders to develop and implement PN improvement plan
Evid
en
ce Completed self-assessment with priorities
identified and work plans that address priorities
Updated self-assessment document Early Childhood Comprehensive System of
Personnel Development plan (EC CSPD)
Early Childhood Comprehensive System of Personnel Development team roster (includes IDEA Part C, Part B 619, public and private early childhood agencies)
Imp
lem
ente
d
as in
ten
ded
? [y
es/n
o]
Tim
elin
e
Yes, implemented as scheduled and ongoing.
Succ
ess
es
&
Ch
alle
nge
s
EC CSPD is for statewide cross-sector early childhood system not just Early ACCESS (IDEA Part C) and Early Childhood Special Education (IDEA Part B 619).
Self-assessment reviewed specifically for items directly related to Early ACCESS (IDEA Part C) during state work team meetings and updated as needed using the Early Childhood Technical Assistance (ECTA) Center resources.
Small groups used to address different parts of the EC CSPD with cross-agency core team responsible for overall plan.
Challenge to balance infrastructure development and shift to evidence-based practices which must happen simultaneously.
IOWA Part C SSIP Phase III Year 1
19
Table 9. Progress on High Quality System Activity 3 of 3 from Logic Model A
ctiv
ity
Review and prioritize remaining ECTA System Framework component for future completion:
Accountability & Quality (AC)
Quality Standards (QS)
Data System (DS)
Finance (FN)
Completed self-assessment on data system (DS) component
Evid
en
ce
Imp
lem
ente
d
as in
ten
ded
? [y
es/n
o]
Tim
elin
e
N/A not scheduled to occur during this reporting period.
Succ
ess
es
&
Ch
alle
nge
s
We are trying to focus on routines in the visits and how to use these skills in all the daily routines that they are doing and different places that they go.
IOWA Part C SSIP Phase III Year 1
20
SECTION C: Data on Implementation and Outcomes
This section will focus on data from implemented activities and connect them to the short, intermediate and
long-term outcomes. First is information explaining the relationship of the Theory of Action, improvement
strategies, logic models and the evaluation plan. It is important to understand that the evaluation questions,
measures and methods of data collection are traced back to the outcomes of the improvement activities. The
remainder of this section is organized around the evaluation questions and measures from the Systems Level
Evaluation Plan for Early ACCESS (submitted in Phase II).
Based on the following evaluation of improvement practices, improvement strategies and infrastructure
development, there are no modification in the outcomes, SiMR or SSIP overall.
Progress on Systems Level Evaluation Plan for Early ACCESS
Connecting the Dots: Theory of Action, Improvement Strategies, Logic Models and Evaluation
The Theory of Action identifies 3 parts of the early intervention system (practice, professional development
and infrastructure) where actions are required that will move the Early ACCESS system to its desired future.
From each of these parts or "strands of action" Iowa identified 3 improvement strategies to use in order to
ensure eligible children and families have improved outcomes. A logic model was created for each
improvement strategy and provides a visual framework for describing the relationship between resources or
inputs, activities, and results or outcomes. Each short, intermediate and long-term outcome on the logic
models is measured using the tools and methods described in the evaluation plan. There are direct
connections between the Theory of Action, improvement strategies, logic models and the items on the
evaluation plan.
How Are We Doing on Evaluating Activities to Achieve Outcomes?
Iowa's improvement activities are designed to impact multiple levels of the Early ACCESS system: Direct
Service Provider Level ; Family Level ; Implementing Agency Level , and State Level . In order to gauge if
activities are having the desired impacts, the evaluation plan is designed to measure all four of these levels.
IOWA Part C SSIP Phase III Year 1
21
Tables are color coded by level and include:
evaluation questions,
measures,
data collection methods include who is responsible for the task,
if implemented as intended, and
success and challenges to the evaluation work.
Following each table, when appropriate, is a description of baseline data, sampling procedures, planned data
comparisons, data management/analysis procedures for assessing progress toward achieving intended
improvements, and results of data collection.
D i r e c t S e r v i c e P r o v i d e r L e v e l o f E v a l u a t i o n
Early intervention service providers are the direct recipients of the professional development (PD) activities
and it is important to evaluate the training effectiveness. The following information demonstrates if the
training is meeting the needs of the providers, what knowledge and skills it has taught, and what change it has
brought in the provider's performance (see Tables 10 through 22).
Table 10. Selected Elements from Direct Service Provider Level Evaluation Plan L2Q5, L2Q5.1
Evaluation Question How do service providers' perceptions of their abilities to implement FGRBI change? (L2Q5)
How do service providers' satisfaction and confidence in service provision change? (L2Q5.1)
Measure(s) Post-then-Pre Evaluation Survey
Data Collection Methods Paper survey completed by each participant at the end each face-to-face PD session Administered by Florida State University
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S1, S2, S10
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes
Successes & Challenges: Simple process. Time allowed complete the form at the end of training each day.
Using a "post-then-pre" or "retrospective pre test" after each face-to-face training allows data to be gathered
one time each day (as compared to a "pre test" done at the beginning of the day and a "post test" done at the
end of the day) which gives a valid picture of the trainings effects (see Figure 11-14). Analysis of results offers
trainers information about provider needs and improves each future training element (i.e. webinars, emails,
online resources, expert coaching sessions, community of practice). The goal is increase net learning which
will reduce the learning lag. The first face-to-face two-day training session occurs shortly after participants are
selected and pre-training videos and forms are complete. The second face-to-face two-day training session
occurs around 6 months later.
IOWA Part C SSIP Phase III Year 1
22
RESULTS
Definitions: Pre-Existing Learning Pre-test Score Net Learning Post-test score minus pre-test score. This is the score attributable to the training. Learning Lag Total minus post-test score
Figure 11. Learning Based Assessment of Training – IA DMM Cohort 4 First Session, Day 1
IA DMM First Face-to-Face Training Cohort 4 - Day 1
Figure 12. Learning Based Assessment of Training – IA DMM Cohort 4 First Session, Day 2
IA DMM First Face-to-Face Training Cohort 4 - Day 2
I can describe rationale for useof caregiver-implementedintervention in everyday
routines and activities
I can identify strategies forbuilding family capacity using
adult learning
I can describe strategies forembedding functional child
and family outcomes ineveryday routines and
activities using evidence basedinterventions
I can describe general andspecific caregiver strategies
Learning Lag 17.40% 20.00% 18.00% 19.60%
Net Learning 16.40% 20.80% 17.40% 16.20%
Pre-Existing Learning 66.20% 59.20% 64.60% 64.20%
66.20%59.20% 64.60% 64.20%
16.40%20.80%
17.40% 16.20%
17.40% 20.00% 18.00% 19.60%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
I can list SS-OO-PP-RRcomponents
I can practice use of aframework for coaching
caregivers that promotessuccess for the triad
I can describe generaland specific caregiver
coaching strategies
I can define qualityindicators that support
implementation of familycentered services in the
natural environment
I can develop an actionplan for next steps with
peer partner
Learning Lag 37.10% 0.80% 17.20% 18.00% 16.00%
Net Learning 17.10% 45.80% 21.40% 20.80% 23.20%
Pre-Existing Learning 45.80% 53.40% 61.40% 61.20% 60.80%
45.80%53.40%
61.40% 61.20% 60.80%
17.10%
45.80%21.40% 20.80% 23.20%
37.10%
0.80%
17.20% 18.00% 16.00%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
This supports comments obtained in provider videos stating this training has offered them a framework for their work.
IOWA Part C SSIP Phase III Year 1
23
Figure 13. IA DMM Cohort 4 Second Session (6 months after 1st Face-to-Face Training), Day 1
IA DMM Second Face-to-Face Training Cohort 4 - Day 1
Figure 14. IA DMM Cohort 4 Second Session (6 months after 1st Face-to-Face Training), Day 2
IA DMM Second Face-to-Face Training Cohort 4 - Day 2
I can describe 3changes in my
knowledge aboutFGRBI
I can describe 3changes in my
knowledge of thecoaching used in IA-
DMM
I can list four stepsfor the problem-
solving process touse with families
I can describe 5 SS-OO-PP-RR strategies
I can describestrateges for
coaching in earlycare and education
Learning Lag 16.80% 13.20% 14.40% 15.20% 27.20%
Net Learning 16.80% 20.00% 34.80% 23.20% 13.60%
Pre-Existing Learning 66.40% 66.80% 50.80% 61.60% 59.20%
66.40% 66.80%
50.80%61.60% 59.20%
16.80% 20.00%
34.80%23.20%
13.60%
16.80% 13.20% 14.40% 15.20%27.20%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
I can describelearning
opportunities foradult learners.
I can list two newways to use video to
support learning.
I can describegeneral and specificcaregiver coaching
strategies
I can define qualityindicators that
supportimplementation of
family centeredservices in the
natural environment
I can develop anaction plan for next
steps with peerpartner
Learning Lag 14.40% 11.60% 17.60% 16.40% 12.00%
Net Learning 24.40% 26.00% 22.80% 19.20% 21.20%
Pre-Existing Learning 61.20% 62.40% 59.60% 64.40% 66.80%
61.20% 62.40% 59.60% 64.40% 66.80%
24.40% 26.00%22.80%
19.20%21.20%
14.40% 11.60%17.60% 16.40% 12.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
strategies
IOWA Part C SSIP Phase III Year 1
24
Tables 11-14 are additional survey data collected after each face-to-face training, used for continuous
improvement purposes.
Table 11. IA DMM Cohort 4 First Session, Day 1
Question Mean/
Average
Very Much (5 pts)
Much (4 pts)
A Fair Amount (3 pts)
A Little (2 pts)
Not at All (1 pt)
1. The facilitation was of high quality. (n=51) 4.21 18 26 7 0 0
2. The information was relevant. (n=52) 4.63 36 16 0 0 0
3. The conversations held were critical for my future work. (n=51)
4.49 29 18 4 0 0
4. Overall, this training was very beneficial. (n=52) 4.27 24 20 7 0 1
Table 12. IA DMM Cohort 4 First Session, Day 2
Question Mean/
Average
Very Much (5 pts)
Much (4 pts)
A Fair Amount (3 pts)
A Little (2 pts)
Not at All
(1 pt)
1. The facilitation was of high quality. (n=49) 4.33 19 27 3 0 0
2. The information was relevant. (n=51) 4.71 38 11 2 0 0
3. The conversations held were critical for my future work. (n=49)
4.57 33 12 3 1 0
4. Overall, this training was very beneficial. (n=48) 4.50 29 15 3 1 0
Table 13. IA DMM Cohort 4 Second Session, Day 1
Question Mean/
Average
Very Much (5 pts)
Much (4 pts)
A Fair Amount (3 pts)
A Little (2 pts)
Not at All (1 pt)
1. The facilitation was of high quality. 4.56 30 18 2 0 0
2. The information was relevant. 4.70 35 15 0 0 0
3. The conversations held were critical for my future work.
4.56 34 10 6 0 0
4. Overall, this training was very beneficial. 4.52 31 15 3 1 0
Table 14. IA DMM Cohort 4 Second Session, Day 2
Question Mean/
Average
Very Much (5 pts)
Much (4 pts)
A Fair Amount (3 pts)
A Little (2 pts)
Not at All (1 pt)
1. The facilitation was of high quality. 4.48 26 21 2 0 0
2. The information was relevant. 4.71 37 10 2 0 0
3. The conversations held were critical for my future work.
4.57 31 16 1 1 0
4. Overall, this training was very beneficial. 4.57 32 14 2 1 0
Three open-ended questions wrap up the evaluations for the face-to-face trainings. Below is each question
and examples of top categories that emerged from responses given by providers.
Best Feature of the IA DMM Professional Development Session was….
Watching example videos and talking through the videos
Discussion and reflection with quads/team members/large group
Learning SS-OO-PP-RR
Adult learning strategies
IOWA Part C SSIP Phase III Year 1
25
Any Suggestions for Improvement?
More on outcomes
Room arrangement
More time to brainstorm, discuss problems
Easier uploading/technology options
Other Comments and Reactions I Wish to Offer
Location was convenient
Sidebar conversations were distracting
Showing video examples of providers at the beginning stages of coaching in FGRBI to compliment the videos of
seasoned providers
Table 15. Selected Elements from Direct Service Provider Level Evaluation Plan L2Q6a-d
Evaluation Question How do service providers shift in their abilities to implement family guided routines based interventions? (L2Q6a-d)
Shifts in various knowledge, skills, and attitudes/dispositions through PD efforts.
Measure(s) Family Routine Categories; Coaching Strategies; FGRBI Key Indicators; SS-OO-PP-RR Framework;
Data Collection Methods IA DMM Session Summary Form Administered by Florida State University
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S1, S10, I1, L1
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes
Successes & Challenges: Video recordings from "door to door" (from the beginning of the early intervention visit through the end) is an improvement to having providers sending selected parts of a recorded early intervention visit. Technology issues have been a challenge.
This systematic change process in early intervention practices over time is key to measuring the impact of the
IA DMM professional development work. These observation data are directly related to meeting the goals of
implementing family-guided routines-based intervention and coaching caregivers.
Service providers are required to submit a self-assessment form with each video that is shared with the expert
coaches at Florida State University (FSU). The self-assessment includes sections to record: routines with 4
categories (i.e. play, caregiving, pre-academic, community and family); specific coaching strategies used;
Family-Guided Routines-Based Key Indicators (12 items); and SS-OO-PP-RR Framework or Setting the Stage-
Observation and Opportunities to practice; Problem solving and Planning-Reflection and Review. In addition,
the expert coaches at FSU complete the IA DMM Session Summary Form which is identical to the self-
assessment each provider completes. Data from the IA DMM Session Summary Forms are used to first, show
baseline from video 1 submitted prior to the beginning of the training, then to show change over the 9 month
training period as videos are submitted and expert coaching sessions occur.
IOWA Part C SSIP Phase III Year 1
26
FSU uses the information as part of the continuous improvement process for the individual participants.
Cohort level data is shared annually in June at the Joint Implementation Team Meeting (includes state and
regional implementation team members) and at regularly scheduled stakeholder meetings.
RESULTS Figure 15. Cohort 1-4 Observation Results, FGRBI SS-OO-PP-RR Key Indicators, Coaching Strategies, Routines Used
As expected, each cohort comes to the training with higher baseline scores. Early ACCESS regional liaisons
and/or supervisors attend training as observers. This helps them gain information to support trainees in the
shift to using evidence-based practices. They also share the information with others who are not yet trained
which piques interest and begins to build the capacity of future trainees. Each new cohort has some influence
from the proceeding cohorts. Overall, trend lines are upwards which indicates improvement.
0
2
4
6
8
10
12
14
Cohort 1 Cohort 2 Cohort 3 Cohort 4
Me
an N
um
be
r o
f In
dic
ato
rs
FGRBI SS-OO-PP-RR Key Indicators
0
10
20
30
40
50
60
70
80
90
100
Cohort 1 Cohort 2 Cohort 3 Cohort 4
Me
an P
erc
en
tage
of
Ke
y In
dic
ato
rs U
sed
FGRBI SS-OO-PP-RR Key Indicators
0
1
2
3
4
5
6
7
8
Cohort 1 Cohort 2 Cohort 3 Cohort 4
Me
an N
um
be
r o
f Sp
eci
fic
Co
ach
ing
Stra
tegi
es
Caregiver Coaching Strategies
0
0.5
1
1.5
2
2.5
3
3.5
4
Cohort 1 Cohort 2 Cohort 3 Cohort 4
Me
an N
um
be
r o
f R
ou
tin
es
Pe
r Se
ssio
n
Routines Use
IOWA Part C SSIP Phase III Year 1
27
A pilot study to check if each cohort continues to implement new practices with fidelity takes place during the
next reporting year. Permanent fidelity checks are built into the system by (1) requiring random samples of
providers to submit a video recorded home visit for assessment, and (2) training internal coaches to replace
the long distance mentors (Florida State University) with local staff. Internal coaches will provide continued
support to providers and ensure fidelity. Details will be submitted in the SSIP Phase III Year 2 report.
Table 16. Selected Elements from Direct Service Provider Level Evaluation Plan L2Q6.1, L2Q6.2
Evaluation Question Are there more service providers participating as internal coaches for their peers? (L2Q6.1)
Are there more service providers participating as coaches to parents/caregivers?
(L2Q6.2)
Measure(s) Training completion count (# and %)
Data Collection Methods Online survey collected from EAGL at least annually + Participant lists from FSU Administered by Iowa Department of Education
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S3, I2
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes
Successes & Challenges: Easy to collect.
Knowing how many people have been trained helps determine future number of trainings required to meet
the needs of the state. Online surveys gather the information. Training rosters provide numbers of trained
providers and their discipline.
RESULTS
Regions vary widely in the percent of providers trained due to their different sizes. Survey results indicate that
the majority of Area Education Agencies (AEA) have less than 50% left to train. This information is used to
determine PD priorities, schedule future cohort training sessions and determine Early ACCESS fiscal needs for
PD. Figure 16 shows how many early interventionists have been trained and the different disciplines that have
participated.
Figure 16. Number of Participants by Cohort and Percentage of Disciplines Trained Cohort 1-4
ECSE59%
SLP18%
PT11%
OT9%
SC2%
Other1%
Cohort 1, 31
Cohort 2, 29
Cohort 3, 28
Cohort 4, 55
# of Participants by Cohort % Disciplines Cohorts 1-4
IOWA Part C SSIP Phase III Year 1
28
Table 17. Selected Elements from Direct Service Provider Level Evaluation Plan L2Q6.3
Evaluation Question Are service providers able to understand and integrate multiple evidence-based approaches for supporting FGRBI and coaching caregivers?
(L2Q6.3)
Measure(s) Anecdotes from regional liaisons
Data Collection Methods Notes from Early ACCESS Leadership Group meetings & Regional Implementation Team updates Administered by Iowa Department of Education
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S6, I5
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes
Successes & Challenges: Information on other trainings are routinely gathered at naturally occurring statewide meetings of stakeholders. AEAs are investing in regional training on caregiver coaching through Rush and Shelden that compliments the IA DMM work. Some AEAs invest in Routines Based Interviews from Robin McWilliam that compliments the IA DMM work. Trainings on coaching from other experts in the early intervention field can confuse some providers.
RESULTS
The Iowa Distance Mentoring Model (IA DMM) of Professional Development (PD) for Early ACCESS operates
along side other PD efforts that are funded by the individual agencies providing services. However, IA DMM is
the only statewide effort that trains to use evidence based practices and uses active implementation
strategies to scale up and sustain the use of family-guided routines-based interventions through coaching
caregivers. For this reason, it is essential that our stakeholders work together to integrate what they learn
through local efforts and the statewide system change.
When agencies report struggles with integration, Florida State University staff bring together other experts via
webinars in order to strengthen everyone's understanding of how different approaches work together and not
against each other (i.e. Shelden & Rush, Robin McWilliam). In addition, other states that are doing similar
work join Iowa webinars. These efforts provide some of the evidence available to show progress in this area.
The goal is to have all early interventionists in Iowa using family-guided routines-based interventions and
caregiver coaching practices to improve outcomes for children and families regardless of which experts
provide training.
IOWA Part C SSIP Phase III Year 1
29
Table 18. Selected Elements from Direct Service Provider Level Evaluation Plan L2Q7
Evaluation Question How do service providers assess the “readiness” of their agency for organizational change? (L2Q7)
Measure(s) Readiness for Organizational Learning and Evaluation Instrument (ROLE); Preskill & Torre (2000)
Data Collection Methods Online survey collected from each participant after their first face-to-face training Administered by external evaluator
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S10, I8, L5
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes
Successes & Challenges: Online process is simple. Good return rates for cohorts 1-4 (93%, 74%, 77%, 76%). Outside evaluation administering and analyzing results. Some providers answered survey based on their work unit with an organization and others on the entire organization, limiting use of results. Data are from only a few people, the Early ACCESS service providers participating in IA DMM not an entire agency or department so caution should be used when interpreting the results.
The Readiness for Organizational Learning and Evaluation or ROLE (Preskill & Torres, 2000) was designed to
help determine the level of readiness for implementing organizational learning, evaluation practices, and
supporting processes. This survey is completed by all cohort service providers after the first face-to-face
training.
The ROLE consists of 75 items grouped into six major areas: (1) Culture, (2) Leadership, (3) Systems and
Structures, (4) Communication, (5) Teams, and (6) Evaluation. Four of these areas – Culture, Leadership,
Systems and Structures, and Communication of Information – are further divided into subareas (see Table 19).
Table 19. ROLE Areas, Subareas, and Number of Items
Culture (27 items)
Leadership (12 items)
Systems & Structures (12 items)
Communication (8 items)
Teams (8 items)
Evaluation (8 items)
Collaboration & Problem Solving
12 items
No Subarea Open & Accessible Work Environment
4 items
Availability – 3 items
No Subarea No Subarea
Risk Taking 5 items
Rewards & Recognition 5 items
Dissemination 5 items
Participatory Decision Making
10 items
Relationship of Work to Organizational Goals
3 items
Service providers are asked to respond to items on a scale of 1 to 5, with 1 meaning “Strongly Disagree,” and 5
meaning “Strongly Agree.” There are five additional items/questions found on this instrument including the
following: a) three yes/no items about teaming practices, and b) two multiple choice items about demographic
IOWA Part C SSIP Phase III Year 1
30
information (job category and length of employment). This report contains the responses from those IA
DMM participants (cohort 1: 2013-cohort 4: 2015) who completed the ROLE instrument.
RESULTS
Preskill & Torres (1999, 2004) have developed a scoring sheet for analyzing the individual items and areas and
suggest that results for all respondents within a department or organization be combined and reported in
summary form. Average scores of 3.5 or higher were considered passing. The authors interpret average scores
below 3.5 as follows: “If an organization were to score low in one or more of the dimensions, this would
indicate that the department or organization isn’t prepared to engage in other kinds of organizational learning
practices” (Preskill & Torres, 1999, p. 11). Scores that fall below a 3.5 are highlighted in red for further review
and discussion by the members of the Regional Implementation Teams (RITs).
RITs explore if the items obtaining scores at or above an overall mean of 3.5 (in white and considered passing)
are indeed representative of their agency. Likewise, they explore if the items obtaining scores below an
overall mean of 3.5 (in red and considered needing attention) are representative of their agency.
The Joint Implementation Team Meeting held June 15, 2016 was used to share data to inform future plans as
well as discuss and develop Regional Implementation Team Action Plans. Stakeholders from the agencies
providing services, state departments, other early childhood organizations, the external evaluator and Florida
State University engaged in discussions and planning based on the data. Conversations continued at Early
ACCESS Leadership Group meetings.
The data below represent the average or mean scores by agency for Culture and Leadership (see Table 20) and
Systems & Structures, Communication, Teams and Evaluation (see Table 21). In addition to the tables on
average scores, analysis of the individual 75 items are used to drill down to specific results of each question.
This helps the agencies target specific items for review which can lead to change with their specific agency.
That information is not provided in this report since the table with the mean scores in each area provide the
overall results.
This program has given me confidence to go into a home visit and be partners with the parents in regards to helping them coach their child in these skills.
IOWA Part C SSIP Phase III Year 1
31
Table 20. Individual AEA/LEA Mean Scores for ROLE Culture and Leadership
AEA/LEA
Culture Leadership
Col
labo
ratio
n
& P
robl
em
Sol
ving
Ris
k T
akin
g
Par
ticip
ator
y
Dec
isio
n
Mak
ing
Total Mean Score
Total Mean Score
A 3.92 3.18 3.37 3.49 3.65
B 4.07 3.88 3.85 3.93 3.72
C 4.09 3.57 3.94 3.87 3.83
D 4.30 3.78 3.89 3.99 3.81
E 3.69 3.15 3.38 3.41 3.45
F 4.13 3.76 3.64 3.84 3.73
G 3.88 3.65 3.63 3.72 3.66
H 4.04 3.76 3.67 3.82 3.69
I 4.22 3.83 3.97 4.01 3.93
J 3.95 3.54 3.84 3.78 3.78
Total # of Mean Scores below
3.5 0 2 2 2 1
Overall there are high scores in the areas of Culture and Leadership. For two agencies, scores fell slightly
below the 3.5 overall mean score. Each has the detailed item results to address concerns.
Note: Agency name/number are de-identified to assure anonymity.
IOWA Part C SSIP Phase III Year 1
32
Table 21. Individual Agency Mean Scores for ROLE Systems & Structures, Communication, Teams and Evaluation
AEA/LEA
Systems & Structure Communication Teams Evaluation
Ope
n &
Acc
essi
ble
Wor
k E
nviro
nmen
t
Rew
ards
&
Rec
ogni
tion
Sys
tem
s &
Pra
ctic
es
Rel
atio
nshi
p of
Wor
k to
Org
aniz
atio
nal
Goa
ls
Total Mean Score
Ava
ilabi
lity
Dis
sem
inat
ion
Total Mean Score
Total Mean Score
Total Mean Score
A 3.11 3.00 3.70 3.29 3.33 3.73 3.53 3.66 3.59
B 3.44 3.35 4.09 3.63 3.75 3.73 3.74 3.80 3.50
C 3.38 3.06 4.29 3.58 3.81 3.80 3.81 4.04 3.50
D 3.04 3.15 3.92 3.37 3.75 3.88 3.82 4.04 3.50
E 3.46 2.80 3.74 3.33 3.23 3.51 3.37 3.53 3.60
F 3.28 3.12 4.33 3.58 3.93 4.02 3.99 4.11 3.45
G 3.16 3.25 4.14 3.51 3.47 3.65 3.56 3.75 3.54
H 3.20 2.90 3.87 3.32 3.40 3.74 3.57 3.96 3.49
I 3.46 3.29 4.33 3.69 3.82 4.05 3.94 4.11 3.53
J 3.36 3.64 4.09 3.54 3.36 3.64 3.50 3.81 3.42
Total # of Mean Scores below
3.5 10 9 0 4 5 0 1 0 3
There is a clear pattern of low scoring in the Open and Accessible Work Environment and Reward and
Recognition Systems and Practices subareas for all agencies. There are also pockets of other issues as
indicated by the red highlights in Communication and Evaluation. Each agency has the detailed item results to
address their specific concerns.
IOWA Part C SSIP Phase III Year 1
33
Table 22. Selected Elements from Direct Service Provider Level Evaluation Plan L2Q8
Evaluation Question What was the most significant change as reported by providers? (L2Q8)
Measure(s) Reflection interview using the following questions: 1) what is working 2) what is tough 3) what I would do differently next time 4) what has been the most significant change so far
Data Collection Methods Video recording of participant reflection interviews Administered by Larry Edelman completed at the second face-to-face training (about 6 months into the training cycle)
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S1, S2, L1
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes
Successes & Challenges: Larry Edelman, early childhood and technology expert, working through Florida State University, has been critical to the success of capturing high quality video reflections of early intervention service providers. Interview data supports results found in other measures. Richer information collected that surveys cannot convey. Reviewing, transcribing and analyzing are very time intensive.
Reflective videos are recorded at the second face-to-face training session which is about 2/3 of the way
through the 9 month training cycle. Peer partners go the recording session as a group and are asked
prescribed questions (see Measures in above table). To date, cohorts 1-3 are under analysis and cohort 4 is
being transcribed.
The process used for analyzing the interviews includes the following steps. First, the three videos were typed
verbatim by one transcriber. Second, each transcription was reviewed by one evaluator. Key responses by the
participants were then highlighted or underlined. Third, the first cohort’s transcriptions were reviewed to
determine if any of the Division for Early Childhood’s Recommended Practices (2014) related to the area of
family were included in the responses. The highlighted transcription was reviewed to determine if any of the
practice characteristics identified on the four performance checklists developed by the Early Childhood
Technical Assistance (ECTA) Center were reported by the participants.
The four performance checklists included the following: 1) Family-Centered Practice Checklist, 2) Informed
Family Decision-Making Practice Checklist, 3) Family Engagement Practices Checklist, and 4) Family Capacity-
Building Practices Checklist. Each practice checklist contained as few as six or as many as nine practice
characteristics. Responses appeared to fall within a few of the characteristics from the Family-Centered or
Family Capacity Building Practices Checklist. At this time, the external evaluator has not conducted this type
of review with the second and third video tapes.
IOWA Part C SSIP Phase III Year 1
34
The fifth step was to summarize themes that emerged within each of the video tapes. This preliminary report
provides the results from video tapes 1 and 2 and information is provided in a bullet format where main
themes are captured with a top line or underline and supporting quotes shared that help illustrate those
themes. A final step would be to aggregate the data across all cohort summaries.
RESULTS
Responses obtained from cohort 1 were brief yet primarily positive. Themes that emerged included: (1)
increased confidence in their ability to partner with parents and other caregivers; (2) increased intentionality
when on visits to address components of the IA DMM model (using routines, asking questions, etc.); and 3)
their need to continue learning and practicing. A few did identify issues including the need for visits to
families to be more frequent than what is currently in place.
Responses from cohort 2 were lengthier. Themes that emerged included: (1) now having a new framework or
a focus to their visit; (2) they have increased the amount of time they are listening and observing families
while encouraging the families or care providers to do most of the communication, problem solving, and/or
decision-making; (3) they specifically discussed the shift away from toy bags compared to Cohort 1; and (4)
they have experienced issues with the technology and/or the time it takes to do the requirements of the
professional development model.
Some of the providers in cohort 2 identified issues or challenges when using this model with particular
families, especially families who were involved with Early ACCESS prior to the training, families who are in
crisis, families where the parents have developmental delays, and/or families and caregivers who have very
young infants that are also medically fragile. Another concern was teaming with other service providers who
have not been trained to coach the caregivers in family-guided routines-based early interventions. A sample
of service provider quotes are in Figures 17 and 18.
Figure 17. Examples of Interview Quotes from Cohorts 1 and 2
I think that DMM has given me a structure. It's helped me organize my own thinking and how I am going to approach the family. It's helped me realize I want the family to be the problem solvers. I want to guide the family to help them realize what's important to them. That they want to change, and then help them, guide them to make choices that they can try and see if they work to make their daily routines better.
The biggest change has been changing the outcome. How I write the outcomes to a more participation based outcome so we look at kind of a big picture thing about what we wanted to accomplish. What they want. What we want the child to participate in as a team and then back track a little bit and pick up the specific skills that they need to work on to be able to accomplish that participation.
I think the biggest change for me is to really look at the families’ routine…and help the family problem solve what will work in their routine…as far as working with their child on strategies or some of the interventions just trying to get their child to learn. And with a particular family that I have had it's just working on one thing at a time, but four or five times within the day within that routine.
IOWA Part C SSIP Phase III Year 1
35
Figure 18. Examples of Interview Quotes from Cohorts 1 and 2
I think the most significant change I've made since starting with DMM is that focused shift from my agenda, to coming in and asking families what their concerns are with their child. What they would like to work on with their child. Not just once, but like, coming in the next visit and saying, how's that going? That reflection. And then what would you like to work on today? What would you like to see happen this week? Kind of taking me out of it. I used to be such a planner and kind of here's the plan for the day. And that collaboration between myself and the families, I think that was probably the biggest change I’ve made with DMM is that collaboration piece with parents.
It's most difficult I think working with children at a child care center or children in the home who have an in-home nurse when the parent is not present to participate. Especially when sometimes the nursing is very inconsistent. I had a family that had a different in-home nurse every visit I went to, so it was much harder to engage them into daily routines and activities when every day they were just trying to get through the demands of their nursing job and just get to know the child for that day and meet their medical needs. So I find that very challenging.
Open-ended questions probably has been the most difficult for me. I tend to want to give information. I want to take care of families. I want to give them all those resources and supports that they need and get them to, give that to them immediately and so kind of stepping back and observing and letting them talk and share and working on those open ended questions has probably been the most difficult for me just to kind of sit back and listen. It's been a roller coaster. I’ve had days where I've felt pretty good about it… It's been a good experience. I think that I've seen some great changes in some of my families and I think probably the best part is just seeing the empowerment that I think this service is giving them. What I've had to kind of accept is that some of
(us) have (had) the training and some of us haven't and recognizing that It’s not going to be this overnight change for some people and recognizing that we just need to give it time for people, for those who it is a big change, to give them time to make that change. And recognizing those that have been doing (some )aspects of the DMM, that it's not a huge change and to allow time, to be patient with those for whom it is a big change.
The biggest change I've made in terms of looking at family focus in my first interview with the family is to ask the parents or the caregiver, what is their style of learning so I can tailor my materials, my response to their style of learning. Not assuming that they are a visual learner like I am. And that has been helping for providing pictures, providing information and parents have said that's been helpful.
I think the challenging part as an OT is the medically fragile children. The referral sources (are) looking at traditional based therapy, but being in a rural area where there are not hospital based therapists for these children to go to…that can be limiting for families who have limited transportation, that we are the only ones in the home. They don't have the option to go to the hospital for medically based or clinically based therapy services. The communication that needs to happen (from) medical providers and what they will describe to the families that will happen (in Early ACCESS) doesn't always have the same face.
IOWA Part C SSIP Phase III Year 1
36
F a m i l y L e v e l o f E v a l u a t i o n
Table 23. Selected Elements from Family Participant Level Evaluation Plan L1Q1
Evaluation Question How confident and competent do IA DMM family participants feel about helping their child develop and learn? (L1Q1)
Measure(s) Early Intervention Parenting Self-Efficacy Scale (EIPSES); Guimond, Wilcox & Lamorey (2008)
Data Collection Methods Survey online or via mail done not sooner than 9 months from initial entry into IA DMM
Administered by external evaluator and Iowa Department of Education
Outcomes Addressed (from Logic Models) (progress reported in report Section E) I3, I4, L2
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes, Fall 2015 completed cohorts 1-3 (C1-3) as scheduled
Successes & Challenges: C1-3 return rate = 44% (40/91) Reached families via email/online and paper surveys Difficulty getting accurate email addresses Period of time between participating in IA DMM and completing survey too long for majority of families Need more timely analysis and sharing results back with stakeholders including parents
The development of the EIPSES was based on existing concerns that few researchers have explored the role of
parenting self efficacy in families of children with disabilities (Gowen, Johnson- Martin, Goldman, &
Appelbaum, 1989; Hastings & Brown, 2002; Pit-ten Cate et al., 2002; Scheel & Rieckmann, 1998). Few existing
instruments allow professionals to focus on understanding whether parents feel competent and confident in
their skills, knowledge, and the ability to make a difference in the lives of their children, especially when their
infant or toddler also participates in early intervention services because of risk or developmental delay. A scale
with this emphasis is important because involvement with early intervention programs may provide
challenges to parents who are raising children with or at risk for developmental delays.
The EIPSES was developed to address the need for viable measures of change in families who are receiving
early intervention. The measure was developed with the intent to quantify parent perspectives about their
ability to facilitate positive child outcomes within the context of early intervention programs and via
interactions with early intervention practitioners.
This EIPSES survey was completed at the end of the cohort 3 training in Fall 2015 and included families from
cohorts 1-3 who participated in 2013 through 2015 (see Table 24). The survey was not administered prior to
training so there is no baseline or comparison data. This was by design as Early ACCESS state staff had to
balance all of the evaluation components with available resources and consider the paperwork already
required by the family at the beginning of the training and IFSP process. This was intended to act as a
feasibility study to make sure the survey and process was administered properly and the information would be
IOWA Part C SSIP Phase III Year 1
37
useful. Eventually, the survey will be completed earlier in the training process. The data from the EIPSES will
be used in identifying strengths and weaknesses regarding caregivers’ perceived abilities in promoting the
development of their children who are receiving early intervention services, information that might then be
used as targeted changes in the families’ intervention programs. This survey will be combined with direct
observations and interviews that will help assess effectiveness or impact of the professional development
efforts on the family.
RESULTS Table 24. Early Intervention Parenting Self-Efficacy Scale for Cohorts 1 through 3, Fall 2015
Early Intervention Parenting Self-Efficacy Scale (EIPSES) 1 to 7 Scale
N = 40 Cohorts 1-3
Range Average/Mean (SD)
Minimum Maximum
Parent Confidence (10 items)
2.40 5.30 3.38 (.69)
Parent Competence (4 items)
4.25 7.00 5.89 (.69)
Parent Self-Efficacy (16 items)
3.44 5.44 4.10 (.47)
Parents report feeling competent in their abilities to promote their child’s development (average score 5.89)
but less confident in their abilities to exert control over their children’s early intervention outcomes (average
score 3.38). In other words, caregivers feel they are able to do the interventions but are less sure that the
interventions are going to produce the outcomes wanted for their child.
Definitions Parent Confidence: The extent to which parents believe child outcomes are a function of environmental influences or constraints, such as family background or the availability of early intervention or community support. Parent Competence: The degree to which caregivers perceive themselves as being personally effective and capable in parenting their child.
I think the biggest change so far in this process has been really involving families more. Observing families more. I spent the last twenty years in this job doing and having families watch me and taking in my toy bag and I'm really trying to not take in toys and utilize more family options and inviting the families to sit down and play and observing them in their natural routines.
IOWA Part C SSIP Phase III Year 1
38
Table 25. Selected Elements from Family Participant Level Evaluation Plan L1Q2a-b, L1Q3
Evaluation Question How confident and competent do IA DMM family participants feel about helping their child develop and learn? (L1Q2a-b)
How confident do family participants feel about working with their child throughout the day? Has coaching changed how effective families feel about helping their child? (L2Q3)
Measure(s) Pre-Interview Survey and Family Interview
Data Collection Methods For a sample of families in DMM, surveys online or via mail done after the EIPSES has been completed. For a sample of families in DMM, after the end of the cohort DMM training cycle.
Administered by Florida State University
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S4, S5, L2
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) No, cohorts 1-3 scheduled for Spring 2016. Solicited participation during this time but evaluation not completed until Fall 2016 (next reporting period).
Successes & Challenges: Families did not respond to invitations to participate in interview. Asked multiple times and ways. Only one family signed consent and participated from cohorts 1-4. Adjusted interview schedule to align more closely with ending of IA DMM training so providers can personally encourage families to participate. Results will be submitted in next reporting period.
Table 26. Selected Elements from Family Participant Level Evaluation Plan L1Q4
Evaluation Question Are families demonstrating increased participation and proficiency in helping their child develop and learn? (L1Q4)
Measure(s) Caregiver Key Indicators
Data Collection Methods Video coding for a sample of families in DMM
Administered by Florida State University
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S5, I3, I4, L2
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes, developed tool during current reporting period for use Summer 2016 (next reporting period)
Successes & Challenges: Instrument had to be developed. Key Indicators developed and piloted.
The primary aim of Part C services is to support the family’s confidence and competence in promoting their
child’s developmental needs. While supporting family capacity is a fundamental goal for services for children
under three, it remains a difficult construct to define and measure. There are limited options for measuring
family capacity building practices in the context of intervention sessions that matches the model that Early
ACCESS providers are being coached to use (FGRBI; Woods, Kashinath, & Goldstein, 2004).
IOWA Part C SSIP Phase III Year 1
39
In order to meet this need, Florida State University developed a tool to describe and quantify the caregiver’s
role in home-based intervention sessions. This tool, called Caregiver Key Indicators (see Figure 19), is intended
to be used in conjunction with the providers’ fidelity measure, and in some ways mirrors the provider fidelity
items. Instead of measuring provider fidelity, though, this tool examines behaviors from the vantage point of
how the caregiver participates in the intervention sessions. The Caregiver Key Indicators uses video
observation of full-length home visits in order to assess whether caregiver-child interactions are primary in the
session, whether the parent participates and practices embedding strategies in routines, whether the
caregiver engages in problem-solving and reflection on the intervention with the provider, and whether the
caregiver helps contribute to an action plan with the provider for the time between visits. The twelve-item
tool offers an overall percentage of indicators that are either present, partially present, or not present.
This tool is a means for providers and program administrators to see the ways in which they are engaging the
family rather than a way to evaluate a family’s participation or to fault them for not participating. Providers
and caregiver’s have a bidirectional relationship, and a family’s engagement and growth in capacity is linked to
what the provider does during each session. Some providers may create opportunities for the caregiver to
practice strategies in routines, but the family may not have a clear role in planning and problem-solving.
Likewise, providers may offer caregivers opportunities to share information and discuss developmental
challenges, but they may struggle to create practice opportunities for the caregivers to try strategies in every
day routines during the session. It is expected that there will be a relationship between the provider’s fidelity
scores and the scores on the Caregiver Key Indicators.
This tool is used to measure participant videos before service providers receive professional development
(PD), during the PD sequence, and after to measure change as a result of the provider’s training. In doing so,
we aim to measure not only what the provider is doing, but how the providers impact the family’s
engagement and participation in intervention sessions.
IOWA Part C SSIP Phase III Year 1
40
Figure 19. Caregiver Key Indicators Form
YES PARTIAL NO 1. Caregiver-child interaction is primary Yes if more than 50 % of time; Partial if 25-50 % of time, No if 24 % or less.
2. Caregiver participates in 2-3 typical/preferred routines from different routines categories? If they only do one type of routine but talk about others, that is scored as partial.
3. Caregiver uses family’s typical materials? 4. Caregiver embeds the child’s functional targets 3-5x in routines? If 3-5 times but only in one routine, score as partial. If 3-5 times in 2 or more routines, then yes. If 2 trials in one routine and three in another, then yes. If less than three times in only one routine, then no.
5. Caregiver practices intervention strategies 3-5x in routines? If 3-5 times but only in one routine, score as partial. If 3-5 times in 2 or more routines, then yes. If 2 trials in one routine and three in another, then yes. If less than three times in only one routine, then no.
6. Caregiver responds to feedback with comments or more practice at least one time per session?
7. Caregiver requests or responds to information and ideas at least one time per session? 8. Caregiver problem solves with interventionist on what is and is not working at least one time
per session?
9. Caregiver reflects on current session at least one time? 10. Caregiver summarizes action plan for intervention between visits?
RESULTS Figure 20 shows the percentage of weighted items observed on video 1 which is the baseline recording done
prior to the service provider's training. This is compared to the percentage of weighted items observed after
training took place. Data shows how the providers impact the family’s engagement and participation in
intervention sessions.
Figure 20. Caregiver Key Indicators Observations Cohort 4 Video 1 and Video 2
DRAFT NOT FOR DISTRIBUTION
0
10
20
30
40
50
60
70
80
90
100
Video1(Range0-91.67) Video2(Range25-91.67)
Percentageofweighteditems
C4CaregiverKeyIndicatorsData
Video 1 (Range 0-91.67) Video 2 (Range 25-91.67)
Per
cen
tage
of
wei
ghte
d it
em
s
C4 Caregiver Key Indicators Data
IOWA Part C SSIP Phase III Year 1
41
I m p l e m e n t i n g A g e n c y L e v e l o f E v a l u a t i o n
Table 27. Selected Elements from Implementing Agency Level Evaluation Plan L3Q9-Q11
Evaluation Question How do partnerships on the regional implementation teams change over the project period? (L3Q9)
What types of network/ collaboration changes occur? (L3Q10)
How and when do they begin to identify "others" who need to be part of their implementation team? (L3Q11)
Measure(s) Level of Collaboration Scale; Frey, Lohmeier, Lee & Tollefson, (2006).
Data Collection Methods Online survey collected from each regional implementation team member annually Collaboration maps from survey results Administered by external evaluator
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S7, L7
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes
Successes & Challenges: Online process worked. Good return rates for each round of data collection (79%, 74%, 63%, 71%) Complex analysis and graphing process. Limited use by implementing agencies.
An increased level of collaboration among agencies providing early intervention services is an implied
objective as it is necessary in order for a statewide system to achieve and sustain long-term outcomes of
improving the results for infants and toddlers served in Early ACCESS. The Level of Collaboration Scale is
designed to measure levels of collaboration among organizations and individuals which is viewed as essential
for success. In addition, a format for visually displaying the results is obtained from using the scale. Even
though analysis and data displays are complex, the "picture" of collaboration for the Regional Implementation
Teams show interesting degrees of relationships.
Data were collected from members of 9 Area Education Agencies (AEAs) and Des Moines Public School District
during 2014 and 2015. Each agency's Regional Implementation Team (RIT) member was asked to participate in
a Levels of Collaboration Survey (Frey et al., 2006), where they were provided with a list of potential
collaborators from their respective AEA. Survey participants were asked to rate the relationship characteristic
of each potential collaborator on a scale of 1 to 5, where 1 corresponds to Networking and 5 corresponds to
Collaboration. For those individuals with whom respondents had no collaboration, they could select 0, “No
Interaction At All”. The Levels of Collaboration Survey, in addition to the definitions for each Relationship
Characteristic (Network to Collaboration), is in Table 28.
IOWA Part C SSIP Phase III Year 1
42
Table 28. Levels of Collaboration Definitions and Relationship Characteristics Five Levels of Collaboration and Their Characteristics
Networking
1 Cooperation
2 Coordination
3 Coalition
4 Collaboration
5
Relationship Characteristics
Aware of organization
Loosely defined roles
Little communication
All decisions are made independently
Provide information to each other
Somewhat defined roles
Formal communication
All decisions are made independently
Share information and resources
Defined roles
Frequent communication
Some shared decision making
Share ideas
Share resources
Frequent and prioritized communication
All members have a vote in decision making
Members belong to one system
Frequent communication is characterized by mutual trust
Consensus is reached on all decisions
Data was sent to The Center for Education Integrating Science, Mathematics, and Computing (CEISMC)
(pronounced like "seismic"), a partnership uniting the Georgia Institute of Technology with educational
groups, schools, corporations, and opinion leaders throughout the state of Georgia. (For more information go
to https://www.ceismc.gatech.edu/). The survey data was re-entered for compatibility with social network
analysis (SNA) software. Specifically, UCINet (Borgatti et al., 2002) was chosen for its ease of use in network
analysis and accompanying network visualization software.
Due to the high turnover of AEA members between 2014 and 2015, it was determined that the most reliable
way to analyze RIT networks was by each year separately. Changes in the strength of ties were explored
between the two years. The RIT networks of each AEA were analyzed by exploring changes in tie strength and
network connectedness within individual AEA Regional Implementation Team, as well as by exploring network
patterns observed within and across individual AEA RITs.
RESULTS
Three (out of 10) individual agency findings are included in this report as examples that will provide enough
information to build understanding of the results. The results for the other 6 AEAs and Des Moines Public
Schools are available in the full report from CEISMC.
AEA "X"
Between 2014 and 2015, AEA X consisted of 13 unique members in the Regional Implementation Team. In
2014, AEA X consisted of 6 members, three of whom left the team following 2014. These 6 members were
connected by a total of 24 ties. In 2015, there were 7 members of AEA X connected by 32 ties, three of whom
were new to the team during that year. Three AEA X members were present in both 2014 and 2015. Figure 21
illustrates the network maps for AEA X during the years 2014 and 2015.
IOWA Part C SSIP Phase III Year 1
43
Figure 21. AEA "X" Network Maps for 2014 and 2015*
*Key: Pink Square: Single Year Member; Orange Square: Member of AEA during 2014 and 2015; Thickness of Lines: Relationship Characteristic Value, as defined within the strength of Collaboration Survey
During 2014, 20 ties, or 83.3% of ties were valued “Coordination (3)” or higher, and 66.7% of ties were valued
as “Coalition (4)” or higher. During 2015, 75% of ties were values as Coordination or higher, and again, 75%
were rated as Coalition or higher (see Figure 22). Overall, AEA X was a very well connected in both 2014 and
2015 by both measures of interest here, including number of ties and tie strength. Interestingly, in 2015, new
members of the AEA are all strongly connected to other members. Overall, few ties in the 2015 network can
be considered weak, indicating high levels of knowledge and resource distribution across the network.
Figure 22. AEA "X" 2014 and 2015 Ties Valued as Coalition or Higher*
2014 2015
AEA "X"
AEA "X" 4+ 2014 2015
IOWA Part C SSIP Phase III Year 1
44
AEA "Y"
Between 2014 and 2015, AEA Y consisted of 16 unique team members. In 2014, AEA Y consisted of 11
members, three of whom left the team following 2014. These 11 members were connected by a total of 82
ties. In 2015, there were 13 members of AEA Y Regional Implementation Team connected by 96 ties, 5 of
whom were new to the group during that year. Eight AEA Y members were present in both 2014 and 2015.
Figure 23 illustrates the RIT network maps for AEA Y during the years 2014 and 2015.
Figure 23. AEA "Y" Network Maps for 2014 and 2015* *Key: Pink Square: Single Year Member; Orange Square: Member of AEA during 2014 and 2015; Thickness of Lines: Relationship Characteristic Value, as defined within the strength of Collaboration Survey
During 2014, 48.8% of ties were valued “Coordination (3)” or higher, and 22% of ties were valued as “Coalition
(4)” or higher. During 2015, 47.9% of ties were values as Coordination or higher, and again, 43.8% were rated
as Coalition or higher (see Figure 24).
Figure 24. AEA "Y" 2014 and 2015 Ties Valued as Coalition or Higher*
AEA "Y" 4+ 2014 2015
AEA "Y" 2014 2015
IOWA Part C SSIP Phase III Year 1
45
AEA "Z"
Between 2014 and 2015, AEA Z consisted of 13 unique Regional Implementation Team (RIT) members. In
2014, AEA Z consisted of 9 members, including five who left the group following 2014. These 9 members were
connected by a total of 60 ties. In 2015, there were 8 members of AEA Z team connected by 36 ties, including
four people who were new to the group during that year. Four AEA Z members were present in both 2014 and
2015. Figure 25 illustrates the RIT network maps for AEA Z during the years 2014 and 2015.
Figure 25. AEA "Z" Network Maps for 2014 and 2015*
*Key: Pink Square: Single Year Member; Orange Square: Member of AEA during 2014 and 2015; Thickness of Lines: Relationship Characteristic Value, as defined within the strength of Collaboration Survey
During 2014, 20% of ties were valued “Coordination (3)” or higher and as “Coalition (4)” or higher. During
2015, 22.2% of ties were valued as Coordination or higher; however, no ties were rated as Coalition or higher.
As previously presented, there are much less ties in the overall network during 2015 than during 2014. In
addition, of the existing ties, relatively few were rated by survey respondents as strong ties. As illustrated in
Figure 26 , there are slightly more strong ties created among recurring members of the AEA during 2015.
However, the ties are still weaker than those in 2014, where 20% of ties were valued by respondents as 4 or
higher.
Figure 26. AEA "Z" 2014 and 2015 Ties Valued as Coalition or Higher*
AEA "Z" 2014 2015
AEA "Z" 4+ 2014 2015
IOWA Part C SSIP Phase III Year 1
46
Summative Patterns
Figures 27 and 28 present all agencies together, during 2014 and 2015, respectively, in an effort to examine
patterns that may occur throughout the AEA network. This summary examines network patterns, but is not
intended to be interpreted as a comparison of individual AEA networks to each other. Figures 27 and 28 are
meant to visualize these patterns in single diagrams, rather than as a comparison of individual agencies.
It is apparent throughout the majority of AEAs that the recurring network actors (orange nodes in the previous
figures) tend to have the strongest ties. As such, those networks with the least amount of turnover tend to
have the strongest overall networks, in terms of strength of ties. This should be expected, as recurring actors
have been contributing members of the AEA for a longer period of time; therefore, they have likely been
afforded a greater opportunity to become familiar with each other, and further collaborate and share ideas.
With few exceptions, this same pattern is not seen among members who were in the AEA for a single year,
represented by the pink nodes in the previous figures. In most cases, these AEA members have weaker
connections to all other nodes within the network. This pattern is important, since according to the Levels of
Collaboration Survey, strong ties (defined here as “Coordination” or higher) indicate AEA members who are
sharing information and resources communicate frequently and share in decision-making. Therefore, a
network in which a greater number of actors are connected by strong ties signifies an AEA in which there is a
greater flow of important information and resources to the whole network.
In addition, recurring members of AEA tend to be more central in the AEA networks. In most cases, those AEA
members most central to the network are again recurring members, or those orange nodes that represent
AEA members present in both 2014 and 2015. In less connected networks, those central actors hold the
greatest ability to move information and resources throughout the network. It is important to note that
network patterns indicate that those with the greatest ability to move resources and information throughout
the majority of AEA networks are those recurring members, which may signify the importance of limiting
member turnover. However, this may also be attributable to characteristics not available as a part of this
study, such as actor job role.
I am guessing the biggest change for me as a speech language pathologist is to leave my toy bag at home. That was kind of a big change. It's always easy to bring novel toys into a home and get kids engaged very easily. So now, you have to work with that a little bit more, but having to use their own toys and the things that they have in their house makes a whole lot more sense and actually you can get kids just as easily engaged that way. So I think that's probably the biggest change for me
IOWA Part C SSIP Phase III Year 1
47
Figure 27. All AEAs Network Maps, 2014*
Figure 28. All AEAs Network Maps, 2015*
*Key: The key for the diagram is as follows: Colors – Each Agency has same color for 2014 and 2015 Shapes: 2014 Only (Circle), Recurring Members (Square) Tie Thickness: Strength of Tie (Thicker = Greater Strength)
IOWA Part C SSIP Phase III Year 1
48
After receiving the results of the Levels of Collaboration Survey, volunteer AEA managers reviewed the data
and results based on a series of questions created by the external evaluator. The questions were about use of
this data. It was the opinion of AEA managers that this data is not useful at the Regional Implementation
Team (RIT) level unless further evidence on the importance of collaboration to successful practice change is
provided. The evaluation team discussed stakeholder feedback and decided one more round of surveys will
be collected in the fall of 2016 to analyze change in the implementation teams across time. This decision is
based on the fact that data from RIT Reports and anecdotal information gathered at Early ACCESS Leadership
Team meetings shows progress in team functioning.
Table 29. Selected Elements from Implementing Agency Level Evaluation Plan L3Q12, L3Q12.1
Evaluation Question How do implementation team members shift in their understanding of Active Implementation Frameworks? (L3Q12)
How do implementation team members shift in their use of Active Implementation Frameworks? (L3Q12.1)
Measure(s) Regional Imp Team (RIT) Reports State Imp Team (SIT) meeting notes State Work Team agendas & notes EAGL agendas & notes
Data Collection Methods RIT report updates collected via Google Docs SIT meeting notes State Work Team meeting notes collected via Google Docs EAGL meeting notes collected via Google Docs Administered by Iowa Department of Education
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S8, S9, S12, S17, I6, I7, I14, I15, L4, L8-L10
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) No, changed measure from Active Implementation (AI) Hub quizzes to more contextual measures and did collect data as scheduled.
Successes & Challenges: Making use of regularly occurring stakeholder meetings. Dedicated one day of the two-day statewide Early ACCESS Leadership Group (EAGL) meetings to FGRBI, coaching caregivers and implementation work. Florida State University participation via technology at EAGL meetings. Use Google Docs to routinely update progress on RIT activities. Face-to-face EAGL meetings used as RIT events where team members across the state who are not part of EAGL join electronically (via Zoom) to participate while other EAGL members who typically are uninvolved with RIT work participate as well. Annual joint face-to-face meeting between RIT and SIT facilitated by Florida State University where data are shared and team plans are reviewed. Even though implementation team work is better, it has been a struggle to maintain well functioning teams. Teams continue to need regular support which is addressed routinely at meetings and through webinars.
Regional Implementation Team Reports are updated via Google Docs 5 times per year at the EAGL meetings.
Table 30 shows the questions that are asked/updated at each meeting by all Area Education Agencies and Des
Moines Public Schools. All agencies record on the same document. This supports communication between
IOWA Part C SSIP Phase III Year 1
49
agencies as they can see what each other has reported. Discussions and actions happen based on the topics in
the reports and is recorded in the Early ACCESS Leadership Group meeting and State Work Team meeting
notes, also completed in Google Docs for easy accessibility by everyone.
RESULTS Table 30 includes examples of information reported and then used to continually improve the work. Table 30: Regional Implementation Team Report Questions & Examples of Data Collected
Who has attended the meetings Mid level management staff, past IA DMM cohort participants, internal coaches, AEA administrators, regional liaisons, instructor of early childhood college courses, community partner health providers, people who are ECSE teachers, occupational therapists, physical therapists, speech language pathologists
How many meetings have you held since August 2016? When are your future meetings? Range from one time per month to quarterly.
Highlights from the meetings
PD planning for cross-discipline meetings, outcome writing, internal coaching plans, initial PSP conversations.
Sharing videos and discussing. Sharing ideas to use with coaching teams. Updates on IA DMM resources available. Setting expectations for next year. Discussions on how we combine coaching, RBI and PSP to move forward.
Participated in April webinars. Met on 5/12 to further plan for FGRBI rollout beginning in 16-17 school year and work on Action Plan.
Cross disciplinary PD planning (new PD implementation beginning fall 2016).
In April and May, we have met to discuss the action plan for sustaining and scaling up DMM, organizing into teams to help support PSP, and having a plan to support the new staff who will be coming on July 1st.
Discussion focused on what PD will be needed for the coming year, particularly as we try to bring along staff who have not been part of a DMM cohort. Areas identified include outcomes, support for service coordinators, family assessments, RBI, and peer coaching.
Barriers or problems addressed or working on at regional level
How to best train new staff, maintaining implementation and working towards fidelity, administration change, PSP implementation.
Support from administrators for those providers that are hard to move forward in personal growth.
Pulling together all 0-3 staff to ensure everyone who has not participated in a Cohort will receive adequate training and support. Planning to have combined 0-3 provider meetings starting in 16-17.
Training staff who spend a small portion of their time in EA - they follow a more traditional model rather than coaching because of lack of training.
Not everyone (particularly SLPs, OTs, PTs, etc.) is putting FGRBI and caregiver coaching strategies into practice. Still having difficulty bringing along staff who have not been DMM participants. We are working on supporting our Regional Directors (middle management) in understanding FGRBI and caregiver coaching. Teams are having difficulty finding sufficient common time to collaborate and meet together.
Help or support needed from the State-level Implementation Team (SIT)
Sustaining FGRBI/rolling out to all AEA 9 providers, moving towards PSP (including training, coaching, learning from other regions).
Keeping the momentum and continuing support to move towards fidelity.
Working with the inter-rater reliability with the SS-OO-PP-RR and ensuring implementation teams have the opportunity to learn about it in depth as well (aka the 32 page manual).
Seeing the types of implementation and results data each AEA is collecting on top of what the state collects for the IA DMM work.
Focusing on service coordinator needs (family assessment, family outcomes, coaching in regard to resources, etc.).
IOWA Part C SSIP Phase III Year 1
50
Information gathered at each EAGL meeting is used to support the entire process of shifting the early
intervention system from child-focused services to a family focus. A database is being created to better
capture the information from these reports and meetings and tie them directly to actions taken. Relying on
Google Docs has become cumbersome with massive amounts of information to manage. Information
gathered helps continuous improvement efforts and plans for the future which are all in line with the State
Systemic Improvement Plan. Nothing indicates that the plan should change. The work is hard; vigilance is
never-ending.
Table 31. Selected Elements from Implementing Agency Level Evaluation Plan L3Q13, L3Q14
Evaluation Question How did systems change to accommodate this initiative? (L3Q13)
What are the barriers that impact agencies/service providers in supporting this model; what policies might impact usage of this model? (L3Q14)
Measure(s) Focus groups and interviews
Data Collection Methods Focus group and phone interviews with the regional implementation teams Administered by Iowa Department of Education
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S8, S11, S16, I6-I8, I13, I14, L5, L6, L8-L11
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) N/A not scheduled to occur during this reporting period.
Successes & Challenges:
Focus groups and interviews were originally scheduled as task for the external evaluator to take place Fall
2016, falling within the next reporting period. A change has occurred. The focus groups/interviews will be
administered by the Iowa Department of Education beginning Fall 2017 and results shared in the appropriate
reporting year.
It's a work in progress. I am learning more information. My main job is working with…the 3-5 teachers in daycare centers and right now I am working with families in home. So I feel like I am learning two-fold. I do have experience with families in home, but I have more experience with the centers and I'm working at this DMM model, putting it into practice with teachers as well...especially the coaching aspect of it and the problem solving aspect of it. So the teachers have more ownership of what they are working on with the child. In home, it definitely is kind of a challenge for me. So I am continuing to work on it.
IOWA Part C SSIP Phase III Year 1
51
State Level Evaluation
Table 32. Selected Elements from State Level Evaluation Plan L4Q15, L4Q16
Evaluation Question How did systems change to accommodate this initiative? (L4Q15)
How did early childhood systems change to accommodate this initiative across the state? (L4Q16)
Measure(s) Level of Collaboration Scale; Frey, Lohmeier, Lee & Tollefson, (2006).
Data Collection Methods Online survey collected from state-level implementation team members Administered by external evaluator
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S7, S16
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) No, this survey have been removed from the evaluation plan
Successes & Challenges: This is a very time intensive and expensive survey to administer and analyze. Results of the state level evaluation would have been used to support 3 short-term outcomes that have other, better measures already being used to show progress. System change is better measured by the Early Childhood Technical Assistance (ECTA) Center self-assessments, documents from team meetings and interviews. For these reasons, this item has been removed from the Systems Level Evaluation Plan for Early ACCESS. (It is still being used at the regional implementation level).
IOWA Part C SSIP Phase III Year 1
52
Table 33. Selected Elements from State Level Evaluation Plan L4Q17, L4Q17.1
Evaluation Question How do state implementation team members shift in their understanding of Active Implementation Frameworks? (L4Q17)
How do implementation team members shift in their use of Active Implementation Frameworks? (L4Q17.1)
Measure(s) Regional (RIT) and State Implementation Team (SIT) Reports State Work Team agendas & notes EAGL agendas & notes
Data Collection Methods RIT and SIT report updates collected via Google Docs State Work Team meeting notes collected via Google Docs EAGL meeting notes collected via Google Docs Administered by Iowa Department of Education
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S8, S9, S12, S17, I6, I7, I14, I15, L4, L8-L10
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) No, changed measure from Active Implementation (AI) Hub quizzes to more contextual measures and did collect data as scheduled..
Successes & Challenges: Making use of regularly occurring stakeholder meetings as well as dedicated implementation team meetings. Dedicated time for regional teams to update reports and specifically state what issues need to move to State Implementation Team. Use Google Docs to routinely update progress on regional and state activities. Annual joint face-to-face meeting between RIT and SIT facilitated by Florida State University where data are shared and team plans are reviewed. Even though implementation team work is better, it has been a struggle to maintain well functioning teams. Teams continue to need regular support which is addressed routinely at meetings and through webinars.
Regional Implementation Team (RIT) Reports are updated via Google Docs 5 times per year at the EAGL
meetings. One question on the RIT Reports asks the regional teams what help or support is needed from the
State-level Implementation Team (SIT). Information from this report is fed directly to the State Work Team
(SWT) (staff from the Iowa Departments of Education, Public Health, Human Services and Child Health
Specialty Clinics) to take 2 possible actions: (1) address the issue at the SWT level, or (3) move the issue to the
SIT level. Even when addressed at the SWT level, information is shared with the SIT at their regularly
scheduled meetings.
The State Work Team meets face-to-face twice a month with regular and frequent contact between meetings.
The SWT is responsible for overall functions of the Early ACCESS system and are all members of the State
Implementation Team. The State Implementation Team includes a larger group of early childhood
stakeholders and meets as needed (fall, winter, spring of this reporting period). A database is currently under
creation to better manage the large volume of written data collected during these meetings.
IOWA Part C SSIP Phase III Year 1
53
RESULTS
Evidence that State Implementation Team is increasing their knowledge of and use of active implementation
frameworks include during the current reporting period:
Development and use of written communication plans.
Rapid cycle problem solving using the Plan Do Study Act (PDSA) cycle used to address barriers that
impact statewide service provision
o Service coordination issues
o Eligibility criteria
Reviews of Regional Implementation Team Reports
Review of State Systemic Improvement Plan
Table 34. Selected Elements from State Level Evaluation Plan L4Q18a-b
Evaluation Question How did systems change to accommodate this initiative?
Governance (GV)
Personnel/workforce (PN)
Other components (L4Q18a-b)
Measure(s) ECTA Self-Assessments Implementation Teams, State Work Team, EAGL, ICEA meeting activities
Data Collection Methods Excel self-assessment forms collected annually for GV, PN Agenda and note documents from each group Administered by Iowa Department of Education
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S11, S13-S19, I9-I16, L5-L11
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes
Successes & Challenges: ECTA self-assessment form is very useful. Intensive TA from the Early Childhood Personnel Center very useful in self-assessment and plan development for Personnel/Workforce component.
DaSy and ECTA Frameworks Self-Assessment Comparison Tool is an Excel-based instrument that provides a
structure for states to record the current status of the state system and set priorities for improvement. It is a
companion to the ECTA and DaSy System Frameworks and was developed with extensive input from Part C
and Section 619 staff from partner states (http://ectacenter.org/sysframe/selfassessment.asp). Stakeholders,
including state staff, completed the self-assessments for the Governance (GV) and Personnel/Workforce (PN)
components completed during the current reporting year. The ECTA Center staff helped with the process.
The PN component was completed for the birth through age 5 early childhood system including Early ACCESS.
GV was completed specific to Early ACCESS.
IOWA Part C SSIP Phase III Year 1
54
RESULTS Figure 29. ECTA System Framework Self-Assessment Governance Results
Initial or baseline results indicate: 4 Quality Indicators have half or more of the quality elements implemented (score 6-7, green bar). 3 Quality Indicators have at least some elements in place and a few fully implements (score 3-5, yellow bar) 1 Quality Indicator has most of the elements not yet planned or in place (score 2, red bar) 0 Quality Indicators have no elements yet to be planned or in place.
Figure 30. ECTA System Framework Self-Assessment Personnel/Workforce Results
Initial or baseline results indicate: 0 Quality Indicators have half or more of the quality elements implemented (score 6-7, green bar). 8 Quality Indicators have at least some elements in place and a few fully implements (score 3-5, yellow bar) 3 Quality Indicator has most of the elements not yet planned or in place (score 2, red bar) 1 Quality Indicators have no elements yet to be planned or in place.
Updating the self-assessment is a fall activity. The first update is Fall 2016 and will be reported in the next
SSIP. The ECTA/DaSy Self-Assessment Comparison Tool (released 6/29/16) will be used to assess progress
toward achieving intended improvements to different aspects of the Early ACCESS infrastructure.
IOWA Part C SSIP Phase III Year 1
55
Table 35. Selected Elements from State Level Evaluation Plan L4Q19
Evaluation Question What is working? What are the positive impacts? What, if any, barriers? What would you do differently? (L4Q19)
Measure(s) Interviews
Data Collection Methods Phone interview with DE Early ACCESS state coordinator and CSPD consultant Administered by external evaluator
Outcomes Addressed (from Logic Models) (progress reported in report Section E) S16, S17, I10-I14, L4, L6, L10, L11
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) Yes, data collection has taken place; analysis scheduled to occur during next reporting period.
Successes & Challenges: Phone interviews simple to do and can be recorded. Qualitative data analysis is time intensive.
Table 36. Selected Elements from State Level Evaluation Plan L4Q20
Evaluation Question How are state IHEs impacted by this initiative? (L4Q20)
Measure(s) Interviews
Data Collection Methods Phone interviews with colleges and universities with pre-service programs that prepare students to be work in the field of early intervention Administered by external evaluator
Outcomes Addressed (from Logic Models) (progress reported in report Section E) I10-I12, L6, L7, L10, L11
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) N/A not scheduled to occur during this reporting period.
Successes & Challenges:
There's technology things that you have to get used to that are frustrating but it's a very worthwhile experience and I think in the end I'll be happy that I did it. The technology things that were challenging were like trying to get things uploaded. Making your camera work and how to set-up the camera at the house to make it effective. I still haven't figured that one out quite yet. The kids not wanting to look at it and feeling like it's right in the middle of everything.
IOWA Part C SSIP Phase III Year 1
56
Table 37. Selected Elements from State Level Evaluation Plan L4Q21
Evaluation Question What lessons learned can we share with other entities and states wanting to develop a focused PD initiative? (L4Q21)
Measure(s) Surveys, interviews, focus groups previously listed in the plan
Data Collection Methods Case study documentation using results of data collected over complete project period Administered by external evaluator
Outcomes Addressed (from Logic Models) (progress reported in report Section E) L4, L6, L7, L10, L11
Implemented as Intended? [yes/no] (detailed timeline in Evaluation Plan) N/A not scheduled to occur during this reporting period.
Successes & Challenges:
I really think that the philosophy and the ideas behind DMM are great. I think that they are what's best for families. It makes a lot of sense. It's just difficult to get from point A to point B. From where we are now to where we want to be. It's a big shift, but I think it will be worth it once we get there.
IOWA Part C SSIP Phase III Year 1
57
SECTION D: Data Quality Issues
There are no concerns or limitations related to quality or quantity of the data used to report progress or
results. Five levels of professional development evaluation data is collected (Guskey, 2000): (1) participant's
reactions; (2) participants' learning; (3) organization support and change; (4) participants' use of new
knowledge and skills; and (5) family outcomes. In addition, no single piece of data is used to assess progress
of the desired changes that will lead to the SiMR. Multiple measures are used at all levels of evaluation.
Collecting data from families means being aware of and sensitive to all that is going on in their life. Balancing
that with the required paperwork for participating in Early ACCESS and the consents to participate in the IA
DMM trainings, which include agreeing to be video recorded, adding "pre-test" surveys would be too much to
require of families and the service providers. This is taken into consideration when no "baseline" data is
available for a family measure.
When a selected measure does not answer an evaluation question, replacement measures are selected. For
example, using the quizzes in the AI Hub for learning Active Implementation Frameworks is not the best
measure for assessing change in actual understanding and use in the early intervention context. Regional and
State Implementation team reporting processes combined with Early ACCESS Leadership Group and Iowa
Council for Early ACCESS meeting notes provide evidence of contextual learning and use AI Frameworks.
Future plans remain the same, to hold regular evaluation team meetings where the evaluation plan is
reviewed and updated as needed.
I think it's a big learning process. I have an open mind and I think it is going to help families do better things for their kids. I especially like the components that we learned first about sitting back and observing what parents are already doing with their kids instead of coming in with their own agenda and saying "this is what you need to do. We need to fix everything because obviously you are not doing it right or I wouldn't be here”. But really looking and observing what they are already doing. Seeing what great things they are already doing and just trying to support and add more and embed interventions to the things that they do with their kids every day.
IOWA Part C SSIP Phase III Year 1
58
Section E: Progress Toward Achieving the Intended Improvements
The focus of Section E is on progress towards meeting outcomes which ultimately will lead to Iowa's State-
Identified Measurable Result (SiMR), increase in families who are confident and competent in helping their
child develop and learn, which then will improve outcomes for infants and toddlers served in Early ACCESS.
Tables are used to share progress towards each outcome identified in the logic models unless it was an
intermediate or long-term outcome where progress will be reported in future SSIPs. When that is the case,
text has been greyed out so you can still see the outcome but no progress is entered. Each table includes:
the outcome,
codes to indicate if it is a short-term (S), intermediate (I) or long-term (L) outcome,
evaluation codes which directly link to an evaluation question, measure and method of gathering evidence on
progress, and
accomplishments towards achieving the outcome.
Note: Data will not be duplicated between this section and Section C: Data on Implementation and Outcomes.
This section will explain progress and refer back to data in Section C tables where appropriate.
Progress on Instructional Practices: Shifting to Family-Guided Routines-Based Interventions and Caregiver
Coaching
Table 38. Evidence-Based Practices Short, Intermediate, Long-term Outcomes
Evidence-Based Practices — Short-term (S); Intermediate (I); Long-term (L) Outcomes
from Logic Models Eval Plan Code
Increased participant awareness, knowledge and skills of FGRBI and coaching caregivers.
Data from the Post-then-Pre evaluations shows participants are learning from the face-
to-face trainings. Participants are coming in with strong pre-existing learning and add to
that foundation during the face-to-face training sessions (see Table 15, Figures 11-15).
Service providers are applying what they have learned and improve performance over
the training period. Evidence is found in increased scores in video recorded
observations of home visits (see Table, 15, Figure 15).
Data from interviews (Table 22, Figures 17-18) supports the data from the Post-then-Pre Survey findings. Many participants made comments about the value of learning the SS-OO-PP-RR framework for coaching caregivers during home visits. Survey data shows the most learning occurred in practicing the use of this framework.
S1
L2Q5 (Table 10), L2Q6a-d (Table 15) L2Q8 (Table 22)
IOWA Part C SSIP Phase III Year 1
59
Table 38. Evidence-Based Practices Short, Intermediate, Long-term Outcomes (cont.)
Evidence-Based Practices — Short-term (S); Intermediate (I); Long-term (L) Outcomes
from Logic Models (continued) Eval Plan Code
Increased participant satisfaction and confidence in service provision.
Data from the second face-to-face training (Table 10, Figures 14-15), held approximately
6 months into the training cycle, indicate that providers are bringing in a large amount of
pre-existing learning, benefiting from the training and still have room for growth
(learning lag).
Data from provider interviews recorded at the second face-to-face training reveal that,
although challenging, using family-guided routines-based interventions and coaching
caregivers is working. The full transcripts are still being analyzed over the next reporting
period for more details. Interviews as well as anecdotal information from face-to-face
trainings point towards the need for continued support beyond the 9 month training
period. Providers want to further develop skills and knowledge with support from
trainers.
Combined with information from all data sources, it is clear that progress is being made
towards meeting this outcome with additional support created to make that happen.
S2 L2Q5.1, (Table 10) L2Q8, (Table 22)
Increased # of providers coaching families.
4 cohorts have been trained as of this reporting period. New cohorts will be added as
scheduled (Figure 16). S3 L2Q6.2 (Table 16)
Increased # of families satisfied with service.
Data collection did not start during this reporting period. Difficulties in getting families
to commit to an interview slowed progress in gather data for this outcome. S4
L1Q2a-b (Table 25)
Increased families' awareness, knowledge and skills of FGRBI and coaching.
Although interviews have not been conducted as of this reporting period, evidence from
the newly developed Caregiver Key Indicator tool shows promising results (Table 26,
Figures 19-20). Through review of recorded home visit, caregiver behavior is observed
and measured. When comparing the results of the Video 1 (baseline) to Video 2, there is
noticeable improvement. This measure was not used beyond the development and
testing during this reporting period. Update will be reported in subsequent SSIP reports.
S5 L1Q3 (Table 25), L1Q4 (Table 26)
Increased understanding of multiple evidence-based approaches for supporting FGRBI and coaching caregivers.
All agencies have brought up the desire to learn more about the Primary Service Provider (PSP) approach to teaming. Efforts to support learning included conversations at the Joint Implementation Team Meeting (regional and state teams) with Rush and Shelden (via Zoom), book studies at some agencies, holding electronic meetings with other states that have or are still using PSP teaming, and many discussions during regularly scheduled stakeholder meetings.
A few agencies are trying PSP teaming and bringing their learnings to the Early ACCESS Leadership Group (EAGL) meetings for discussion.
Florida State University staff join conversations at EAGL meetings.
Agencies who have invested in Rush and Sheldon coaching training bring learnings to the EAGL meetings for discussion.
Webinars facilitated by Florida State University bring in Robin McWilliam, Dathan Rush and M'Lisa Shelden to help Iowa service providers integrate the multiple sources of support.
S6 L2Q6.3 (Table 17)
IOWA Part C SSIP Phase III Year 1
60
Table 38. Evidence-Based Practices Short, Intermediate, Long-term Outcomes (cont.)
Evidence-Based Practices — Short-term (S); Intermediate (I); Long-term (L) Outcomes
from Logic Models (continued) Eval Plan Code
Increased provider proficiency in and use of coaching caregivers in FGRBI with fidelity.
Fidelity checks are done during the next reporting period and will be in next SSIP. I1
L2Q6a-d (Table 15)
Increased # of providers who are internal coaches.
Internal coach training occurs during next reporting period. I2 L2Q6.1 (Table 16)
Increased family participation in helping their child develop and learn.
As reported in S5 above, the newly developed Caregiver Key Indicator tool shows
promising results (Table 26, Figures 19-20). It is premature to say we are making
progress on this intermediate outcome.
I3 L1Q1 (Table 23) L1Q4 (Table 26)
Increased family proficiency in using FGRBI. I4 L1Q1, L1Q4
Increased integration of multiple evidence-based approaches for supporting FGRBI and coaching caregivers.
I5 L2Q6.3
Increased quality of practice by early intervention professionals. L1 L2Q6a-d, L2Q8
Increased families who are confident and competent in helping their child develop and learn (Iowa's Part C SIMR).
L2 L1Q1, L1Q2b, L1Q3, L1Q4, APR C4
Improved outcomes for infants and toddlers with disabilities. L3 APR C3
IOWA Part C SSIP Phase III Year 1
61
Progress on Implementation Practices: Using Active Implementation Frameworks Table 39. Implementation Strategies Short, Intermediate, Long-term Outcomes
Implementation Strategies—Short-term (S); Intermediate (I); Long-term (L) Outcomes
from Logic Models Eval Plan Code
Increased # of strategic partnerships.
The Levels of Collaboration Scale measures different characteristics of collaboration: (1)
networking, (2) cooperation, (3) coordination, (4) coalition, and (5) collaboration and
results are displayed as maps of these relationship characteristics (Figures 21-28) in the
Regional Implementation Teams (RIT). In addition, the maps show the changing numbers
of people in the in the networks. Over time we will be able to see if the members
increase and if the relationship move nearer to collaboration.
Information from the RIT Report updates supports the above data. Teams share who has
attended the meetings. Comparisons between updates will help identify changes in
strategic partnerships. Over time, there has been an increase in mid level management
participation as well as increases in early interventionists other than teachers such as
occupational therapists (OT), physical therapists (PT), speech language pathologists (SLP).
Other evidence of progress towards increasing partnerships includes the numbers of
invitations to present Early ACCESS work to groups and councils (e.g. teachers of the
deaf/hard of hearing and audiologists, Early Head Start/Head Start, Early Hearing
Detection and Intervention Advisory Council).
S7 L3Q9, L3Q10, L3Q11 (Table 27)
I would like more time to talk with my peer about the things that we are doing sometimes we kind of do things on the fly as far as trying to watch our videos. I'm lucky to have a peer that's very tech savvy and can help on that end of things because there's been some learning that's had to happen for technology for me…getting the files sent to Florida State for example. The first one said it was going to take seven hours, so my peer had to help me send those things off and we couldn't figure out why the file was so long and why it was going to take so long. So we had to have our technology consultant come in and help with those things. Learning that the IPad is not the thing to use to video tape because that was partly the size issue.
IOWA Part C SSIP Phase III Year 1
62
Table 39. Implementation Strategies Short, Intermediate, Long-term Outcomes (cont.)
Implementation Strategies—Short-term (S); Intermediate (I); Long-term (L) Outcomes
from Logic Models (continued) Eval Plan Code
Increased awareness and buy-in for evidence-based early intervention practices and implementation practices.
Occurs naturally as the number of partnerships increases the awareness of the work
increases.
Buy-in for implementing FGRBI and using Active Implementation Frameworks takes
time and deliberate actions to foster. Written administrative and participant
agreements have been modified as needed to increase buy-in. Creating written and
recorded products helps buy-in. Sharing information with the many groups that
implementation team members and State Work Team have contact with all
contribute to meeting this outcome.
Progress on building understanding, appreciation and buy-in for this work is slow,
steady and never ending. Iowa is in year 4 of training and shifting to statewide
implementation of evidence based practices. The need for improving awareness and
buy-in is still very apparent.
S8
L3Q12 (Table 29), L3Q13-14 (Table 31), L4Q17 (Table 33)
Increased knowledge and use of the 5 Active Implementation (AI) Frameworks.
Steady progress is being made in building knowledge and use of AI Frameworks.
Several approaches have been used including direct teaching through modules on
the AI Hub (http://implementation.fpg.unc.edu/) and presentations by State Work
Team members, Mary Beth Bruder from the Early Childhood Personnel Center and
Juliann Woods from Florida State University.
Using AI Frameworks and especially the continued development of implementation
teams takes a few key people to champion the work. Webinars helps support
growth in understanding and practice (Table 4, Figure 8). Most helpful is doing the
implementation work and then putting it in context of the frameworks. Simply
teaching what the frameworks are just introduces the concepts. It must be practiced
in the context of the system change to become a natural way for doing business.
Iowa is not there yet.
S9
L3Q12-12.1, (Table 29) L4Q17-17.1 (Table 33)
Increased readiness for implementation of EBPs.
At the very beginning of the IA DMM work, all agencies completed both practice and
technology needs assessments in order to develop professional development
components that align with participant and agency needs. From that point forward,
feedback from each cohort is used to continue alignment with needs, supporting
progress on this outcome.
Administrator, provider and trainer agreements evolve each year. Learning from the
previous years, agreements are tailored to address known barriers (e.g. time,
technology, data collection) up front in order to be better prepared for a training
cycle.
Each new cohort has higher levels of baseline skills observed during the first recorded
home visit done prior to any training (Figure 15). This shows that providers have
increased readiness for implementation of evidence-based practices.
S10
L2Q5 (Table 10), L2Q6a-d (Table 15), L2Q7 (Table 18)
IOWA Part C SSIP Phase III Year 1
63
Table 39. Implementation Strategies Short, Intermediate, Long-term Outcomes (cont.)
Implementation Strategies—Short-term (S); Intermediate (I); Long-term (L) Outcomes
from Logic Models (continued) Eval Plan Code
Increased support for PD at regional & state levels.
Interviews to gather data are not scheduled to occur during this reporting period
(Table 31).
ECTA System Framework self-assessments were done during this reporting period so
baseline data is available for change comparison next year.
Meeting notes will show that support from regional administrators has increased over
time with some negotiating over caseloads and contract hours to support the extra
work required as part of IA DMM.
At the beginning of this school year, Fall 2016, 3 agencies—separately from one
another—informed the state that service providers came to beginning-of-the-year
meetings and said they were all on board with the shift to using family-guided
routines-based intervention and caregiver coaching. Providers understand that this
shift is here to stay and that it is the "right" work to be doing. This is the end of year 4
and beginning of year 5 of implementation. Shift happens…and it takes time and
intense dedication to the process. These comments indicate a tipping point has been
reached. Providers and administrators realize this is not "one more thing" the state is
requiring today only to be gone tomorrow. However, this is not a statewide
sentiment and, although progress is evident, we still have a long way to go.
S11
L3Q13-14, (Table 31) L4Q18a-b
(Table 34)
Increased use of data related to implementation and intervention activities.
Using data is a constant conversation at Early ACCESS Leadership Group (EAGL), State
Work Team (SWT), Iowa Council for Early ACCESS (ICEA) and implementation team
meetings. Each audience has a different level of desire to use data and then
individuals within the groups have differences as well. Striking the right balance with
the right audience on data use has proven to be tricky.
Every June (since 2014) a Joint Implementation Team meeting is held with members
of the State Implementation Team (SIT), Regional Implementation Teams (RIT), State
Work Team (SWT), Florida State University and the external evaluator. Data sharing is
a natural part of these meetings so that the next round of training can be built from
what is learned. At the June 2016 event, a "data placemat" was used to assist in data
conversation and planning (see Figure 31 on following page).
EAGL meetings are used to share, examine, and collect regional and state level data.
In addition, these meetings are used to talk about what data would be useful for RIT
members to have. ICEA meetings do the same. The SWT serves as the group that
data flows through to the rest of the state.
Analyzing and sharing data more timely is a work in progress. Iowa does not have a
full-time data manager dedicated to Part C. Duties are shared with the part-time data
manager and the remaining State Work Team members.
S12
L3Q12.1 (Table 29), L4Q17.1 (Table 33)
IOWA Part C SSIP Phase III Year 1
64
Figure 31. Data Placemat (2-sided, 8.5 x 14) from June 15, 2016 Table 39. Implementation Strategies Short, Intermediate, Long-term Outcomes (cont.)
Implementation Strategies—Short-term (S); Intermediate (I); Long-term (L) Outcomes
from Logic Models (continued) Eval Plan Code
Active Implementation Stages, Drivers, and Improvement Cycles are carried out by skilled individuals with the expertise to help individuals, organizations, and systems successfully use evidence-based practices.
I6 L3Q12, L3Q12.1, L3Q13, L3Q14, L4Q17, L4Q17.1
Increased capacity for identifying, implementing, scaling up and sustaining evidence-based practices and programs for continuous improvement of the Early ACCESS system.
I7 L3Q12, L3Q12.1, L3Q13, L3Q14, L4Q17, L4Q17.1
Increase in skilled providers who feel supported at multiple levels to use evidence-based practices.
I8 L2Q7, L3Q13, L3Q14
AI Frameworks are used to make a persistent and consistent effort to bring about change and improve outcomes for children and families.
L4 L3Q12, L3Q12.1, L4Q17, L4Q17.1, L4Q19, L4Q21
Early ACCESS system is aligned to support practitioners’ efforts to use evidence-based practices.
L5 L2Q7, L3Q13, L3Q14, L4Q18a-b
State and regional policies promote implementation science and best practices. L6 L3Q13, L3Q14, L4Q18a-b, L4Q19, L4Q20, L4Q21
Diverse stakeholders' voices and experiences influence and guide implementation efforts. L7 L3Q9, L3Q10, L3Q11, L4Q18a-b, L4Q20, L4Q21
Reliable and valid data is regularly available for decision making at all levels: practitioners, managers of provider organizations, state departments, policy makers and funders.
L8 L3Q12.1, L3Q13, L3Q14, L4Q17.1, L4Q18a-b
A proactive and thoughtful style of solving problems is used routinely. L9 L3Q12.1, L3Q13, L3Q14, L4Q17, L4Q18a-b
IOWA Part C SSIP Phase III Year 1
65
Progress on High Quality Infrastructure: ECTA System Framework
Table 40. High Quality Infrastructure Short, Intermediate, Long-term Outcomes Governance
Governance Infrastructure
Short-term (S); Intermediate (I); Long-term (L) Outcomes from Logic Models Eval Plan Code
Staff and stakeholders increased knowledge of structures and partnerships that are needed to support an effective, efficient statewide service delivery system.
Focus groups and interviews with Regional Implementation Teams are not scheduled
to occur during this reporting period. Data will be shared in next reporting period.
ECTA System Framework Self-Assessment has been scored once (baseline) and an
update will be shared in next reporting period.
Data collection has taken place for state staff interviews; analysis is not scheduled to
occur during this reporting period.
Results from the baseline assessment analysis for the Governance Component
indicated a high priority was Quality Indicator 5: State and regional and/or local system
entities are designed to maximize meaningful family engagement in the development
and implementation of the system. A multi-agency task team of 6 people, half being
parents of children with disabilities, has formed to organize a system wide self-
assessment specific to family engagement. Activities are scheduled to occur during the
next reporting period.
S16
L3Q13-14 (Table 31) L4Q18a-b (Table 34) L4Q19 (Table 35)
Staff and stakeholders use knowledge and skills to implement plans.
The infrastructure improvement plan submitted during Phase II (Memorandum of
Agreement Action Plan) is used to ensure activities happen within the designated
timelines as much as possible.
Balancing infrastructure development and a shift to evidence-based practices which
must happen simultaneously is challenging. Having a written agreement
(Memorandum of Agreement) that binds 4 signatory agencies (Iowa Department of
Education, Department of Public Health, Department of Human Services, Child Health
Specialty Clinic) together as responsible for the Early ACCESS systems allows for
progress to be made no matter how small the steps may be at times.
S17
L3Q12, L3Q12.1 (Table 29) L4Q17, L4Q17.1 (Table 33) L4Q18a-b (Table 34) L4Q19 (Table 35)
IOWA Part C SSIP Phase III Year 1
66
Table 40. High Quality Infrastructure Short, Intermediate, Long-term Outcomes Governance (cont.)
Governance Infrastructure
Short-term (S); Intermediate (I); Long-term (L) Outcomes from Logic Models (continued) Eval Plan Code
Effective and well-coordinated governance infrastructure that provides enforceable decision-making authority to effectively implement the EA system.
I13 L3Q14, L3Q13, L4Q18a-b, L4Q19
Leadership advocates and leverages sufficient fiscal and human resources to support quality services.
I14
L3Q12, L3Q12.1, L3Q13, L3Q14, L4Q17, L4Q17.1, L4Q18a-b, L4Q19
Mechanisms are in place that facilitate clear communication, collaboration and relation-building with stakeholders and partners at all levels.
I15 L3Q12, L3Q12.1, L4Q17, L4Q17.1, L4Q18a-b
Early ACCESS system encourages, supports and requires implementation of effective practices.
L10
L3Q12, L3Q12.1, L3Q13, L3Q14, L4Q17, L4Q17.1, L4Q18a-b, L4Q19, L4Q20, L4Q21
EA system is self-sustaining with adequate resources to address the needs of individuals and organizations in the system.
L11 L3Q13, L3Q14, L4Q18a-b, L4Q19, L4Q20, L4Q21
Table 41. High Quality Infrastructure Short, Intermediate, Long-term Outcomes Personnel/Workforce
Personnel/Workforce Infrastructure
Short-term (S); Intermediate (I); Long-term (L) Outcomes from Logic Models Eval Plan Code
Staff and stakeholders increased knowledge of a CSPD.
Iowa received intensive technical assistance from the ECTA Center to create an Early
Childhood Comprehensive System of Personnel Development (CSPD) for people serving
children ages birth to five. Early ACCESS reviewed the larger plan and specifically
noted items that directly impact early intervention personnel. The vision for personnel
development was create as a statewide vision for all early childhood entities and the
visioning activity was done as an Early Childhood Iowa subcommittee activity.
Information about what an early childhood CSPD still needs to be shared around the
state even though a broad representation of stakeholders were involved in the
visioning process and development of the improvement plan for Personnel/Workforce
infrastructure improvement.
S13 L4Q18a-b (Table 34)
Staff and stakeholders increased knowledge of personnel available in the state to meet service needs.
Information on staff shortages in different disciplines that service infants and toddlers
with disabilities has been gathered and shared with a few. Further distribution will
occur during the next reporting period (see Table 38 on following page).
S14 L4Q18a-b (Table 34)
Staff and stakeholders use knowledge and skills to implement plan.
A core team of public and private early childhood agencies are responsible for moving
the work forward. Re-assessing will happen in the fall and change will be reported in
the next reporting year.
S15 L4Q18a-b (Table 34)
IOWA Part C SSIP Phase III Year 1
67
Table 41. High Quality Infrastructure Short, Intermediate, Long-term Outcomes Personnel/Workforce (cont.)
Personnel/Workforce Infrastructure
Short-term (S); Intermediate (I); Long-term (L) Outcomes from Logic Models (continued) Eval Plan Code
Effective and well-coordinated personnel/workforce infrastructure that guides planning, development, implementation and evaluation of a CSPD.
I9 L4Q18a-b
Personnel/workforce infrastructure ensures that infants and toddlers with disabilities and their families are provided services by knowledgeable, skilled, competent, and highly qualified personnel.
I10 L4Q18a-b, L4Q19, L4Q20
EA CSPD promotes both effective practices and the implementation of legal requirements. I11 L4Q18a-b, L4Q19, L4Q20,
EA has increased coordination between preservice programs and inservice training to ensure consistency of practices.
I12 L4Q18a-b, L4Q19, L4Q20
Early ACCESS system encourages, supports and requires implementation of effective practices.
L10
L3Q12, L3Q12.1, L3Q13, L3Q14, L4Q17, L4Q17.1, L4Q18a-b, L4Q19, L4Q20, L4Q21
EA system is self-sustaining with adequate resources to address the needs of individuals and organizations in the system.
L11 L3Q13, L3Q14, L4Q18a-b, L4Q19, L4Q20, L4Q21
Table 42. Licensed Area Education Agency and District Staff that Could Retire in Next Five Years (Based on Rule of 88 or Retirement at Age 65.
Retiring
FTE 2014 0-2
FTE 2014 3-5
FTE 2014 3-21
TOTAL PEOPLE
Total FTE % leaving
PK teacher 32 99.13 26.56 471.00 6.79%
OT 24 17.54 77.12 94.66 25.35%
PT 18 16.1 51.36 67.46 26.68%
Sch SW 38 18.52 209.83 228.35 16.64%
Sch Psych 62 5.52 286.87 292.39 21.20%
SLP 154 49.78 470.12 519.90 29.62%
Rather than based on numbers of retiring staff, the FTE from each AEA were used to calculate the actual percentages (based on number of staff in DE October Count data from recent 2014-2015 were used). Of the programs with greater than 25% loss, there is only one program in Iowa graduating OT, three programs graduating SLP, and four programs graduating PT. Drake and Allen College have started OT programs but graduating classes are still 4-5 years away.
It's been really good for me to be insightful about my own practice and just being more thoughtful about each visit when I go in, who is this child, who is this family, where were they, where are we going, that kind of thing.
IOWA Part C SSIP Phase III Year 1
68
Table 43. High Quality Infrastructure Short, Intermediate, Long-term Outcomes Accountability & Quality, Quality Standards, Data System, Finance
Accountability & Quality, Quality Standards, Data System, Finance Infrastructure
Short-term (S); Intermediate (I); Long-term (L) Outcomes from Logic Models Eval Plan Code
Staff and stakeholders increased knowledge of Accountability & Quality, Quality Standards, Data System, and Finance components.
S18 L4Q18a-b
Staff and stakeholders commitments to completing self-assessments by agreed upon deadlines.
Scheduled to occur in Fall 2017. S19
L4Q18a-b (Table 34)
Improvement plans are developed for all components. I16 L4Q18a-b
Early ACCESS system encourages, supports and requires implementation of effective practices.
L10
L3Q12, L3Q12.1, L3Q13, L3Q14, L4Q17, L4Q17.1, L4Q18a-b, L4Q19, L4Q20, L4Q21
EA system is self-sustaining with adequate resources to address the needs of individuals and organizations in the system.
L11 L3Q13, L3Q14, L4Q18a-b, L4Q19, L4Q20, L4Q21
SECTION F: Plans for Next Year
In 2016-2017, IA DMM will continue as planned:
Cohort 5 training for groups of 3 new providers or trios;
Cohort 5 training for first group of internal coaches;
Webinars for cohorts, implementation teams, Community of Practice;
FaceBook and website maintenance;
Meetings of implementation teams
Engagement of stakeholder groups
Use of task teams made up of diverse stakeholders to address needs as they arise
Evaluation will continue as written in the Systems Level Evaluation Plan for Early ACCESS with one change. We
will no longer hire an external evaluator and the State Work Team staff will take on those responsibilities.
Details and timelines are in the plan.
Systems are in place for addressing barriers to the work. Protocols and communication between trainers,
regions and the state are well established.
There is no additional support required at this time.
IOWA Part C SSIP Phase III Year 1
69
References
Borgatti, S.P., Everett, M.G., Freeman, L.C. (2002). Ucinet for Windows: Software for Social Network Analysis
[Computer Software]. Harvard, MA: Analytic Technologies. Frey et al. (2006). Measuring collaboration among grant partners. American Journal of Evaluation 27, 383-392.
DOI: 10.1177/1098214006290356 Gowen, J.W., Johnson-Martin, N., Goldman, B. D., & Appelbaum, M. (1989). Feelings of depression and
parenting competence of mothers of handicapped and nonhandicapped infants: A longitudinal study. American Journal on Mental Retardation, 94, 259-271.
Guskey, T. R. (2000). Evaluating Professional Development. Thousand Oaks, CA: Corwin Press. Hastings, R.P., & Brown, T. (2002). Behavior problems of children with autism, parental self-efficacy, and
mental health. American Journal on Mental Retardation, 107, 222-232. Pit-ten Cate, I., Kennedy, C., & Stevenson, J. (2002). Disability and quality of life in spina bifida and
hydrocephalus. Developmental Medicine and Child Neurology, 44(5), 317-322. Preskill, H. & Torres, R.T. (2000). Readiness for Organizational Learning and Evaluation Instrument (ROLE),
based on Preskill, H. & Torres, R. T. (1999). Evaluative Inquiry for Learning in Organizations. Thousand Oaks, CA: Sage. For more information about the survey, please contact Hallie Preskill at [email protected]. Reliability data for the instrument are can be found in: Preskill, H., Torres, R. T., & Martinez-Papponi, B. (1999). Assessing an organization’s readiness for learning from evaluative inquiry. Paper presented at the American Evaluation Association annual conference. Orlando, FL, November.
Scheel, M.J., & Rieckmann, T. (1998). An empirically derived description of self-efficacy and empowerment for
parents of children identified as psychologically disordered. American Journal of Family Therapy, 26, 15-27.
Woods, J., Kashinath, S. & Goldstein, H. (2004), Effects of embedding caregiver implemented teaching
strategies in daily routines on children’s communication outcomes. Journal of Early Intervention, 26, 175-193
One of the most significant things I have identified is that I would typically go in with a mindset of what I was going to do, which handout I would give the family, what intervention I would model and I was comfortable with that. So now I am having to be flexible and be really open to whatever they want to work on and whatever they want to talk about. That's hard.
Early ACCESS, IDEA Part C Early Intervention System Iowa Department of Education, March 11, 2107 Page 70
Inputs Outputs Short-term (direct results of activities and outputs)
Intermediate (changes in the actions or behaviors)
Long-term (anticipated results based on goals)
Outcomes
A. Design and implement professional development plan to build and expand capacity of providers in coaching caregivers and family-guided routines-based intervention (FGRBI)
Increased provider proficiency in and use of coaching caregivers in FGRBI with fidelity (I1) Increased # of providers who are internal coaches (I2) Increased family participation in helping their child develop and learn (I3) Increased family proficiency in using FGRBI (I4)
B. Develop and implement plan for tracking other related professional development that occurs within individual Area Education Agencies and signatory agencies that supports coaching caregivers and FGRBI
Comprehensive list of statewide professional development and capacity building opportunities in the Part C system Common data collection tools for FGRBI and coaching caregivers
Increased quality of practice by early intervention professionals (L1) Increased families who are confident and competent in helping their child develop and learn (Iowa's Part C SIMR) (L2)
Improved outcomes for infants and toddlers with disabilities (L3)
Activities
Increased integration of multiple evidence-based approaches for supporting FGRBI and coaching caregivers (I5)
Field tested coaching curriculum, coaching implementation competencies and learning modules Fidelity checklists Timeline for cohort participation Cohort peer mentoring pairs with technology supported action plans Participant videos of coaching and performance feedback Number of individuals accessing webinars, CoP, social media Number & type of print materials, EB practice guides, coaching curriculum, online learning modules
Increased participant awareness, knowledge and skills of FGRBI and coaching caregivers (S1) Increased participant satisfaction and confidence in service provision (S2) Increased # of providers coaching families (S3) Increased # of families satisfied with service (S4) Increased families' awareness, knowledge and skills of FGRBI and coaching (S5)
Increased understanding of multiple evidence-based approaches for supporting FGRBI and coaching caregivers (S6)
DE, IDPH, DHS, CHSC consultants, admin & written collaboration agreements Regional EA staff, admin Family & provider stakeholders State, regional, local admin & directors of related EC initiatives (e.g. ECI, EHS, MIECHV, CCRR) Institutes of higher education State, national consults (e.g. Woods, Maude, Edelman, Trivette) & TA Centers DEC Recommended Practices & other evidence based intervention practices Implementation science frameworks Standards, competencies, & policies Technology Data systems Federal & state early intervention/special education funds; regional PD funds
Early ACCESS Logic Model: New Instructional Practice
Evaluation: Measures/indicators, data collection, analysis and interpretation, feedback for program improvement, assessment & validation
External Factors/Context: staff turnover; other professional development efforts related to improving practice and outcomes; Area Education Agency system; signatory agency shared leadership structure
Appendix A, SSIP Phase III Year 1
Early ACCESS, IDEA Part C Early Intervention System Iowa Department of Education, March 11, 2107 Page 71
Inputs
DE, IDPH, DHS, CHSC consultants, admin & written collaboration agreements Regional EA staff, admin Family & provider stakeholders State, regional, local admin & directors of related EC initiatives (e.g. ECI, EHS, MIECHV, CCRR) Institutes of higher education State, national consults (e.g. Woods, Maude, Edelman, Trivette) & TA Centers DEC Recommended Practices & other evidence based intervention practices Implementation science frameworks Standards, competencies, & policies Technology Data systems Federal & state early intervention/special education funds; regional PD funds
Outputs Short-term (direct results of activities and outputs)
Intermediate (changes in the actions or behaviors)
Long-term (anticipated results based on goals)
Outcomes
A. Establish collaborative planning & implementation teams & networks
AI Frameworks are used to make a persistent and consistent effort to bring about change and improve outcomes for children and families (L4)
Early ACCESS system is aligned to support practitioners’ efforts to use evidence-based practices (L5)
State and regional policies promote implementation science and best practices (L6) Diverse stakeholders' voices and experiences influence and guide implementation efforts (L7)
Reliable and valid data is regularly available for decision making at all levels: practitioners, managers of provider organizations, state departments, policy makers and funders (L8)
A proactive and thoughtful style of solving problems is used routinely (L9)
Activities
D. Maintain communication with stakeholders, PD participants, EA leadership team, and council
Educational trainings/outreach events with stakeholders and public Documents, meeting notes, communication with stakeholders (e.g. IHEs, EA leadership group, ICEA, signatory agencies)
PD vision and plan developed with stakeholder input Administrator & provider participation agreements Completed AI Hub modules
Active Implementation Stages, Drivers, and Improvement Cycles are carried out by skilled individuals with the expertise to help individuals, organizations, and systems successfully use evidence-based practices (I6)
Increased capacity for identifying, implementing, scaling up and sustaining evidence-based practices and programs for continuous improvement of the Early ACCESS system (I7)
Increase in skilled providers who feel supported at multiple levels to use evidence-based practices (I8)
B. Design and implement plan to build capacity of state and regional teams in Active Implementation Frameworks
C. Assess readiness in the Area Education Agencies
Needs assessment Assess current EI policies, practices, etc. Assess and acquire resources needed to do the work
Regional and state teams with diverse stakeholder membership
PD components that align with the identified needs of participants Acquired or repurposed resources needed to do the work
Increased # of strategic partnerships (S7) Increased awareness and buy-in for evidence-based early intervention practices and implementation practices (S8)
External Factors/Context: staff turnover; other professional development efforts related to improving practice and outcomes; Area Education Agency system; signatory agency shared leadership structure
Increased knowledge and use of the 5 Active Implementation Frameworks (S9) Increased readiness for implementation of EBPs (S10) Increased support for PD at regional & state levels (S11) Increased use of data related to implementation and intervention activities (S12)
Evaluation: Measures/indicators, data collection, analysis and interpretation, feedback for program improvement, assessment & validation
Early ACCESS Logic Model: New Implementation Strategies
Early ACCESS, IDEA Part C Early Intervention System Iowa Department of Education, March 11, 2107 Page 72
Early ACCESS Logic Model: New High Quality Part C System
Inputs
DE, IDPH, DHS, CHSC consultants, admin & written collaboration agreements Regional EA staff, admin Family & provider stakeholders State, regional, local admin & directors of related EC initiatives (e.g. ECI, EHS, MIECHV, CCRR) Institutes of higher education State, national consults (e.g. Woods, Maude, Edelman, Trivette) & TA centers DEC Recommended Practices & other evidence based intervention practices Implementation science frameworks Standards, competencies, & policies Technology Data systems Federal & state early intervention/special education funds; regional PD funds
External Factors/Context: staff turnover; other professional development efforts related to improving practice and outcomes; Area Education Agency system; signatory agency shared leadership structure
Outputs Short-term (direct results of activities and outputs)
Intermediate (changes in the actions or behaviors)
Long-term (anticipated results based on goals)
Outcomes
A. Complete ECTA self-assessment for Governance (GV) component; engage stakeholders to develop and implement GV improvement plan
Effective and well-coordinated governance infrastructure that provides enforceable decision-making authority to effectively implement the EA system (I9) Leadership advocates and leverages sufficient fiscal and human resources to support quality services (I10) Mechanisms are in place that facilitate clear communication, collaboration and relation-building with stakeholders and partners at all levels (I11) Early ACCESS system
encourages, supports and requires implementation of effective practices (L10)
EA system is self-sustaining with adequate resources to address the needs of individuals and organizations in the system (L11)
Activities
Improvement plans are developed for AC, QS, DS and FN components (I16)
Effective and well-coordinated personnel/workforce infrastructure that guides planning, development, implementation and evaluation of a comprehensive system of personnel development (CSPD) (I12) Personnel/workforce infrastructure ensures that infants and toddlers with disabilities and their families are provided services by knowledgeable, skilled, competent, and highly qualified personnel (I13) EA CSPD promotes both effective practices and the implementation of legal requirement (I14) EA has increased coordination between preservice programs and inservice training to ensure consistency of practices (I15)
B. Complete ECTA self-assessment for Personnel/Workforce (PN) component; engage stakeholders to develop and implement PN improvement plan
C. Review and prioritize remaining ECTA System Framework component for future completion:
Accountability & Quality (AC) Quality Standards (QS) Data System (DS) Finance (FN)
Completed GV self-assessments with areas identified for improvement Coordinated plan to address needed improvements in GV component
Prioritized list for addressing all remaining system components Timeline for completing all remaining self-assessments
Staff and stakeholders increased knowledge of structures and partnerships that are needed to support effective, efficient statewide service delivery system (S13) Staff and stakeholders use knowledge and skills to implement plans (S14)
Staff and stakeholders increased knowledge of Accountability & Quality, Quality Standards, Data System and Finance components (S18) Staff and stakeholders commitments to completing self-assessments by agreed upon deadlines (S19)
Completed PN self-assessments with areas identified for improvement Coordinated plan to address needed improvements in PN component
Staff and stakeholders increased knowledge of a comprehensive system of personnel development (CSPD) (S15) Staff and stakeholders increased knowledge of personnel available in the state to meet service needs (S16) Staff and stakeholders use knowledge and skills to implement plan (S17)
Evaluation: Measures/indicators, data collection, analysis and interpretation, feedback for program improvement, assessment & validation
Iowa Department of Education pg. 73 Evaluation Plan Updates: 1/23/14; 8/20/14; 9/23/14; 2/6/15; 4/7/15; 8/18/15; 1/26/16; 2/9/16, 2/24/16, 2/25/16, 3/9/16, 2/9/17
Systems Level Evaluation Plan for Early ACCESS Plan updated 1/23/14; 8/20/14; 9/23/14; 2/6/15; 4/7/15; 8/18/15; 1/26/16; 2/9/16, 2/24/16, 2/25/16, 3/9/16, 2/9/17
Project periods are Federal Fiscal Years (FFY) 10/1 through 9/30 of each year with the exception of Project Year 1 which was 4/1/13 – 9/30/13.
System Level Evaluation Questions
Measures & Formative or Summative
Data Collection Methods Cohort Year 1
2012-2013 Year 2
2013-2014 Year 3
2014-2015 Year 4
2015-2016 Year 5
2016-2017 Year 6
2017-2018 Year 7
2018-2019
I. Program
Recipients Family Participants
L1Q1: How confident and competent do IA-DMM family participants feel about helping their child develop and learn? I3, I4, L2
Early Intervention Parenting Self-Efficacy Scale (EIPSES); Guimond, Wilcox & Lamorey (2008) Summative
Survey online or via mail done not sooner than 9 months from initial entry into IA-DMM By external evaluator and DE
1 Fall 2015
2 Fall 2015
3 Fall 2015
4 Summer 2016
5 Summer 2017
6 Summer 2018
7 Summer 2019
L1Q2a: Were family members satisfied with this type of intervention? What worked? What barriers? S4
Pre-interview survey Formative
For a sample of families in DMM, surveys online or via mail done after the EIPSES has been completed. By FSU
1 Spring 2016
2 Spring 2016
3 Spring 2016
4 Fall 2016
5 .Fall 2017
6 Fall 2018
7 Fall 2019
L1Q2b: Were family members satisfied with this type of intervention? What worked? What barriers? S4, L2 L1Q3: How confident do family participants feel about working with their child throughout the day? Has coaching changed how effective families feel about helping their child? S5, L2
Family interviews Formative & Summative
For a sample of families in DMM, after the end of the cohort DMM training cycle. By FSU
1 Spring 2016
2 Spring 2016
3 Spring 2016
4 Fall 2016
5 Fall 2017
6 Fall 2018
7 . Fall 2019
L1Q4: Are families demonstrating increased participation and proficiency in helping their child develop and learn? S5, I3, I4, L2
Caregiver Key Indicators Summative
Video coding for a sample of families in DMM By FSU
1
2
3
4 Summer 2016
5 Summer 2017
6 Summer 2018
7 Summer 2019
Red text = Outcome # from Logic Models Appendix B, SSIP Phase III Year 1
Iowa Department of Education pg. 74 Evaluation Plan Updates: 1/23/14; 8/20/14; 9/23/14; 2/6/15; 4/7/15; 8/18/15; 1/26/16; 2/9/16, 2/24/16, 2/25/16, 3/9/16, 2/9/17
System Level Evaluation Questions
Measures & Formative or Summative
Data Collection Methods Cohort Year 1
2012-2013 Year 2
2013-2014 Year 3
2014-2015 Year 4
2015-2016 Year 5
2016-2017 Year 6
2017-2018 Year 7
2018-2019
II. Direct Service
Provider Early ACCESS service providers
L2Q5: How do service providers' perceptions of their abilities to implement FGRB Interventions change? Shifts in various knowledge, skills, and attitudes/dispositions through PD efforts. S1, S10
Post-then-Pre Evaluation Survey Formative & Summative
Paper survey completed by each participant at the end of each face-to-face PD session By FSU
1 Fall 2013 Winter 2013
2 Spring 2013 Fall 2013
3 Fall 2014 Spring 2015
4 Fall 2015 Spring 2016
5 Fall 2016 Spring 2017
6 Fall 2017 Spring 2018
7 Fall 2018 Spring 2019
L2Q5.1 How do service providers' satisfaction and confidence in service provision change? S2
Post-then-Pre Evaluation Survey Summative
Paper survey completed by each participant at the end of 2nd face-to-face PD session By FSU
5 Spring 2017
6 Spring 2018
7 Spring 2019
L2Q6a: How do service providers change in their abilities to implement FGRB Interventions? Shifts in various knowledge, skills, and attitudes/dispositions through PD efforts. S1, S10, I1, L1
Family Routine Categories Formative & Summative
IA-DMM session summary form By FSU
1 Summer 2014
2 Summer 2015
3 Summer 2016
4 Summer 2017
5 Fall 2017
6 Fall 2018
7 Fall 2019
L2Q6b: How do service providers change in their abilities to implement FGRB Interventions? Shifts in various knowledge, skills, and attitudes/dispositions through PD efforts. S1, S10, I1, L1
FGRBI Key Indicators Formative & Summative
IA-DMM session summary form By FSU
1 Summer 2014
2 Summer 2015
3 Summer 2016
4 Summer 2017
5 Fall 2017
6 Fall 2018
7 Fall 2019
Red text = Outcome # from Logic Models
Iowa Department of Education pg. 75 Evaluation Plan Updates: 1/23/14; 8/20/14; 9/23/14; 2/6/15; 4/7/15; 8/18/15; 1/26/16; 2/9/16, 2/24/16, 2/25/16, 3/9/16, 2/9/17
System Level Evaluation Questions
Measures & Formative or Summative
Data Collection Methods Cohort Year 1
2012-2013 Year 2
2013-2014 Year 3
2014-2015 Year 4
2015-2016 Year 5
2016-2017 Year 6
2017-2018 Year 7
2018-2019
II. Direct Service
Provider (cont.)
Early ACCESS service providers
L2Q6c: How do service providers change in their abilities to implement FGRB Interventions? Shifts in various knowledge, skills, and attitudes/dispositions through PD efforts. S1, S10, I1, L1
SSOOPPRR Framework Formative & Summative
IA-DMM session summary form By FSU
1 Summer 2014
2 Summer 2015
3 Summer 2016
4 Summer 2017
5 Fall 2017
6 Fall 2018
7 Fall 2019
L2Q6d: How do service providers change in their abilities to implement FGRB Interventions? Shifts in various knowledge, skills, and attitudes/dispositions through PD efforts. S1, S10, I1, L1
Coaching Strategies Formative & Summative
IA-DMM session summary form By FSU
1 Summer 2014
2 Summer 2015
3 Summer 2016
4 Summer 2017
5 Fall 2017
6 Fall 2018
7 Fall 2019
L2Q6.1 Are there more service providers participating as peer internal coaches? I2
Training completion count (# and %) Formative
Online survey collected from EAGL at least annually + Participant lists from FSU By DE
Summer 2017 Summer 2018 Summer 2019
L2Q6.2 Are there more service providers participating as coaches to caregivers? S3
Training completion count (# and %) Formative
Online survey collected from EAGL at least annually + Participant lists from FSU By DE
Summer 2016 Summer 2017 Summer 2018 Summer 2019
L2Q6.3 Are service providers able to understand and integrate multiple evidence-based approaches for supporting FGRBI and coaching caregivers? S6, I5
Anecdotes from regional liaisons Formative
Notes from EAGL meetings & Regional Imp Team updates By DE
Summer 2014 Summer 2015 Summer 2016 Summer 2017 Summer 2018 Summer 2019
L2Q7: How do service providers assess the “readiness” of their agency for organizational change? S10, I8, L5
Readiness for Organizational Learning and Evaluation Instrument (ROLE); Preskill & Torre (2000) Formative
Online survey collected from each participant after their first face-to-face training By external evaluator
1 Fall 2013
2 Spring 2014
3 Fall 2014
4 Fall 2015
5 Fall 2016
6 Fall 2017
7 Fall 2018
Red text = Outcome # from Logic Models
Iowa Department of Education pg. 76 Evaluation Plan Updates: 1/23/14; 8/20/14; 9/23/14; 2/6/15; 4/7/15; 8/18/15; 1/26/16; 2/9/16, 2/24/16, 2/25/16, 3/9/16, 2/9/17
System Level Evaluation Questions
Measures & Formative or Summative
Data Collection Methods Cohort Year 1
2012-2013 Year 2
2013-2014 Year 3
2014-2015 Year 4
2015-2016 Year 5
2016-2017 Year 6
2017-2018 Year 7
2018-2019
II. Direct Service
Provider (cont.)
Early ACCESS service providers
L2Q8: What was the most significant change as reported by providers? S1, S2, L1
Reflection interview using the following questions: 1) what is working, 2) what is tough, 3) what I would do
differently next time, 4) what has been the most
significant change so far Summative
Video recording of participant reflection interviews By Larry Edelman completed at the second face-to-face training (about 6 months into the training cycle)
1 Spring 2014
2 Fall 2014
3 Spring 2015
4 Spring 2016
5 Spring 2017
6 Spring 2018
7 Spring 2019
Red text = Outcome # from Logic Models
Iowa Department of Education pg. 77 Evaluation Plan Updates: 1/23/14; 8/20/14; 9/23/14; 2/6/15; 4/7/15; 8/18/15; 1/26/16; 2/9/16, 2/24/16, 2/25/16, 3/9/16, 2/9/17
System Level Evaluation Questions
Measures & Formative or Summative
Data Collection Methods
Year 1 2012-2013
Year 2 2013-2014
Year 3 2014-2015
Year 4 2015-2016
Year 5 2016-2017
Year 6 2017-2018
Year 7 2018-2019
III. Implementing Agency Regional Implementation Teams for each AEA and DMPS
L3Q9: How do partnerships on the regional implementation teams change over the project period? S7, L7 L3Q10: What types of network/ collaboration changes occur? S7, L7 L3Q11: How and when do they begin to identify "others" who need to be part of their implementation team? S7, L7
Level of Collaboration Scale; Frey, Lohmeier, Lee & Tollefson, (2006). Summative
Online survey collected from each regional implementation team member annually beginning Summer 2014 Collaboration maps from survey results By external evaluator
Summer 2014 Fall 2015 Fall 2016 Fall 2017 Fall 2018 Fall 2019
L3Q12: How do implementation team members shift in their understanding of Active Implementation Frameworks? S8, S9, S17, I6, I7, I14, I15, L4, L10
Regional Imp Team (RIT) updates SWT agenda/notes EAGL conversations Formative
RIT Update documents SWT meeting notes EAGL meeting notes By DE
Fall 2016 Fall 2017 Fall 2018 Fall 2019
L3Q12.1: How do implementation team members shift in their use of Active Implementation Frameworks? S9, S12, S17, I6, I7, I14, I15, L4, L8, L9, L10
Regional Imp Team (RIT) updates SWT agenda/notes EAGL conversations Formative
RIT Update documents SWT meeting notes EAGL meeting notes By DE
Fall 2017 Fall 2018 Fall 2019
L3Q13: How did systems change to accommodate this initiative? (same as L3Q14) L3Q14: What are the barriers that impact agencies/service providers in supporting this model; what policies might impact usage of this model? S8, S11, S16, I6, I7, I8, I13, I14, L5, L6, L8, L9, L10, L11
Focus group and interviews Formative
Focus group with the regional implementation team with phone interviews with members unable to participate-to be completed in project years 3 and 5 By DE
Fall 2017 Fall 2018 Fall 2019
Red text = Outcome # from Logic Models
Iowa Department of Education pg. 78 Evaluation Plan Updates: 1/23/14; 8/20/14; 9/23/14; 2/6/15; 4/7/15; 8/18/15; 1/26/16; 2/9/16, 2/24/16, 2/25/16, 3/9/16, 2/9/17
System Level Evaluation Questions
Measures & Formative or Summative
Data Collection Methods
Year 1 2012-2013
Year 2 2013-2014
Year 3 2014-2015
Year 4 2015-2016
Year 5 2016-2017
Year 6 2017-2018
Year 7 2018-2019
IV. State State Implementation Team
L4Q17: How do state implementation team members shift in their understanding of Active Implementation Frameworks? S8, S9, S17, I6, I7, I14, I15, L4, L9, L10
Regional & State Imp Team reports Implementation Webinars Formative & Summative
RIT & SIT Update documents State Work Team agendas & notes EAGL agendas & notes Webinar participation lists By DE
Fall 2016 Fall 2017 Fall 2018 Fall 2019
L4Q17.1: How do implementation team members shift in their use of Active Implementation Frameworks? S9, S12, S17, I6, I7, I14, I15, L4, L8, L10
Regional & State Imp Team updates State Work Team meetings EAGL meetings Formative & Summative
RIT & SIT Update documents State Work Team agendas & notes EAGL agendas & notes By DE
Fall 2016 Fall 2017 Fall 2018 Fall 2019
L4Q18a: How did systems change to accommodate this initiative?
Governance (GV) Personnel/workforce (PN) Other components
S11, S13-19, I9-I16, L5-L11
ECTA Self-Assessments Formative & Summative
Excel self-assessment forms collected annually for GV, PN By DE Fall 2015 Fall 2016 Fall 2017 Fall 2018 Fall 2019
L4Q18b: How did systems change to accommodate this initiative?
Governance Personnel/workforce Other components
S11, S13-S19, I9-I16, L5-L11
Imp Team and State Work Team EAGL, ICEA meetings Formative & Summative
Agenda and note documents from each group. By DE Fall 2017 Fall 2018 Fall 2019
L4Q19: What is working? What are the positive impacts? What, if any, barriers? What would you do differently? S16, S17, I10, I11-I14, L4, L6, L10, L11
Interviews Formative
Phone interview with DE Early ACCESS state coordinator and CSPD consultant By external evaluator
Winter 2013 Spring 2014 Winter 2014 Winter 2015 Winter 2016 Winter 2017 Winter 2018
L4Q20: How are state IHEs impacted by this initiative? I10, I11, I12, L6, L7, L10, L11
Interviews Summative
Phone interviews with colleges and universities with pre-service programs that prepare students to be work in the field of early intervention By external evaluator
Spring 2017 Spring 2018 Spring 2019
Red text = Outcome # from Logic Models L4Q15 and L4Q16 deleted. Discontinued using Level of Collaboration Scale for State level evaluation Fall 2015.
Iowa Department of Education pg. 79 Evaluation Plan Updates: 1/23/14; 8/20/14; 9/23/14; 2/6/15; 4/7/15; 8/18/15; 1/26/16; 2/9/16, 2/24/16, 2/25/16, 3/9/16, 2/9/17
System Level Evaluation Questions
Measures & Formative or Summative
Data Collection Methods
Year 1 2012-2013
Year 2 2013-2014
Year 3 2014-2015
Year 4 2015-2016
Year 5 2016-2017
Year 6 2017-2018
Year 7 2018-2019
IV. State State Implementation Team (continued)
L4Q21: What lessons learned can we share with other entities and states wanting to develop a focused PD initiative? L4, L6, L7, L10, L11
Surveys, interviews, focus groups previously listed in the plan Formative & Summative
Case study documentation using results of data collected over complete project period By external evaluator
Fall 2017 Fall 2019
References Frey, B., Lohmeier, J.H., Lee, S.W. & Tollefson, N. (2006). Measuring collaboration among grant partners. American Journal of Evaluation, 27(3). 383-392 Guimond, A.B., Wilcox, M.J. & Lamorey, S.G. (2008) The Early Intervention Parenting Self-Efficacy Scale (EIPSES). Journal of Early Intervention, 30(4), 295-320 Preskill, H. & Torres, R. T. (1999). Evaluative Inquiry for Learning in Organizations. Thousand Oaks, CA: Sage.
Red text = Outcome # from Logic Models