Implementing and Validating a Revised TQRIS: A North

Carolina Update

Edna Neal Collins

Jani Kozlowski

Smart Start National Conference

May 1, 2013

QRIS Advisory Committee

• Met September 2009 – June 2012

• Diverse group of stakeholders• For-profit, non-profit, and government sponsored ECE providers

• State and local early education partners

• Analyzed the current Quality Rating & Improvement System (QRIS) with the goal of creating recommendations that will guide the evolution of the system over the next 10 years.

• Drafted short-term and long-term recommendations reflecting a statewide vision for quality early care and education systems and programs. 

• Recommendations directed to DCDEE, the Child Care Commission and other key system partners.

Considerations

All components of a QRISStandardsAccountability and assessmentsProgram and provider outreach & supportFinancial incentives linked to compliance with standardsConsumer education

Specific areas of emphasisCultural competenceInclusion of children with disabilitiesAlignment with Foundations

Systems that support the QRIS and programs

Advisory Committee Work Groups

Ad hoc group on QRIS Structure

Education and Professional Development

Evaluation of the Overall System

Finance

Infant-Toddler

Program Assessment

Program Standards

Recommendation Process

HomeworkStudied North CarolinaLooked at research and national reportsLooked at other statesListened to resource persons

Extensive review of recommendationsWork group ideasAnalysis (who affected, resources, timeframe, RTT,

etc.)Reviews by ChairsReviews in whole group

QRIS Structural Considerations

• Raise the floor and raise the ceiling.• Encourage continuous improvement.• Distinguish different levels and logical

“stair steps”.• Allow for child care provider choice.• Make it as simple as possible.• Integrate work group recommendations.• Provide a starting point for discussion.

Conceptual Framework for QRIS

Program & Environment

Ratios &Group Size

Education & Professional

Development

POLICIESAdmin Practices,Family Engagement,Cultural Responsiveness,Inclusion, PD Plans

SPACE,SAFETY,

COMPLIANCE, etc.

Block Requirements (increase with levels)

Core Requirements (maintained at all levels)

1st Level(Basic Licensing Requirements)

Conceptual Framework for QRIS

Program & Environment

Ratios &Group

Size

Education &

PD

POLICIESAdmin Practices, Family Engagement,Cultural Responsiveness, Inclusion, PD Plans

SPACE, SAFETY,

COMPLIANCE, etc.

1st

Level

Block Requirements (increase with levels)

Core Requirements (maintained at all levels)

Basic Licensing Requirements

2nd

Level

Specialization

Could Include CSEFEL

Infant Toddler

School Age

STEM The Arts ETC.

All requirements of Level 1, plus points that lead toward the Level 3 standards. Points must be earned in at least 2 of these 4 categories

(Program & Environment; Ratios & Group Size; Education & PD; Specialization)

Conceptual Framework for QRIS

Program & Environment

Ratios &Group

Size

Education &

PD

POLICIESAdmin Practices, Family Engagement,Cultural Responsiveness, Inclusion, PD Plans

SPACE, SAFETY,

COMPLIANCE, etc.

1st

LevelBlock Requirements (increase with levels)

Core Requirements (maintained at all levels)

Basic Licensing Requirements

2nd

Level

Specialization

Could Include CSEFEL

Infant Toddler

School Age

STEM The Arts ETC.

All requirements of Level 1, plus points that lead toward the Level 3 standards. Points must be earned in at least 2 of these 4 categories

(Program & Environment; Ratios & Group Size; Education & PD; Specialization)

3rd

Level*

Program &

Environment

Ratios &

Group Size

Ed/ PD

Requirements in P&E, Ratios/Group Size, Ed/PD, PLUS Specialization

* ERS or other quality measure required at this level. Also, it is expected that program accreditation will be added to the model during the pilot phase, most likely at levels 3-5.

Conceptual Framework for QRIS

Program & Environment

Ratios &Group

Size

Education &

PD

POLICIESAdmin Practices, Family Engagement,Cultural Responsiveness, Inclusion, PD Plans

SPACE, SAFETY,

COMPLIANCE, etc.

1st

Level

Block Requirements (increase with levels)

Core Requirements (maintained at all levels)

Basic Licensing Requirements

2nd

Level

Specialization

Could Include CSEFEL

Infant Toddler

School Age

STEM The Arts ETC.

All requirements of Level 1, plus points that lead toward the Level 3 standards. Points must be earned in at least 2 of these 4 categories

(Program & Environment; Ratios & Group Size; Education & PD; Specialization)

3rd

Level*

Program & Environment

Ratios &Group Size

Ed/ PD

4th

Level*

All requirements of Level 3, plus points that lead toward the Level 5 standards.

Points must be earned in at least 2 of these 4 categories (Program & Environment; Ratios & Group Size; Education & PD; Specialization)

Potential Programs of Distinction starting at 4th Level

Requirements in P&E, Ratios/Group Size, Ed/PD, PLUS Specialization

* ERS or other quality measure required at this level. Also, it is expected that program accreditation will be added to the model during the pilot phase, most likely at levels 3-5.

Conceptual Framework for QRIS

Program & Environment

Ratios &

Group Size

Education &

PD

POLICIESAdmin Practices, Family Engagement, Cultural Responsiveness, Inclusion, PD Plans

SPACE, SAFETY, COMPLIANCE,

etc.

1st

Level

Block Requirements (increase with levels) Core Requirements

(maintained at all levels)

Basic Licensing Requirements

2nd

Level

Specialization

Could Include CSEFEL

Infant Toddler

School Age

STEM The Arts ETC.

All requirements of Level 1, plus points that lead toward the Level 3 standards. Points must be earned in at least 2 of these 4 categories

(Program & Environment; Ratios & Group Size; Education & PD; Specialization)

3rd

Level*

Program & Environment

Ratios &Group Size

Ed/ PD

4th

Level* All requirements of Level 3, plus points that lead toward the Level 5 standards.

Points must be earned in at least 2 of these 4 categories (Program & Environment; Ratios & Group Size; Education & PD; Specialization)

Potential Programs of Distinction at the 4th and 5th Levels

5th

Level*

Program &

Environment

Ratios &

Group Size

Ed/ PD

Requirements in P&E, Ratios/Group Size, Ed/PD, PLUS Specialization

Requirements in P&E, Ratios/Group Size, Ed/PD, PLUS Specialization

* ERS or other quality measure required at this level. Also, it is expected that program accreditation will be added to the model during the pilot phase, most likely at levels 3-5.

Executive Summary

Process Recommendations

• Consider data about how programs meet current requirements.

• Map current and proposed requirements onto the proposed conceptual framework.

• Collect a lot of input from stakeholders about both the requirements of the new system and how it should be implemented.

• “Test” the proposed revisions using existing data and targeted data collection.

• Project how programs would be distributed across a proposed new structure or structures before finalizing the revised TQRIS.

Implementation Plan

Executive Summary available at http://ncchildcare.dhhs.state.nc.us/PDF_forms/QRIS-Advisory-Committee-Executive-Summary.pdf

Spreadsheet of specific recommendations provided to DCDEE

DCDEE staff coordinate feedback, validation, and implementation process. Integrate recommendations to create a coherent

system. Refer some specific recommendations directly to

Child Care Commission Refer system-level recommendations to system

partners.

Race to the TopEarly Learning

Challenge

Building on the Recommendations

RttT-ELC

• U.S. Department of Education and U.S. Department of Health & Human Services

• Total federal funding of $500 million

• Award Period: January 2012 – December 2015

• 9 winners: NC, MA, WA, DE, OH, MD, MN, RI, CA

• NC Award: $69,991,121

• Primary focus: School readiness for children at risk

RttT-ELC and the QRIS

• QRIS as a key component of a system to promote school readiness for all children.

• QRIS-related activities in NC grant• Validating and implementing a revised TQRIS

• Developing a new program quality measure for use in a TQRIS

• Strengthening TA and PD related to possible QRIS quality indicators

• Strengthening the workforce

• Expanding the QRIS & moving programs to higher levels

What is TQRIS Validation?

Validation of a QRIS is a multi-step process that assesses the degree to which design decisions about program quality standards and measurement strategies are resulting in accurate and meaningful ratings.

Zellman, G. L. & Fiene, R. (2012). Validation of Quality Rating and Improvement Systems for Early Care and Education and School-Age Care, Research-to-Policy, Research-to-Practice Brief OPRE 2012-29. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

TQRIS Validation

(Zellman & Fiene, 2012)

1. Examine the validity of underlying constructs.• Expert and empirical support• “Stakeholder validity”

2. Examine the measurement strategies and psychometric properties of measures.

• Do they perform as expected?• Do the “cut scores” differentiate levels appropriately?

3. Assess the outputs of the rating process.

4. Relate the ratings to children’s developmental outcomes (and other important outcomes).

Pilot Study and Validation Study

• Study the impact of recommended changes.

• Use results to develop blocks, points, weighting, and standards for each level.

QRIS Advisory Committee

Recommendation for Pilot Study

• Identify quality features that distinguish between programs at higher levels.

• Relate program features and quality ratings to children’s progress, especially children with High Needs.

Race to the Top Early Learning

Challenge - TQRIS Validation

Study

Pilot/Validation Study

Revised TQRIS Pilot and Validation Study (funded by RttT-ELC)

• Study the impact of recommended changes.

• Use results to develop blocks, points, weighting, and standards for each level.

• Identify quality features that distinguish between programs at higher levels.

• Relate program features and quality ratings to children’s progress.

• Study the impact of.

Validation Study Phase 1

• Gather information from stakeholders to inform the creation of alternative TQRIS models for testing.

• Generate 3-4 alternative models for testing, consistent with the Conceptual Framework, including Core Requirements, Blocks, Points, and Specializations.

• Test 3-4 alternative models using existing data, targeted data collection, and additional stakeholder input.

• Use results of this process to choose one model for further testing and validation in Phase II.

Validation Study Phase 2 (subject to approval)

• Validate the proposed new Block levels.• Examine WHICH quality indicators do the best job of

distinguishing among programs at the higher levels of quality.

• Test reliability and validity of a few new components.• Conduct a pilot study, including assessment of

relationship between quality ratings in the revised TQRIS and child outcomes.

• Generate data needed to make adjustments to the revised TQRIS before implementation.

Recent Work

• DCDEE Staff mapped individual recommendations onto conceptual framework.

• QRIS Advisory Committee Work Group Chairs gave feedback.

• Contract for Phase I of Validation Study awarded to FPG.

• Stakeholder interviews completed.

• Provider survey launched.

• Focus groups in progress.

Challenges to Discuss

• Provider buy-in

• Balancing consistency and flexibility

• Supporting continuous quality improvement

• Supporting and recognizing the highest quality

• Weaving in all aspects of quality

• The role of accreditation