Implementing RCTs: What Agency Administrators Need to Know

OPRE Report # 2021-113

Sarah Hurley, Managing Director of Data Science, Youth Villages

Trevor Williams, Youth Services Administrator, Colorado Department of Human Services, Division of Child Welfare

2

Session Agenda

1. Learning objectives

2. Why do you need to conduct or support Randomized

Controlled Trials (RCTs)?

3. What conditions are necessary to conduct an RCT?

4. How do you manage an RCT?

I N T R O D U C T I O N

Poll:How many of you have experience

implementing an RCT?

4

Learning Objectives

Participants will be able to explain why RCTs are considered the gold standard of research design.

Participants will be able to determine whether an RCT is possible under certain conditions.

Participants will be able to design a plan for ensuring sound RCT management.

I N T R O D U C T I O N

Why should you do an RCT?

6

Two Key Features Make RCTs the Gold Standard for Establishing Evidence of Program Impacts—First:

Random Assignment = Assigning participants to treatment and comparison groups using a coin toss or lottery system

Random assignment is the method most likely to create comparable groups:

randomly assigned groups are nearly always comparable on characteristics you can measure and

likely to be comparable on characteristics that are difficult to measure.

W H Y D O A N R C T ?

7

Two Key Features Make RCTs the Gold Standard for Establishing Evidence of Program Impacts—Second:

Intent-to-Treat Analysis:

Once random assignment occurs, outcomes are evaluated based on group assignment rather than treatment received

Impacts are more difficult to determine when:

those assigned to the treatment group do not get treatment;

those assigned to the control group receive treatment.

W H Y D O A N R C T ?

8

Considerations for Random Assignment

Think carefully about where to place random assignment within the referral, recruitment, and consent processes.

Depending on sample size, the split between treatment and control groups may not be 50/50.

Consider options among various types of random assignment.

W H Y D O A N R C T

Types of Random Assignment

9

W H Y D O A N R C T ?

Simple Random Sample

X O X OX O X OX O X OX O X OX O X O

Treatment Group – 7 X; 3 OControl Group – 4 X; 6 O

Stratified Random Sample

X X O OX X O OX X O OX X O OX X O O

Treatment Group – 5 X; 5 OControl Group – 5 X; 5 O

Instead of sampling individuals, you can randomly assign teams, offices, schools, or other entities

Would anyone like to share examples of differences in RCT methodology to meet local needs?

10

What conditions are necessary to conduct an RCT?

12

The Program or Service Must Be Well Defined

The theory of change connects the program or service to the outcomes.

The program manual or model specifies the “who, what, when, where, and how” of the activities.

The manual or model may include tools to monitor program fidelity.

N E C E S S A R Y C O N D I T I O N S F O R R C T

13

Conditions that Support a Random Assignment Study

Not all who need a program can receive it due to limited resources.

The impact of the program is unknown.

Under these conditions, an RCT is a fair way to distribute a limited resource AND will establish rigorous evidence of

program effectiveness.

N E C E S S A R Y C O N D I T I O N S F O R R C T

14

Infrastructure for Conducting Research—Human Resources

Staff who know your agency, program, and/or target population AND who “speak research”

Staff to manage your agency’s participation in the study

Staff to implement the study (e.g., recruit subjects, get informed consent, gather baseline and/or follow-up data)

N E C E S S A R Y C O N D I T I O N S F O R R C T

15

Infrastructure for Conducting Research—Administrative Data

Data to identify potential study subjects and establish eligibility

Data to describe prior contact with the child welfare system

Service data for treatment and control groups

Linked administrative data to measure outcomes

N E C E S S A R Y C O N D I T I O N S F O R R C T

16

Strong Collaborative Ties with Key Stakeholders Are Critical

Child welfare agency

Service providers or program developers

Evaluators

Funders

Other key stakeholders

N E C E S S A R Y C O N D I T I O N S F O R R C T

Poll:Do you feel that conditions in your

agency/environment would support doing an RCT to test the effectiveness of your program?

17

What are the potential ethical issues in RCTs?

Up to half of study participants will bedenied entry into the program.

How do you justify that?

20

Potential Hidden Costs of Providing Programs with Limited Evidence

The potential of harm exists.

The program may not be any better than business as usual.

Change can take resources away from existing programing and/or strain relationships with stakeholders and staff.

N E C E S S A R Y C O N D I T I O N S F O R R C T

21

Collaborate with Stakeholders to Secure a Strong Commitment to Random Assignment

Secure agreement from agency decisionmakers including the legal department on evaluation questions, methods, random assignment, and data use.

Engage frontline staff and study participants to understand the study and the benefits to rigorous evaluation of impacts.

Be prepared to explain the study, including the justification for random assignment, to legislators, media, and other relevant stakeholders.

N E C E S S A R Y C O N D I T I O N S F O R R C T

How do you manage an RCT?

Invest in Planning for an RCT

Engage stakeholders to create a plan.

Clarify the responsibilities of each party.

Determine the resources required from each party.

Schedule touchpoints to share information and celebrate study progress.

M A N A G I N G A N R C T

23

24

Dedicate Specific Resources to the RCT Project

Identify a project manager/coordinator.

Ensure staff dedicate sufficient time to the project.

Allocate adequate financial resources.

M A N A G I N G A N R C T

25

Focus on Training for Staff Involved with the Study

Leadership staff need to be knowledgeable about the “why” behind the RCT.

Frontline staff involved in study recruitment and/or data collection need clear direction on study protocols.

Supervisors and local leaders also need to understand study protocols to fix barriers.

A plan is needed for staff turnover.

M A N A G I N G A N R C T

26

Provide Relevant Study Information to Other Stakeholders

IT staff providing study data need to understand the study’s aim and purpose.

External stakeholders need training on study aspects relevant to their work.

M A N A G I N G A N R C T

27

Strategies for Monitoring Study Implementation

Determine what issues to measure and monitor based on your group’s role in the RCT.

Schedule recurring metric review meetings with relevant staff and leadership.

Course-correct as necessary.

Celebrate victories.

M A N A G I N G A N R C T

Is anyone willing to share strategies for managing an RCT?

What are common challenges in implementing an RCT?

30

The Challenge of Study Recruitment

The expectation:

C O M M O N C H A L L E N G E S

The reality:

Examples of Management Tools for RCT Recruitment

31

C O M M O N C H A L L E N G E S

• Daily updates to leadership staff

• Incentives when milestones were achieved (e.g., recruiting participant number such as 500 or 1,000, big incentive for number 1,300)

• Estimates of effort required to reach study recruitment goals

32

Managing Implementation of an RCT

Focus study recruitment on the identified targetpopulation.

Ensure the treatment group gets treatment and thecontrol group does not.

The program is implemented as described in the manual,without any changes or tweaks along the way.

C O M M O N C H A L L E N G E S

33

Manage Stakeholder Expectations

RCTs are not built in a day.

RCTs measure program impact but may not answer other questions of interest.

RCTs don’t always find impacts.

C O M M O N C H A L L E N G E S

Does anyone care to share other challenges they have faced in implementing an RCT?

34

Lessons Learned

Go in with eyes wide open.

Take time to prepare.

Be ready for the unexpected.

Be kind to yourself and your stakeholders.

C O N C L U S I O N

35

Thank you for your commitment to building evidence of what works to help children, families, and young adults!

This report is in the public domain. Permission to reproduce is not necessary. Suggested citation: Urban Institute et al. (2021). Slide Deck Session 7: Implementing RCTs: What Agency Administrators Need to Know - Child Welfare Evidence-Building Academy. OPRE Report 2021-113, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.