Using Data to Assess and Improve SNAP E&T - SNAP to Skills · in using performance data to demonstrate programmatic achievements, challenges, and stakeholder return-on-investment\爀屲in

$Page 1: Using Data to Assess and Improve SNAP E&T - SNAP to Skills · in using performance data to demonstrate programmatic achievements, challenges, and stakeholder return-on-investment\爀屲in$
Using Data to Assess and Improve SNAP E&T

September 25, 2019

Presenter

Presentation Notes

Julie and Nola introduce themselves (or is David introducing? Clarify)

2

Agenda

1. Welcome and Overview – David Kaz and Julie Strawn2. National Perspective: Loretta Robertson, FNS Office of Employment and

Training Shelia Lewis and Christina Eckenroth, US

Department of Labor3. State Perspective: Andrew Prihar, State of Pennsylvania DHS Shavana Howard, State of Washington DSHS

4. Q&A

• What type of outcome data and other information should States consider collecting and analyzing to see if SNAP E&T is achieving good outcomes for participants?

• Where can they get this data and through which mechanisms? How can States use the data to tell their stories, show return on investment, and continuously improve their programs?

• How can they use that information to identify and manage effective third-party service provider partnerships?

Overview

3

Presenter

Presentation Notes

Thank you, David. Good morning (or afternoon depending on your time zone), I’m Julie Strawn with Abt Associates and will moderate the webinar. READ SLIDE We have a terrific slate of speakers to address these topics today. Our first speaker, Loretta Robertson, will speak about how the Food and Nutrition Service’s Office of Emp and Trg will use data reported by the States internally and externally, including sharing information on new national initiatives to make better use of that data both at the federal level and to share it back to the States. Our next speakers, Shelia Lewis and Christina Eckenroth, from the US Department of Labor, will share different methods for using State performance data to effectively communicate a State’s story as well as a new DOL/ED tool, the Maximizing Performance Data Development Model ,and state experiences from a recent project that explored strategies for using performance data to demonstrate programmatic achievements, challenges, and stakeholder return-on-investment. They will be followed by Andrew Prihar, from the state of Pennsylvania Dept of social services, who will describe how PA is making effective use of data to understand SNAP E&T outcomes and make program improvements, including setting and monitoring performance expectations for third-party partners. Their data system, the Commonwealth Workforce Data System (CWDS), is integrated with TANF and WIOA, as well as the State education agency, allowing them to capture an especially robust set of data. Our final speaker, Shavana Howard, will share how Washington State is collecting and using data from its well-established SNAP E&T program, known as BFET, to improve program outcomes. She will also cover the work DSHS completed with SJI to survey all third-party providers on ways to improve and streamline overall program administration. We will then have time for questions. And now I’ll turn it over to Loretta to start us off.

4

Loretta RobertsonSNAP Office of Employment and

Training

SNAP E&T Data Life Cycle

State Plan

Participation Report (FNS-583)

Annual Outcome Report

FNS Data Initiatives

FNS-583 Participation Reports

Data Driven Process Improvement Contract

Conduct Assessment of the Collection, Analytics, Validation and Reporting of SNAP E&T Data.

Data Driven Process Improvement Contract

Organizing, analysis and preparation of summary reports of FY 2018 data from Annual Outcome Reports

Two Deliverables• State Annual Report Summary – provides overview of State’s E&T program;

Employment and Earnings Outcomes; Participant characteristics; and State component measures for three of the State’s components.

• Technical Assistance Table – information from all three data sources (State Plan, FNS-583 and Annual Outcome Reports) to give a comprehensive overview of the State’s E&T Program. Includes: Key Program Characteristics – type; areas served; target population; pledge State status; total

program budget Program Data measures – comparing State Plan, FNS-583 and Annual Reports Component participation – comparing State Plan to actual participation Outcome Measures Identifies any technical issues with data and areas for possible program improvement

Assessment of SNAP E&T Data Collection

Contractor: Mathematica Three year project that will:

• Identify and describe the current State and Federal systems that collect, validate, and analyze E&T data;

• Assess the current and future E&T data needs of Federal, Regional, and State staff; and

• Recommend a plan to improve how Federal, Regional and State staff collect and use data for E&T program improvement and reporting.

Key Tasks include:• Conducting in-person meetings with National and Regional staff• Site visits in 7 States to observe their SNAP E&T business, data collection and

analysis processes; and• Creating process maps showing how SNAP E&T data flow through State,

Regional and Federal agencies.

Maximizing WIOA and Non-WIOA Performance Data CohortA c o l l a b o r a t i o n o f T h e D e p a r t m e n t o f L a b o r ’s E m p l o y m e n t a n d Tr a i n i n g A d m i n i s t r a t i o n ( E TA ) a n d T h e D e p a r t m e n t o f E d u c a t i o n ’s O f f i c e o f C a r e e r, Te c h n i c a l , A d u l t E d u c a t i o n ( O C TA E ) a n d R e h a b i l i ta t i o n S e r v i c e s A d m i n i s t r a t i o n ( R S A )

I want stakeholders

to understand

it

What story is this telling about our programs?

How do I analyze it? Data

The Participants: Cohort State Teams

• Arkansas• Indiana• Kentucky• Massachusetts• Minnesota• Virginia

11

Presenter

Presentation Notes

made up of agency representatives from the WIOA core programs


The Work

The participants explored methods for using performance data to demonstrate:

Programmatic achievements,

Challenges, and

Stakeholder return-on-investment (ROI)

Presenter

Presentation Notes

Different methods for using state performance data to effectively communicate a state’s story.

Goals of the Cohort

1. Strengthening state interagency partnerships and infrastructure

2. Identifying and exploring approaches to using WIOA and non-WIOA performance data to tell their state’s story

3. Developing a national resource that will assist states in using their performance data to demonstrate programmatic outcomes (aka: Performance Data Development Model)

4. Advancing a state-specific action plan

Developed solutions to issues affecting:

The alignment of program definitions,

The development and utilization of state-specific performance indicators,

State and local level understanding of the value of WIOA performance data, and

System-wide efforts to measure state/local performance in real time.

Cohort Outcomes

16

1. Maximizing Performance Data

Development Model

Self-Assessment Tool

Tools & Resources

Facilitation Guide

2. State Action Plans

Maximizing Performance Data Development Model

“Diagnose” where they are as organizations (Self-Assessment Tool);

Determine next steps for making progress (Resources and Tools).

Resourceto helpstates:

17

Presenter

Presentation Notes

in using performance data to demonstrate programmatic achievements, challenges, and stakeholder return-on-investment in using performance data to demonstrate achievements, challenges, and ROI, drawing from targeted resources and tools

Maximizing Performance Data:Self-Assessment Tool

• Four multiple-choice questions focused on: –Leadership and Partners–Using and Integrating Data–Resource Needs–Data Governance

• Set of five answer options corresponding to five levels

18

Key Questions

Leadership and Partners: To what extent do your state’s leadership and partners see the importance of data-informed decision making, to what degree do they BEGIN with data?

Using and Integrating Data: To what extent is your state using and integrating data to inform policy and operational decisions?

Resource Needs: To what extent has your state identified and met resource needs to enable you to effectively use data for decision making at all levels?

Data Governance: How would you describe your state’s data governance structure (i.e., formal approach to securing, managing, and sharing data)?

Maximizing Performance Data:Self-Assessment Tool

20

Maximizing Performance Data:Template for Next Steps

21

Maximizing Performance Data: Facilitation Guide

Step-by-step guidance for applying the Maximizing Performance Data Development Model, including convening a state team to: Complete Self Assessment Tool Determine concrete next steps for

making progress within the state

22

Maximizing Performance Data: Facilitation Guide

23

Team building: Build the concept that the four Core partners are in the same “family” and share the same broad goals.Data synthesis: Create more reports and dashboards using partners’ combined dataRegular examination of data by Core stakeholders (and other required partners as available)Integrate the referral process among the four WIOA Titles Establish a Workforce Data Trust for Data Governance of WIOA data

24

State Action Plans

State Action Plans• Develop a data analytics team to

meet the data analytics needs of stakeholders

• Implement management systems focused on coordination

• Optimize Data Analysis Systems

• Visualization Optimization and Automation

• Build a business intelligence capacity

• Identify and share analytic resources

• Develop data use licensing agreements

• Present WIOA performance data in a more integrated way to the public/stakeholders.

• Create opportunities for local areas and locations to view and utilize their specific performance data

Resources https://www.workforcegps.org/events/2019/06/25/16/01/Maximizing-WIOA-and-non-WIOA-Performance-Data

https://www.workforcegps.org/events/2019/06/25/16/01/Maximizing-WIOA-and-non-WIOA-Performance-Data

Using Data to Access and Improve SNAP E&T

Andrew Singh PriharSNAP E&T Policy Mgr.

PA Dept. of Human Services

9/24/2019 27

Agency Level – Strategic Planning

• State Labor Agency Data– Unemployment Rates– Regional High-Priority Occupation List

• Number of job openings• Average wage of position

• ABAWD Data– Track # of ABAWDs, by county

• Focus of PA’s initial 50/50 Partnerships– Look at waived counties, too

• How likely is the county to lose waiver in next year?

• Consider Expanding Scope of Existing Partnerships

9/24/2019 28

Presenter

Presentation Notes

We start our use of data in Pennsylvania early – integrating it into our decision-making process regarding which third-party organizations to partner with through SNAP E&T. We regularly review unemployment rates for each county and many cities, provided by BLS and our State Labor Agency. We also review the official high-priority jobs list published by each local workforce development board (WDB) to see which jobs – and thus what type of trainings – are in greatest demand. While we don’t exclusively base our partnership decisions on this data, it is one element we consciously consider. We also track unemployment rates to determine which counties are likeliest to lose waivers from the ABAWD time limit; we match this data with our internal MIS data on number of individuals not exempt from the time limit in each county, then focus our E&T efforts on the counties with the most at-risk ABAWDs. For example, while Philadelphia retained a county/city waiver when the ABAWD time limit returned in 2016, Pittsburgh did not. While Philadelphia is our largest city, we focused most of our initial 50/50 efforts on Pittsburgh, to ensure ample opportunities were available for able-bodied adults. In conjunction with the above-mentioned approaches, analyzing participation data can help states determine when to expand the scope of an existing partnership to new tracks or initiatives. This allows states to focus expansion funds on the highest-performing current providers.

Agency Level – Estimating Funding

• Participant Reimbursements as an entitlement– Issued by local County Assistance Offices (CAOs)– Annual/Lifetime limits– Matched with 50% budgeted state funds– Known as Special Allowances, or SPALs– Comparable level of services across programs

• SPAL Trending Report run monthly– Numeric and visual representations of YTD spend– Predictable “spikes” at beginning of fall/spring semesters– Breaks down by child care/non-child care SPALs

9/24/2019 29

Presenter

Presentation Notes

In a similar vein, data can also be used to project costs for a State’s E&T program – something which is especially important in Pennsylvania, where some portions of our E&T program are funded through dedicated state appropriations, and others through third-party match. In Pennsylvania, participant reimbursements are typically issued by the CAO as special allowances for supportive services, or SPALs. We match federal funds with dedicated state dollars to fund this SPAL program. We find this to be a compelling alternative to having each third-party partner issue their own participant reimbursements, because it ensures a uniform level of supportive services are issued, regardless of whether or not their individual 50/50 program can financially afford to issue the full gamut of supports. It also ensures participant reimbursements are not issued (or not issued) arbitrarily, and that participants receive proper notice and have due process to challenge denials. Operating the SPAL program is complex, and requires us to accurately estimate funding needs for what is essentially (from the participant’s perspective) an entitlement program. As our SNAP E&T program continues to grow, this becomes even more difficult – as past expenditures can only be so predictive of future growth. By regularly analyzing data on our SPAL program expenditures in the past year, growth from the prior year, and the expected effect of any policy changes occurring in the coming year, we are able to make well-informed annual estimates for our Budget office of how much in state and federal participant reimbursement funds we’ll need to operate the program in the coming year. While we’ve occasionally had to tweak this number mid-year in our state plan, our estimates since implementing this procedure have largely been in the ballpark of the final actual number, saving us from having to make much larger state plan and budgetary adjustments.

Case Level – Regular Data Sharing

• Third-Party Partner (Case Manager)– Responsible for CWDS system data entry

• Component Participation– Start and End Dates– Hours

– Needs to know when individuals lose SNAP eligibility

• Certification Office (IM Caseworker)– Responsible for determining (continued) SNAP eligibility– Responsible for issuing supportive services (SPALs)– Responsible for enforcing time limits on some components

• Job search• Job retention• Vocational education

9/24/2019 30

Presenter

Presentation Notes

Another important aspect of leveraging data is ensuring that it is collected accurately at the ground level. In Pennsylvania, all partners are required to use the Commonwealth Workforce Development System (CWDS) computerized case management system. CWDS is accessible to varying degrees by third-party partners, county assistance office (CAO) caseworkers, and the local WDB. Data from CWDS also synchronizes nightly with our eligibility MIS. This allows the third-party partner case manager and the CAO caseworkers share vital information, and ensures everyone is working off the same figures. Each component is tracked in CWDS – start and end dates are logged and weekly hours of participation are entered. This is crucial for CAO caseworkers to be able to issue SPALs that are reasonable, necessary, and directly related to participation in the E&T component. This data may also be used by CAO caseworkers to ensure that ABAWDs are meeting the work requirement – this ensures that ABAWDs who are meeting the requirement aren’t inadvertently cut off from food assistance due to the time limit – an administrative mistake that would also cause them to lose eligibility for SNAP E&T services. Finally, if a SNAP E&T participant’s SNAP benefits close for any other reason, or if TANF opens, the MIS sends that information over to CWDS, putting the client in “hold” status and alerting the third-party partner that services provided to that participant are no longer eligible for reimbursement under SNAP E&T.

Contract Level – Data Available

• CWDS Reports Run Monthly– For each contract

• Participant count• Graduation percentage• Employment percentage

– Reports can be accessed by DHS and Third Party Partner

• Monthly Participant Rosters– Sent by each provider– Accompanies invoice

• Invoice Data– Amount billed YTD

9/24/2019 31

Presenter

Presentation Notes

In Pennsylvania, the core way we use data in our SNAP E&T program is to monitor the performance of each third-party partner individually. CWDS includes a built-in reporting function, which allows the state agency to run both standard and ad-hoc custom reports. We can also grant this ability to third-party partners upon request. Our standard monthly report includes the number of individuals currently participating in each program at the end of the month, the number of individuals who exited each program in the current fiscal year, and of those “leavers,” how many attained positive outcomes – graduating from the program, obtaining employment, or both. We also require providers to send us a roster of all of their SNAP E&T participants each month with their invoice. This allows us to grasp how many individuals are being served by each program, and at what cost.

Contract Level – Program Integrity

• “Trust, but verify”

• Compare outcomes to contract benchmarks– Corrective action if/when necessary

• Annual in-person monitoring visits– Compare observations and interviews to data collected

• Compare invoices to participants served– CWDS vs. participant roster– Determining budget adjustments– Corrective action if/when necessary

9/24/2019 32

Presenter

Presentation Notes

SNAP E&T was originally signed into law by President Ronald Reagan, as part of the Food Stamp Act of 1985. So its only fitting that we in Pennsylvania refer back to President Reagan’s favorite Russian proverb – Doveryai, no Proveryai (“Trust, but Verify”) – to ensure the integrity of our third-party programs. By running monthly reports through CWDS, we are able to get an accurate view of which contracts are under- and overperforming expectations. The outcomes tracked in our monthly report align to those included in partner contracts, and so both the state agency and the partner can regularly check to see how close a given partner is to meeting their benchmarks for the year. Data reported can also be compared with our annual in-person monitoring visits to each third-party partner. Both a case review and client interviews provide our state agency an opportunity to ensure partners are correctly entering data, and not painting a rosier picture than they should. Luckily, both our in-person monitoring and our data have confirmed the wonderful success our 50/50 program has enjoyed. Should an individual provider ever begin to underperform, however, data could serve as the backbone of our rationale for corrective action. Our team also compares participant rosters submitted with each invoice to the actual participants listed in CWDS, ensuring that providers are not billing for individuals not enrolled in SNAP E&T, those not receiving SNAP benefits, or those who are receiving TANF. On the rare occasion that an ineligible individual is included on the roster– for example, someone who is open in CWDS, but in hold status – we reach out to the provider to have them submit a revised invoice and roster which does not bill for the individual, thus ensuring federal SNAP E&T funds are being spent as intended.

Program Level – Outcome Reporting

• CWDS Codes– Activity termination codes align with component measures

listed in PA’s State Plan– All three program groups use CWDS

• SNAP KEYS• SNAP Work Ready• SNAP 50/50

• CWDS data syncs with eligibility MIS– Allows CWDS records to be matched to SSNs

• State Labor Agency MOU

9/24/2019 33

Presenter

Presentation Notes

In Pennsylvania, we’ve aligned the termination codes used in CWDS, our state case management system, with those used for component outcome measures in our SNAP E&T state plan. When paired with an MOU with our state labor agency to access quarterly wage data, this makes it simple for us to compile our annual Outcome Report for submission to USDA. We simply use the data originating in CWDS to run reports of each quarter of SNAP E&T leavers – since the data from CWDS is sent over to our MIS, we simply pull the data from our MIS, allowing us to immediately link to each participant’s social security number and other identifying data. This file can then be sent over to our state labor agency to obtain quarterly wage data, as well as parsed by activity termination code to develop the component measure part of the report. Additionally, Pennsylvania is able to use our CWDS data to go beyond FNS’s required outcome reporting – as mentioned, we can run monthly reports, break down our internal data by county and by individual third-party partner, and match it with SPAL data. In addition, our state agency retains the ability to do a “deep dive” when necessary, scouring the case management system for data and details on individual participants and their situations when a question from the eligibility office or E&T provider arises.

34

BFET Using Data to Improve Outcomes

Building Strong Partnerships that Transform Lives and Communities in Washington StateShavana Howard, BFET Administrator

The Goal of BFET

The Basic Food Employment & Training (BFET) program is a group of agencies partnering together to provide employment, education, and training services to Basic Food recipients.

The goal of the program is to assist Basic Food recipients in obtaining a livable wage leading towards self-sufficiency.

35

Presenter

Presentation Notes

The BFET program helps the federally funded Basic Food recipients improve their employment prospects and wage potential through voluntary participation in job search, job training, education, or workfare activities. The goal is to assist Basic Food recipients in obtaining a livable wage leading towards helping them reach their full potential.

Brief History of BFET 2005 – 2007: The Birth of BFET BFET began as a pilot program in 2005 in the White Center community with:

o 1 Community collegeo 5 Community-based organizations

BFET grew in 2006 – 2007o 4 Community colleges joined in 2006o 2 Community colleges joined in 2007o 183 participants in 2006o 258 participants in 2007o 381 participants by the 4th quarter in 2007

2007 – 2012: The Infancy of BFET Strong emphasis on growth in:

o Quantity of contractors• Increase from 15 contractors in 2007 to 53 contractors in 2012• Increase in BFET participants from 381 in the 4th quarter of 2007 to 2142 in the 1st quarter of 2012

o Quality of services provided• The BFET program is focused on outcomes (How many of our participants receive employment with

a livable wage?)• BFET providers continually share best practices on how to achieve this outcome• Overall 58% entered employment during this period

Presenter

Presentation Notes

This snapshot of our history shows how important it has been to capture data for growth and funding.

BFET is an important part of Washington State’s comprehensive workforce development system serving the needs of low-income individuals, displaced workers, and employers.

TODAY

Individuals Served

Stimulating Washington’s

Economy

Building Self sustaining people &

Communities

• More than 84,000 individuals receiving food benefits participated in the BFET program since it began in 2005.

• The program has brought in over 20 million dollars each year to local CBOs and community & technical colleges to deliver BFET services.

• For participants exiting the BFET program, 60 percent enter employment and receive an average 10 percent wage increase over a year.

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=&url=http://www.clker.com/clipart-wrong-cross.html&psig=AFQjCNF5gCy6S5pueSf3Bg6nEIaeys0DQQ&ust=1500740980204452

38

Presenter

Presentation Notes

The data explains the story of how DSHS is intentional about our mission of transforming lives. It allows DSHS to target specific areas, demographics for expansion. As you can see the impact on the economy is huge, not only did people benefit from the added nutrition, but the BFET program is transforming lives with increasing wages! Future BFET service expansion depends the availability of partners with the necessary local funds as third party match to draw down federal funds, program and service needs in an area, and the overall BFET strategic plan. The Department continues efforts to recruit CBOs and tribal nations in unserved or underserved parts of the state to provide wraparound services to support participant success.

Three Types of Program Measures

39

1 • FNS National Reporting Measures

2 CSD Program Measures

3 Provider Success Measures

Presenter

Presentation Notes

BFET offers services in 34 of 39 counties in Washington State. DSHS partners with 30 CBOs in FFY18 and 35 CBOs in FFY19 (with multiple services locations) along with 34 Community and Technical Colleges. ORIA contracts with 12 CBOs, with some overlap with non-ORIA contracted CBOs, across the state to administer services to increase access to E&T services for refugees and immigrants.

FNS Annual Reporting MeasuresBFET Participants: In unsubsidized employment during the 2nd quarter after completion In unsubsidized employment during the 4th quarter after completion Median quarterly earnings in unsubsidized employment during the 2nd quarter after completion Number and percent completed training, education, work experience or on-the-job-training (OJT)

componentso Number and percentage who received BFET services and in unsubsidized employment upon

completiono Number and percentage that obtain a recognized credentialo Number and percentage in an education or training component that leads to a recognized credentialo Assess skills acquisition of BFET participants

Total number and percentage of:o Voluntary versus mandatory participantso Participants having achieved a high school degree, or equivalent, prior to participating in the BFET

program;o Able Bodied Adults without Dependents (ABAWDs);o Limited English Proficiency (LEP) participants;o Male versus female participants; ando Participants belonging in the following age ranges: 16-17, 18-35, 36-49, 50-59, and 60 or older.

ABAWD Data

40

CSD Program Measures

BFET collects data for 5 strategic reasons

Develop the annual Legislative Report Evaluate success of BFET services in meeting the agency

mission Evaluate provider contracted performance Measure success in meeting ESA and CSD strategic

measures Evaluate progress of the BFET Strategic Plan

41

Provider Success Measures

Number and percentage thato Entered employmento Retained employment (up to 1 year)o Receive recognized credentialso Gained skills

Starting wages Increase of wages Number and percentage with BFA closing due to employment (over

income or request closure) Evaluate contracted performance

42

Program Administration

43

Improved Outcomes

Increased Accountability

Performance MeasuresBudgets

Contract Renewals

Presenter

Presentation Notes

The BFET Providers require specific data to evaluate the success of their BFET program. This data is used to evaluate their success and identify performance gaps. Their needed information is:

DSHS BFET SURVEY

44

SJI (Seattle Jobs Initiative)

Independent

Obtain Objective Feedback

Improve Service Delivery

Benefits of Tracking Data

45

Justification of funds

Benefits of Tracking Data

Growth of th program

What services are needed

Growth of the program

Growth of the program

Justifying funds

Where services are

needed

Program success and

integrity

Data Tracked for Program Oversight Number of unduplicated participants Federal funds

o Broken down by 50/50 admin cost & support services Non-federal funds Average distribution of participants by county Average distribution of participants regionally Annual comparisons Average women served per month Average men served per month Participants by ethnicity Participants by age Activity by components used Employment outcomes

o Average number of women working per month, average age, and wageo Average number of men working per month, average age, and wage

Contract performance reports Contract monitoring

46

BFET Impacts

Expands capacity by leveraging funding already out in the communityImpact on the DSHS mission/visionBuilds community relationships between

providersNational recognition

Presenter

Presentation Notes

our state can look at Trends for future year goals and budget


Thank You!

[email protected]

49

Presenter

Presentation Notes

We cant to it alone. It takes building CBO and partner collaborations to develop a strong network of support

QUESTIONS?

50

Documents

Using Data to Assess and Improve SNAP E&T - SNAP to Skills · in using performance data to demonstrate programmatic achievements, challenges, and stakeholder return-on-investment\爀屲in