DEI Evaluation Synthesis Report (Year 1) December 13, 2011 In partnership with Altarum Institute Berkeley Policy Associates Mathematica Policy Research

DEI Evaluation Synthesis Report (Year 1)

December 13, 2011

Social Dynamics, LLC

In partnership with

Altarum Institute Berkeley Policy Associates

Mathematica Policy Research

Social Dynamics, LLC 1

To contact the DEI Evaluation Team:

Douglas Klayman, Ph.D. (Project Director)Anne Chamberlain, M.A. (Project Manager)

1-855-990-1105 (Toll Free)[email protected]

[email protected]

Overview of Topics

3Social Dynamics, LLC

1. Overview of the DEI Evaluation 2. What is the DEI Evaluation Synthesis Report2. What is a ‘Getting To Know You Site Visit’?3. What We Learned About…

• State Workforce Structure: System and Policies• Ticket to Work Program• One-Stop Center Service Processes• Disability Resource Coordinators• Partnerships and Collaboration• Working with Customers with Disabilities• WIA Performance Measures• Initial Steps Toward Systems Change


4. Team Contact Information 5. Recap Year 2 Plans

Overview of Topics (cont.’)


• DEI is a collaborative between ODEP and ETA. • ETA is jointly funding and administrating the DEI in collaboration

with the Office of Disability Employment Policy. • ODEP provides national leadership on disability employment

policy by: • developing and influencing the use of evidence-based

disability employment policies and practices, • building collaborative partnerships, and • delivering authoritative and credible data on employment

of people with disabilities.• ETA’s FPOs are located Regional offices and serve as the Grant

Officer’s technical representative to the DEI cooperative agreement

• Regional FPOs conduct desk and on-site monitoring of the grant and are the primary federal contact with the grantee.

1.0 Overview of the DEI Evaluation


1.0 Overview of the DEI Evaluation (cont.’)

• DEI is a comprehensive evaluation, with impact, outcome, and implementation components.

• Over the course of the grant, data will be collected through the DEI Data System, and regular site visits and phone interviews.

• For more information, visit our DEI Evaluation website:

http://www.socialdynamicsllc.com/deigranteessecure.html

2.0 What is the DEI Evaluation Synthesis Report?


• First annual report for the Evaluation of the Disability Employment Initiative (DEI). The entire report is available at: https://disability.workforce3one.org/command/view.aspx?look=2001127955535929318&mode=info&pparams=

• Purposes of the report:

• To inform ODEP and ETA of the progress made to-date on DEI evaluation activities during Year 1 of the DEI.

• To provide useful information on DEI start-up (on both the program and the evaluation side) for Round 2 DEI states.

3.0 What is a ‘Getting to Know You Site Visit’?


• ‘Getting to Know You Site Visits’ (GTKY) were conducted in spring 2011.

• GTKY Site Visits included discussions with state, WIB and One-Stop staff, and their partners on the following:

• Challenges to implementation of the DEI• Structure of the existing workforce development system• Resources available to individuals with disabilities• Status of the implementation of the DEI in each Round 1

state• Structure of WIA services and the Wagner-Peyser

program, lists of partners and collaborators, and the collection of service brochures and project descriptions, and other relevant documents.

3.0 What is a ‘Getting to Know You Site Visit’? (cont.)


• GTKY site visits included a site visit plan that was developed by the Evaluation Liaisons and the DEI State Lead.

• Unlike the more structured site visits planned for future years, the GTKY site visits centered on informal conversations with staff about their initial implementation of DEI, local workforce development systems and plans for gathering and providing the data needed for the evaluation. Town Hall meetings also were conducted.

• GTKY site visits also provided an opportunity to introduce the evaluation team to the grantees.

4.0 What We Learned


4.1 State Workforce Structure: System and Policies •Federal WIA policy affects DEI implementation in many ways. For example,

• Both state and local workforce agencies are subject to performance measurement requirements to ensure the accountability of their WIA program.

• If more than one agency providing employment services for a customer wishes to claim “credit” for an employment outcome and no mechanism exists for shared credit, there is potential for conflict or even a disincentive to collaborate.


4.1 State Workforce Structure: System and Policies (cont.)

• Several grantees addressed this concern by making a state-level decision that WIA, VR, or ES staff could share the credit for assisting people with disabilities in obtaining or maintaining employment.

•States and local WIBs have negotiated various roles for ES within their One-Stop Centers. For example,

• NY requires that all UI recipients visit a One-Stop Center for reemployment services (vs. states that require only a percentage of UI claimants to do so).

• In AK, ES staff operate the Centers• In ME, ES operates separately from the rest of the center

DKlayman (Dell)

What do you mean? People with certain characteristics or just a percentage UI recipients?


4.1 State Workforce Structure: System and Policies (cont.)

•In several states, WIA and VR complement each other and leverage each other’s resources when they collaborate at the WIB and One-Stop levels.

•Grantees emphasized the importance and value of colocation of VR counselors. Colocation fosters better communication between employment counselors on both sides. “Just being able to walk a customer over and introduce him or her to the other agency’s counselor,” can help overcome any reluctance on the part of the customer to seek additional services.

•Colocation is challenged by:• Cost of office space (VR lacks funds)• District administrators interpret VR legislation as restricting

$ spent on space where non-disabled customers are also served.

DKlayman (Dell)

Explain in one sentence. Do you mean they provide workforce services outside of One-Stops?


4.1 State Workforce Structure: System and Policies (cont.) •A key factor in determining the roles of VR and WIA in collaborating to provide services is the availability of VR case service funds that can be leveraged to pay for needed services.

•States vary in the amount of resources available to serve IWD, and states with insufficient funds must implement an Order of Selection, prioritizing individuals with the most significant disabilities. This results in waiting lists.

•Due to Order of Selection, some states reported that they expect to see increasing numbers of customers coming to them from VR. They worry that VR staff might have less time for providing consultation and support to Center staff about disability issues.

DKlayman (Dell)

Do some states do this and some not?

4.0 What We Learned


4.2 Ticket to Work

•AK, AR, KS, and ME are seeking EN status at the state level, while other Round 1 states have been actively working toward gaining EN status at the WIB level.

•Although most Round 1 grantees are still early in their EN learning curve, several were in the process of applying to become an EN in Spring 2011.

4.0 What We Learned


4.3 One-Stop Center Service Processes •Considerable variation exists in the processes by which One-

Stop Centers serve customers from “intake” to exit (and, sometimes, follow-up):

• Which staff is involved?• In which phases of the service process?• How and when are internal and external referrals made?• How are decisions made about going beyond core

services?

•These variations can have significant implications both for the progress of the demonstration and for evaluating its effectiveness.


4.3 One-Stop Center Service Processes (cont.) •For example, an important consideration in evaluating systems

change outcomes is the process by which IWD enter the public workforce development system and make their way into the WIA intensive and training services that have the potential to substantially impact their employment outcomes.

•How customers become involved with a One-Stop Center, which staff are involved in helping them identify employment goals and needed services, and how their disability might be identified and taken into consideration during the service process are all critical to understanding the impact of DEI on serving IWD.

•Level of training and expertise of One-Stop staff, and the extent and timing of contact with staff, varies considerably, too, and can also have a significant impact on customers’ experiences.


4.3 One-Stop Center Service Processes (cont.)

•Some One-Stops have 1-3 people to help out in the resource room including staff from other agencies. At other sites, customers may have little interaction with staff other than to conduct a job search or inquire about available resources.

•In some states all customers who visit a One-Stop have the option to meet one-on-one with a career consultant, caseworker, or a case manager.

•In all states, most customers enter as WIA intensive customers and go through a one-on-one assessment.



•In these initial one-on-one assessments, staff determines which service track the customer needs: (a) Job Search Ready, if the customer just needs help with job leads; or (b) Career Development Services, if the customer needs more help with his/her résumé, interviewing, etc.

•Assistive technology and staff trained in its use, was available in

each One-Stop Center visited.

•There are some differences in how One-Stop Centers are increasing their focus on serving customers with disabilities.

•In some states, certain staff members have been trained to have special expertise in the use of assistive technology and resources for IWD.



•An important consideration for the evaluation is the definition of “exit” from One-Stop Center services, as well as the process by which an individual exits.

•Variations are important: (a) because the definition of those who leave services is integral to determine the denominator for outcome measures (such as 8/10 exiters found employment); and (b) we must maximize the consistency with which grantees collect follow-up measures.

•In all states, customers have “soft” exits if they go 90 days without service. This may represent the majority of exits. In other states, exits fall into three categories: positive (employment), neutral (hospitalization, incarceration), and negative (dropout).



•In other states, such as KS and NY, staff identify the customers who are approaching 90 days post-service and—depending on customer flow in the Center and staff availability— follow up by phone and/or letter, to determine if the customer wishes to engage in additional services.

•DE, a youth state, reported that there is a loss of students visiting the Center following graduation from high school despite the VR involvement in the high schools. Since many youth do not report having a disability, there is often a break in services due to the student “dropping off the grid” post-graduation. Finding and tracking students, can be difficult in a state as small as DE due to migration across state lines.

4.0 Findings from the Synthesis Report


4.4 Disability Resource Coordinators (DRCs)

•Beyond basic requirements that DRCs facilitate TTW and partnerships, states’ definition of the DRC responsibilities, assignment of this position, and determination of stakeholders served varies considerably:

• In ME, DRCs will focus on employer/community outreach, while VRs’ ‘RC1’ position will work with customers with disabilities.

• In DE, the DRC is employed by VR and will work specified hours in the center, but then also work specifically with schools & correctional facilities.

• In AK, the DRC is a rotating position.

4.0 What We Learned


4.5 Partnerships and Collaboration

Several DEI grantees reported that they collaborate with other agencies within the workforce system.

• NY receives financial support from VR.

• DE reported strong and ongoing relationships with a comprehensive set of entities including ETA, the WIB, the Secretary of Labor, and Youth Rehabilitative Services.

• VA One-Stops provide VR services on-site.

• In both ME and NY, VR staff is only at the One-Stops once per week or per month.


4.5 Partnerships and Collaboration (cont.)

•Staff at all of the Centers visited reported working with partners that provide education, health care, veterans’ services, family and child welfare assistance, youth services and employers.

•Several states invite employers to come to the Center to recruit applicants and conduct interviews on-site, partner with staffing agencies and employing staff members specifically devoted to business and employer outreach.

•One-Stop Center leverage financial resources as part of their working relationship with various partners.


4.5 Partnerships and Collaboration (cont.)

•Some state-level staff also mentioned using funds from the Medicaid Infrastructure Grant (MIG) to support DEI.

•IL uses a “network system,” whereby they can identify which One-Stop Centers have available funds in order to refer customers, in case their own Center does not or cannot provide needed WIA services (universal core services).

•VA receives financial support from the local community college for a classroom on-site that is available for core and intensive services.

•NY Makes Work Pay partners the State Office of Mental Health and Syracuse and Cornell Universities. MIG- funded, the partnership works on asset development, inter-agency collaboration, and employer assistance in hiring and retaining workers with disabilities.


4.5 Partnerships and Collaboration (cont.) •AK’s Mental Health Trust Authority funds initiatives such as the

“You Know Me” disability awareness campaign, as well as supported employment services.

•AR and ME work closely with the Social Security Administration,

not only because of TTW, but because of its Work Incentive Planning and Assistance (WIPA) projects, which assist SSA beneficiaries with disabilities in seeking and maintaining employment and taking advantage of SSA work incentives.

•Each of the more than 100 WIPA projects across the U.S. employs a Community Work Incentives Coordinator (CWIC).

4.0 What We Learned


4.6 Working with Customers with Disabilities •How Centers approach asking customers about disabilities varies

considerably.

•Center staff and management offered estimates of (disability) disclosure rates that ranged from “less than half” to “one in ten.”

•Most staff felt that non-visible disabilities, especially mental health/substance abuse disabilities, are the least likely to be disclosed.

•Customers are more likely to disclose disabilities once they’ve developed a rapport with their case manager. Physical space for private conversations, and customers’ understanding that they may be eligible for additional services, also helps.


4.6 Working with Customers with Disabilities (cont.)

•For youth with disabilities, fear of stigma or the desire not to be “labeled” are particularly strong challenges.

•Youth are not always able to obtain documentation (such as an IEP) for their disabilities, esp. if they have relocated since school.

•In light of the difficulty of obtaining disability documentation, staff may not record disabilities for youth, particularly if the customer already qualifies for WIA youth services based on income alone.

•Some staff mentioned that customers were concerned that information provided on state-wide systems could be accessed by employers.

4.0 What We Learned


4.7 WIA Performance Measures •One key concern about serving IWD is whether they are likely to

achieve the same level of success as other customers, or whether serving IWD might reflect poorly in the WIA performance measures.

•Some Centers keep customers out of the WIA data system unless they seem likely to succeed, in order to maintain the Center’s performance measures.

•By contrast, one LWIB described itself as being “sanctioned for serving too many customers with disabilities,” explaining that its low employment outcomes/performance measures were a consequence of working with a large population of CWD.

4.0 What We Learned


4.8 Initial Steps toward Systems Change •DEI is a systems change initiative, the premise of which is to

use proven systems-change strategies to make lasting, impactful change in the employment outcomes of IWD. We expect to see the emergence of systems change towards the end of the DEI grant period.

•Systems change will look different in each grantee state. Each

has a unique culture, unique strengths and challenges, and has chosen unique approaches to improve its workforce system.

•Already we have seen indications of systems change, or the readiness for it.


4.8 Initial Steps toward Systems Change (cont.)

•AK’s rotating DRC is one example of the emergence of systems change.

•In several locations DEI was referred to by LWIB or Center staff as “the DRC grant” or “the DEI program.” Staff (mis)perceived DEI as a stand-alone “program” that was implemented by a particular person.

•There was also sometimes the misperception that DEI was a funding source to maintain a DPN who would now be called a DRC. It is important that all workforce system staff are “on the same page” in understanding what they are doing, and why.

5.0 Contact Information

Round 2 Evaluation Liaisons

Robert Bleimann, Ph.D. (HI, SD, WA)[email protected]

Douglas Klayman, Ph.D. (CA, TN, WI, OH)[email protected]

1-855-990-1105 (Toll Free)


6.0 Year 2 PlansRound 1 States

•Finalize Data Sharing Agreements•Implement and monitor the DEI Data System •Assist states with quarterly WIASRD and W-P data submissions to the evaluation team•Conduct mid-term telephone interviews

Round 2 States

•Complete random assignment process•Conduct Site Visits•Assist states in preparing for the DEI Data System •Begin Data Sharing Agreement process

Social Dynamics, LLC

Documents

DEI Evaluation Synthesis Report (Year 1) December 13, 2011 In partnership with Altarum Institute Berkeley Policy Associates Mathematica Policy Research