1 Performance Measurement & Reporting For Employment And Training Programs

1

Performance Measurement & ReportingPerformance Measurement & ReportingFor Employment And Training ProgramsFor Employment And Training Programs

2

AgendaAgenda

• Quick review of Agenda and Ground Rules for the Session

• This morning’s session:– Foundation for Performance– Revisions to existing reporting

requirements– Data sources– Calculating outcomes on the

performance measures

3

Ground Rules for Today & Tomorrow

1) Cell phones on mute

2) Questions, Answers and Parking Lot issues– Flip Charts

3) The session is designed to provide information on the definitions of the common measures as outlined in TEGL 28-04 and on the revised reporting instructions. We cannot debate whether or not common measures are “good or bad.”

4

A Foundation for Performance A Foundation for Performance ManagementManagement

5

Improving Performance Results

• Reporting performance is a fundamental element of improving customer services and good public administration

• Performance information should be easily understood by all customers, stakeholders, and operators of the workforce investment system

• Establishing common measures and standardizing customer data collection improves the comparability and understanding of performance results

• Ensuring the accuracy and timeliness of performance results is necessary for demonstrating system integrity and value

6

Establishing Common MeasuresEstablishing Common Measures• President’s Management Agenda• Affects six agencies (29 programs)

– Department of Labor– Department of Education– Department of Health and Human Svcs.– Department of Veterans Affairs– Department of the Interior– Department of Housing and Urban Development

• TEGL 28-04 issued on April 15, 2005– Rescinds TEGL 15-03– Policy applies to DOL funded programs only– July 1, 2005, implementation for W-P, VETS and

WIA; October 1, 2005, implementation for TAA

7

The Common MeasuresThe Common Measures• Adult measures:

– Entered employment– Employment retention – Earnings increase

• Youth and lifelong learning measures:– Placement in employment or education– Attainment of a degree or certificate– Literacy or numeracy gains

• Designations of adult or youth are spelled out in each program’s eligibility requirements

8

Benefits of Common Measures• Focus on the core purposes of the workforce

system; employment for adults and skill gain for youth

• Break down barriers to integration resulting from different definitions, data and reports for each workforce program

• Resolve questions raised by GAO and other oversight agencies regarding the consistency and reliability of data

• Reduce confusion among our customers and stakeholders who want to know about results

9

The Intersection of Common The Intersection of Common Measures and Revised Reporting Measures and Revised Reporting

Requirements forRequirements forWagner-Peyser, VETS, TAA & WIA Wagner-Peyser, VETS, TAA & WIA

ProgramsPrograms

10

• Common measure outcomes will be collected through revised reports

• Common measure definitions replace existing program-specific performance measure definitions where possible

• Efficiency measures to be computed at Federal level only

Roll-Out Of Common MeasuresRoll-Out Of Common Measures

11

Wagner-Peyser and VETS Program MeasuresWagner-Peyser and VETS Program Measures


Wagner-PeyserWagner-Peyser Performance Measures

Job seeker entered employment rate

Job seeker employment

retention at six months

Job seeker customer

satisfaction scoreEmployer customer satisfaction score

Adult Common Performance

MeasuresAdult entered

employment rate

Adult employment retention rate

Adult earnings increase

12



Job seeker employment

retention at six months

Wagner-PeyserWagner-Peyser Performance Measures

Job seeker customer

satisfaction scoreEmployer customer satisfaction score



employment rate



Adult entered employment rate

Job seeker entered employment rate

Wagner-Peyser and VETS Program MeasuresWagner-Peyser and VETS Program Measures


PY05 will be the baseline year for these measures. Goals will be negotiated beginning with PY06.

PY05 will be the baseline year for these measures. Goals will be negotiated beginning with PY06.

13

TAA TAA Performance Measures

Entered employment rate

Employment retention rate at

six months

Earnings replacement rate



employment rate



TAA Performance MeasuresTAA Performance Measures


14

Adult earnings increaseEarnings

replacement rate



six months

TAATAA Performance Measures



employment rate





TAA Performance MeasuresTAA Performance Measures


States are responsible for achieving National Goals for the Trade program.

States are responsible for achieving National Goals for the Trade program.

15

WIA AdultWIA Adult Performance

MeasuresEntered employment rate


six months

Earnings gain

Employment and credential rate



employment rate



WIA Adult Program MeasuresWIA Adult Program Measures


16


Earnings gain



six months

WIA AdultWIA Adult Performance

Measures



employment rate






WIA Adult Program Measures WIA Adult Program Measures


Note: States have negotiated performance goals for entered employment, retention, earnings gain and credential for PY05 and PY06.

Note: States have negotiated performance goals for entered employment, retention, earnings gain and credential for PY05 and PY06.

17

WIA Dislocated WorkerWIA Dislocated Worker

Performance MeasuresEntered

employment rate


six months

Earnings replacement




employment rate



WIA Dislocated Worker Program MeasuresWIA Dislocated Worker Program Measures


18

Adult earnings increaseEarnings

replacement



six months

WIA Dislocated WorkerWIA Dislocated Worker

Performance Measures



employment rate






WIA Dislocated Worker Program MeasuresWIA Dislocated Worker Program Measures


Note: States have negotiated performance goals for entered employment, retention, earnings gain and credential measures for PY05 and PY06.

Note: States have negotiated performance goals for entered employment, retention, earnings gain and credential measures for PY05 and PY06.

19

WIA Youth Program MeasuresWIA Youth Program Measures

Roll-Out Of Common MeasuresRoll-Out Of Common MeasuresYouth Common Performance Measures

Placement in employment or education/ advanced training (PY05)

Attainment of a degree or certificate (PY05)Literacy and numeracy gains (PY05/06)

States have not negotiated performance goals for the youth common measures, however states will be expected to begin reporting on these measures in PY05 and PY06.

States have negotiated performance goals for WIA programs according to statutory measures for PY 2005 and PY 2006

Younger Youth Skill attainment rate Diploma attainment rate Retention rate

Older Youth Entered employment Employment retention Earnings change Employment and credential rate

20

Revised Reporting RequirementsRevised Reporting Requirements

• ETA received emergency approval in April 2005 to incorporate common measures into the WIA, W-P, VETS, and TAA program reporting systems

• Remember, where possible the definitions for common performance measures are replacing “old” definitions

• These changes are effective on July 1, 2005 for WIA W-P and VETS programs, and October 1, 2005, for the TAA program

The Department published three 60-day public comment Information Collection Requests in the Federal Register as part of the process to obtain full approval of these revisions. The public comment period ends September 12, 2005.

• Comments should be sent to: Dr. Esther R. Johnson, Administrator Office of Performance and TechnologyDepartment of Labor/Employment and Training Administration200 Constitution Avenue NW Room S5206Washington, DC 20210

• Electronic copies should be sent to [email protected]

21

Revised Reporting RequirementsRevised Reporting Requirements

What the revised reporting accomplishes• Facilitates the collection and reporting of

information, including the common performance measures

• Where possible, standardizes reporting elements across programs

• Continues to communicate program performance results and holds states accountable for federal funds

22

Revisions to Reporting Requirements

WIA Performance Reporting System Quarterly Report

• Collects aggregate counts of self-service participants, the number of participants in training, and youth (by age and school status)

• Collects current WIA youth measures plus common measures• Separate reporting of common measures for Adults, Dislocated Workers, and

NEG participants • Rolling four-quarter reporting methodology to align with W-P employment

services Reporting System• Eliminated customer satisfaction reporting on a quarterly basis

Annual Report• Collects similar aggregate counts of participants and exiters as the Quarterly

Report• New report table to collect youth common measures

WIASRD• Collects additional data on participant services and outcomes to calculate

common measures• States do not submit WIASRD files on participants who receive only self-

services and informational activities• States must report co-enrollment of individuals in Wagner-Peyser or TAA

programs

23

Revisions to Reporting Requirements

Wagner-Peyser Reporting System (W-P & VETS) Aligns performance reporting cohorts with WIA reporting

cohorts Eliminates all customer satisfaction reporting Collects aggregate counts on participants, exiters, and

those receiving workforce information services Collects aggregate outcomes for the adult common

measures Collects aggregate data on services to Homeless Veterans

(VETS 200 C report only) Eliminates reporting on each category of veteran (e.g.,

campaign, disabled) by age grouping

Trade Act Participant Report (TAPR) Collects data elements to calculate common measures Collects information on participant services as authorized

under the TAA Reform Act of 2002

24

Important: Reporting Reminder!! Reporting for WIA, W-P, and VETS for PY

2004 (which ended June 30, 2005) has not changed and does not reflect common performance measures– No change to 4th quarter ES 9002 or VETS 200

reports, due in August– No change to 4th quarter WIA 9090, due in August,

or the WIA Annual or WIASRD, due in October Reporting for TAA for FY 2005 (which will end

September 30, 2005) has not changed and does not reflect common performance measures– No change to 4th quarter TAPR, due in November

25

Important: Data Validation Update

• ETA is upgrading data validation and reporting software to reflect changes in the WIA reporting system– Revised import file format will be available in early

August 2005• Based on revised WIASRD specifications• Modified edit checks/coding procedures• Conversion rules from old-to-new WIASRD format

– Projected release of revised WIA data validation and reporting software is Mid-October 2005

• All data validation and reporting documentation will be available on the web at http://www.doleta.gov/performance

26

Key ConceptsKey Concepts

27

Program ParticipationProgram Participation• A participant is a person who is

determined eligible to participate in a program and receives a service funded by the program in a physical location

• The date of participation is the date of the first service after eligibility determination

• Individuals who receive services that are accessed by virtual means outside of the program’s facility need not, but may, be treated as participants

___________________

28

Program ParticipationProgram Participation• Generally, all participants will be taken into

account when computing performance outcomes under the common measures

• Two types of participants may be excluded from the performance calculations:

• Participants who do not voluntarily disclose valid Social Security numbers

• Participants who are excluded from the calculations by law

____________________

29

“Participant” – Wagner-Peyser Act

• Wagner-Peyser employment services– Universal Access– No formal determination of eligibility is

required

• DVOP/LVER services– veterans, eligible persons, transitioning

service members who meet eligibility criteria

30

Clarification of “Self-directed Job Search” Exclusion

• Application to W-P employment services:– Exclusion is limited to the act of conducting a

job search, job bank browsing or modifying job search requirements.

– Some self-directed job search activities should be considered reportable service (e.g. building resumes, accessing workforce information, and enrolling in online job search mini courses.)

31

At a minimum, data needed from self-service participants: social security number (SSN) employment status at the point of

participation.

Upon request of a customer’s SSN, the customer must also be offered the opportunity to provide Equal Employment/demographic information.

Minimum Data Needed from Self-Service Participants

32

Program ExitProgram Exit

Exit occurs when . . .• Participant does not receive a service or partner

service for 90 consecutive calendar days;• Participant has no planned gap in service; and• Participant is not scheduled for future services

Date of exit is applied retroactively to the last day of service funded by the program or partner program

Must have the capability to track partner services in order to extend the date of exit

No More Hard Exit

33

Program ExitProgram Exit

• Days included in any planned gap in services do not count when determining the exit date– Delay before the beginning of training– Health condition or providing care for

a family member with a health condition– Temporary move from the area– All gaps in service must be documented

34

“Exiter” – Wagner-Peyser Act

Significant change from former Wagner-Peyser performance measures, which calculated performance outcomes based on registration quarter.

35

Global ExclusionsGlobal Exclusions• Exclusions from all common measures:

– Persons institutionalized (exceptions are detained Responsible Reintegration of Youthful Offenders and disabled individuals residing in institutions)

– Persons unable to continue due to health/medical or family care reasons

– Deceased – Reservists called to active duty– Relocated to a mandated residential program (youth only)

• To be excluded from performance calculations, the condition(s) must be expected to last 90 days or more

36

Data SourcesData Sources

37

• Unemployment Insurance wage records– Includes state wage records and additional sources

such as WRIS and FEDES– The first source for employment and retention

measures– The only source for earnings increase

• Supplemental data may be used to determine employment and retention

Outcome Data CollectionOutcome Data Collection

38

• Administrative records are the source for outcomes that are not employment- based

• Administrative data may be obtained through automated data exchanges

• Administrative record data (except automated data exchanges) and data collection methods must be documented and subject to audit

Outcome Data CollectionOutcome Data Collection

39

Calculating OutcomesCalculating Outcomes Adult MeasuresAdult Measures

40

Summary of Major Policy Changes

TEGL 15-03Old Policy

TEGL 28-04New Policy

ADULT MEASURES

Entered Employment Rate •No change in methodology from WIA or TEGL 15-03•Permits grantees to use wage records and supplemental data

Employment Retention Rate

Measurement at 1st, 2nd, and 3rd quarters

•No change in methodology from TEGL 15-03•Permits grantees to use wage records and supplemental data

Earnings Increase Measure

Separated into two distinct indicators of performance

1.Percent change pre- to post-program Qtr 1

2.Percent change post-program Qtr 1 to Qtr 3

•Replaces the two-part earnings increase measure identified in TEGL 15-03 with a single Six Month Earnings Increase measure (pre- to post-program)•Wage records are the only data source•Grantees that do not have access to wage records can

use supplemental data as an interim means of reporting

Program Efficiency

Identified as a core common measure

•Eliminates program efficiency as a reportable measure at the grantee level

41

AdultAdult

Entered EmploymentEntered Employment

• Exclude adults employed at the time of participation (including those individuals who have taken a job after being laid-off or dislocated)

• Include adults employed at the time of participation who are on layoff notice

• Wage records or supplemental data may be used to determine employment in the quarter after exit• Supplemental data must be documented

• Employment at participation is based on information from the individual – not wage records

Computing Outcomes

Number employed in quarter after exit

Number of exiters

Of those not employed at participation:

42

YES

YES Numerator

Denominator

Excluded YES

Employed in quarter after exit?

NO

Employed at participation?

AdultAdult

Entered EmploymentEntered Employment

NO

43

AdultAdult

Employment RetentionEmployment Retention

• Based on those employed in quarter after exit• Wage records and supplemental data

are acceptable data sources– Supplemental data must be documented

• Employment in the 1st, 2nd and 3rd quarters after exit does not have to be with the same employer

• Must be employed in both the 2nd and 3rd quarters after exit to be retained

Computing Outcomes

Of those employed in the quarter after exit:

Number employed in both the 2nd and 3rd quarters after exit

Number of exiters

44

Employed in quarter after exit?

YES

Employed in 2nd and 3rd qtr. after exit?

YESYES Numerator

Denominator

ExcludedNO

AdultAdult

Employment RetentionEmployment Retention

NO

45

AdultAdult

Earnings IncreaseEarnings Increase

• Only wage records may be used for determining pre-program and post- program earnings

• Includes same population as the adult retention measure

• Likely to be negative for dislocated workers• False, erroneous and misleading earnings data

may be excluded by the grantee

-Number exiters

Of those employed in the quarter after exit:Earnings in 2nd

and 3rd quarter after exit

Earnings in 2nd and 3rd quarter prior to participation

Computing Outcomes

46

YES

NO

Employed in qtr. after exit?

Excluded

Supplemental data used in Q1?

Included

NO

Supplemental data used in Q3?

YES

AdultAdult

Earnings IncreaseEarnings Increase

NO

YES

YES

47

Calculating OutcomesCalculating Outcomes YouthYouth MeasuresMeasures

48

Youth Measures Requirements

Reporting for PY 05:– 7 current WIA youth measures (4 OY and 3 YY) –

negotiate on these for accountability– First 2 youth common measures (placement and

certificate)– Literacy/Numeracy common measure

encouraged, but not required (required in PY 06)– Literacy/Numeracy only include new youth

enrolling after July 1 of the year in which the State implements it

– No negotiation on common measures in PY 05

49

Youth and Lifelong Learning Common Measures

• Will include all youth, no distinction between younger/older

1) Placement in Employment, Education, or Training

2) Attainment of a Degree/Certificate

3) Literacy/Numeracy Gains

50

Summary of Major Policy ChangesTEGL 15-03

Old PolicyTEGL 28-04New Policy

YOUTH MEASURES

Placement in Employment or Education• Excluded individuals in secondary

school at exit from performance calculations

• Individuals in secondary school at exit are now included in the measure

• Permits grantees to use wage records and supplemental data

Attainment of Degree or Certificate• Excluded individuals in secondary

school at exit from performance calculations

• Individuals in secondary school at exit are now included in the measure

• Clarifies the definition of Diploma• Includes degrees/certificates awarded by

tribally-controlled colleges and universities

Literacy and Numeracy Gains• Included all in-school and out-of-school

youth who were basic skills deficient

• In-school youth are now excluded from the measure

• Provides additional guidance on providing reasonable accommodations for testing youth with disabilities

Program Efficiency • identified as a core common measure

• Eliminates Program Efficiency as a reportable measure at the grantee level

51

Youth And Lifelong LearningYouth And Lifelong Learning

Placement In Employment Or EducationPlacement In Employment Or Education

• Excludes youth in employment, the military, or post-secondary education at participation

• Includes youth in secondary education at exit• Employment, military and education status at

participation are based on information from the individual

Number of participants in employment or the military or

enrolled in post-secondary education and/or advanced

training/occupational skills training in the quarter after exit

Number of exiters

Of those who are not in any of the following at participation - post-secondary education, employment, or the military:

Computing Outcomes

52



• Employment and military service in the 1st quarter after exit may be based on wage record data or supplemental data• Supplemental data must be documented

• Similar to current YY Retention measure in that it recognizes the same outcomes – improvement from OY Entered Employment

• However, it includes first quarter after exit only – no concept of retention or earnings change in common measures for youth programs

53

NO

YES

YES Numerator

Denominator

NO

Did the qualifying outcome occur in the 1st quarter

after exit?



In post-secondary education, employment, or the military at

participation?ExcludedYES

54

Youth And Lifelong Learning Youth And Lifelong Learning

Attainment Of Degree Or CertificateAttainment Of Degree Or Certificate

• Education refers to secondary school, post-secondary school, adult education, or any organized program of study

• Includes youth in secondary school at exit• Outcome can be attained during participation• Diploma is any credential the State Education

Agency accepts as equivalent to a high school diploma

Of those enrolled in education at participation or any time during the program:

Number attaining a diploma, GED or certificate by the end of the 3rd quarter

after exit

Number of exitersComputing Outcomes

55

Youth And Lifelong Learning Youth And Lifelong Learning

Attainment Of Degree Or CertificateAttainment Of Degree Or Certificate

A certificate is awarded in recognition of an individual’s attainment of measurable technical or occupational skills (emphasis added) necessary to gain employment or advance within an occupation. These technical or occupational skills are based on standards developed or endorsed by employers. Recognition of generic pre-employment and/or work readiness skills are not considered certificates.

• Defining “certificate”…

Computing Outcomes

• Definition applies only to youth attainment of degree or certificate common measure

56

Definition of Certificate (cont.)Awarding Institutions include:

– A State educational agency– Institution of higher education– Professional, industry or employer organization or a

product manufacturer– Registered apprenticeship program– Public regulatory agency– Department of Veterans Affairs– Office of Job Corps– Indian Tribe Higher Education Institution

57

Definition of Certificate (cont.)

Clarification of definition:

– If a state educational system or employer organization granted certificate status to very specific standards for a work readiness program, this would be acceptable as a certificate under the measure

– For example, the U.S. Chamber of Commerce’s recently announced work readiness credential would be allowable under the certificate measure

58

NO

Enrolled in education at participation or any time

during the program?ExcludedNO

YES

Attained diploma, GED or certificate by the end

of the 3rd quarter after exit?

Youth And Lifelong LearningYouth And Lifelong Learning Attainment Of Degree Or CertificateAttainment Of Degree Or Certificate

YES Numerator

Denominator

YES

59

Comparison to Current Measures

• This measure is similar to a combination of the YY Diploma Rate and OY Credential Rate, but with a tighter definition of certificate

• Narrower denominator than current credential measure: includes only those in education

• Does not have employment component as the current credential measure does

60


Literacy Or Numeracy GainsLiteracy Or Numeracy Gains

• Excludes youth who are not basic skills deficient

• Excludes youth who are in-school• Includes individuals with learning disabilities • To be in the numerator, an individual must advance

one or more Adult Basic Education (ABE) or English as a Second Language (ESL) functioning levels

Number who completed a

year of participation

Of those out-of-school youth who are basic skills deficient:

Number who increase one or more educational functioning levels

Number who exit before completing

a year of participation

+

Computing Outcomes

61

• Educational skill gain measurement is consistent with the National Reporting System for Adult Education

• Tests cross-walked with ABE and ESL levels include:– Comprehensive Adult Student Assessment

Instrument (CASAS)– Test of Adult Basic Education (TABE)– Adult Basic Learning Examination (ABLE)– Student Performance Levels for ESL– Basic English Skills Test (BEST) for ESL– WorkKeys (for the top 3 ABE levels)



62

• Requires the use of the same standardized test for both pre and post assessments

• Individuals should be pre-tested within 60 days of the participation date

• Individuals with disabilities should be accommodated according to guidelines associated with the assessment tool and state law or policies

• Individuals should be post-tested by the end of one year of participation and compared to pre-test results obtained during the initial assessment



Computing Outcomes

63

• If an individual continues to be basic skills deficient after the first 12 months of participation, then s/he should continue to receive training in literacy and/or numeracy skills

• Participants should be post-tested and included in the measure at the completion of the 2nd year

• Only included in measure a second time if they complete a full 2nd year in program



Computing Outcomes

64

Literacy/Numeracy-- Comparison to Current Measures

• Creates specific focus on basic skills rather than broad focus on work readiness, occupational skills, and basic skills– Does not mean work readiness and

occupational skills no longer important (impact of those skills will be seen in placement rate)

• Standard for success much higher than with less-defined skill attainment rate

65

For Further Information…For Further Information…

ETA’s Performance And Results Website

http://www.doleta.gov/performance

ETA’s Performance And Results E-MAIL ADDRESS

[email protected]

National Reporting System for Adult Education Website

http://www.nrsweb.org

Documents

1 Performance Measurement & Reporting For Employment And Training Programs