National Governors Association

Work-Based Learning Policy Academy TA Webinar Series:

Measurement

Friday, January 27, 201612:00-1:30 PM EST

Geoff KingNGA Center for Best Practicesgking@nga.org

National Governors Association

Housekeeping

• Today’s workshop is being recorded

• Participant phone lines are muted

• Submit questions through chat box

• If you are having technical difficulties with the webinar platform, please submit questions to Kimberly Hauge at KHauge@nga.org.

National Governors Association

Webinar Overview

• Measurement Overview and Review of Work to Date

• WBL State Highlights:

• Montana: Siri Smillie, Education Policy Advisor, Office of the Governor

• Utah: Kimberlee Carlile, Cluster Pathways Specialist, Utah Governor's Office of Economic Development

• Work-Based Learning Data Collection Tool Review

• Role of Longitudinal Data Systems in Measurement Progress

• Rachel Zinn, Director, Workforce Data Quality Campaign

• Questions & Answers and Wrap-Up

National Governors Association

Measurement Overview and Review of Work to Date

Geoff King, Senior Policy Analyst, NGA Center

National Governors Association

Towards an Assessment Plan

Input:The Level of Participation in

WBL

Input:The Level of Quality of Existing

WBL Efforts

Measure the Baselines of…

Outcome:Assessment of Progress for Youth and

Employers

7

• States track data on a broad range of WBL activities; but participant totals and funding levels are often missing or incomplete.

• Data on youth WBL participation is not readily available for many programs, particularly those outside CTE and others focused on youth.

• Outcomes data is not available for most WBL programs.

Key Themes from Baseline Data

8

Other WBL Policy Academy State Highlights

Total Youth WBL Participation (non-apprenticeship)

• 2 states developed estimates (IA = 15,651 and UT = 8,237)

Programs that Can Support Youth WBL

• 2 states developed program inventories capturing all local initiatives (NH = 144 and WA = 220)

• Iowa developed a state program inventory (14 total programs)

Programs that Meet High-Quality Standards

• Utah assessed secondary pathways that include WBL against high-quality criteria (containing components of high wage, high skills and high demand employment)

• 31 of 76 pathways or about 41% met Utah’s criteria

Baseline Data of Note: Other WBL

9

Key Themes from Assessment Plans

Building Data Capacity and Identifying Progress Indicators

• Additional baseline data collection and benchmark setting

• More outcomes data sought

• More systemic definitions and processes will be helpful

Measuring Increases in Quality of WBL Programs

• Quantitative outcomes and qualitative data (including high-quality criteria) proposed

• Leveraging existing data systems and working groups

Measuring Increases in Scale of WBL Programs

• Program inventories used to set baselines

• IN goal – all residents age 16-29 experience at least one kind of measurable, quality WBL

National Governors Association

F I G U R E 1 : F R A M E W O R K F O R H I G H - Q U A L I T Y , D E M A N D -D R I V E N W O R K - B A S E D L E A R N I N G

National Governors Association

Additional Highlights from the States

• Tennessee has developed a framework for high-quality work based learning in the state’s “Work-Based Implementation Guide,” which clearly outlines expectations along with an evaluation and assessment process to ensure that work-based learning experiences are of high quality for the participant.

• South Carolina has incorporated WBL experiences into their high school report cards; and captures a range of WBL success measures for different types of WBL experiences.

National Governors Association

State Highlight: Montana

Siri Smillie, Education Policy Advisor, Office of the Governor

Montana WBL Assessment Plan

• Scan of existing data systems at OPI, DLI, OCHE, COM

• State collects some data related to• quantity (participation)

• quality (outcomes)

What do we know?

Career Awareness

• Number visits to the Montana Career Information System (MCIS) site

• Number of active portfolios on MCIS

• Number of sites using MCIS

Career Exploration

• Number of participants in Career and Technical Education Student Organizations (CTSO)

• Number of Jobs for Montana’s Graduates (JMG) programs

• Number students enrolled in a JMG course

• JMG outcomes one year after graduation

Career Preparation

• Number of high school students enrolled in CTE courses

• Number of high school students enrolled in CTE dual enrollment courses

• Number of post-secondary students (in the Montana University System) receiving credit for internships

• Post-secondary CTE concentrators

• Students served by the TAACCCT grants (III and IV)

• TAACCCT IV program completion

• TAACCCT IV program retention

• SSN of post-secondary students with the possibility to match with wage outcomes

Career Training

• Number of registered apprentices• Number of occupations with active registered apprentices• Number of apprentice sponsors• Number of current workers accessing OJT funding• Number of Employees trained through the Worker Training Grant program• Number of Incumbent Worker Training Grants awarded• Number of Employees completing Incumbent Worker Training• Retention of apprentices• Wages and wage growth of apprentices, post-apprenticeship • SSN of apprentices and OJT participants with the ability to match with

wage outcomes

Reflections

LIMITS: WHAT WE HAVE

• Mostly quantitative

• Not much available statewide data for Career Awareness and Career Exploration

• Lack of appetite for additional data collection

OPPORTUNITIES: DIVING DEEPER

• Sector-specific analysis

• Example: health care• Area Health Education Centers

• HealthCARE MT

• DLI Apprenticeships

How can we communicate what we know?

Baseline report delivered to:

• Governor

• Lt. Governor

• Superintendent of Public Instruction

• Montana State Legislature

• Commissioner of Labor and Industry

• Director of the Department of Commerce

• Commissioner of Higher Education

• Chief Business Development Officer

• Board of Public Education

• Board of Regents

• State Workforce Innovation Board

Next steps

• Develop rubric of Montana Work-Based Learning Quality Indicators to include:

• Clear partnership agreement (participants served, education/intermediary organizations involved, employer investment, partner satisfaction)

• Authentic work experience (participants served, retention, wages, supervisor meets regularly with participant)

• Structured learning (learning component and outcomes are clearly outlined, job description aligns with learning outcomes, learning outcomes are aligned with career pathway)

• Recognition of skills (participant demonstrates skill attainment through final assessment, participant receives credential recognizing skills and experience, employer completes assessment of overall satisfaction)

• Determine short and long-term goals informed by baseline data report

• Sector-specific deep dive

National Governors Association

State Highlight: Utah

Kimberlee Carlile, Cluster Pathways Specialist, Utah Governor's Office of Economic Development

National Governors Association

WBL Data Snapshot Tool

Geoff King, Senior Policy Analyst, NGA Center

National Governors Association

Data Snapshot Tool Overview

• Based on WBL RFP Guidance

• Goals:

• Help Policy Academy States build on existing baseline data and analysis plans, and establish a framework that can be used to report the scale of WBL.

• Help States report WBL progress to policymakers and better tell the story of the work they are engaged in – a key factor in sustainability.

National Governors Association

Additional Outcomes Data Reporting

Data on quality and outcomes limited and more program specific; but some information is available.

• Quantitative outcome measures: oEmployment, retention, and earnings data (workforce system)

oProgram completion, credential, and degree attainment

oReturn-on-investment (ROI) analyses

• Qualitative measures of program quality:oAssessments against high-quality criteria

oSurveys of employers and participants

National Governors Association

Data Snapshot Tool Next Steps

• Today: Review and Feedback (during Q&A)

• Additional feedback submitted via e-mail or check-in calls by Friday, February 9, 2017

• Revised tool shared in February, with a goal of obtaining additional WBL data from grantees by mid-March; including additional outcomes/ quality data.

• Additional data reviewed at next cross-state meeting for peer learning, used to describe policy academy outcomes and help states plan next steps for sustainability.

National Governors Association

Role of Longitudinal Data Systems in Measurement Progress

Rachel Zinn, Director, Workforce Data Quality Campaign

National Governors Association

Q&A and Discussion

Please enter any questions and comments in the chat box or unmute to chime in.

Thank you for your participation!

National Governors Association

ResourcesWBL Policy Academy Cross-State Meeting Resources – Iowa – October, 31 2016

• Measurement Overview Presentation Slides

• Tennessee Measurement Presentation Slides – “Tennessee’s Revitalized Work-Based Learning Model” – Chelsea Parker, Tennessee Department of Education

• Tennessee Department of Education, “Work-Based Learning Implementaion Guide” (May 2016). Note that pp. 55-60 provide details on assessing skills and evaluating programs.

• Workforce Data Quality Campaign Presentation Slides – “WBL Data – A Work in Progress” – Rachel Zinn

NGA Publications

• K.Hauge and B.Parton. State Strategies to Scale Quality Work-Based Learning (Washington, D.C.: National Governors Association Center for Best Practices, October 31, 2016).

Other

• AIR PowerPoint – “Work-Based Learning: Promoting a Well-Rounded Education for All Students”, January 10, 2017. Features information from Job’s for the Future’s Pathways to Prosperity Network and state examples from South Carolina and West Virginia. Note relevant slides on measurement in South Carolina – slides 37-39.

• The Benefits and Costs of Apprenticeship: A Business Perspective. (Washington, D.C.: Case West Reserve University and United States Department of Commerce, November, 2016).

National Governors Association

Resources (cont.)

Workforce Data Quality Campaign Resources

• http://www.workforcedqc.org/resources-events/resources/2016-mastering-blueprint-state-progress-workforce-data

• http://www.workforcedqc.org/resources-events/resources/data-policy-toolkit

• http://www.workforcedqc.org/resources-events/resources/mapping-postsecondary-and-workforce-information-gaps-state-data-systems

• http://www.workforcedqc.org/resources-events/resources/registered-apprenticeship-data-faqs

• http://www.nationalskillscoalition.org/resources/publications/file/10-4-NSC-YouthWorkBasedLearning_v4.pdf

National Governors Association

Upcoming Work-Based Learning Sessions

• Wednesday, February 8th, 3:00-4:00pm Eastern: WBL Learning Network Call

• Friday, February 17th, 12:00-1:30pm Eastern: Enhancing the Quality of Work-Based Learning (2nd in Series)

• Cohort-Wide TA Webinars continue monthly from March -May