National Research Center for Career and Technical Education

The work reported herein was supported under the National Research Center for Career and Technical Education, PR/Award (No. VO51A070003) as administered by the Office of Vocational and Adult Education, U.S. Department of Education. However, the contents do not necessarily represent the positions or policies of the Office of Vocational and Adult Education or the U.S. Department of Education and you should not assume endorsement by the Federal Government.

Pradeep Kotamraju, Ph.D.Deputy Director

Principal Investigator

Oscar Aliaga Ph.D.Program Director

Emily DickinsonGraduate Assistant

Year 2

Year 3

Year 4

Technical Skills

Crosswalk Validation

NCES Data Analysis

ROI

DevelopedInventory Template

PopulatedInventory Template with Data Submitted by States

Linked 3rd Party

Industry Certification to LMI

Figure 1:The Work So Far

Built a Common Postsecondary

CTE Data Dictionary

Analyzed Data To Test Common

Postsecondary CTE Data Dictionary

Jointly with the Center for

Education and the Workforce

(CEW), Georgetown U,

Producing a Report on 2018

Employment Projections by

Career Clusters

Working on NCES Restricted Sample

Survey Data to Analyze CTE Engagement,

Achievement, and Transitions

Community College

Completions Using IPEDS

Data; Published in Community

College Journal of Research &

Practice

Developing an ROI Guidebook

Neutral Intermediary

Chaired NSWG on POS

Measurement, Accountability, and Evaluation

Produced a Brief on ROI for the

USA TVET Consortium

Made a Presentation on ROI at the ACTE

Annual Conference

An NRCCTE Report: Common Postsecondary

CTE Data Dictionary

Technical Skills

Crosswalk Validation

NCES Data Analysis

ROI

Produce a Final

Inventory

Conduct Two Sub-Analyses (one on Soft

Skills and the Other on Technical Skills and Connect to

CTE Engagement, Achievement,

and Transitions Using State-level Data

Figure 2:The Work For Year 5 (and Beyond?)

Develop and Validate Existing

OVAE/Career Clusters Crosswalk

State-Level Analysis on

2018 Employment Projections by Career Clusters/ Pathways

Continue Work on NCES Restricted Sample Survey Data to Analyze

CTE Engagement, Achievement, and

Transitions

Refining the ROI Guidebook

Neutral Intermediary

Produce Presentations,

Reports, Monographs,

Journal Article Submissions

Preliminary Analysis of

Using Existing Data Sets to

Develop ROI for CTE

As requested by Center Partners and by External

Stakeholders

In Center Years 2 and 3 (August 1, 2008-July 31, 2010), data collection began and continued in which states were asked to submit information that corresponded with the elements of the template. Fourteen states submitted information that has been reformatted and entered into the template. Analysis of the information points to the following: The top seven program areas (based on two-digit CIP codes) are IT,

Health, Business, Manufacturing, Construction, Communications, and Agriculture.

Third-Party Certification/Technical Exams, State-Developed Program of Study Exams, and State-Developed End-of-Program Exams appear to be most commonly used in states.

Outstanding issues to be resolved include: Status of Postsecondary TSAs (no information has been provided as

yet to the Center). Program of Study level assessments – the extent to which the

postsecondary partner has a role and input. The shape, form and extent of the relationship of states with third-

party TSA organizations.

4

Technical Skills Assessment

In Center Year 4(August 1, 2010-July 31, 2011), the NRCCTE was requested by the Office of Vocational and Adult Education (OVAE), U.S. Department of Education (USDE) to connect industry skill certifications, currently being offered within different state educational programs, to available labor market information (LMI).

The analysis required obtaining data on industry skill certifications, educational program and CIP codes, labor market information, and a crosswalk connecting CIP codes and occupational (SOC) data. The methodology used:

Converted the data states submitted to NRCCTE as part of the technical skills assessment (TSA) inventory template project into a database that could be used in the analysis.

The template data was collected in a Microsoft Excel spreadsheet. The spreadsheet information was reformatted so that it could be imported into a Microsoft Access database.

5


The crosswalk information, which was only available as a Microsoft Excel spreadsheet, was also reformatted for import into the same Microsoft Access database.

The wage information was downloaded as a Microsoft Excel workbook from the U.S. Department of Labor and appropriate wage information was reformatted for import into the same Microsoft Access database. A database with all connected information was then assembled.

The database contains 139 third-party, industry-based technical skill certifications, 111 secondary education programs, 56 CIP codes, and 110 SOC codes for the five states.

6


7

CIP Family

Arts, Design, Entertainment,

Sports and Media

Construction and Extraction

Healthcare Practioner and

Technical

Healthcare Support

Installation, Maintenance,

and Repair

Office and Administrative

Support

Personal Care and Service

Production Protective Services

Sales and Related

Transportation and Material

Moving

Total Industry Skill

Certification

AGRICULTURE, AGRICULTURE OPERATIONS, AND RELATED

SCIENCES.1 1 2 1 5

BUSINESS, MANAGEMENT, MARKETING, AND RELATED

SUPPORT SERVICES. 4 2 2 8

COMMUNICATIONS TECHNOLOGIES/

TECHNICIANS AND SUPPORT SERVICES.

1 2 10 13

COMPUTER AND INFORMATION SCIENCES AND

SUPPORT SERVICES. 22 22

CONSTRUCTION TRADES. 112 7 18 8 145ENGINEERING TECHNOLOGIES AND ENGINEERING-RELATED

FIELDS.1 1

FAMILY AND CONSUMER SCIENCES/HUMAN SCIENCES. 3 3

HEALTH PROFESSIONS AND RELATED PROGRAMS. 1 6 2 9

HOMELAND SECURITY, LAW ENFORCEMENT,

FIREFIGHTING AND RELATED PROTECTIVE SERVICES.

4 4

MECHANIC AND REPAIR TECHNOLOGIES/

TECHNICIANS. 5 31 24 1 61

PERSONAL AND CULINARY SERVICES. 5 2 7

PRECISION PRODUCTION. 22 22TRANSPORTATION AND MATERIALS MOVING. 10 1 2 13

VISUAL AND PERFORMING ARTS. 4 4

All CIP Family 7 127 1 7 38 33 5 81 4 3 11 317

Tables 1: Relating CIP and Occupation Families: No. of 3rd Party Industry Certifications

8

CIP FamilyArts, Design,

Entertainment, Sports and Media

Construction and Extraction

Healthcare Practioner and

Technical

Healthcare Support

Installation, Maintenance,

and Repair

Office and Administrative

Support

Personal Care and Service

Production Protective Services

Sales and Related

Transportation and Material

MovingTotal Wage

AGRICULTURE, AGRICULTURE

OPERATIONS, AND RELATED SCIENCES. $19,630 $17,190 $16,010 $15,680 $17,128

BUSINESS, MANAGEMENT, MARKETING, AND RELATED

SUPPORT SERVICES.$24,835 $22,920 $15,730 $21,162

COMMUNICATIONS TECHNOLOGIES/

TECHNICIANS AND SUPPORT SERVICES. $20,210 $18,195 $18,860 $19,088

COMPUTER AND INFORMATION SCIENCES AND SUPPORT SERVICES. $23,925 $23,925CONSTRUCTION TRADES. $25,453 $29,249 $19,390 $29,448 $25,885

ENGINEERING TECHNOLOGIES AND

ENGINEERING-RELATED FIELDS. $24,620 $24,620

FAMILY AND CONSUMER SCIENCES/HUMAN

SCIENCES. $16,230 $16,230HEALTH PROFESSIONS AND

RELATED PROGRAMS. $21,610 $19,322 $20,410 $20,447HOMELAND SECURITY, LAW ENFORCEMENT, FIREFIGHTING AND

RELATED PROTECTIVE SERVICES. $23,990 $23,990

MECHANIC AND REPAIR TECHNOLOGIES/

TECHNICIANS. $28,922 $23,005 $22,344 $27,220 $25,373PERSONAL AND CULINARY

SERVICES. $23,298 $16,890 $20,094

PRECISION PRODUCTION. $21,820 $21,820TRANSPORTATION AND MATERIALS MOVING. $26,104 $29,500 $18,855 $24,820

VISUAL AND PERFORMING ARTS. $24,620 $24,620

All CIP Family $22,270 $26,826 $21,610 $18,256 $26,127 $22,563 $16,120 $21,517 $23,990 $15,705 $25,174 $21,833

Table 2: Relating CIP and Occupation Families: Average Entry-Wage for 3rd Party Industry Certifications

The NRCCTE convened a workgroup consisting of researchers, data experts, and policymakers who met on January 12th and 13th, 2010, at the University of Louisville. The purpose of this meeting was to capture the evolving multi-state, multi-institutional collaborative efforts seeking greater consistency and clarity in Perkins secondary and postsecondary data collection and reporting, The goal of this meeting was to continue to move toward a common data crosswalk that links educational programs to occupations with career clusters and career pathways and create a foundation for more standardized accountability requirements in later iterations of the Perkins legislation. Concluding the meeting, participants recommended that the Center undertake the following tasks: Document and validate the decision rules that connect occupations to educational

programs, career clusters, and career pathways in several of the existing crosswalks.

Analyze and discuss the different units of analyses that are currently operational within several crosswalks, and outline for a plan for incorporating programs of study as a unit of analysis for crosswalks.

Develop a plan for disseminating the procedures, processes and products to data experts, researchers and policymakers.

The project continues into Year 4 and is being jointly led by the University

of Louisville and the NASDCTEc. Jointly with the Center for Education and the Workforce (CEW),

Georgetown U, Producing a Report on 2018 Employment Projections by Career Clusters

9

Crosswalk Validation Project

Traditionally, only the total number of CTE credits enrolled and completed either across all areas or within one area are used to identify the extent of participating or concentrating in CTE. A three-credit threshold is used to divide CTE students into participants and concentrators

The NRCCTE has begun exploring, examining, and analyzing credits taken within an occupational area, across all occupational areas, and the number of occupational areas in which high school graduates participate and concentrate

The main source is the US Department of Education sample survey data, specifically the 2005 NCES High School Transcript Studies (HSTS) data and the Education Longitudinal Study (ELS) data

10

The NRCCTE CTE TYPOLOGY

11

Conventional Way of Defining A CTE student

Different denominations: concentrator, participant, investor …

“Traditional” classification in large surveys:Academic trackVocational trackNeither and both (dual)—as defined in the NCES

sample surveyNCES: Fulfillment of an occupational area if 3 or

more credits are taken in that area (“occupational concentrator”)

At the State Level: different levels, different goals

A Methodological Rationale for

Measuring, Accounting, and Evaluating CTE

The purpose is to understand, the level, mix, and intensity of CTE course-taking within and across different occupational areas.

The NRCCTE has expanded the participant-concentrator dichotomy by creating a taxonomy that more accurately represents CTE course-taking.

Using the Classification of Secondary School Courses (CSSC) different parts of the high school graduates’ transcripts are analyzed

Reformulating CTE Course Taking

14

• Following along the lines of NCES, the basic steps are as follows:

Identify CTE and Academic courses Divided CTE into 13 occupational areas Divided Academic into different subject areas,

including Math and Science Categorized Math & Science courses by three

levels: Less than Basic, Basic, and Advanced Identified Academic and CTE courses which are

in sequence and further divided by No Sequence/Sequence

Determined the last grade year Math & Science course taken



Dividing CTE course taking patterns of high school graduates (students) into three facets: Identify how many credits high school graduates

takes and completes within each occupational area

Determine the total number of CTE credits enrolled and completed across all occupational areas

Count the number of occupational areas in which each graduate enrolls and completes courses

A crosstab has been built that relates the above three facets creating the following (shown on the next slide)

16


No CTE Course Taking

Concentrator

Participant

0 is No CTE Course Taking; 1,2, and 3 are classified as Participants; 4 is classified as Concentrators

17


No CTE Course Taking

Concentrator

Participant

0 is No CTE Course Taking; 1,2, and 3 are classified as Participants; 4 is classified as Concentrators

18

CTE Credit Categorization %

No CTE credits 8.0

More than 0 and less than 1 CTE credits, no occupational area fulfilled7.5

1 CTE credit, no occupational area fulfilled 10.7

More than 1 and less than 3 CTE credits, no occupational area fulfilled29.9

3 CTE credits, no occupational area fulfilled 7.6

More than 3 CTE credits, no occupational area fulfilled 19.6

3 CTE credits, 1 occupational area fulfilled 0.8

More than 3 credits, at least 1 occupational area fulfilled 16.0

Frequencies of CTE Credit Categorizations (n=2,698,609)

19

0 1 2 3 4 5 6 7 Weighted n

Female 9.0 8.1 11.4 30.8 7.6 18.4 0.9 13.8 1,378,033Male 6.9 6.8 9.9 28.9 7.6 20.8 0.7 18.3 1,320,577

White 8.3 7.4 10.1 27.8 7.1 21.2 0.8 17.4 1,633,287Black 8.0 5.7 8.9 33.5 7.8 17.9 1.3 16.9 379,133Hispanic 7.7 9.2 12.2 31.6 8.8 17.7 0.8 12.1 421,325Asian 10.3 9.8 17.0 34.9 7.2 12.0 0.3 8.4 122,181Other 7.5 9.8 21.0 70.2 16.5 46.8 1.3 26.7 142,684

SES-Q1 (Low) 5.9 6.0 9.7 29.5 7.8 20.6 0.9 19.6 640,172SES-Q2 5.7 6.6 8.6 28.9 7.2 23.6 0.5 19.0 687,844SES-Q3 8.7 6.4 10.3 30.7 8.2 19.5 0.8 15.3 703,344SES-Q4 (High) 11.7 11.0 14.1 30.4 7.1 14.5 1.0 10.2 667,249

Group 0: No CTE creditsGroup 1: More than 0 and less than 1 CTE credits, no occupational area fulfilledGroup 2: 1 CTE credit, no occupational area fulfilledGroup 3: More than 1 and less than 3 CTE credits, no occupational area fulfilledGroup 4: 3 CTE credits, no occupational area fulfilledGroup 5: More than 3 CTE credits, no occupational area fulfilledGroup 6: 3 CTE credits, 1 occupational area fulfilledGroup 7: More than 3 credits, at least 1 occupational area fulfilled

Frequencies of Gender, Race and Socioeconomic Status by CTE Credit Categorizations for Public High School Students

Students taking more than 3 CTE credits, with or without a focus on an occupational area, are 36% to 59% less likely to drop out of high school than those taking fewer CTE credits.

Students who have taken 3 or more CTE credits, fulfilling the requirements of an occupational area or not, are at least 43% more likely to attend 2-year colleges than those taking fewer CTE credits.

Based on high school graduate transcripts that describe CTE and academic course taking, and using conventional definitions of college, career, and college and career readiness, early estimates for each are 42%, 28%, and 18%, respectively.

20

Preliminary Results

21

Kotamraju, P. (2011, June). A new taxonomy of high school career and technical education (CTE): Describing the engagement, achievement, and transitions of CTE students. Paper presented at the 15h Annual Career Clusters Institute, Atlanta, GA.

Kotamraju, P. (in press—expected publication date, Summer 2011). Vocational education and training quality and evaluation: Its place in the U.S. community college. In A. Barabasch & F. Rauner (Eds.), The art of integration: Work and education in America. Berlin, Germany: Springer Verlag Press.

Kotamraju, P. (2011). Meeting the 2020 American Graduation Initiative (AGI) goal of increasing postsecondary graduation rates and completions: A macro perspective of community college student educational attainment. Community College Journal of Research and Practice, 35(1-2), 1-18.

Kotamraju P., Richards, A., Wu, J., & Klein, S. (2010, April). A common postsecondary data dictionary for Perkins accountability. Louisville, KY, and Berkeley, CA: National Research Center for Career and Technical Education, University of Louisville, and MPR Associates, Inc. Retrieved from http://136.165.122.102/UserFiles/File/Tech_Reports/Postsecondary_Data_Dictionary_Report_WEB.pdf

22

Stakeholder Contacts: Publications

Kotamraju, P. (2011, May). CTE Measurement, Accountability and Evaluation: The NRCCTE’s Comprehensive Strategy for Technical Assistance. Paper To Be Presented at the National Association of Career and Technical Education Information annual Meeting, Philadelphia, Pennsylvania.

Kotamraju, P. (2011, May). A New Taxonomy for Career and Technical Education (CTE): Measuring Engagement, Achievement, and Transitions of CTE Students. Paper To Be Presented at the National Association of Career and Technical Education Information annual Meeting, Philadelphia, Pennsylvania.

Kotamraju, P. (2011, April). Getting Timing and Sequencing of Math and Science Course-Taking Right: A Possible Answer to the College and Career Readiness of High School Graduates Concentrating on Career and Technical Education (CTE). Paper presented at 2011 Council for the Study of Community Colleges Annual Conference. New Orleans.

Aliaga, Oscar. (2011, April). Career and Technical Education Graduates and Enrollment in 2-Year Colleges. Paper presented at the annual meeting of the Council for the Study of Community Colleges. New Orleans, Louisiana.

Kotamraju, P. (2011, April). The college and career readiness of U.S. high school graduates: From concept to measurement. Paper to be presented at the annual convention of the American Educational Research Association. New Orleans, LA.

23

Stakeholder Contacts: Paper Presentations

Kotamraju, P. (2010, December). A tool kit for measuring CTE effectiveness using return on investment and other related techniques: A basic introduction. Paper presented at the annual convention of the Association for Career and Technical Education, Las Vegas, NV.

Kotamraju, P. (2010, October). Meeting the 2020 AGI completion goal: The role of the community college. Paper presented at the annual convention of the National Council of Workforce Education, Washington, DC.

Kotamraju, P. (2010, June). Building a Technical Skills Inventory Database One State at a Time,” 14th Annual Career Clusters Institute, Denver, Colorado.

Kotamraju, P. (2010, May). A Cross-State Comparison of Postsecondary CTE Student Graduation Rates and Completions: Determining the Efficacy of Using IPEDS Data for Perkins Reporting,” National Association of Career and Technical Education Information Annual Meeting, St. Louis, Missouri.

Kotamraju, P. (2010, May). Building a Useable Inventory Template for Collecting State and Local Information for Meeting the Technical Skills Accountability. National Association of Career and Technical Education Information Annual Meeting, St. Louis, Missouri.

Kotamraju, P. (2010, April). A Cross-State Comparison of Postsecondary Student Graduation Rates: A Macro Perspective of Educational Attainment,” Annual Meeting, Council for the Study of Community Colleges, Seattle, Washington.

24


Kotamraju, P. (2009, December). The Minnesota FastTRAC Project: Solving the Career Pathways Puzzle. 15th Annual Data Quality Institute, Baltimore, Maryland.

Kotamraju, P. (2009, December). Building a Technical Skill Inventory Database One State at a Time,” 15th Annual Data Quality Institute, Baltimore.

Kotamraju, P. (2009, May). A Repository Template of Technical Skills Assessment: Collecting, Accessing, Reporting and Using Information and Resources for Meeting the Perkins Technical Skill Attainment Indicator,” National Association of Career and Technical Education Information Albuquerque, New Mexico.

Kotamraju, P. (2009, May). Building a Data Dictionary Template from State Administrative Record Data Systems,” Albuquerque, New Mexico.

25


26

Correspondence should be sent to:

Pradeep Kotamraju, NRCCTE, College of Education and Human Development

Room 351 University of Louisville, Louisville, KY 40292.

Phone: 502-852-7714. Email: [email protected].

Documents

National Research Center for Career and Technical Education