Measuring Career Readiness and Student Growth in Career and Technical Education

Nadja Young, SAS InstituteSteve Gratz, Ohio Department of Education

Michael Womochil, Colorado Community College SystemTammy Howard, North Carolina Department of Public Instruction

Developing Career Field Technical Content StandardsSteve Gratz, Ph.D., Senior Executive DirectorCenter for Student Support and Education Options

Accelerate Process: Career-Technical Assurance Guides

Content Advisory Committees

Willing to commit time

Distribution of expertise

Thought leaders

Recognize training and expertise

Database of Learning Outcomes

and Postsecondary Pathways

Converts Learning Outcomes to Standards and Competencies

Maintain fidelity

Minimize re-wording

Ensure consistency

O*NETData

Create Competency

Models

Serves as Supporting Information

for Standards

Content Advisory Committees

Reviews results of converting learning outcomes into standards and competencies

Recommends missing technical content

Assessments

Focus on the learning outcomes identified by the postsecondary faculty

Created for single, secondary courses

Professional Development

Give teachers the background knowledge to deliver the content

Industry and postsecondary faculty serve as content experts

Major Concern

Promotion, marketing, roll-out and communication

education.ohio.gov

Follow Superintendent Ross on Twitter

Social Media

@OHEducation

ohio-department-of-education

Ohio Families and EducationOhio Teachers’ Homeroom

OhioEdDept

storify.com/ohioEdDept

Colorado Technical Assessments

Michael WomochilProgram Director

Colorado Community College System

Colorado Technical Assessments

• cPAS - Career Pathway Assessment System• End of program student assessments.

– Completion of CTE program of study (2+ years)• Multi-state consortium involved in

development– Colorado, Kansas, Mississippi– Managed by Center for Educational Testing and

Evaluation at University of Kansas

cPASS Assessments• Built from CTE content standards• Two components to each assessment

– Sit-down computer assessment• Standard questions & technology enhanced items• 100 questions / assessment• Additional modules (20) question for some areas

– Career Competency Qualifications (CCQ)• Performance evaluation component• Five or more for each assessment• Student selects which ones they complete ( 3 or more)

cPASS Assessments Operational

• General CTE– Based on College & Career Readiness Skills– Academic foundations & career ready skills

• Comprehensive Agriculture– Contains content from 6 of 7 AFNR pathways– Add on module for Power, Structural & Technical

cPASS Assessments In development

• Animal Systems (operational F15)• Plant Systems ( add on module for green

industry) (operational S16)• Manufacturing Production (operational F15)• Design and Construction (Pilot )• General Business ( final item writing)• Finance (Final Item Writing)

End of Course Assessments

• Driven by Teacher Effectiveness legislation.– 50% of teacher evaluation based on evidence of

student growth.– Created demand for pre & post assessments at

course level.– Utilizing Precision Exams from Utah.– Evaluated each assessment for alignment to

Colorado content standards.– Three levels of recommendation for use.

End of Course Assessments

• Demand for use has lessened due to delay in implementation of student growth requirement.

• If fully implemented we’ll work with Precision to develop Colorado specific course assessments– Aligned with end of program (cPASS) assessments

Future Projects

• Instructors trained in assessment item writing and performance evaluation development

• Creation of additional performance assessments for areas where there are no current measurement tools

• Development of industry recognized credentials for students completing CTE programs and indicating mastery via assessment system

Future Projects

• Recognition of student content mastery by PS institutions for admissions and scholarships.

• Creation of formative item test bank aligned to new National AFNR Career Technical Core.

North CarolinaCareer Technical Education

Assessments: Using Assessment and Data for

Continuous Growth and Improvement

Tammy Howard, PhD

Director, Accountability Services

North Carolina Department of Public Instruction

General Overview of the System

• 215 Courses at the State Level

• 215+ End of Course Assessments

• Ability to Deliver Assessments Online or Paper/Pencil

• System to Capture and Validate Credentials and Certifications

26

Vision and Concept

• System Initially Designed in the Early 1990’s

• Originally Conceptualized as a Method to Reward Teachers for Continuous Improvement

• Recognized Nationally as Model for Formative and Summative Assessment

• Aligned to Carl D. Perkins Career and Technical Education Act Technical Attainment Performance Indicators

27

Assessment Models• Types of Test Item Banks

– Field Test

– Classroom (Teacher – Formative)

– Secure (Accountability – State End-of-Course)

• Types of End-of-Course Summative Assessments

– Developed

– Adopted

– Industry-Based

• Pre- and Post-Assessment Model

• EVAAS

28

Curricular Alignment

• All Courses in NCCTE are required to have:– “Blueprints” of Essential Instructional Standards

Aligned to:– Instructional Content

Aligned to:– Formative Assessment

Aligned to:– Summative Assessment

29

Validity, Reliability, and Analysis

• Validity Required for All Secure and Classroom Test Items

• Reliability Study Required for Each New Assessment

• Data Analysis Performed at State and District/Local Level

• State-level Item Analysis

• Scale Scores Determined After Reliability and Item Analysis

30

Growth Analysis• In the early 1980s, the EVAAS approach to measuring

growth was founded at the University of Tennessee Knoxville by Dr. William Sanders

• Revolutionized the way educators and policymakers viewed schooling effectiveness and the ability of students to make growth

• In 2006, EVAAS was implemented statewide as a school improvement resource.

• In 2012, EVAAS became a formal part of North Carolina’s teacher evaluation and accountability after recommendation by WestEd and UNC researchers.

31

How is growth measured?

• Predictive-based model– Based on students’ prior testing history

• all previous tests for which correlation has been established

• Students must have three prior test scores in any grade/subject

– What is the difference between students’ expected score and observed score?

• Use all available testing history for each student to minimize impact of measurement error

• Use standard errors to address uncertainty inherent in any growth model and protect against misclassification

32

Inclusion in Growth Model (EVAAS)Each CTE assessment must have sufficient:• Stretch in the scales

– To measure growth of both high- & low-achieving students

– No floor or ceiling effects• Reliability in the scales

– Preferably have at least 40-50 test questions per subject/grade/year

• Number of districts/schools/teachers administering test – Preferably 25 at each level

33

Inclusion in Growth Model (EVAAS)Each CTE assessment must have sufficient:• Correlation with curricular objectives• Predictive relationship

– Results of each CTE test must be related or correlated with results of other tests from different grades/subjects

– i.e. prior math or reading test scores and the CTE test score were not strongly related for Carpentry, but they were for Biomedical Technology

34

EVAAS Reporting• Dozens of reports for use in school improvement

– Reflective analytics, such as value-added and diagnostic reports for districts, teachers and schools

– Proactive analytics, such as student projections– Comparison reports, such as value-added summary and

scatterplots

• Roster verification for the student-teacher linkages in teacher value-added reports

• Help supports, such as video clips, online ticketing system, and help pages

• Available through a secure web application with customized access

35

Sampling of CTE Assessments Receiving EVAAS Reporting

• 50 CTE assessment met criteria for inclusion in growth model

Source: http://www.dpi.state.nc.us/docs/effectiveness-model/evaas/resources/vam-assessments.pdf

36

37

38

39

40

41

For More Information….

Contact:

Trey Michael - trey.michael@dpi.nc.gov

Tammy Howard - tammy.howard@dpi.nc.gov

Tom Tomberlin - thomas.tomberlin@dpi.nc.gov

42