Chair Academy Perkins Iv Sandy

Perkins IV Perkins IV Assessment of Assessment of Technical Skill Technical Skill

AttainmentAttainment

Wisconsin Technical College System

Sandy SchmitWTCS Foundation – WIDS

Judy Neill

Overview of the Wisconsin Technical College System

What does Carl Perkins 2006 (Perkins IV) require?

How does curriculum fit in? How is WTCS using performance

assessment? What will WTCS require for 3rd Party

Assessments? How can WIDS help?

Overview

The Wisconsin Technical The Wisconsin Technical College System (WTCS)College System (WTCS)

16 Technical Colleges

349 unduplicated Career and Technical Education Programs

1179 Duplicated Career and Technical Education Programs

390,272 enrolled students in 2008, 69,632 full time equivalent

23,617 Degrees granted in 2008

WTCS

How do you know when How do you know when your students “get it”?your students “get it”?

Assessment?Assessment?

Summative Summative Assessment?Assessment?

Carl Perkins IV?Carl Perkins IV?

What does Carl Perkins What does Carl Perkins 2006 (Perkins IV) 2006 (Perkins IV)

require?require?

“Student attainment of challenging career and technical skill proficiencies, including student achievement on technical assessments that are aligned with industry-recognized standards, if available and appropriate.” (U.S. Department of Education, 2006).

The Law

Focuses on industry aligned skill sets.

Students know they are acquiring skills that have value & portability.

Increases odds that students will be successful in their employment.

Ensures that WTCS is educating students to meet the needs of employers.

Big Picture Perspective

Increased accountability to improve teaching and learning

Accreditation (NCA, AQIP, industry)

Carl D. Perkins IV legislation and funding

Drivers to Process

StudentsFeedback about skills needed on the jobAdded value for education

CollegesImprovement of teaching and learningSupports overall mission

EmployersAssurance that WTCS graduates have attained technical skills needed on the job

Benefits

Carl D. Perkins 2006

Newly revised Career and Technical Education (CTE) Law

Key Concept - Demonstrate that program graduates have attained the knowledge and skills that are industry validated career and technical skills

Preparation for postsecondary education and employment

Preparation not only to get a ‘job,’but ‘academic and technical preparation’

Increased emphasis on achievement of a degree, certificate or credential

Perkins IV Areas of Emphasis

Increased Accountability

New accountability provisions are a reminder that:

Funds are not an entitlement

Use of funds is flexible & responsive to the accountability data

Separate secondary and postsecondary measures

Performance levels at state and local levels

Negotiations and redirection of funds

Increased Accountability

Indirect Measures Data that support

inferences about the quality of teaching and learning and give insight into why achievement is high or low:• Enrollment• Retention rates• Course completion• Student/graduate/

employer satisfaction surveys

• Placement statistics

Direct Measures Data that provide

evidence that students have achieved the intended outcomes:• Performance

assessments with rubrics• Portfolios• Artifacts• Performances• Outcome referenced

tests

Increased Accountability for Direct Measurement of Learning

Denominator: number of CTE concentrators who took technical courses during the three year reporting period

Prior accountability required only indirect measurement:

Numerator: Number of CTE concentrators who passed 80% of the technical courses they attempted during the three year reporting period

Measure =


•Denominator: Number of CTE concentrators who took technical skill assessments during the reporting year

New OVAE Guidance for Perkins IV Measurement requires direct measures of skill attainment:

Numerator: Number of CTE concentrators who passed technical skill assessments that are aligned with industry-recognized standards, if available and appropriate, during the reporting year

Measure =


Department of Education goal is to have end of program outcomes assessments that are:• Direct measures of outcomes• Independent• 3rd-Party• Employer validated

By 2015 most WTCS assessments should be external or approved assessments.


WTCS TSA Assessments External TSA Assessment

(ex. NCLEX, Barber/Cosmetology, etc.)

Indirect Assessments

Wisconsin Standards for Assessment - WTCS


WTCS Assessment StandardsWTCS approved assessment that objectively measures student attainment of industry recognized skills upon graduation.

May be developed and/or implemented by the state system.

May be developed and/or implemented locally according to system guidelines and with industry (advisory committee) approval.

Must be valid and reliable.

External Assessment Standards Indirect Assessment Standards



WTCS Assessment Standards External Assessment Standards (ex.

NCLEX, ASE, Barber/Cosmetology)

External, third-party assessment that objectively measures student attainment of industry recognized skills upon graduation.

Indirect Assessment Standards



WTCS Assessment Standards External Assessment Standards Indirect Assessment Standards

Non-assessment related indicators such as indirect measures.GPA and/or Course CompletionProgram Completion – Graduation

Teacher-developed exams that do not meet WTCS standard (not WTCS approved)



WTCS will move toward a balance of WTCS TSA Assessments and External TSA Assessments

WTC

S Assessm

ent

External A

ssessment

Indirect

2012-2013

New Accountability for Direct Measurement of Learning

How does curriculum fit How does curriculum fit in?in?

Plan•Establish outcomes•Build assessments•Design learning

DO• Deliver• Assess• Document

CHECK• Analyze assessment

data• Propose

improvements

ACT•Adjust design

•Adjust delivery

•Adjust assessment

Assessment Process and Curriculum

WIDS Model 3 of 4 components answer assessment questions

What do they need to be able to achieve?

Who are the learners?

How will we know and show when they’ve achieved it?

How will they get there?

WHO

WHAT

HOW

Program Outcomes Measurable, observable, and field-

specific skills – major outcomes• Number 5-7 per program (guideline not rule)

Originate from:• Current DACUMs• Accrediting Agencies• National (or other) Skill Standards• Advisory Committees

Threaded through courses Performance verified with

summative assessment of skill performance

Integrating Outcomes

Linked Competenci

es(performance assessments)

Exit Learning Outcomes

Program Outcomes(summative assessment)

Core Abilities

(summative assessment)

Gen Ed Outcomes(summative assessment)

Job Task Analysis(DACUM)

Job Task Analysis(DACUM)

External Standards

(Research-based)

External Standards

(Research-based)

Linked Competenci


Linked Competenci


Formative assessment of program outcome

Summative assessment of learning outcomes

External

Program-Level

Course-Level

Drives Learning Design

Assesses Learning Results

Drives Learning Design

Assesses Learning Results

External

Program-Level

Course-Level

Dental Hygiene Example

Graduates are competent in applying ethical, legal and regulatory concepts to the provision and/or support of oral health care services.

Incorporate into dental hygiene practice professional laws, regulations and policies established by the licensing state and regulatory agencies

Respond to a request to perform a task that is not legally permitted to be delegated to a dental hygienist.

ADA Dental Hygiene Education Standard

Dental Hygiene Program Outcome

Dental Hygiene CompetencyFormative assessment of program outcome

Summative assessment of Learning outcomes

Summative/Formative Assessment

Sound Assessment

Valid Outcomes based on standards

(industry) Measures intended outcomes Measures application and critical

thinking

Reliable Performance assessment based on

consistent rubrics, scoring guides, and rating scales

Consistent process - each learner is assessed in same way as other learners

Fair Valid Reliable Learners informed of expectations

up front! Feedback to learners

Standards for WTCS Standards for WTCS Approved TSA Approved TSA AssessmentsAssessments

WTCS will move toward a balance of WTCS TSA Assessments and External TSA Assessments

WTC

S Assessm

ent

External A

ssessment

Indirect

2012-2013

New Accountability for Direct Measurement of Learning

is valid and reliable;meets WTCS standards/guidelines; is approved by industry (advisory

committee); is developed and/or implemented

collaboratively by the WTCS colleges; ordeveloped and/or implemented locally

according to system guidelines and with industry (advisory committee) approval.

WTCS Assessment Standards

WTCS approved assessment objectively measures student attainment of industry recognized skills upon graduation and:

Available in your handout packet

WTCS Standards forWTCS Standards for33rdrd-Party External -Party External

AssessmentsAssessments

3rd Party TSA Also referred to as:

• External• Direct Method• Credential• License

Generally controls entrance into an occupation

Validity / Reliability Alignment to Standards Availability of data Value to the Student Development /

Administrative Costs Pooling / Sharing Resources

Considerations for selecting 3rd Party External Assessments

Can we obtain the results/data?

Are cost and administration feasible?

Does it add value or control entry to the student/occupation?

Is it valid, reliable, reputable, recognized?

Is it summative, cumulative or a partial measurement of skills?


Does the assessment align with program outcomes?

Does the assessment measure what needs to be measured?

Does the data help improve and strengthen our programs?


WTCS 3rd Party Assessments Currently in Place

Occupations regulated by the Department of Regulation and Licensing• Barber/Cosmetology • Funeral Director • LPN • Registered Nurse• Dental Hygienist• Massage Therapist• Occupational Therapy Assistant• Physical Therapy Assistant• Respiratory Care Practitioner• Veterinary Technician• Wisconsin Registered Interior Designer

Guidelines for Selecting 3rd Party Assessments

This is a guiding document to assist districts in determining direct technical skill assessment appropriateness.

Districts will work with the state office staff in the determination.

Keep in mind this is for a small number of programs.


Concerns: Only measures part of the program

outcomes-”All Code-No Application” Does not indicate job performance Does it help employers? Focused only on technical skills,

what about academic skills and soft skills?


3rd Party Assessment - Example

Automotive Service Excellence (ASE) Long-standing credential of

automotive technicians A series of 8 certification

areas/assessments Would it work for Perkins?

Logistics Professional CPL/DL-International Society of Logistics

CPIM/CSCP-Association for Operations Management

PLS/CTL/DLP conferred by the American Society of Transportation & Logistics


Federal Aviation Administration (FAA) Airframe and Powerplant Mechanic Certificate

Administered by the FAA Written, practical and oral examination

Required for entry


How Can WIDS Help?How Can WIDS Help?

The Learning & Assessment Design Technology Tool Kit

Tra

cDat

B

lack

board

iWebfo

lio

eFo

lio

Quest

ionm

ark

People

Soft

B

anner

Data

tel

Create/maintain Program Outcome Summaries for all new and existing programs.

Develop/maintain Course Outcome Summaries for all courses.

Design Performance Assessment Tasks for course level (formative) and summative assessments.

Use WIDS Analyzer to build Learning Outcome Matrices that document how outcomes are linked with programs and courses.

Present WIDS documentation as a central feature in accreditation.

How can WIDS help?

WIDS Model and Software Tools:

Facilitate collaborative development of:• DACUM or Industry Review Focus

Groups• Program Outcomes• Program Outcome Summaries• Summative Performance

Assessments Produce

• Industry Approval Reports• Analyzer Reports• Updated WIDS files and reports

How can WIDS help?

WIDS Team

How do we communicate WTCS long-standing commitment to industry input and standards?

How do we ensure that we use quality assessments? (That we don’t regress to testing regurgitation of information?)

How do we ensure that we actually are doing what our model and assessment plans imply?

How do we collect and report data?

Challenges

Current ProjectsCurrent Projects

Automotive Technology/Technician NATEF end of program evaluation

Accounting WIDS

Early Childhood WIDS

Criminal Justice WIDS

Welding WIDS

Current Projects