Testing & Credentials in CTE:Alignment is ImperativeAlignment is Imperative

J T A tiJames T. AustinCETE

NACTEI, May 21-22 2008

1

CONTENT AND PRESENTERCONTENT AND PRESENTERCredentialing within CTE covered two ways

Basic concepts: secondary, post-secondaryImplementation: steps, align, evaluate

Jim Austin (CETE, 1997-Present)Industrial Organizational Psychologist Industrial-Organizational Psychologist

Virginia Tech PhD (1987)

Testing to evaluate persons, programsg p , p gCredentialing presentations: 2000, 04, 05

2

TRENDS: EDUCATIONAL TESTINGTRENDS: EDUCATIONAL TESTINGTests sample domains, test-taker responds, items scored, scores “used” for purposes

Technical skills definition: top tier Q at Data Quality Institutes (DQI; 2005 2007)Data Quality Institutes (DQI; 2005-2007)

Classroom AND standardized testingClassroom AND standardized testingDrivers: teachers or districts, states, nationsAlignment of two testing levels lacking

N d f “ t lit “Need for “assessment literacy “

3

CTE: NATIONAL INITIATIVESCTE: NATIONAL INITIATIVESInitiatives: States Career Cluster (NASDCTEc) & College and Career Transitions (CCTI)Perkins IV legislation: accountability ramp-up

P f t d ti l/SEA/LEA Program of study: national/SEA/LEA sourcesEnd-of-program OR end-of-course (EOP-EOC)Industry-based credentials a way to measure Industry-based credentials a way to measure technical skill in tune with business

Transitions emphasis in CCTI & P-O-S4

CTE: STATE OF STATESCTE: STATE OF STATESWhat are other states doing? Other districts?

Ohio trying to build seamless adult systemOhio Skills Bank, Stackable Certificates

Benchmarking crucial for testing-assessment leadership (look around)assessment leadership (look around)Need to scrutinize what others are doing

Who are your comparable states? DistrictsNeighbors, thought leaders, NACTEI

5

TESTING PROJECT SAMPLERTESTING PROJECT SAMPLERSampling of CETE projects: experience

Statewide testing systems & Webxam testing/data collection portal

OCTCA hosts WFL tests also customized systemOCTCA hosts WFL tests, also customized system

Occupational analyses with verificationAligning standards or test items (CTE, ABE)Aligning standards or test items (CTE, ABE)Certification testing and systemsWork Readiness: State & national levelsTest Dev Workshop: capacity building

6

CTE Standards‐based Educational Reform:  “What is what”

Covered AND Prescribed

Academic, Skill Standards

Teaching & Instruction

Tested AND

Covered ANDT d AND

State Testing, 

Instruction AND Prescribed

Tested AND Prescribed

C d AND T t d

g,Credentials

7

Covered AND Tested

TECHNICAL SKILL & ATTAINMENTTECHNICAL SKILL & ATTAINMENTFramework (3WH) Measurements( )

WHOWHAT

Local (LEA) State (SEA)

WHENHOW

Third-party National/federal

WHY test: Perkins IV but mostly

Credential fits with third-party, national modelIV, but mostly

learnersnational model

8

WHO IS TESTED?WHO IS TESTED?Student definitions

ConcentratorNumerator / Denominator

How are special needs CTE students tested and reported?

Effects of POS on who is tested?

9

WHAT IS TESTED?WHAT IS TESTED?Definition: Knowledge / Competency portions

Relationship of Technical Skill Attainment to Other Indicators: Academic Completion Other Indicators: Academic, Completion

Distinguishing foundational/generic and ti l Skill occupational Skills

Clarifying clusters, pathways, P-O-S

Testing aligned content that is instructed!!10

WHEN TO TEST?WHEN TO TEST?Temporal: Most testing is EOP, but EOC

id b di i bili (d illd )provides better diagnostic capability (drilldown)

Importance of showing gain (baseline – final) Importance of showing gain (baseline final) or even maintenance (multiple post-tests)

Example: Career-Based Intervention in OHF di f F d Ed t d l Funding from Fed-Ed to develop non-cognitive measures usable by practitioners at levels: local (LEA) and state (SEA)

11

HOW TO TEST?HOW TO TEST?Development or selection: LEA vs SEA locus

Evaluating suitability & value to student

Defining & establishing validity / reliability Defining & establishing validity / reliability Aligned to skill standards (academics?)

Development / administrative costsPooling / sharing resourcesN tion l Item B nk o le ingho eNational Item Bank or clearinghouse

12

HOW TO TEST: Alignment ModelHOW TO TEST: Alignment ModelOutput Results Impact

Content

Assessment

I t ti Instruction

13

Global Skills Xchange; www.gskillsxchange.com/

CREDENTIALS: BASIC CONCEPTSCREDENTIALS: BASIC CONCEPTSWhat is a credential?

Who earns a credential? How do they it?earn it?

What career outcomes are associated What career outcomes are associated with credentials (tipping points)?

Are there standards for certifications?14

CREDENTIALS: BASICSCREDENTIALS: BASICSWhat is a credential?

License or certification granted by a body [ t i ti ti t ][government, association, corporation, etc]

Government does licenses, much wider for certsNational Organization for Competency Assuranceg p yis professional group for certifications (voluntary)Council on Licensure, Enforcement, and Regulation is similar group for licensingg g p g

15

CREDENTIALS: BASICSCREDENTIALS: BASICSType of Body Example Body Example Credential

State regulatoryboard

State Board: Well-driller, Cosmetology

Licensed well-driller, cosmetologist

Professional i ti

National Board for R i t C

CPFT, CRT, NPS, RPFT, RRTassociation Respiratory Care RRT

Trade assoc National Automotive Technical Education Foundation

Automotive Service Excellence; E O P TestsFoundation E-O-P Tests

Business firm Microsoft, Cisco MSCE / MCP; CCNA

F d ti ECDL F d ti ECDL ICDL b i Federation ECDL Foundation ECDL, ICDL basic digital ITAC

16

CREDENTIALS: BASICSCREDENTIALS: BASICSWho earns credentials? How are they earned?

Credentials awarded by decision rules (3-E)Education (HS, AA, BS, etc.)Experience (time, quality)Examination score (knowledge, performance)( g p )

Usage implies validation for all 3E, but exams get most scrutinyG ti b d ti l t tt t Granting body stipulates attempts, recertification, etc. 17

CREDENTIALS: BASICSCREDENTIALS: BASICSWhat labor market outcomes are associated with earning credentials (tipping point)?

D d ti l i ti Do credentials increase compensation, occupational level, & job security?

How to know? Marketing versus evidence!!gPerceived Value of Certification Tool (PVCT) is an 18-item measure with intrinsic & extrinsic parts

Washington state research indicates “one Washington state research indicates one year + credential” is positive tip point

18

CREDENTIALS: BASICSCREDENTIALS: BASICSAre there quality standards for credentials?

You bet!! Accredited credentials offer best l b t till j t i f CTEvalue, but still just one piece for CTE

Standards for certifications promulgated by organizations for audit purposes organizations for audit purposes

NOCA (2005) has 21 standards [10-18 on test]ISO/ANSI (2003) has 6 standardsA k t ti l d b ? dit d?Ask potential vendors: members? accredited?

19

CREDENTIALS: CTE PROCESSCREDENTIALS: CTE PROCESSHow could CTE stakeholders use credentials?

H th ti l l f d ti l Hypothetical example of credential use

How should CTE stakeholders use How should CTE stakeholders use credentials?

A process for using credentials20

CREDENTIALS: CTE PROCESSCREDENTIALS: CTE PROCESSHow could CTE stakeholders use credentials?

What are models-in-use?Exploratory in which SEA allocates $$ to early adopters, who report back to SEA on lessons learned (Ohio allocated $$, no data back yet)Implemented, for example Virginia, with process outlined & applied (web-based look-ups)Aspirational/future to synthesize what is known and make improvements

21

CREDENTIALS: CTE PROCESSCREDENTIALS: CTE PROCESSHypothetical example of use of credentials

Cross-cutting example: CTE TEACHERSState licensing involves 3E but mostly exams (Praxis II-III; NES) at basic levelNational Board for Professional Teaching Standards (NBPTS) certifies advanced knowledge using portfolio & assessment center

Carry logic backwards to CTE teacher prep y g p p(CPTP pathway in Education-Training Career-Field is great for transitions!!) 22

CREDENTIALS: CTE PROCESSCREDENTIALS: CTE PROCESSHow should CTE stakeholders use credentials?

Aspirational/future model meets Perkins & b dmoves beyond

Synthesize what is known into defensible stepsPrescriptive approachp pp

Describe, use, then improveSampling states: CETE doing telephone interviews to ID themes & needed services, addressing needs to ID themes & needed services, addressing needs in quarterly Centergram & other formats

23

CREDENTIALS: CTE PROCESSCREDENTIALS: CTE PROCESSHow should CTE stakeholders use credentials?

Overall, an evolved process from Mahlman-A ti (2003) i t d l li k bl t Austin (2003) nine-step model, linkable to

P-S transitions emphasis [CCTI]Standards for certifications [NOCA ANSI]Standards for certifications [NOCA, ANSI]Levels of career clusters initiative [SCCI]

24

CREDENTIALS: CTE PROCESSCREDENTIALS: CTE PROCESSProcess for using credentials

1. Database state / local CTE programs (levels)2. Identify / import licenses & certifications 3. Conduct preliminary screen for usability

A Virginia process one modelA. Virginia process one modelB. Generic model of usability (need rubric)

4 Survivors receive enhanced screening4. Survivors receive enhanced screeningA. Options for rigor 25

CREDENTIALS: CTE PROCESSCREDENTIALS: CTE PROCESSProcess for using credentials

5. Investigate perceived / actual valueA. Stakeholders include business, students

6. Disseminate project results in multiple waysA Searchable web interface through portalA. Searchable web interface through portalB. Database download for savvy tech userC PDF to print remotely for late adoptersC. PDF to print remotely for late adoptersD. Word/WP/HTML for manipulation 26

CREDENTIALS: CTE PROCESSCREDENTIALS: CTE PROCESSProcess for using credentials?

What is enhanced screening? Varies, but ld i l d could include ….

Alignment of practice analysis to skill standards in g p yfull GSX model, and statewide CT2 articulationStudy perceived/actual value (surveys, focus groups, salary data)g p y )Field test w/ CTE providers (districts, CTPD)

27

SUMMARY: CREDENTIALSSUMMARY: CREDENTIALS

Credentials complement standardized tests, to offer access alternatives as “parallel universe”

Basic & use/process features addressedBasic & use/process features addressed

Application experience awaits a project with Application experience awaits a project with career pathways in Ohio

Not with CTE, but through WIA!!

Have to start somewhere! Why not now? 28

Digital (ICDL, IC3),CLUSTERCLUSTER first aid/CPR, food

safety, OSHA 10-

PATHWAYPATHWAYLID

ER

yhr, tools, plants

PATHWAYPATHWAY

VE

L SL

??

LE

V

OCCUPATION / OCCUPATION / SPECIALIZATIONSPECIALIZATION

ASE, licenses, NCCER, STNA,

29A+, CDA

DEPARTING THOUGHTS ……..DEPARTING THOUGHTS ……..Hope that your goals were met

Four themes for credentials – definition, technical quality, CTE usability, & fairness

Email with additional feedback/questions toaustin 38@osu eduaustin.38@osu.edu

30

DEVELOPING TESTS: STEPS1. Determine Test Purpose2. Define Content Domain (Practice / Job Analysis, Skill Standards)

3. Create/Evaluate Test Blueprint4. Write, Manage, & Evaluate Items (and item writers)g ( )5. Set Cutoff Scores6. Develop, Evaluate, Refine Test Forms7. Use Reliability, Validity, Fairness as Quality Yardsticks

8. Interpret & Report Scores to Stakeholders

31

9. Maintain Testing System10. Evaluate Tests: Own & Others

P S d P S dPost-Secondary Education: Academic

Post-Secondary Education: Technical

Academic Content Standards (NCLB)

Career-Technical Secondary

Industry-Recognized Skill Standards (

B i P tiBasic Preparation Academic Content Standards

Career Awareness Middle School N/A

32

Table Four: Selected Usability-Technical Quality Indicators with Criteria

Sample Evaluative Standard Example Levels (finalized by panel)Sample Evaluative Standard Example Levels (finalized by panel)

Alignment with Ohio Career Field Technical Content Standards (CFTCS)

d/ P f S d (U bili )

At least 60% alignment (at pathway level)

and/or Program of Study (Usability)Cost to Student, to LEA or to SEA (Usability)

Price points with comfort levels (tradeoffs)

D A il bili f S d / CTE M i f i il biliData Availability for Student / CTE System (Usability)

Maximum for maximum availability

Credential Time-span (Usability) At least 3 to 5 yearsReliability Evidence (Technical Quality) Technical report; level of reliability

estimateValidity Evidence (Technical Quality) Technical report; quality of evidence

Cutoff Evidence (Technical Quality) Technical report section; quality of procedures & controls for bias

33