Upload
roy-anderson
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
Elizabeth L. Shoenfelt, Ph.D.Alicia Turner, M.A. CandidatePatricia Slack, M.A. Candidate
David Normansell, M.A. Candidate
Department of PsychologyWestern Kentucky University
ASSESSMENT of the CONTENT VALIDITY
of the WATER TRAINING INSTITUTE (WTI)
CURRICULUM
OVERVIEW
Content Validation – Training Program Evaluation Methods
Criterion Phase Participants Procedure
Content Phase Participants Procedure
Analysis
Background: WTI
2 year associate program2 certification programs (Water and
Wastewater)Curriculum: general requirements and 2
specialized tracks (Water and Wastewater)4 projected entry level jobs that WTI graduates
will most likely enter: o Water Treatment Operatoro Wastewater Treatment Operatoro Distribution Systems Operatoro Collection Systems Operator
Background: NSF Grant
Grant evaluation includes an assessment of the content validity of the WTI Curriculum
Will examine the 4 courses currently in place to ensure they are preparing students with the Knowledge Skills and Abilities (KSAs) they need to successfully enter the workforce in any of the 4 identified entry level jobs
Content Validity
CONTENT VALIDITY: The extent to which the material taught in the training course reflects the actual KSAs required for effective job performance
The more similar the WTI training program content is to the job, the more effective it should be in preparing WTI program graduates.
Content Validation – Training Evaluation Methods
Matching Technique (Ford & Wroten, 1984) Analyzed the effectiveness of a Police department training program by identifying Knowledge, Skills, and Abilities (KSAs) considered important to job performance and the KSAs that currently taught in the training program (i.e., time spent)
• Used a matching matrix to graph the data to identify Hits and Misses• Hit: training emphasis reflects training needs (i.e., high importance KSA receives high emphasis in program)• Miss: training emphasis does not reflect training needs
• o Deficiency: high importance KSA receives low emphasis
• o Excess: low importance KSA receives high emphasis
Matching Matrix
Content Validation – Training Evaluation Methods
Linking Technique (Teachout, Sergo, & Ford, 1997) Used the matching matrix but also linked training emphasis to difficulty of learning
More effective picture: High Importance KSA with Low Emphasis – is it a Deficiency or does it have Low Difficulty of Learning?
Current Evaluation
Will examine the content validity of the WTI curriculum
Will use combination of the Matching and Linking Techniques
2 Phases: Criterion (Job)Content (WTI Curriculum)
Criterion Phase: Procedure
Developed preliminary Job Knowledge Surveys (JKS) based on materials received from the Associate Director of the Center for Water Resource Studies at WKU that included the KSAs needed for the 4 entry level jobs
For each element of job information, JKS included ratings for: a) Time Spent on job
b) Importance to jobc) Difficulty of Learningd) When learned (before hire/formal trn/on the
job)e) Should it be taught in WTIf) Is it needed for certification
Job Knowledge Survey: Sample
Criterion Phase: Participants
WTI Steering Committee Members
Knowledgeable individuals about Water and Wastewater Industry for both Kentucky and Tennessee.
From a variety of organizations from Kentucky and Tennessee such as Kentucky Rural Water Association and Tennessee Association of Utility Districts
JKSs were distributed to steering committee members during their WTI workshop/conference in Louisville, KY. The members pilot tested the surveys and provided feedback to refine surveys
Criterion Phase: Procedure
The JKSs were refined based on Steering Committee feedback
Steering Committee members collectively identified 40 water and wastewater incumbents to serve as job content Subject Matter Experts (SMEs) to complete the JKSs(Approximately 10 per entry level job)
The JKS were mailed to the Steering Committee Members for distribution to the identified job incumbent SMEs with a return deadline of October 30, 2009
The JKS data have been entered.We are now in the process of analyzing the JKS data.
Content Phase: Participants
Second Phase is a content analysis of the WTI Curriculum
We will develop Course Content Surveys to determine KSAs taught and training emphasis (time)
WTI Curriculum Content SMEs:o Bowling Green Community College WTI Instructoro Teaching Assistant for the Center for Water Resource
Studies at WKUo The 4 current WTI students also will complete the Course
Content Surveys
Content Phase: Procedure
The SMEs will develop a list of KSAs that are taught in each of the current 4 courses through a “brainstorming” panel discussion
At the end of the discussion, Curriculum SMEs will be provided with the course syllabi and any relevant training materials to further identify KSAs taught
The KSAs identified will be used to develop a Course Content Survey. Each KSA taught in the WTI course will be rated on the time spent teaching the KSA.
The SMEs (including students) will complete the survey
Analysis
• Matching Matrix will be used to compare: • Importance ratings vs. Time Spent Training ratings• Difficulty of Learning ratings vs. Time Spent Training
ratings
• Identify hits and misses in the courses:• Hits – appropriate emphasis in curriculum• Deficiencies – where more emphasis is needed• Excesses – where time on topic can be devoted
elsewhere
Matching Matrix
Formative Evaluation
Data from the Content Validation can be used:
• To revise current WTI courses (if needed) to ensure appropriate emphasis of job-related KSAs
• In developing other WTI courses to ensure they contain appropriate emphasis of job-related KSAs
References
Primary References
Bownas, D., Bosshardt, M., & Donnelly, L. (1985). A Quantitative Approach to Evaluating Training Curriculum Content
Sampling Adequacy. Personnel Psychology, 38. 117-131.
Ford, J., & Wroten, S.(1984). Introducing New Methods for Conducting Training Evaluation and for Linking Training
Evaluation to Program Redesign. Personnel Psychology, 37. 651-665.
Sproule, C. Rationale and Research Evidence Supporting the Use of Content Validation in Personnel Assessment.
Retrieved from International Personnel Assessment Council Website: http://www.ipacweb.org
Teachout, M., Sego, D., & Ford, J. (1997). An Integrated Approach to Summative Evaluation for Facilitating Training
Course Improvement. Training Research Journal, 3. 169-184.
Questions?
Content Validity Ratio ((N imp – N not-imp) / N)