Nationally Normed Testing via ACS Exams:
Research and Practice
Thomas Holme Iowa State University
ACS DivCHED Examinations Institute
ChemEd 2012, Adelaide
A fundamental challenge
• Teaching is, at once, inherently personal and inescapably corporate.
• At present, the corporate interests in student learning are often articulated in terms of assessment.
Exams Institute?
• How is it that chemistry in the US has an Exams Institute?
A Short History
1921
Division of Chemical Education
A Short History
1930 Committee on Examinations and Tests Formed to construct or exploit the development of
objective test construction and train teachers how to construct tests for their own classes
A Short History
1930
Committee on Examinations and Tests
“…only specialists in the field can write good tests because they are the persons who know what is significant and important, rather than the test experts who know the forms and techniques of test construction but not the subject matter”
Ted Ashford on Ralph Tyler’s position on testing
A Short History
1930
Committee on Examinations and Tests
The Committee was subsidized by:
General Education Board of the Cooperative Test Service
Carnegie Foundation for the Advancement of Teaching
Dr. Ben Wood
A Short History
1934
A group of five Committee members released the first general chemistry test in three forms
A Short History
1946
Ted Ashford appointed as Chair of the Committee
A Short History
1984
Committee on Examinations and Tests renamed to Examinations Institute
Board of Trustees appointed to oversee operation of the Institute
A Short History
1987
Dwaine Eubanks appointed Director
Examinations Institute moves to Oklahoma State University
A Short History
2002
Tom Holme appointed Director
Examinations Institute moves to University of Wisconsin – Milwaukee
- (2008) Moves to Iowa State
Reason Exams Institute exists?
• ~ P.T. Barnum
• There’s a sucker born every 2 or 3 decades
• “on average”
The key constituencies
• Practitioners
– Often motivated by practicality
• Chem Ed Researchers (TUES grant recipients)
– Often motivated by validation challenges
Exam development
• Chair is named • Committee is recruited • First meeting - sets content coverage • Items are written and collated • Second meeting - editing items, setting trials • Trial testing in classes - provides item stats • Third meeting - look at stats and set exam • Meetings are held in conjunction with ACS National
Meetings (or BCCE) – Partial reimbursement to volunteers
Gen Chem Exams
• Full Year Exam (2009, 2011)
• First Term Exam (2005, 2009)
• Second Term Exam (2006, 2010)
• 1st Term Paired Questions (2005)
• 2nd Term Paired Questions (2007)
• Conceptual (1st term, 2nd term, full year)
• Full year - brief exam (2002, 2006)
• All exams carry secure copyright – Released – not published
Norms and reporting
• Norms are calculated on voluntary return of student performance data
• We have an interactive web site for score reporting for exams that do not yet have enough data to report a norm.
• People often use norm (percentile) to help students who transfer to other programs.
Exams as artifacts of teaching
• Because ACS Exams have been around for a long time, they provide an additional artifact of what the community values.
• Consider Organic Chemistry Exams – Analyze cognitive demands
• Vast majority of items on organic exam qualify as conceptual understanding
– Analyze chemistry content. • Content coverage shows modest fluctuation, but few
fundamental shifts in the past 20 years.
Then and now
• In addition to historic value, the current efforts of the Exams Institute also shed light on research and practice.
• First a bit on curricular practice.
• Then some a research example.
Criterion referencing for program assessment
• Requires criteria
• At the college level, they don’t exist.
• Build a consensus content map.
• Similar to using backward design1.
1: Understanding by Design, Grant P. Wiggins, Jay McTighe
Anchoring Concept
• Use “big ideas” or anchoring concepts to organize content across disciplines.
• Build levels with finer grain size down to the point where exam items are generally written.
Levels of criteria map
• Anchoring Concept
• Enduring Understanding
• Sub-disciplinary articulation
• Content details
Process for setting map (so far)
• Begin from EMV conference ideas • Focus Group (Mar08): Level 1 + Level 2 • Workshop (Jul08): Level 2 + Level 3 (General) • Focus Group (Aug08): Level 2 + Level 3 (Organic) • Workshop (Mar09): Level 3 + Level 4 (General) • Focus Group (Aug09): Level 2 + Level 3 (Organic) • Workshop (Mar10): Alignment (General) • Focus Group (Mar 10): Level 2 + Level 3 (Physical) • Focus Group (Jul 10): Level 3 Organic • Focus Group (Dec 10): Level 3 + Complexity Organic • Focus Groups (Mar 11): Level 3 (Analytical, Biochem, Physical) • Focus Groups (Aug 11): Level 3 + Complexity (Organic) • Focus Groups (Mar 12): Level 4, Alignment, Complexity (General,
Organic)
Example of comparing content
Access?
• The Gen Chem version has been published. – Uses Authors Choice
so it should be downloadable.
• Organic is expected to be publishable by fall of this year.
Item Alignment
• Look at current items from ACS Exams and align them to Level 3/4
• Process guided by psychometric experts.
• Can include both skills and content
• Ultimately can help define specifications for future ACS Exams.
Comparison of gen and org
Enjoy the ride…
• This project shows how discussions around testing/benchmarking could ultimately lead to curricular forcing.
• The discussions remain “grass roots”
• The Exams Institute provides the playground and occasionally has to decide what rules to follow, but doesn’t push an “ACS” agenda.
But, testing…are you sure?
• Are we really that confident in our measurements?
Research: Teachable moments
• Because a sizeable fraction of the Chem Ed community uses (or at least trusts) ACS Exams, the characterization of the exams allows an avenue to educate about assessment issues.
Recently taught topics
• Role of item complexity
• Item characteristic curves
• Item Order Effects
• Answer Order Effects
• Differential Item Functioning (DIF)
• Partial credit / polytomous scoring
Content vs. construct
• Tests demand that students complete tasks
• Each item is a task
• Students need knowledge within the content domain (chemistry)
• Students need knowledge about how to organize their efforts (test taking)
• Cast this understanding in terms of item complexity.
The Information Processing Model
• Consider the task in terms of the Information Processing Model of Johnstone and coworkers.
Johnstone, CERP, 7, 49-63 (2006)
Estimating Task Complexity (Johnstone and coworkers)
• Count up the pieces of information needed to accomplish the task
• Compare to student performance
Estimating task complexity elsewhere
Paas & Van Merriënboer’s 9-point scale (1994).
1 ~ very, very low mental effort 2 ~ very low mental effort 3 ~ low mental effort 4 ~ lower than average mental effort 5 ~ average mental effort 6 ~ higher than average mental effort 7 ~ high mental effort 8 ~ very high mental effort 9 ~ very, very high mental effort
Data for each chemistry exam item
• Performance data (difficulty index)
• Expert-rated objective complexity
• Mental effort (hypothesized to represent the subjective complexity)
Three constructs of task complexity
• Complexity treated as a psychological experience. – Subjective complexity
• Complexity treated as a function of objective task
characteristics. – Objective complexity
• Complexity treated as an interaction between task and
person characteristics.
Principle component analysis
Communalities
1.000 .700
1.000 .629
1.000 .704
error rate
complexity (rating)
mental effort (rating)
Initial Extraction
Extraction Method: Principal Component Analysis.
Total Variance Explained
2.033 67.768 67.768 2.033 67.768 67.768
.537 17.910 85.678
.430 14.322 100.000
Component
1
2
3
Total % of Variance Cumulativ e % Total % of Variance Cumulativ e %
Initial Eigenv alues Extract ion Sums of Squared Loadings
Extract ion Method: Principal Component Analy sis.
Component Matrixa
.837
.793
.839
error rate
complexity (rating)
mental ef fort (rating)
1
Component
Extraction Method: Principal Component Analysis.
1 components extracted.a.
Factor analysis
• Factor Analysis finds a single factor with all PCA loading factors above 0.75
• Hypothesis: This factor represents the complexity of multiple-choice chemistry items.
• Principal axis factoring and maximum likelihood factoring both reveal a single factor as well.
Take home message
• Depending on the model used for factor analysis the amount of variance changes (from 51% to 67%).
• Nonetheless, for Gen Chem half of the variance of student performance can be explained by the latent variable of task complexity.
• Not arguing against complex content – arguing for awareness of complexity when making measurements
• Current work is looking at Organic. – Key challenge is assigning task complexity.
What can researchers use?
• High quality individual instruments
– DUCK
– Paired question exams
– On-line laboratory assessment
• Includes full motion video and novel item constructs
• A support system for program assessment.
– Criterion referencing
What’s a DUCK
• Diagnostic of Undergraduate Chemistry Knowledge.
• Fundamentally inter-disciplinary
• Scenario Based.
• 15 Scenarios, each with 4 items.
• Taken at/near the end of the undergraduate curriculum. – Sometimes in a capstone course, sometimes as an
extra.
What have we learned?
• Analysis of performance from US students on current DUCK.
Other advantages?
• Having the Exams Institute provides a venue for looking at the interaction of measurement and learning in chemistry.
• Complexity
• Interdisciplinarity
• Measurement factors – Item order effects
– Differential Item Functioning
– Coming: Human-computer interaction
Acknowledgements
• Kristen Murphy (UWM)
• Jeff Raker (ISU)
• Kim Linenberger (ISU)
• Mike Slade (ISU)
• Heather Caruthers (ISU)
• Anna Prisacara (ISU)
• Jessica Reed (ISU)
• John Balyut (ISU)
• April Zenisky (Umass)
• Karen Knaus (UC-Denver)
• Jacob Schroeder (Clemson)
• Mary Emenike (Rutgers)
• Megan Grunert (W. Michigan)
• Chris Bauer (New Hampshire)
NSF: DUE-0618600, 0717769, 0817409, 0920266
[email protected] @Tom_Holme
Current Collaborators Prior Collaborators