Michigan Assessment Michigan Assessment ConsortiumConsortium
Common Assessment Common Assessment Development SeriesDevelopment Series
Putting TogetherPutting TogetherThe Test BlueprintThe Test Blueprint
Developed and Narrated byDeveloped and Narrated by
Bruce R. Fay, PhDBruce R. Fay, PhDAssessment ConsultantAssessment Consultant
Wayne RESAWayne RESA
SupportSupport
The Michigan Assessment Consortium The Michigan Assessment Consortium professional development series in professional development series in common assessment development is common assessment development is funded in part by the funded in part by the Michigan Michigan Association of Intermediate School Association of Intermediate School AdministratorsAdministrators in cooperation with in cooperation with MDE, MSU, Ingham & Ionia ISDs, MDE, MSU, Ingham & Ionia ISDs, Oakland Schools, and Wayne RESA.Oakland Schools, and Wayne RESA.
What You Will LearnWhat You Will Learn
Test blueprints…what they are and Test blueprints…what they are and why you need themwhy you need them
The components of a test blueprintThe components of a test blueprint Criteria for a good test blueprintCriteria for a good test blueprint Test blueprint examplesTest blueprint examples
““If you don't know where If you don't know where you're going, any road will you're going, any road will take you there.”take you there.”
George Harrison (1943 - 2001)George Harrison (1943 - 2001)
"Any Road", Brainwashed, 2002"Any Road", Brainwashed, 2002
Assessment with a PurposeAssessment with a Purpose
Educational assessment is not something Educational assessment is not something incidental to teaching and learning. It is an incidental to teaching and learning. It is an equal partner with curriculum and equal partner with curriculum and instruction. It is the critical “3instruction. It is the critical “3 rdrd leg” through leg” through which both students and teachers receive which both students and teachers receive feedback about the effectiveness of the feedback about the effectiveness of the teaching and learning process in achieving teaching and learning process in achieving desired learning outcomes. Assessment desired learning outcomes. Assessment closes the loop.closes the loop.
Closed–LoopClosed–Loop(Feedback) Systems(Feedback) Systems
Home Heating System(Teaching & Learning)
Desired Temperature(Learning Target)
Actual Temperature(Test Results)
C – I – A AlignmentC – I – A Alignment
Requires thoughtful alignment – Requires thoughtful alignment – ensuring that the items on a test fairly ensuring that the items on a test fairly represent the…represent the… Intended learning targets (intended Intended learning targets (intended
curriculum)curriculum) Actual learning targets (taught Actual learning targets (taught
curriculum)curriculum)
Test what you teach, teach what you testTest what you teach, teach what you test
Target–Level AlignmentTarget–Level Alignment
Relative importance of those targetsRelative importance of those targets Level of cognitive complexity Level of cognitive complexity
associated with those targetsassociated with those targets
Useful feedback requires Useful feedback requires tests that are…tests that are… Reliable (consistent; actually measure Reliable (consistent; actually measure
something)something) Fair (Free from bias or distortions)Fair (Free from bias or distortions) Valid (contextually meaningful or Valid (contextually meaningful or
interpretable; can reasonably support interpretable; can reasonably support the decisions we make based on the decisions we make based on them)them)
Test BlueprintsTest BlueprintsThe Big IdeaThe Big Idea
A simple but essential tool, used A simple but essential tool, used to:to: Design tests that can meet the Design tests that can meet the
preceding requirementspreceding requirements Define the acceptable evidence to Define the acceptable evidence to
infer mastery of the targetsinfer mastery of the targets Build in evidence for validityBuild in evidence for validity
The Test Blueprint orThe Test Blueprint orTable of Test SpecificationsTable of Test Specifications
Explicitly “map” test items to:Explicitly “map” test items to: Learning TargetsLearning Targets Levels of ComplexityLevels of Complexity Relative ImportanceRelative Importance
Provides “common” definition of the Provides “common” definition of the testtest
Learning Targets &Learning Targets &Standards FrameworksStandards Frameworks Standards as structured hierarchical Standards as structured hierarchical
frameworks. Michigan’s is:frameworks. Michigan’s is: StrandsStrands StandardsStandards DomainsDomains Content ExpectationsContent Expectations
Detailed curriculum is usually left to Detailed curriculum is usually left to local districts or classroom teachers.local districts or classroom teachers.
A Simple Taxonomy ofA Simple Taxonomy ofCognitive ComplexityCognitive Complexity
Norm Webb’s Depth Of Knowledge (1997)Norm Webb’s Depth Of Knowledge (1997)(highest to lowest)(highest to lowest)
Extended ThinkingExtended Thinking Strategic ThinkingStrategic Thinking Skill / concept use / applicationSkill / concept use / application RecallRecall
Putting it all together…Putting it all together…A Basic Test BlueprintA Basic Test Blueprint
Table (matrix) format (spreadsheet)Table (matrix) format (spreadsheet) Rows = learning targets Rows = learning targets (one for each)(one for each)
Columns = Depth of KnowledgeColumns = Depth of Knowledge Cells = number of items and points Cells = number of items and points
possiblepossible
Summary InformationSummary Information
Number of items and points possible:Number of items and points possible:
Row Margins = for that targetRow Margins = for that target
Column Margins = for that level of Column Margins = for that level of complexitycomplexity
Lower Right Corner = for the testLower Right Corner = for the test
Example 1 – Basic BlueprintExample 1 – Basic Blueprintfor a test with 5 learning targetsfor a test with 5 learning targets
LearningTargets
Recall# (pts)
Use# (pts)
Strategic# (pts)
Extended#(pts)
Target Totals# (pts)
Target 1 3 (3) 2 (2) 5 (5)
Target 2 1 (1) 2 (2) 2 (4) 5 (7)
Target 3 2 (2) 1 (3) 3 (5)
Target 4 3 (3) 1 (2) 4 (5)
Target 5 2 (4) 1 (4) 3 (8)
Level Totals# (pts)
6 (6) 10 (14) 4 (10) 20 (30)
Is this reasonable?Is this reasonable?Rule of Thumb Criteria…Rule of Thumb Criteria…
At least 3 items per target for reliabilityAt least 3 items per target for reliability Appropriate:Appropriate:
Distribution of items over targetsDistribution of items over targets Levels of complexity for Levels of complexity for
targets/instructiontargets/instruction Distribution of items over levels of Distribution of items over levels of
complexitycomplexity
(all items are NOT at the lowest or highest level)(all items are NOT at the lowest or highest level)
Professional JudgmentProfessional Judgment
Like all things in education, the Like all things in education, the development of assessments and the development of assessments and the use of the results are dependent on use of the results are dependent on professional judgment, which can be professional judgment, which can be improved through…improved through… ExperienceExperience CollaborationCollaboration Reflection on methods Reflection on methods Results Results
Limitations…Limitations…
Shows total points for each target/level Shows total points for each target/level combination, but not how those points combination, but not how those points apply to each itemapply to each item
Doesn’t show item typesDoesn’t show item types Doesn’t indicate if partial credit scoring Doesn’t indicate if partial credit scoring
can/will be used (but may be implied)can/will be used (but may be implied) But…it was easy to construct, is still a But…it was easy to construct, is still a
useful blueprint, and is much better than useful blueprint, and is much better than not making one!not making one!
Add details on item type Add details on item type and format to ensure…and format to ensure…
Appropriate match to learning targets and Appropriate match to learning targets and associated levels of complexityassociated levels of complexity
Balanced use within tests and across tests Balanced use within tests and across tests over timeover time
Specification of test resources, i.e. –Specification of test resources, i.e. – Calculators, dictionaries, measuring tools…Calculators, dictionaries, measuring tools…
Track on same or separate spreadsheetTrack on same or separate spreadsheet
Common item types Common item types include…include…
Selected-response Selected-response Multiple-choiceMultiple-choice MatchingMatching
Constructed-response Constructed-response Brief (fill-in-the-blank, short answer, sort a list)Brief (fill-in-the-blank, short answer, sort a list) Extended (outline, essay, etc.)Extended (outline, essay, etc.)
PerformancePerformance ProjectProject PortfolioPortfolio
Complexity vs. UtilityComplexity vs. Utility
Your test blueprint could get Your test blueprint could get complicated if you try to account for complicated if you try to account for too much in one spreadsheet.too much in one spreadsheet.
Make sure your test blueprint covers Make sure your test blueprint covers the basics, is not a burden to create, the basics, is not a burden to create, and is useful to youand is useful to you
The following example is slightly more The following example is slightly more complicateded, but still workablecomplicateded, but still workable
TargetCode
Item#
Item type SR
Item type CR-B
Item type CR-E
DOKRecall
Pts
DOKUse
Pts
DOKStrategic
Pts
Trgt Tots(pts)
1.1.1.1 1 x 1
1.1.1.1 2 x 1
1.1.1.1 3 x 2 4
1.2.3.4 4 x 3
1.2.3.4 5 x 5 8
etc etc
Col Tots 5 2 2 1 1 7 5 12
Example 2 – BlueprintExample 2 – Blueprintwith Explicit Items and Item Typeswith Explicit Items and Item Types
Beyond the Test BlueprintBeyond the Test Blueprint
Answer key (selected-response items)Answer key (selected-response items) Links to scoring guides & rubricsLinks to scoring guides & rubrics Specs for external test resourcesSpecs for external test resources Item numbering for alternate test formsItem numbering for alternate test forms
ConclusionsConclusions
Destination & Road MapDestination & Road Map Alignment/balance of items/types for…Alignment/balance of items/types for…
learning targets (curriculum /content)learning targets (curriculum /content) size (complexity) of targetssize (complexity) of targets cognitive level of targetscognitive level of targets relative importance of targetsrelative importance of targets Spec or document other aspects of the Spec or document other aspects of the
testtest