Building Effective Assessments

Agenda

Brief overview of Assess2Know content development

Assessment building pre-planning

Cognitive factors

Building the assessment

Building Effective Assessments Using

Formative Assessment Content

Assess2Know

Reading grades 2 – 11

Mathematics grades 2 – 11

Science grades 3 – 11

Social Studies grades 9 – 11 NEW

Formative Assessment Content

High Quality Content

Rigorous development process

Tight alignment with Michigan State Standards

Highly trained and experienced assessment development staff

Formative Assessment ContentRigorous Requirements

Clearly and simply written; no “trick questions”

Appropriate vocabulary for the grade level being tested

Functions independently; does not provide information that may be used to answer other items

Requires students to read the passage in order to answer the question

Does not use “all of the above” or “none of the above”

Presents answer options in a logical order

Formative Assessment Content

Reading

Passages are commissioned works by well-known authors

Lexiles and Flesch-Kincaid

Genres

Bloom’s Taxonomy, Cognitive Difficulty Levels

Maps, Charts, Graphs, Images, and Primary Sources

Flesch-Kincaid Formula

FKRS = (0.39 x ASL) + (11.8 x ASW) - 15.59

• FKRS: Flesch-Kincaid Readability Score• ASL: average sentence length in words or

average number of words in sentence (number of words divided by the number of sentences)

• ASW :average number of syllables per word (the number of syllables divided by the number of words)

Assess2Know Lexiles

• Lexile measures are considered to be the most accurate measure of readability because they are the only formula which applies a psychometric formula to the assignation for establishing readability. 

• Since 2005, Metametrics assigns each A2K passage a Certified Lexile Measure.

Assess2Know Reading Items

• To determine grade level placement of passages, Riverside uses readability guidelines to place passages at each grade level. 

• In addition, passage structure, interest level, topic, and word count are among some of the things used to evaluate where a passage should be placed for grade level benchmarking. 

• District level committees review passages and items and if they had felt a passage was not appropriate, Riverside replaces it and the items at the review.  – Advantage to onsite committee reviews because educators in

the field are affirming what is appropriate for each benchmark.

Assess2Know Readability Limits

Grade Flesh-Kincaid Lexile

  minimum maximum minimum maximum

3 2.0 4.0 300L 850L

4 3.0 5.0 500L 950L

5 4.0 6.0 650L 1050L

6 5.0 7.0 750L 1075L

7 6.0 8.0 850L 1100L

8 7.0 9.0 950L 1150L

9 8.0 10.0 1000L 1200L

10 9.0 11.0 1050L 1300L

11 10.0 12.0 1100L 1300L

12 11.0 13.0 1100L 1300L

Formative Assessment Content

Mathematics, Science, and Social Studies

Real world context

Unique

Scenario Based Science Items

Bloom’s Taxonomy, Difficulty levels

Use of Tools

Maps, Charts, Graphs, Images, and Primary Sources

Research Efforts

• The Research and Measurement Services Department analyzes data from multiple users of the Assess2Know Benchmark item bank from across the country to monitor the performance of the items in the bank.

• Goal is to ensure that the item bank provides users with strong quality items with a range of difficulty.

• Results look at average ranges for statistics such as p-values, discrimination statistics, and point-biserials for each item.

• This information is used to plan for refresh and future development of our item banks.

High Quality Assessment

Why do we give assessments?

Importance of Designing High Quality Assessments

• Assessments measure what is being taught in the classroom

• Align to the district curriculum– Curriculum Maps– Scope and Sequence

• Results of the assessment are used to report on student achievement – Used to make decisions about student

growth, interventions and program placement

Characteristics of High Quality Assessments

• Design curriculum based assessments that measure student achievement to the state standards

• Report assessment results at the individual student, class, school and district level

• Provide achievement information that can be compared from year to year

• Construct assessments that are valid and reliable

Characteristics of High Quality Assessments

• Assessment design allows for a variety of assessment types: selected response, open ended items, constructed response and rubrics

• Delivered in a timely manner

• Easy to administer

Purpose of Assessment

• Evaluate students overall achievement and growth in a content domain

• Diagnose students strengths and weaknesses in and across content domain

• Plan educational interventions and to design individual instruction

Purpose of Assessment

• Place students in appropriate educational program

• Select applicants into programs with limited enrollment

• Certify individual achievement or qualifications from Standards for Educational and Psychological Testing

What Do We Ask Ourselves Before Building an Assessment?

• What do we want to measure?• What information do we want to

gather?• How long do we want the

assessment to take?• How will we determine which

standards we want to assess?• How will we use the results?

Guiding Principles to Keep in Mind When Building Formative

Assessments• A minimum of three items for every

standard you want to measure.• Spread items measuring an individual

standard across the assessment.• Range of overall assessment difficulty

and for each standard measured.• Begin assessment with easier items,

build to more difficult and conclude with easier items.

Guiding Principles to Keep in Mind When Building Formative

Assessments

• How many passages will be used for reading assessments?

• What item types will be used?

• How many items per passage?

• Total number of items? Can this be completed in the allotted time?

Guiding Principles to Keep in Mind When Building Formative

Assessments• Spread items with graphics across test.• Readability levels for reading passages;

genres needed.• A minimum of five items per passage.• Even spread of answer choices across

assessment when possible; spread them out across the form.

• Check items for appropriateness to use online, paper, or both.

• Check for clueing

Formative Assessment Content

Level 1 Basic Skills recall of information such as fact, definition, term, or simple

one-step procedure.

Level 2 Conceptual Understanding includes the engagement of some cognitive processing

beyond recalling or reproducing a response. A conceptual understanding item requires students to make some decisions as to how to approach the problem or activity and may imply more than a single step.

Level 3 Extended Reasoning requires problem solving, planning, and/or using evidence.

Items require students to develop a strategy to connect and relate ideas in order to solve the problem while using multiple steps and drawing upon a variety of skills.

Cognitive Difficulty Levels

Importance of Content Complexity

• Vastness of Content

• Alignment

• Validity

• Clarity

• Teacher Guidance

• Truth in Advertising

Level One: Basic Skills

Recall of information such as fact, definition, term, or simple one-step procedure.

• Support ideas by reference to details in text• Use dictionary to find meaning• Identify figurative language in passage• Solve a one step word problem• Perform a specified procedure

Level 2: Conceptual Understanding

Includes the engagement of some cognitive processing beyond recalling or reproducing a response. A conceptual understanding item requires students to make some decisions as to how to approach the problem or activity and may imply more than a single step.

• Predict logical outcome• Identify and summarize main points• Represent a situation mathematically in more

than one way.• Interpret a visual representation

Level 3: Extended ReasoningRequires problem solving, planning, and/or using evidence. Items require students to develop a strategy to connect and relate ideas in order to solve the problem while using multiple steps and drawing upon a variety of skills.

• Determine effect of author’s purpose on text elements

• Summarize information from multiple sources• Provide a mathematical justification• Describe, compare and contrast solution methods

Building Effective Assessments

Questions