PowerPoint Presentation

A collaboration of 3 nonprofits through a contract award from

If you want to go fast go alone, if you want to go far, go together. African Proverbwww.vermontpln.org1

ASSESSMENT DESIGN

2Introductions

Introductions3Understand the CCSS and SBAC connections in terms of Curriculum, Instruction, and AssessmentExplore an Assessment Design Process and Sample AssessmentsUse an Assessment Design Process to construct quality assessments

Apply the process and tools for developing interim reading comprehension assessments. 4Common Core and Smarter Balanced ConnectionsThe Curriculum-Instruction-Assessment System

9:005Assessment is an integral part of the teaching process.

Teaching may be conceptualized as a process including quality curriculum, instruction/learning and assessment.

There should be a reciprocal relationship between curriculum, instruction, and assessment.The Curriculum-Instruction-Assessment SystemAssessment data should provide objective feedback about student learning and the effectiveness of instruction. This data can drive decision making and promotes learning. 6Types of AssessmentsIntermediate assessment to monitor achievement over a group of standardsShould predict achievement on summativeData should be used to inform re-teaching prior to summative assessmentAssessment yielded evidence about students status that guides instructional decisions and adjustments. Typically targeted to a single standardMay be embedded in a lessonProvides immediate evidence of student progressA comprehensive measurement of learning (outcomes) following instructionProvides information about what students have and have not masteredTypically administered at the end of a unit, semester, or year7

8

When engaging in Common Core based Close Reading activities for a sustained period of time, it is helpful for both teacher and students to be able to assess their engagement with the text via annotations. This rubric provides such an opportunity, and focuses heavily on Reading Standards two and four.

9

This slide contains an example of a formative assessment from a US History Unit. First, after instruction, and practice, students are asked to describe the political cartoon, and then identify the authors purpose. This is a preliminary, scaffolded stage in assessing skill development towards a Document Based Question that would assess multiple Common Core ELA Literacy in History Standards.

10

This rubric primarily assesses students mastery of ELA History Writing Standard 1.a. Once students have mastered recounting the claims and counterclaims from a piece of text, they are ready to begin to analyze and synthesize multiple texts.11www.smarterbalanced.orghttp://sbac.portal.airast.org/practice-test/

Green Box>> Sign In>> Choose Grade Level >> Choose Test>> Choose Setting-Select>> Start my Test

What do you notice about the structure of the tests?What do you notice about the types of questions?How are the CCSS standards addressed?

Sample AssessmentsLook at entire performance task part A and part B. Just type letters in the box to proceed to final page.Look at the first reading selection and corresponding questions for the reading/writing assessment.

*Performance task requires integration and synthesis of information from multiple sources*Variety in question types requiring evidence from text*Questions address clusters of standards.

12How does assessment fit into CCSS aligned units of instruction?

What are the big ideas?

How are text sets used?

How do the unit lessons connect?

How are students assessed?

Engage NYHANDOUT 1http://www.engageny.org/sites/default/files/resource/attachments/8m1.1.pdf13Shifts of the Common Core: English Language ArtsRegular practice with complex text and academic language across content areasUsing evidence from literary and informational text to support claims and conclusions during reading, writing, and discussion, across content areasBuilding content knowledge from informational text during reading, writing, and discussion, across content areas

14Key Shift 1: Complex Text

Reading passages on tests should have appropriate grade complexity Close reading, rather than skimming, is required.Passages should be of high quality and worthy of close reading. Most of the items assessing vocabulary should focus on words that matter mostacademic vocabulary--and the use of context to determine their meaning. Source: ccsso.org

Only when assessment developers start with a complex text are they able to develop complex, challenging items. When passages are not complex, they lack full development of ideas, and thus they lack the complexity needed for CCSS-aligned test questions, which require students to locate and use evidence from the text. Vocabulary items may include:Figurative language (the meaning or intended purpose, not the identification of labels for the kinds of figurative language) Words that impact the tone of the text Words that have diverse meanings in different places in the text Grade-appropriate use of strategies like roots and affixes

15Key Shift 2: Evidence

Items require students to respond to and draw evidence from texts

Items should enable and require students to linger over the specifics, leading back to the text for close reading.

Source: ccsso.org

Key Shift 3: Building Knowledge

Reading passages should include: strong, cogent examples of fiction, non-fiction, and informational text. Students should be reading like scientists, like historians, etc.

Interim/benchmark assessments should include: effective sequences of quality text-dependent questions to build content knowledge and comprehension from the text.

Source: ccsso.org

17Assessment Design Process

10:00When we examined the item bank selections earlier this year, we realized the selections and items did not meet the criteria set forth by the new Common Core Standards for ELA/Literacy. Our goal now is to apply the new criteria and process to develop quality assessments that may be shared with colleagues around the state.18The Common Core ConnectionThe first step of assessment design is clearly defining the content area (domain) and standards to be measured. For our purposes, we will focus on English Language Arts and Literacy in History/Social Studies, Science, and Technical Subjects

19

Claims A claim is a statement about what a student knows or is able to do. After carefully analyzing the Common Core State Standards and thinking about what students must know and be able to do in order to be prepared for college and career paths, Smarter Balanced has identified four claims specific to English Language Arts that focus on what students are expected to be able to do at each grade level.

20

SBAC and CCSS ConnectionsHANDOUT 2HANDOUT 2Discuss which standards connect to claims 1 and 2. Claims are dealing with multiple standards across the curriculum.Get out of your boxshift to--collaborating /planning with colleagues, integrating content areas.Claims are the bridge from the standards to the assessment

21SMARTER BALANCED ELA LITERACY CONTENT SPECIFICATIONS

Clearly define the knowledge, skills, and abilities that students will demonstrate. For Smarter Balanced, these are called assessment targets.These targets are all connected to the clusters we looked at earlier!

Assessment Targets

23Assessment Targets for Claim 1LiteraryKey DetailsCentral IdeasWord MeaningReasoning and EvidenceAnalysis within or across textsText Structures and FeaturesLanguage UseInformational8. Key Details9. Central Ideas10. Word Meaning11. Reasoning and Evidence12. Analysis within or across texts13. Text Structures and Features14. Language Use

Students can read closely and analytically to comprehend a range of increasingly complex literary and informational textsAssessment Targets for Claim 2Students can produce effective and well-grounded writing for a range of purposes and audiences.

Determine Text Complexity MEASURES

CCRA.R.10 Read and comprehend complex literary and informational texts independently and proficiently.

R.CCR.10

Common Core ConnectionRemember, assessment selections should be on grade level for students to read independently.

27Three Part ModelDimensions of Text Complexity

Quantitative measures

Qualitative measures

Reader and Task considerations

It is not just the length of words, sentences and paragraphs we must consider when examining complexity. Text may contain complex concepts/ideas, ironic/archaic/figurative language, and complex text structures.A readers background knowledge, motivation, and interests must also be considered.

28Quantitative Measures

Quantitative Measures Comparison

Lexile

ATOS

Source: www.corestandards.orgHANDOUT 3 (front)HANDOUT 3This chart contains current grade band levels for a variety of quantitative measure sourcesCAUTION:-Poetry and Drama cant be measured on quantitative scales-qualitative measures should override the quantitative measures30

Quantitative Measures For Santorio SantorioFlesch-Kincaid and Reading MaturityFlesch-Kincaid 6.51-10.34 Reading Maturity: 7.04-9.57For the purpose of assessment, we must be sure the text is within the appropriate grade band (so the student can read the piece independently)BE SURE TEXT IS IN PROPER FORMAT FOR MEASURING (i.e., plain text, word, pdf)31ATOS Analyzer

Show participants how to analyze text with Santorio text.32Santorio Santorio and the ThermometerEXAMPLE from www.achievethecore.org

HANDOUT 7READ THE TEXT33

Results

HANDOUT 734Qualitative Measures

What other factors make the text complex?Source: www.fisherandfrey.comThese are the qualitative dimensions of text complexity.36

Qualitative Measures

Choose appropriate rubric for the type of textSource: ccsso.orgHANDOUT 4

ANNOTATEWHICH COMPONENTS MAKE THIS TEXT COMPLEX?HANDOUT 4These are the same dimensions in a rubric format.37Qualitative Analysis of Santorio Santorio and the Thermometer

HANDOUT 3 (back)HANDOUT 338Reader and Task Considerations

Reader and TaskMotivation

Knowledge and experience

Purpose for reading

Complexity of task assigned regarding text

Complexity of questions asked regarding text

HANDOUT 5HANDOUT 541

Remember, this is an assessment. If complexity levels are high on the prior knowledge scales (in the qualitative and reader/task considerations), it will not be a good fit for an assessment selection. We want students to be able to glean the necessary content and learning from the passage itselfnot be assessed on prior knowledge. 42Assessment Questions

Selected Response ItemsStudents to select appropriate response(s) from options provided.Strengths and Weaknesses of Selected Response ItemsStrengthsAllow for large numbers of itemsCan be scored efficiently, objectively, and reliablyWeaknessesDifficult to writeNot able to assess all objectives (e.g. writing)Subject to random guessingAdapted from Reynolds, Livingston, and Willson (2009)45Lizards are fascinating creatures. There are over 3,000 known species, including monitors, skinks, geckos, chameleons, and iguanas, and they vary greatly in appearance. The largest lizard, the Komodo dragon, can grow over ten feet long, and the smallest, the Jaragua lizard, can fit on a dime. Skinks usually have smooth scales like snakes, iguanas have mohawk-like crests running down their backs, and the moloch is covered with spikes from head to tail. Lizards vary in color from shades of gray and brown to bright red or green, spotted or striped. Most have four legs but some are legless and easily confused with snakes (Hint: if it has external eardrums and eyelids its a lizard). Geckos can walk up walls. Chameleons not only change color but also have prehensile tails, similar to those of monkeys, that wrap around branches and their eyes can move in different directions.

What is the best way to revise the highlighted sentence to match the language and style of the paragraph?

Geckos are able to adhere to flush surfaces because setae on their footpads facilitate van der Waals forces between the setae structures and the surface. Geckos are awesome because they have sticky toes that allow them to climb windows like Spiderman.Geckos have the remarkable ability to walk up walls thanks to tiny hair-like structures on their toes that cling to smooth surfaces.Geckos scurry up walls like tiny dancers gliding effortlessly across a stage, their movements as natural as a well-rehearsed ballet. Formats and Components of Selected Response ItemsTraditional Distractor AnalysisA. This option uses too much scientific language to fit with the rest of the paragraph.B. This option is more informal than the rest of the paragraph.C. KEY. This option uses formal, non-technical language that fits well with the rest of the paragraph.D. This option uses too much figurative language to fit with the rest of the paragraph. STIMULUSLizards are fascinating creatures. There are over 3,000 known species, including monitors, skinks, geckos, chameleons, and iguanas, and they vary greatly in appearance. The largest lizard, the Komodo dragon, can grow over ten feet long, and the smallest, the Jaragua lizard, can fit on a dime. Skinks usually have smooth scales like snakes, iguanas have mohawk-like crests running down their backs, and the moloch is covered with spikes from head to tail. Lizards vary in color from shades of gray and brown to bright red or green, spotted or striped. Most have four legs but some are legless and easily confused with snakes (Hint: if it has external eardrums and eyelids its a lizard). Geckos can walk up walls. Chameleons not only change color but also have prehensile tails, similar to those of monkeys, that wrap around branches and their eyes can move in different directions. STEMWhat is the best way to revise the highlighted sentence to match the language and style of the paragraph?OPTIONSGeckos are able to adhere to flush surfaces because setae on their footpads facilitate van der Waals forces between the setae structures and the surface. Geckos are awesome because they have sticky toes that allow them to climb windows like Spiderman.Geckos have the remarkable ability to walk up walls thanks to tiny hair-like structures on their toes that cling to smooth surfaces.Geckos scurry up walls like tiny dancers gliding effortlessly across a stage, their movements as natural as a well-rehearsed ballet. (Question)HANDOUT 6The Smarter Balanced Assessment System uses both traditional and non-traditional selected response items. This item is a traditional Grade 9 selected response that is sometimes referred to as a multiple choice item. This item contains a stimulus, a stem and four options. The stimulus is the passage that sets the context for the item.The stem is the statement of the question to which the student responds. The stem is sometimes also referred to as a prompt.The options are possible answers from which the students must select. Options should be arranged according to a logical order such as numerically, alphabetically, or the order in which information is presented in the stimulus. There are four different ways to respond to this item but only one correct answer.

Distractors are incorrect answer options and the key is the correct answer. The distractor analysis explains why the correct answer is correct.46Selected Response Example

OPTIONS

Take a look at the samples from Santorio. 47Selected Response Example

Selected Response Example

OPTIONS/STUDENT RESPONSES

DISCUSS HOW THESE ITEMS DIFFER FROM TRADITIONAL FORMATS. 49Constructed Response ItemsStudents create or construct an appropriate response.Strengths and Weaknesses of Constructed Response ItemsStrengthsMay be easier to writeCan assess higher order cognitive abilities and complex task performanceEliminate random guessingWeaknessesMore time for students to complete, so, fewer items can be included on testMore difficult to score reliablyVulnerable to construct irrelevance factorsAdapted from Reynolds, Livingston, and Willson (2009)

51Constructed Response FormatThe Shepherds Boy and the Wolf

A Shepherd's Boy was tending his flock near a village, and thought it would be great fun to trick the villagers by pretending that a Wolf was attacking the sheep: so he shouted out, "Wolf! Wolf!" and when the people came running up he laughed at them because they believed him. He did this more than once, and every time the villagers found they had been tricked, for there was no Wolf at all. At last a Wolf really did come, and the Boy cried, "Wolf! Wolf!" as loud as he could: but the people were so used to hearing him call that they took no notice of his cries for help. And so no one came to help the boy, and the Wolf attacked the sheep.

In a few sentences, explain what lesson the reader can learn from the shepherds boy. Use details from the story to support your response. STEMIn a few sentences, explain what lesson the reader can learn from the shepherds boy. Use details from the story to support your response. STIMULUSThe Shepherds Boy and the Wolf

A Shepherd's Boy was tending his flock near a village, and thought it would be great fun to trick the villagers by pretending that a Wolf was attacking the sheep: so he shouted out, "Wolf! Wolf!" and when the people came running up he laughed at them because they believed him. He did this more than once, and every time the villagers found they had been tricked, for there was no Wolf at all. At last a Wolf really did come, and the Boy cried, "Wolf! Wolf!" as loud as he could: but the people were so used to hearing him call that they took no notice of his cries for help. And so no one came to help the boy, and the Wolf attacked the sheep. Like selected response items, constructed response items have a stimulus and a stem.

52Constructed Response Example

Recall the culminating writing taskWhat was Santorio like as a person? Make a claim about what kind of person he was and defend your claim using evidence from the text. Write your response below.

StimulusStem

Here is the constructed response sample from the Santorio assessment.53Discuss the types of selected and constructed response items you viewed in the SBAC practice test earlier today.

54Constructing Quality Items based on essential understanding

Text-dependent questions that scaffold learning:Require careful examination of text

Require students to respond with support from text

Range from simple to complex (but based on importance of building knowledge or interpretations)

Go beyond basic facts and recall

Promote deep comprehension and retention of content

Common Core supporters have been emphasizing the idea that teachers should ask text dependent questions. This term is from reading assessment, and it refers to the idea that test questionsusually multiple-choice questionsshould not be answerable without the text being available. A quality set of questions allow students to interact with the text and each other to build deeper understandings than would develop from a cursory reading.They foster a habit of re-reading and locating evidence to support their responses.56Words/PhrasesSentencesParagraphSegmentsSelectionAcross Texts

to build gradual, overall understanding.A variety of questions on a variety of levels Questions should address the myriad levels within text.57Variety in levels of cognitive rigor

2009 Karin K. Hess: Hess Cognitive Rigor Matrix: Permission to reproduce is given when authorship is fully cited [khess@nciea.org] For full article, go to www.nciea.org

HANDOUT 9Refer to this document when constructing questionsThe Cognitive Rigor matrix combines two common taxonomies that categorize levels of cognition.

58

Guidelines for constructing effective text-dependent questionsHANDOUT 1059Analyzing questions

Culminating Task/Question related to Essential Understanding

What was Santorio like as a person? Make a claim about what kind of person he was and defend your claim using evidence from the text. Write your response below.

Text based questions provide scaffolding for students to be able to reach a deep understanding and respond to this question.

Which CCSS anchor standards apply?Ask Participants: Which anchor standards apply to this question?Text based questions provide scaffolding for students to be able to reach a deep understanding and respond to this question.61

Read the grade level standards being measured. Remember, these connect to the anchor standards we looked at earlier.

Lets look at the sample questions from Santorio and determine how they connect to the Common Core Standards for the 6-8 grade band and the Depths of Knowledge levels. We will look specifically at 7th grade standards for informational text and 6-8 standards for Reading in Science and technical subjects.

62Match Standards to Questions

HANDOUT 7 AND 863

Headings in Standards

Peer Review ChecklistPeers use checklist to examine items and ensure all criteria are met.

HANDOUT 11HANDOUT 1165What is MTSS?MTSS is a coherent continuum of evidence based, system-wide practices that support:a rapid response to academic and behavioral challenges,frequent data-based monitoring instructional decision-making in order for each Vermont student to achieve high standards.Why MTSS?

vtMTSS Framework: A Tiered System of SupportIn a classroom of 30 studentsMTSS: A tiered system of supportALL 30 students (100%) receive high quality Tier 1 instruction.24-27 students (80-90%) are successful with just Tier 1 instruction.1-3 students (5-10%) require targeted Tier 2 supplemental instruction to be successful.1 student (1-5%) require intensive Tier 3 support in order to be successful.

69Colorin Colorado

The Common Core for English Language LearnersWhat is the role of informal, classroom-based assessment in guiding instruction for students with limited English proficiency?

Center on InstructionThe Center on Instruction offers reports, professional learning modules, practice guides, presentations and online courses, including research-based practices in instruction and assessment for English Language LearnersGoogle Translator

Babylonhttp://www.babylon.com

36 Languages

Reaching ALL Students

Cindys Slide73This collection of easy to read books provides students with significant cognitive disabilities access to the content of the Exemplar Texts listed in Appendix B of the Common Core State Standards at an appropriate complexity level. The open source, accessible texts were created in Tar Heel Reader (tarheelreader.org) http://www.dynamiclearningmaps.org/unc/texts/index.html

Lunch

Sample AssessmentsCactus Jam (3-5)Nature By Design(6-8)Inaugural Address (9-12)

Provide electronic sample text for 3-5, 6-8, 9-12 for practiceHave participants:determine text complexity level using electronic sampleslook at the final question and determine the essential understandingexamine questions and match to appropriate standards, clusters, and DOK. Enter on Assessment Design Template Worksheet76Read text and determine complexityDimensions of Text Complexity

Quantitative measures

Qualitative measures

Reader and Task considerations

77Take the Test!Analyze the Questions

Determine the essential understandingNote the standards and the structure/complexity of the questions

Peer Review ChecklistDetermine if the assessment questions meet these criteria?

79APPLY THE PROCESS TO YOUR OWN TEXT

1:4580

Try it OutAssessment Design TemplatePeer Review

Text Complexity AnalysisSample Text SelectionsHANDOUT 12

Developing Effective Text Dependent Questions Guide and DOK levels

1:00HANDOUT 12Lets quickly look at how you will apply the process.81Materials Evaluation Toolkit

Toolkit for Evaluating Alignment of Instructional and Assessment Materials to the CCSSNon-Fiction and Informational Text Resourceshttp://kids.nationalgeographic.com/kids/http://portal.acs.org/portal/acs/corg/content?_nfpb=true&_pageLabel=PP_SUPERARTICLE&node_id=1758&use_sec=false&sec_url_var=region1&__uuid=fa36d8e4-96dd-441d-920f-f2dad28ca741https://www.studentnewsnet.com/index.php?fuseaction=home.our_worldhttp://www.dogonews.com/http://whyfiles.org/http://www.washingtonpost.com/lifestyle/kidsposthttp://www.cnn.com/studentnews/http://www.ipl.org/div/news/http://www.pbs.org/newshour/extra/http://learning.blogs.nytimes.com/http://textproject.org/students/fyi-for-kids/http://theweek.com/http://kellygallagher.org/resources/articles.htmlhttp://www.readworks.org/passages/great-lakeshttp://newsela.com/articles/?category=kidsNY Times Learning Networkhttp://www.mybookezzz.net

83Special Thank you to the Professional Learning TeamJoey HawkinsDiana LeddyLindy JohnsonMarion DeweyJoe RiversMaggie EatonJuliet HalladayAlysia BackmanChristina SuarezJanna OsmanJenn BaroneLaura KingPat FitzsimmonsCindy MoranLori DolezalElementary Literacy Assessment CoordinatorLori.dolezal@state.vt.usPlan for Professional Development

2:30Participants construct a plan and gather materials (text selections, templates, etc.) to present in their PLCsShare out ideas on ELMO?85Join the Exchange

ShareDiscussInquireI will construct an Assessment Design Workshop group in the Vermont Education Exchange. This will be the house for the new assessments designed by educators. They can share their work, hold discussions, and generate questions in this forum.86ReferencesReynolds, Cecil R., Livingston, R.B., & Willson, V. (2009). Measurement and Assessment in Education (2nd edition). Boston MA: Pearson.

www.achievethecore.org (Student Achievement Partners)

http://programs.ccsso.org/projects/common%20core%20resources/documents/Informational%20Text%20Qualitative%20Rubric.pdf (Qualitative Rubric for Informational Text: Council of Chief State School Officers and Student Achievement Partners)

http://programs.ccsso.org/projects/common%20core%20resources/documents/Literary%20Text%20Qualitative%20Rubric.pdf (Qualitative Rubric for Literary Text: Council of Chief State School Officers and Student Achievement Partners)

http://www.ccsso.org/Documents/2013/March%2019%20CCSS%20DRAFT%20Rubric%20for%20Interm.pdf(Rubric for interim Assessments)

http://www.renlearn.com/atos/analyze.aspx?type=3 (ATOS analyzer)

http://www.smarterbalanced.org/

Hess, K (2009) Hess Cognitive Rigor Matrix. www.nciea.org

www.corestandards.org (The Common Core Standards)

http://www.colorincolorado.org/common-core/

http://www.centeroninstruction.org/topic.cfm?s=1&k=ELL