16.10 (NESA) Accurate & Effective Formative Assessment … · (1) Student investment occurs when assessment and self-regulation have a symbiotic relationship. (6) The communication

www.tomschimmer.com [email protected]

Accurate & Effective Formative Assessment

Tom Schimmer

www.allthingsassessment.info @TomSchimmer

__________________________________________________________________________________________________________________________________________________________________________________________________ © Tom Schimmer (2016) – Accurate & Effective Formative Assessment @TomSchimmer [email protected]

1

Accurate, Balanced, & Hopeful Assessment

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

“The formative and summative purposes of assessment can be so intertwined that they are mutually supportive rather than conflicting. Unless this is done, formative assessment cannot achieve its full potential to improve learning,

-Paul Black (2013)

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

Expectations about the likelihood of eventual success determine the amount of effort people are willing to put in. Those who are convinced that they can be successful in carrying out the actions required for a successful outcome - who have the self-efficacy - are likely to try harder and persist longer when they face obstacles.”

-Rosabeth Moss Kanter Confidence, p. 39

“It’s not mistakes that cause winners to lose, it’s panic. Panic is a sudden, anxious feeling of loss of control, and panicking can make a small fumble worse, by causing people to lose their heads and forget to think clearly…Panic is the enemy of good decision making under pressure.”

Confidence, p.68

Assessment'prac,ces'must'build!hope,!efficacy,!and!achievement'for'learners'and'teachers.'In'this'learning'environment,'the'following'tenets'

ground'all'of'our'assessment'policies'and'prac,ces:

(Erkens,)Schimmer,)&)Vagle,)2017)

(1) Student investment occurs when assessment and self-regulation have a symbiotic relationship.

(6) The communication of assessment results must generate productive responses from learners and all the stakeholders who support them.

(4) Assessment purposes (formative and summative) must be interdependent to maximize learning and verify achievement.

(3) Instructional agility occurs when emerging evidence informs real-time modifications within the context of the expected learning.

(2) Assessment architecture is most effective when it is planned, purposeful, and intentionally sequenced in advance of instruction by all of those responsible for the delivery.

(5) The interpretation of assessment results must be accurate, accessible, and reliable.

Duration of Learning

Summative Assessment Verify'that'le

arning'

has'occurred

Formative AssessmentKeep'lea

rning'on'track;'

continual'improvem

ent


2

Describe the characteristics and attributes of a confident learner. What does it look, feel, and sound like when a student is on an academic winning streak? Current/past examples? What does the opposite look, feel, and sound like? Current/past examples?

Accuracy

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

Rank the essential elements from 1-3, with 1 being the element that gets the most attention and 3 being the element that gets the least attention. Explain your 1-3 order.

Assessment Accuracy

Why? What?

How?

Formative*vs**

Summative?

Intended*learning*&**success*criteria*

Thoughtful*Assessment*Choices*


3

Formative v. Summative Assessment Reflect on the idea of establishing a balanced assessment system to answer he following questions:

1. What are the unique aspects/characteristics of FORMATIVE and SUMMATIVE Assessment? 2. What aspects/characteristics overlap between FORMATIVE and SUMMATIVE Assessment?

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

Analyzing StandardsNicole Dimich Vagle (2015) “Design in 5”

1.#Circle#the#verbs#(and#other#words#that#indicate#what#students#should#do)#in#the#standards.#

• "“These"words"lead"us"to"the"cognitive"level"required"and"the"type"of"work"in"which"our"students"need"to"engage.”"(pg.#25)"

2.#Underline#key(concepts,#vocabulary(words,#and#contextual(information.#

• “Consideration"of"concepts"and"vocabulary"also"helps"inform"instruction"and"the"type"of"academic"language"pertinent"to"the"grade"level"and"content"area.”"(pg.#26)

Summative Assessment

Formative Assessment


4

Webb’s “Depth of Knowledge” Chart

Use the depth of knowledge chart to identify curricular underpinnings within the standards

Webb, N. L., Alt, M., Ely, R., & Vesperman, B. (2005) Webb alignment tool (WAT) training manual. Madison, WI:

Wisconsin Center for Educational Research.

Thoughtful Assessment Choices Under what circumstances or conditions would each of the following assessment methods be considered most appropriate?

Performance Assessment

Under&what&circumstances&or&conditions&would&each&of&the&following&assessment&methods&be&most&appropriate?

Selected Response

Constructed Response


5

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

Common%Guidelines %Selected%Response%Items

1. Use&a&complete&question&stem.&

2. Word&the&question&stem&positively.&

3. Avoid&using&AOTA&&&NOTA.&

4. Keep&answers&choices&equal&in&length.&

5. Three&(3)&answer&options&is&sufficient

Constructed Response Thomas P. Hogan (2013)

Completion*

Fill,in*

Short,Answer

Demonstrations*

Short*Projects*

Brief*written*assignments

Extended*Performances*

Term*papers,*long*projects*

Portfolios

Short, Simple

Long, Extended

CONSTRUCTED RESPONSE SOME IMPORTANT QUESTIONS

• Is$there$a$more$effective/efficient$way$for$you$to$access$the$

learning$information$you’re$seeking?$

•Are$the$students$familiar$with$the$rubric/success$criteria$and$

how$it$will$be$applied?$

• SAME$RUBRIC?$Are$there$any$shifts$in$emphasis$that$students$need$

to$be$familiar$with?$

• DIFFERENT$RUBRIC:$Have$you$identified$the$differences$between$

the$“familiar”$and$“unfamiliar”$rubrics?$

• Is$my$question$clear$and$thorough$enough$to$avoid$any$

confusion$about$what$content$to$include?

Performance*Assessment:*Intent *(Lane,*2010)

• Generalization?*Then*adequate*sampling*across*the*domain*or*discipline*is*required.**

• Finite*performance?*Then*more*speciEic*tasks*&*criteria*are*more*appropriate.*

• Formative?*Then*a*much*narrower*scope*is*more*Eitting.


6

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

Effective Feedback “The result of decades of research on feedback and learning offers near unanimous agreement on the importance of feedback in improving student learning.” “It appears that as researchers we know less about formative feedback than would have been predicted. Our degree of not knowing, however, must be appreciated as the accumulation of knowledge resulting from 30 years of research on feedback.”

-‐Maria Araceli Ruiz-‐Primo & Min Li (2013)

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

(1) Does it elicit a Productive Response?

“To compound matters further, even when feedback is well designed it will only have an effect if it’s acted upon by the learner.”

-‐Dylan Wiliam (2013)

Performance*Assessment:*Clarity *(Mislevy,*Steinberg,*&*Almond,*2003)

• We*must*be*clear*on*both*the*content*and*the*cognitive*processes*that*are*being*assessed.*

• What*performances*will*reveal*a*level*of*proFiciency*with*that*content*and/or*cognitive*processes.*

• What*tasks*will*most*likely*lead*to*those*performances.

Limitations of Feedback Research(Maria Araceli Ruiz-Primo & Min Li, 2013)

1. High'percentage'of'studies'focus'on'feedback'in'one'experimental'session.'(Long&Term+effects?)+

2. Most'feedback'studies'for'math/science'provided'no'background,'guidance,'or'direction.'(Instructional+bene8its?)+

3. Many'studies'provided'little'information'about'the'the'speci@ic'tasks'OR'tasks'were'arti@icial.'(Link+to+speci8ic+type+of+learning?)

Five%Keys%to%Effective%Feedback

1.%%%Does%it%elicit%a%productive+response?%

2.%%%Does%it%identify%what’s+next?%

3.%%%Is%it%targeted%to%the%learner?%

4.%%%Is%it%strength4based?%

5.%%%Does%it%cause+thinking?


7

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

“To compound matters further, even when feedback is well designed it will only have an effect if it’s acted upon by the learner.” “When we try to determine what kind of feedback works we are [often] asking the wrong question. What matters is what response the feedback triggers in the recipient.”

-Dylan Wiliam

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

(2) Does it provide clarity on what’s next?

(3) Is it targeted to the learners’ levels?

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

Potential)Impact)of)Scores

• High&score&OR&atypically&high?&

• Unnecessary*

• Low&score&OR&atypically&low?&

• Undesirable

• Motive"is"in"reference"to"whether"the"learner"needs"the"feedback."

• Opportunity"is"that"learners"receive"the"feedback"at"time"when"they"are"most"able"to"use"it."

• Means"is"a"question"of"whether"the"learner"has"the"willingness"and"capacity"to"act"upon"the"feedback.

Motive - Opportunity - Means(Valerie Shute, 2008)

The Three Levels of Feedback (John&Hattie,&2012)

Task&Feedback&8Describes&how&well&the&student&has&performed&on&a&task.&8Right/Wrong,&acquiring&specific&information,&building&knowledge.&

Process&Feedback&8Describes&the&process&underlying&or&related&to&tasks.&8Strategies&to&detect/learn&from&errors.&8Ways&to&establish&a&relationship&among&ideas.&

Self8Regulation&Feedback&8Describes&how&learners&can&monitor,&direct,&and&regulate&their&own&actions&as&they&work.&8Foster&the&willingness&and&capacity&to&seek&and&effectively&deal&with&feedback.&8Encourage&students&to&self8assess&and&self8correct.

Novice

Proficient

Competent

Content

Strategies

Understandings


8

(4) Is it strength-‐based?

(5) Does it cause thinking?

“Feedback is most effective when it is the right kind, delivered in the right way, at the right time, and to the right person.

-‐Heidi Andrade (2013)

Right kind? Right way?

Right time? Right person?


9

Self-Assessment and Self-Regulation

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

Which of the three phases of self-regulation of learning do you utilize the most? The least?

Which of the three phases do you think you could most easily add into your instructional routines? (Assuming you don’t already use/maximize all three).

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

Prerequisites• Clear'learning'goals'

• Clear'success'criteria'

• Clear'interpretation'of'the'criteria.'

• Classroom'culture'where'being&wrong'is'supported.'

• Classroom'culture'with'a'norm'of'working'together.'

• Students'who'have'some'proficiency.'

• ??

Now we’re ready!

Self%Regulation.of.Learning Zimmerman,)B.J.)(2002).)Becoming)a)self7regulated)learner:)An)overview.)Theory'into'Practice,'41(2))64770.

• Forethought)Phase)

• Performance)Phase)

• Self7Reflection)Phase

Task AnalysisSelf-Motivation Beliefs

Self-Control Self-Observation

Self-Judgment Self-Reaction

Assessment & Self-Regulation

Assessment''(FA/SA)'as'INPUT

Monitoring/Reflecting''Performance

Assessment''(FA/SA)'as'OUTPUT

Monitoring/Reflecting''on'Performance

It’s not as clean as “IF/THEN”

Potential flaws of self-assessment

(Dunning, Heath, & Suls, 2004)

A"tendency"for"humans"to"be"overly"optimistic"about"their"own"abilities.

A"tendency"for"humans"to"believe"they"are""above"average.

A"tendency"for"humans"to""neglect"crucial"information.

A"tendency"for"humans"to"have""deficits"in"their"information.

Double Whammy!


10

Which of the four potential limitations of self-assessment do your students display most often?

How have you tried to mitigate this concern?

Peer Assessment “Peer assessment is generally an arrangement for classmates to consider the level, value, or worth of the products or outcomes of learning of their equal-status peers. However, it can also extend to learning behavior or social behavior and sometimes encompass both academic products and associated behavior.”

-Keith J. Topping (2013)

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

Formative Assessment

Summative Assessment

Qualitative)detail)about)areas)of)strength)and)that)which)needs)strengthening;)where,)why,)&)how.

(+)$More$available$(.)$May$not$be$as#accurate

Overall,)holistic)judgment)of)quality)that)considers)overall)strengths)and)weaknesses.)

(+)$More$reliable$(.)$Validity/Reliability$concerns

Grades?

Peer$Assessment$Typology (Keith$Topping,$2013)

• Qualitative?$Quantitative?$Both?$• Single$sample?$Multiple?$Varied?$• Balanced$feedback?$Only$one$(+/L)?$• Opportunity$to$act?$$• Individual?$Pairs?$Groups?$• Reciprocal?$OneLway?$Group?$• Deliberate$matching?$Random?$• Same$year?$Different$year?$• Same$ability?$Different$ability?

Qualitative? Quantitative? Both?• Qualitative"assessment"would"focus"on"peers"providing"

descriptive*feedback"on"what’s"next"to"improve"the"quality"of"the"work."

• Quantitative"assessment"would"focus"on"peers"making"a"holistic"judgement"about"the"overall"quality"of"the"work."

• Both"means"a"holistic"judgement"is"made"and"descriptive*feedback"is"provided.

Which one? How often? Most effective? Would like to try?


11

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________

Single Sample? Multiple? Varied?• Single'sample"means"a"one)off"peer"assessment"

opportunity."

• Multiple'samples"means"peer"assessment"occurs"once"with"multiple"samples"at"the"same"time"or"several"times"with"one"sample"per;"between"the"same"student,"or"with"different"students."

• Varied"means"the"work"sample"formats"would"be"different"given"the"assessment"targets"and"standards.

Which one? How often? Most effective? Would like to try?

Cross Curricular Competencies

Critical Thinking!

Collaboration! Communication!

Creativity!


12

References Andrade, H. (2013). Classroom assessment in the context of learning theory and research. In J. H. McMillan (Ed.), Sage handbook of research on classroom assessment (pp. 17-‐34), Thousand Oaks, CA: Sage Publications. Black, P. (2013). Formative and summative aspects of assessment: Theoretical and research foundations in the context of pedagogy. In J. H. McMillan (Ed.), Sage handbook of research on classroom assessment (pp. 167-‐178), Thousand Oaks, CA: Sage Publications. Brookhart, S. (2013). Classroom assessment in context of motivation theory and research. In J. McMillan (Ed.), Sage handbook of research on classroom assessment. Thousand Oaks, CA: SAGE Publications Inc. Brown, G.T.L. & Harris, L. R. (2013). Student Self-‐Assessment. In J. H. McMillan (Ed.), Sage handbook of research on classroom assessment (pp. 367-‐393), Thousand Oaks, CA: Sage Publications. Chappuis, J. (2009). Seven strategies of assessment for learning. Portland, OR: Pearson Assessment Training Institute. Dunning, D., Heath, C., & Suls, J.M. (2004). Flawed self-‐assessment: Implications for health, education, and the workplace. Psychological Science in the Public Interest, 5(3), 69-‐106. Goleman, D., McKee, A., & Boyatzis, R. (2002). Primal Leadership: Learning to Lead with Emotional Intelligence. Boston, MA: Harvard Business Review Press. Hattie, J. (2012). Know thy Impact. Educational Leadership, 70(1), 18-‐23. Hattie, J. (2009). Visible Learning. New York: Routledge. Hogan, T.P. (2013). Constructed-‐response approaches for classroom assessment. In J. H. McMillan (Ed.), Sage handbook of research on classroom assessment (pp. 275-‐292), Thousand Oaks, CA: Sage Publications. Kanter, R. M. (2004). Confidence: How winning streaks and losing streaks begin and end. New York, NY: Crown Business. Lane, S. (2010). Performance Assessment: The state of the art. Stanford, CA: Stanford University, Stanford Center for Opportunity Policy in Education. Mislevy, R.J., Steinberg, L.S., & Almond, R.G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1(1), 3-‐62. Ruiz-‐Primo, M. A., & Li, M. (2013). Examining formative feedback in the classroom context: New research perspectives. In J. H. McMillan (Ed.), Sage handbook of research on classroom assessment (pp. 215–232). Thousand Oaks, CA: SAGE Publications. Schimmer, T. (2016), Grading from the inside out: Bringing accuracy to student assessment through a standards-‐based mindset. Bloomington, IN: Solution Tree Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.

Topping, K. (2013). Peers as a source of formative and summative assessment. In J. H. McMillan (Ed.), Sage handbook of research on classroom assessment (pp. 395-‐412). Thousand Oaks, CA: SAGE Publications. Vagle, N. D. (2015). Design in Five: Essential Phases to Create Engaging Assessment Practice.

Bloomington, IN: Solution Tree.

Wiliam, D. (2013). Feedback and Instructional Correctives. In J. H. McMillan (Ed.), Sage handbook of research on classroom assessment (pp. 197-‐214), Thousand Oaks, CA: Sage Publications. Webb, N. L., Alt, M., Ely, R., & Vesperman, B. (2005) Webb alignment tool (WAT) training manual. Madison,

WI: Wisconsin Center for Educational Research. Zimmerman, B.J. (2002). Becoming a self-‐regulated learner: An overview. Theory into Practice, 41(2) 64-‐70.

Documents

16.10 (NESA) Accurate & Effective Formative Assessment … · (1) Student investment occurs when assessment and self-regulation have a symbiotic relationship. (6) The communication