Upload
constance-harvey
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
EXPLORING INTER-RATER RELIABILITY
Professional Learning Team Conference
June 2014
Desired Outcomes:Each participant will leave the session
knowledgeable about inter-rater reliability
What it is Why it’s important, and How to use evidence to consistently
and accurately rate teachers’ performance
Goals for Inter-Rater Reliability 2014-2015
Demonstrate familiarity with district rubric
Describe proficient practices based on district rubric descriptors
Provide feedback to educators based on evidence collection
Calibration of evaluators
PARTICIPANTS WILL:
Establish a common vocabulary around inter-rater reliability
Be able to explain levels of performance with look-fors
Gain a deeper understanding of proficient level of performance aligned to standards
GOALS:
CHINESE PROVERB:
What I hear, I forget 5% retention after 24 hours
What I see, I remember 30% retention rate if use of
audio-visual and demonstration
What I do, I understand 80% retention rate with
discussion and practice by engagement
NORMSShare experiences to enrich others
Ask questions
Learn by doing
Set aside any preconceived notions about evaluating educators
Apply your own work
LEARNING TARGET:Participants will become familiar with look fors of effective practice and be able to use that knowledge to assess performance in three focused areas:
Student Engagement
Questions, Prompts & Discussion
Assessment
NATIONAL FOCUS ON TEACHER EFFECTIVENESS AND EVALUATION
EFFECTIVENESS:
Teacher effectiveness is the single most important school-related factor affecting student learning.
EVALUATION:
However, most schools fail to evaluate teachers in a meaningful way, with 94% of teachers receiving the highest ratings.
Teacher evaluations in five urban school districts, based on data taken from a report by The New Teacher Project: http://widgeteffect.org/downloads/TheWidgetEffect.pdf.
Teacher evaluations in five urban school districts, based on data taken from a report by The New Teacher Project: http://widgeteffect.org/downloads/TheWidgetEffect.pdf
a systemic approach to calibrate observations for consistent and fair professional practice ratings
INTER-RATER RELIABILITY:
Learning Experience:
At your table define and describe
Student Engagement
Capture your collective thoughts on chart paper
Educational author and former teacher, Michael Schmoker shares in his book,
Results Now, a study that found of 1,500 classrooms
visited, 85 percent of them had engaged less than
50 percent of the students
In other words, only 15 percent of the classrooms had more than half of the class at
least paying attention to the lesson
You will see students...
Paying attention (alert, tracking with their eyes) Taking notes (particularly Cornell) Listening (as opposed to chatting, or sleeping) Asking questions (content related, or in a game) Responding to questions (whole group, small group, four
corners, Socratic Seminar) Following requests (participating, Total Physical
Reponses (TPR), storytelling, Simon Says) Reacting (laughing, crying, shouting, etc.) Reading critically (with pen in hand Interacting with other students
Michael Schmoker
WHAT DO "ENGAGED" STUDENTS LOOK LIKE?
Activities aligned with the goals of the lesson
Student enthusiasm, interest, thinking, problem-solving…
Learning tasks that require high-level student thinking and are aligned with lesson objectives
Students are highly motivated to work on all tasks and are persistent even when the tasks are challenging
Students actively "working," rather than watching while the teacher "works“
Suitable pacing of the lesson: neither dragging nor rushed, with time for closure and student reflection
ENGAGING STUDENTS IN LEARNING
When determining the level of performance for Engaging Students in Learning,
consider the quality of activities, assignments, and resources the instructor
uses to pull students into the lesson
It is also important to understand how the lesson uses student grouping and pacing
when determining the level of performance for
this component
STUDENT ENGAGEMENT
Learning Experience:
At your table define and describe
Assessment
Capture your collective thoughts on chart paper
is to support student learning and
to communicate that learning to others
THE PURPOSE OF CLASSROOM ASSESSMENT
In order to support student learning classroom assessment needs to:
involve students deeply in the assessment process
provide specific, descriptive feedback during the learning, and
include evaluative feedback as required to communicate and report progress over time
ASSESSMENT:
To ensure success of all:
1) students need to know what they already know
2) know what needs to be learned and 3) know what success looks like
Students also need to learn how to guide their own learning through being involved in setting
and using criteria, giving themselves feedback for learning (self-assessment), setting goals,
collecting evidence and communicating that evidence of
learning to others
ASSESSMENT:
The teacher pays close attention to evidence of student understanding
The teacher poses questions specifically created to elicit evidence of student understanding
The teacher circulates to monitor student learning and to offer feedback
Students assesses their own work against established criteria
The teacher adjusts instruction in response to evidence of student understanding (or lack of it)
INDICATORS THAT MAY BE USED AS A GUIDE FOR LOOKING FOR EVIDENCE OF ASSESSMENT
BREAK
Work with your District partners to identify the elements in your rubric that focus on
Student EngagementQuestion, Prompts & Discussion
Assessment
Learning Experience:
While observing the video, collect evidence that you will later use to rate the teacher’s
performance using your district rubric
Learning Experience:
You will use your district’s rubric to rate the teacher’s performance
in the following areas:
Student EngagementQuestion, Prompts & DiscussionAssessment
Remember…
Video
Math Lesson: CC Standard 7.G.6
https://www.teachingchannel.org/videos/surface-area-lesson#
LUNCH
In just a moment you will be asked to review the data
you’ve collected, cluster and code it for easy interpretation, and rate the
teacher’s performance using your district’s rubric
DATA
DATA
DATA
Evidence
Interpretation
Judgment
WHAT IT LOOKS LIKE TOCLUSTER AND CODE
ANOTHER METHOD OF EXAMINING YOUR EVIDENCE
Student: Um, how he has the equation. CM: Exactly, how he has the equation so every calculation starts with his equation. I like that too, good. Kay-Wen, keep it moving. Student: He does it step by step. CM: What do you mean? Student: Well, for the circumference, see how he switched the d – 4:11 CM: Into a…? Student: Where is it? CM: Here. Into the diameter of 60, so that’s one step, that step’s called substitution. Very good, I like that, Kay-Wen. What else do we like, Corrine? Student: I like how it’s organized and that there’s no mistakes and he’s not skipping steps. CM: Absolutely, did any of you guys have to do a double take, like look at it like ugh, what does that say? No, it’s very neat and organized, it’s clearly stated, very good. Taylor, what else? Student: Um, his answers are circled so that we know where to look for the actual answer.
Feedback that guides performance
Select student at randon to participate
Prompts
Feedback to reinforce behavior rather than instruction
Fade 10:44 Student: We’re using a basic strategy, but I think it might work. CM: Let me, let me see. First of all, start your strategy off, what was your strategy? Student: Our strategy was um, height by circumference would equal the surface area of the cylinder and the reason is because – CM: The whole cylinder? Which part? Student: This part. CM: Which part? Student: The part around. CM: Okay, is that it?
Teacher circulates, checking understanding and asking clarifying question to deepen understanding Assessing learning
ANOTHER METHOD OF EXAMINING YOUR EVIDENCE
With an elbow partner, use the verbatim transcript of the teacher’s observation to identify and organize
evidence to accurately and fairly rate the teacher’s performance in the
areas of
Student engagementQuestions, Prompts and Discussion
Assessment
Look at your evidence for each component. Where does the preponderance of evidence align with levels of performance?
Match your data with the rubric performance level description
Assign a performance level
Use your clicker to rate this teacher’s performance using the numbers
Unsatisfactory
Basic Proficient
Distinguish
TESTIDENTIFY YOUR FAVORITE
CANDY BAR
1. Snickers2. Milky Way3. Kit Kat4. Trix
Vikings
Beavers
Boxers
Ducks
64%
0%
9%
27%
1. Unsatisfactory2. Basic3. Proficient4. Distinguish
Unsatisfa
ctory
Basic
Proficie
nt
Distinguish
0%
27%
64%
9%
STUDENT ENGAGEMENT
DISCUSS
1. Unsatisfactory2. Basic3. Proficient4. Distinguish
Unsatisfa
ctory
Basic
Proficie
nt
Distinguish
0% 0%
88%
13%
QUESTIONS, PROMPTS & DISCUSSION
DISCUSS
1. Unsatisfactory2. Basic3. Proficient4. Distinguish
Unsatisfa
ctory
Basic
Proficie
nt
Distinguish
0%
18%
82%
0%
ASSESSMENT
DISCUSS
ONE – TWO PUNCH
One thing you heard today that you want to
remember
Two things you heard that you want to act on
before the next PLT Conference
John Doe1234 Paradise Lane
Anywhere, OR 97890
QUESTIONS
Thank you for your active
participation and cooperation