Student Learning Objectives: Our Last, Best Hope ( for improving classroom assessment, instruction, & learning)

SCOTT MARION & ELENA DIAZ-BILELLO*

CENTER FOR ASSESSMENT

ACEE MEETING

JANUARY 17, 2014

*THANKS TO OUR COLLEAGUES, CHARLIE DEPASCALE & JERI THOMPSON FOR MANY CONTRIBUTIONS TO THIS

PRESENTATION

Student Learning Objectives: Our Last, Best Hope

(for improving classroom assessment, instruction, & learning)

Center for Assessment ACEE Meeting (1/17/14)

2

Advance Organizer

Intro •A discussion of SLOs with a focus on deeper learning

SLO •The roll of assessments within SLOs•Evidence Centered Design (ECD)

Quality •The Challenge of SLO targets


3

The business of schools is to invent tasks, activities, and assignments that the students find engaging and that bring them into profound interactions with content and processes they will need to master to be judged well educated.

Schlechty, P.C. (2001) Shaking up the schoolhouse. San Francisco: Jossey-Bass


4

What Do You Think?

1. How do teachers think about learning goals for students? Be honest!

a. They follow the sequence of content and lessons in the text

b. They create learning goals for lessons and units, but not necessarily sequenced throughout the year

c. They create long-term learning goals and sequence instruction to help students reach the goals

d. Some combination of the above


5

Defining Student Learning Objectives

SLOs are content- and grade/course-specific measurable learning objectives that can be used to document student learning over a defined period of time. They include:1. A learning goal focused on a “big idea” of the discipline2. Assessment(s) to measure students’ learning of the goal3. A description of the instruction and materials used to

provide students with an opportunity to learn the goal4. Targets for students and aggregate targets for teachers

We focus on 1, 2, & 4 today


6

It’s About the Learning

Yes, SLOs have gained popularity because of teacher evaluation

SLOs provide a vehicle for thinking about critical and long-term learning goals and how students progress towards those learning goals

We haven’t had many large-scale initiatives that have provided this sort of dedicated focus on the learning process for both teachers and students


7

Our last, best hope

SLO are our last, best hope for reining in large-scale assessment-based accountability.

Since the mid-1980s, large-scale, standardized assessment in K-12 education has increased inexorably on virtually every conceivable metric (time, frequency, cost, consequences, uses, etc.)

Ironically, SLO represent the first crack in the accountability door that has been closed tight on classroom- and school-based measures of student and teacher performance.

The Challenge of the “Learning Goals”

We have not seen evidence that teachers and other educators can generate high quality SLOs without significant practice and training

Identifying meaningful learning goals appears to quite difficult

We can draw on work such as Wiggins & McTighe’s Understanding by Design and the assessment specifications being developed by both large scale assessment consortia But this is still a huge challenge!


8


9

Learning with understanding is facilitated when new and existing knowledge is structured around major concepts and principles of the discipline.

Learners use what they already know to construct new understandings.

Metacognitive strategies and self-regulatory abilities facilitate learning.

Learners’ motivation to learn and sense of self affect what is learned.

Participation in social practice is a fundamental form of learning.

How People Learn Bransford, Brown, & Cocking (1999), NRC


10

Deep Understanding

There is a close relationship between truly understanding a concept and being able to transfer knowledge and use it in new situations

Deep understanding is flexible, connected, and

generalizable

Deep understanding is challenging to achieve, so it must be focused on the critical components of the discipline


11

All SLOs are not the same

Some (many) SLOs are being implemented around the country simply as an accountability or compliance mechanism

We need to ensure SLOs are designed to promote deeper learning

Like Understanding By Design (UBD, Wiggins & McTighe), learning goals for SLOs should lead to enduring understandings Core idea of the discipline Useful (critical) for continued learning in that and other

disciplines Measureable and able to judge progress along the way


12

Learning Goal Examples

Please refer to the handout for examples of learning goals. We think some of these are really strong and others are less so, but still pretty good.

We’ve also distributed the rubric, developed as part of the Center for Assessment SLO Toolkit (www.nciea.org) that can be used to evaluate the learning goals.

We don’t have time today to go through this in detail, but wanted you to have access to the examples.

http://www.nciea.org/


13

But There’s Too Much Stuff!

Legitimate complaints from teachers that there is too much content to cover, too much processes involved

Not enough time to learn all required content and skills deeply

Hopefully newer standards have trimmed the amount of content and skills, but…

We need to establish structures, professional learning opportunities, and permission for teachers to facilitate deeper learning

SLOs may serve as a promising structure


14

The Goal and the Journey

Identifying the long-term learning goal for teachers and students helps provide a sense of structure of the discipline and can support greater understanding of learning theory for teachers and metacognition for students

But the goal is not enough…Teachers need a clear understanding of the

important markers along the way to the goal

15

Learning Progressions Knowing What Students Know (2001)

Learning progressions, based on professional judgment and empirical test, can be a powerful tool to support teacher learning.

“Progress maps describe skills, understandings, and knowledge in the sequence in which they typically develop: a picture of what it means to ‘improve’ in an area of learning.” (Masters & Forster, 1996).

A criterion-referenced growth model.

Learning progressions or Progress Maps provide an underlying model of learning to coherently link classroom and large-scale assessments.



16

Learning Progressions

Learning progressions provide a great organizing framework for:

Clarifying the learning goals Establishing targets for student performanceIdentifying the set of assessments that can be

used to both progress monitor and evaluate students on learning goals

Facilitating conversations with and among teachers about student work

17

Example: A semester long SLO framed as a progression

What students should know and be

able to do after instruction

[SLO]

What students should know and

be able to do prior to instruction

The “Messy Middle”:

1. Upper anchor (the SLO learning goal): Students will critique and compare the techniques and styles of two artists located in two time periods. Students will write up their critique and provide their comparative analysis in a report.

2. Provides evidence supporting a point of view based on information gathered from the image content, the intended impact of seeing the work of art, the artists intended meaning, and interprets the work to communicate a deeper connection with the artists.

3. Makes connections between the elements observed, the meaning and purpose perceived and the information gathered regarding works from each artistic period and movement.

4. Compares the elements and multimedia language used in an image used by each artist.

5. Compares the visual evidence produced using either traditional or new forms of materials and tools used by each artist.

6. Lower anchor: Identifies methods of representing space including isometric and aerial perspectives used by each artist.


18

Discussion Questions

In what ways is the SLO process different from current practices in the classroom?1. What do teachers consider now in planning instruction for

the year?2. Would a typical teacher be able to describe at the

beginning of the year the level or type of performance that would be required to earn a final grade of A, B, or C in the course?

a. To what extent are those grades tied to specific knowledge and skills, status, growth, or other factors?

3. What is the current balance between focus on a. content and students, b. process and outcomes, c. short-term activities and long-term goals?


19

Learning and Assessment

Meaningful assessment scores depend on tight linkages among learning targets and assessment design

As we have discussed, the learning goals should expect students to learn rigorous content and use disciplinary skills to apply this content knowledge

In other words, we want students to develop deep understanding of important knowledge and skills


20

Assessing for Deep Understanding

Students cannot develop deep understanding unless they are provided opportunities on both learning and assessment tasks.

In other words, if low-level assessment items are the focus, it is unlikely that teachers will feel the need to teach students to think deeply.

A major component of 21st Century skills is the ability to solve novel problems—this requires deep understanding!

Assessment conveys what’s important to learn (a signal) as well as providing an opportunity to check on students’ understanding and evaluate achievement


21

Developing an SLO: The role of assessment

Where does assessment enter into the process of designing an SLO? Is assessment the driving force in the

development of the SLO? Or Are assessment decisions the outcome of prior

decisions on content, knowledge, skills, and required evidence?


A “normal” sequence

22

Begin with Content Standards

and Curriculum Materials

Identify Priority Knowledge and

Skills

Determine and Describe Desired

Performance

Identify the assessment(s) to

collect evidence of the Desired

Performance


What happens when the normal sequence is turned upside down?

23

Begin with Content Standards

and Curriculum Materials

Identify Priority Knowledge and

SkillsDetermine and

Describe Desired Performance

Identify the assessment(s) to collect evidence of the Desired Performance


Beginning with the Assessment

24

Begin with a

state or district

assessment

Determine what is measured

on the assessme

nt

Develop an SLO

In the worst-case scenario, it will be necessary to force-fit an SLO to a misaligned assessment

In the best-case scenario, there will be alignment between the content standards, curriculum, instruction, and the assessment


25

Beginning with the Assessment?

Advantages

Increased likelihood for a “quality” assessment

Opportunity for common performance expectations across classrooms

Reduces the burden of test development

Reflects the “reality” of common content standards

DisadvantagesMore likely to result in

evidence being gathered from a single assessment

Increased likelihood for the assessment and the SLO to be perceived as external and separate from instruction

Potentially less sensitive to differentiated needs of students in a particular class


26

How Many Assessments?

If the learning goal is really a big idea of the discipline, it is hard to imagine that it can be validly measured with a single assessment

The number of assessments is likely contingent upon: The scope of the learning goal The scope of the assessment(s)


27

How Many Assessments?

We need to tie this back to the evidence model

What type and number of assessments will provide the required evidence to support claims that students have mastered the learning goal? Does this include claims about generalizability too?

If we have multiple assessments, we need to be very thoughtful about combining the results. A simple average will often not be the best approach. What else should we consider?


28

Assessment Quality for NTSG?

Some have proposed classifying assessments evaluating teachers’ contributions to student learning in non-tested subjects and grades according to the following scheme: Type I: Statewide standards-based assessments Type II: Commercially available standardized

summative and interim assessments Type III: Locally-created assessments

What’s wrong with this picture?

Technical Quality


29

Many states and others are beginning to examine the technical quality of measures used in educator evaluation determinations

Most of what we have seen focuses on traditional aspects of assessment quality

This is a good start, but do we have to consider any other dimensions for measurement of SLOs?


30

Criteria for Quality Assessments

Critical criteria and standards for the quality of educational assessments: Validity Reliability Fairness (inclusivity and equitability)

Additional criteria often include: Manageability or practicality Relevance Transparency

Darling-Hammond and colleagues (2013) recently posited: Assessment of higher-order cognitive skills High-fidelity assessment of critical abilities Standards that are internationally benchmarked Use of items that are instructionally sensitive and educationally valuable Assessments that are valid, reliable, and fair


31

Dilemma for the use of SLO

There are few locally-developed assessments that meet the technical criteria for quality assessments

There are few local assessment practices and policies in place that support high quality assessment

It is not practical to develop and maintain external, standardized assessments (i.e., state assessments) in for every course – although some states and large districts have adopted this approach

Even if a high quality, external assessment were available for each course, it would only partially fulfill the assessment needs of a good SLO


32

Solution

Increase general assessment literacy to support informed decision-making regarding assessment

Improve understanding of Which aspects of each criterion are critical and must

receive attention What steps to improve technical quality are important

and practical for different levels (e.g, state, district, classroom) and assessment formats (e.g. performance tasks, multiple-choice, essay)

The impact of decisions, actions, and inaction related to certain quality criteria


33

Assessment Review Tools to Support Quality SLOs

Many states, districts and the Center have developed review tools to help users judge the quality of tasks and assessments that are created or selected locally

For any of these tools, guidance is still required to help users understand quality criteria and standards

The vast majority of these tools are publicly available on most websites

Also…


34

Accessing Assessments for SLOs

With the growth of the technological tools, there is Increased opportunities for sharing assessment instruments

and practices Increased opportunities for sharing information and

providing support regarding assessment decisions Increased opportunities for state-supported assessment

tools (in contrast to state-mandated assessments)Large-scale assessment has reached a tipping

point or saturation point. It will become obvious that large-scale assessment, alone, cannot meet the assessment demands of the Common Core or the Darling-Hammond et al criteria.


35

Performance Targets

Student targets: How well students are expected to perform on the assessment(s) tied to the learning goal

Teacher targets: How well students are expected to perform in the aggregate to contribute to decisions about educator evaluation


36

Major Approaches for Student Targets

Achievement or Status Focus on the level of achievement attained at the

end of the interval (e.g., proficiency, mastery, college-and-career readiness)

Growth Focus on a change in performance over the

course of the interval The interval may be within a year (fall to

spring), or The interval may be across years (spring to

spring)

37

Status v. Growth: Is it a false dichotomy?

If not false, then perhaps not quite a true dichotomy

In some cases, status (or achievement) at the end of an interval is a proxy for growth.

Implied growth is reflected in status when there is an assumption that the student had not reached a particular level of achievement prior to the beginning of the interval. That assumption may be stronger in some cases or courses

than others. That assumption may be stronger for a group of students

than for an individual student.


The Problem of “Growth”

Many want to set the student and teacher targets using some sort of growth framework (pretest/posttest)

In most cases, this makes very little senseYes, I know what the RTTT requirements

say…They are wrong!If these requirements were followed literally,

value-added and student growth percentile models could not be used—not growth models!


38


40

Pretest/Posttest: Two Options

Administer the same test twice

Advantages “Easy” to see improvement

between tests Only need one test

Disadvantages Familiarity with the test

items Danger of teaching to the

test or learning the test Potential loss of

generalizability

Administer two different tests

Disadvantages Differences in content or

difficulty between tests Need multiple tests

Advantages Less exposure of test items Greater opportunity for

targeted assessment at each point in time

Increased support for claims of generalizability


Dangers of Using Different Tests When Thinking They Are the “Same” (An example)

41

Set A Set B

Popham (1978). Criterion-Referenced Measurement, Prentice-Hall

89x94

87x69

99x99

97x89

30x10

20x20

10x10

10x20

43

Gain Scores: A simple difference

The Siren Song of Simplicity Gain scores are inherently appealing They appear simple to compute and to interpret

The Illusion of Precision A test score, whether expressed as a number correct or a

percentage, seems very precise. The difference between two scores conveys a sense of precision,

truth, and objectivity.Unfortunately, it’s just not true!

For a detailed explanation, see: Marion, S.F., DePascale, C., Domaleski, C., Gong, B., & Diaz-Bilello,

E. (2012, May). (2012). Considerations for analyzing educators’ contributions to student learning in non-tested subjects and grades with a focus on Student Learning Objectives. www.nciea.org.


http://www.nciea.org/


44

Gains Scores: Two popular approaches

Fixed Gain Determining a fixed number of points for all students,

or a subgroup of students, to gain from pretest to posttest. May be based on a norm such as the average gain or a

criterion such as the number of points needed to remain at the Proficient level from one grade to the next.

“Half the Distance” Determining an individual “gain target” for each

student based on cutting in half the gap from the pretest score to a fixed point (e.g., 100%).

45

In case it’s not bad enough…

Measurement error influences both the pretest and posttest when setting and evaluating performance targets.

This could be reflected in: Where you set the performance target for individual students,

or whether you establish a target range rather than a fixed point.

Where you set the bar for the number (percentage) of students expected to meet the target.

When setting performance targets: It is easier to notice gross than fine changes in performance It is easy to classify performance far away from a benchmark

or cut score.Center for Assessment ACEE Meeting (1/17/14)


50

Implications of Measurement ErrorIn addition to all of the other caveats

You must consider the presence of measurement error on both the pretest and posttest when setting performance targets.

This could be reflected in Where you set the performance target for individual students, or

whether you establish a target range rather than a fixed point. Where you set the bar for the number (percentage) of students

expected to meet the target.When setting performance targets or scoring an SLO:

It is easier to notice gross than fine changes in performance It is easy to classify performance far away from a benchmark or

cut score.

A “Rough Conditioning” Approach for SLOs

Using prior performance information (e.g., last year) or some early assessments in the current year, we can group students into 3-4 “performance” groups

SLO targets would then be differentiated according to the students’ starting group.

At least two ways to differentiate targets: Different levels of achievement (e.g., basic, proficient) Different proportions of students reaching the same

target (e.g., 80% of Level 3 students will achieve target, 65% of Level 2 students will achieve goal)


51


52

Learning trajectory approach

Use prior performance data or early assessments in current year directly related to the learning trajectory, establish starting points for each student

Evaluate performance of students relative to specific assessments anchored to each level of the trajectory and determine the location of students on the trajectory

Options for evaluating movement on the trajectory: Assign points to movement using ratio scale Assign points to movement using pseudo-continuous scale

Key difference from rough conditioning approach: underlying points have content-referenced meaning


53

Failure is not an option

Regardless of how teacher evaluation evolves, teachers making informed decisions about instruction, setting rigorous yet realistic goals for students, and making accurate judgments about student performance (i.e., the SLO process) is the essence of education.

If we cannot reach the point where we have confidence in the accuracy of performance information generated at the classroom and school level then what is the point…

Documents

Student Learning Objectives: Our Last, Best Hope ( for improving classroom assessment, instruction, & learning)