Advancing Assessment Literacy Data Gathering I: Establishing Outcomes

Advancing Assessment Literacy

Data Gathering I:

Establishing Outcomes

Data Gathering I:


Part One: Beginning the Search

Advancing Assessment Literacy Modules: Data Gathering I - February 2008 3

Rearview Mirror Effect

Planning for the future on the basis of events past is the school planning problem

Stephen White (2005) identified as

the Rearview Mirror Effect.

White, S. H. (2005). Beyond the numbers: Making data work for teachers and school leaders. Englewood, CO: Advanced Learning Press.


Four Debilitating Characteristics of the Rearview Mirror Effect

1. Responding to a rapidly changing reality based on past events.

• Non-anticipatory• Doesn’t seek information about the current

reality from students and parents• This is the wait and see approach to

education – “We’ll see if this literacy thing works before we adopt it for ourselves”.




1. Responding to a rapidly changing reality based on past events.

• What do you anticipate will need to be addressed in the future regarding student learning?

• How much do you know about the current reality of students, parents and teachers?




2. Waiting for the road to reveal itself, by depending on and waiting for biannual assessments.

• This is a common concern regarding the Assessment for Learning results – the data comes too late to matter and the decisions we make don’t affect the students who were tested.




2. Waiting for the road to reveal itself, by depending on and waiting for annual or biannual assessments.

• In what ways might the road be made more clear between annual or biannual assessments?• What is known about the function of

assessments such as Assessment for Learning and CAT3?




3. Focusing on a single dimension of the highway; that is, a focus only on what students do.

• We must look at the interrelationship inherent within problems, processes and systems.

• We must look at data from a wide variety of sources to inform planning.




3. Focusing on a single dimension of the highway; that is, a focus only on what students do.

• What other types of data might need to be gathered and looked at to paint a more detailed picture of student achievement?




4. A wistful looking back to a time when things were simpler.

• “I wish I could go back to just teaching, rather than constantly assessing.”




4. A wistful looking back to a time when things were simpler.

• Is teaching today the same as at the beginning of your career?• What changes have you noticed?• What things have improved? Stayed the same?

Deteriorated?



Beginning the Search:Looking at Archival Data

You have been supplied with three or more years of archival data.

Example: Course Averages by Grade – final grades out of a possible 100.

Grade 7 2004 2005 2006

ELA 78 74 75

Math 75 80 68


Beginning the Search:Looking at Archival Data

• What patterns or trends seem to be emerging from the data?

• Craft general statements based on what you see; for example:– Student performance in English seems

to be consistent.– Math performance was increasing but

has decreased significantly since 2005.


Beginning the Search: Affinity Diagramming

• As a table group, choose one of the statements that you believe is most pressing.

• Brainstorm hunches and hypotheses that might explain why this situation exists.

– Put each on a sticky note. (Avoid one word responses.)

– Brainstorm at least 20 hunches/hypotheses.

– For example, “Math achievement is low because teachers are struggling with the format of the curriculum guide.”

Bernhardt, V. L. (2004). Data analysis for continuous school improvement. Larchmont, NY: Eye on Education.

15


• When you are done brainstorming:– Look over the sticky notes and sort the items into

categories.– Create a title for each category.

Title

Title

Title




• This process allows for the identification of a variety of seemingly unrelated reasons and can lead to the revelation of interesting interrelationships.

• From your Affinity Diagram, select the category or categories that contain the majority of hunches and hypotheses.


17

Identifying Effects and Drivers: Interrelationship Diagramming

• Within each category it is important to separate drivers (causes) from effects.– For example, introduction of a breakfast program (driver) will

improve student concentration (effect).

Wellman, B. & Lipton, L. (2004). Data-driven dialogue. Mira Via, LLC.

Arrows moving away from boxes indicate drivers,while arrows moving into boxes indicate effects.

A sample interrelationship diagram is on the following slide.

BreakfastProgram

StudentConcentration

Driver Effect


Division

• Division Standards

19

Creating anInterrelationship Diagram

Take the sticky notes from the longest list and place them in a circle with the topic as a header above:

TOPIC (The title of the category)




• Select one item in the category as a starting point.• Ask two-way questions to determine whether this item is

a driver or effect of each of the other items. For example, “Is provision of a breakfast program a driver of student concentration or is student concentration a driver of providing a breakfast program?”

• Draw arrows from the drivers to the effects.• Continue in this way through each of the items.• NO two-headed arrows! Decide which item dominates

the other.




• DRIVERS

– Drivers are indicated by the number of arrows going away from an item.

– Count the number of arrows going away from each item.

– Rank the drivers from highest to lowest.

• EFFECTS

– Effects are indicated by the number of arrows pointing towards an item.

– Count the number of arrows pointing towards each item.

– Rank the effects from highest to lowest.



Creating HypothesesBased on Drivers

• Select the top 3-5 drivers.• Create a hypothesis for each. For

example,– “Math achievement dropped because

teachers are struggling with the format of the curriculum guide.”

– “Math achievement has been decreasing because their background understanding is insufficient.”


Testing Validity

• The hypotheses that have been generated are based on the experiences and perceptions of the group.

• Before committing to action regarding these hypotheses, it is necessary to further check their validity to ensure that they are truly drivers of student learning in the areas you have identified.

• Literature regarding these hypotheses will be gathered and shared at your next meeting.

Data Gathering I:


Part Two: Testing the Validity of Hypotheses


Testing the Validityof Hypotheses

• Hypotheses were created based on the drivers identified at our last meeting.

• Today, we are going to check the validity of those hypotheses against current literature in those areas.



• Divide into equally sized groups based on the number of hypotheses to be tested.

• Each group will receive the hypothesis and one or more current articles pertaining to that hypothesis.



The task:• Read the articles related to the hypothesis you’ve been

assigned.• How is this hypothesis reflected in the literature? What key

elements are present or missing?• Is the hypothesis supported by the literature? (i.e. Is it really an

issue?)• How might the hypothesis need to be changed to reflect current

research in this area?• Is this hypothesis valid? Is it truly a driver of student learning in

this area?• Be prepared to present your findings to the larger group.


Presentations

• Read your hypothesis to the group.

• Present the findings from your reading.

• Give your recommendation regarding the validity of this hypothesis.

• Once all have been presented, decide which hypotheses you would like to explore.


Converting Hypothesesto Goal Statements

• Once hypotheses have been identified as valid and worthy of pursuit, they must now be converted into goal statements.

• For example:– “Math achievement is low because teachers are

struggling with the format of the curriculum guide.”

will now be converted to:– “Math teachers will be supported in their

understanding and actualization of the curriculum guide in order to improve student learning.”


Identified Outcomes

• The goal statements that you have created are now the identified general outcomes for your staff and students.

• These statements are broad and will require further unpacking to flesh out the indicators of success as you work toward achieving your goals.

• At a later date, these general goal statements will serve as the foundation of a goal statement such as SMART goals.

Documents

Advancing Assessment Literacy Data Gathering I: Establishing Outcomes