DATA DRIVEN DECISION MAKING A Leadership Seminar for Pennsylvania Urban Academy June 23-34, 2004

DATA DRIVEN DECISION MAKING

A Leadership Seminar for

Pennsylvania Urban Academy

June 23-34, 2004

Some Basic Understandings

In solving student achievement problems, you can’t blame the kids!

• First of all, whatever the circumstances are, no one really cares. They want you to do your job – no blame and no excuses.

• Second, whatever circumstances they bring are at least partially out of their control. I’m sure they didn’t wish those circumstances on themselves.

• Finally, our job is to overcome circumstances so we can do the job we’re authorized to do.

You can’t blame the parents either!

• First of all, no one in public policy cares about their role in shaping student achievement. They have no public accountability for their child’s academic learning.

• Second, if they could do better, most of them probably would. By and large, at least one parent wishes for their child something better than they currently have.

• Blaming them for their child’s academic failure only allows the school to try and deflect its own professional responsibility, and that won’t work.

Nobody has all the answers!• None of us is genetically encoded to solve the

almost intractable problems of educating all children to high levels of performance.

• However, everybody knows something, and the “thing” one person knows might be exactly what another person needs to know.

• The “trick” is to figure out how to tap into other people’s experiences. That can start by your encouraging them to tap into your own.

Beating a dead horse harder will not bring the horse to life!

• If something isn’t working as it should, spending more time and energy doing the same thing is ineffective.

• Remember the organizational definition of insanity.

• Most good athletic coaches adjust to their talent, to their successes, and to their past failures. The difference between consistent winners and all others is in their passion for learning how to do ordinary things differently.

There is no safety net in the business of school improvement.

• If you’re trying to navigate the process of change as a risk-free enterprise, you’re in the wrong business.

• Remember the Gretsky principle: You will miss 100% of the shots you never take.

There are not enough resources – and there never have been!

• Any thinking that the most possible solution lies in “more” (of anything) will be a problem before it ever gets off the ground.

• Most plausible solutions will require reallocating existing resources to achieve heretofore elusive results.

• The basic question is: How can you get the best results with the resources you know you can control? If you can’t control them, you might as well not have them.

Data change nothing – People change everything!

• Information is inert. It’s only purpose is to be useful. Otherwise, it just sits somewhere, occupying space.

• Three kids are sent to you because of their directing profanity towards their teacher. Will all three of them get the same treatment from you? What will differentiate punishments?

• School improvement has less to do with what you know than what you do with what you know.

Ignorance is no excuse!• If doing heavy instructional leadership lies

outside your comfort zone, not doing it is no longer an option. Either learn how or delegate and empower it to someone else.

• If a teacher teaching in a tested grade level lacks the ability to manage curriculum content mastery, either help him/her learn how to or find this person another less obtrusive assignment.

Shift happens!• Whatever you think current reality is doesn’t make

it tomorrow’s reality. Decisions others make have a way of altering your status quo, sometimes in profound ways.

• Consider the effect of any serious demographic shift in your school’s population on the school’s overall achievement profile.

• Are there any probable consequences if you had to replace a gifted and experienced 5th grade teacher with a lateral entry rookie who got into teaching because her dot.com company was downsized?

So, what is this fuss about data all about?

How do you decide whether or not you can afford to put a whole new roof on the house or simply shingle over the old ones?

What factors will influence your decision as to whether or not your 15 year old daughter can go to a friend’s Halloween party and spend the night there?

You see, while the information you have is important, it is only important to help you make the best decision possible under whatever circumstances exist. When circumstances or the data change, so might your decision.

Let’s look at some examples of data and you tell me the relative value of each.

Assume you’re the coach of a championship caliber high school basketball team. As you begin planning for the next season, you have these data. Your goal is nothing less than the Conference Championship.

Team’s record this past season was 16-10

Finished in 2nd place in the Western Division of the Conference

Record for games on the road was 5-8 (Division average was 6-7)

The team lost in the 2nd round of the Division Playoffs

The team averaged 12.6 turnovers per game (Conference champion was 8.2)

Team’s free throw percentage was 70.3% (Conference champion was 75.2%)

Team averaged 24 rebounds per game, 6 were offensive (Conference champion was 27)

Team committed 46% of its fouls in the 4th quarter

Team averaged 2 hours of practice per game, 75% on offense

Each team member shot about 30 free throws per practice

Team’s scoring average was 66.5 points per game; they gave up an average of 63.6 points per game

The starting guards averaged about 27 minutes of playing time per game, out of possible 36 minutes

Let’s look at some other examples of data and you tell me the relative value of each.

Assume you’re the principal of a high school committed to getting better. You and your school improvement team have these data from last year. You want nothing less than to be regarded as the highest performing high school in the region.

11th Graders scored at the 56th percentile in reading and 62nd percentile in math on the PSSA

62% of the students taking Algebra I scored at proficiency or higher

68% of the 11th grade Asian students taking the Reading Exam scored at proficiency or higher

The school’s overall proficiency percentage composite was 58.4%

The analysis of teacher-assigned grades over the previous academic year showed 44% of all students received A’s and B’s, while 22% received D’s or F’s.

The teacher turnover rate for the last three years has averaged 22% each year.

25% of last year’s teachers did not hold tenure.

About 40% of the teachers either are not familiar or they ignore the Pennsylvania’s Curriculum Standards.

In the reading test, 48% of the students missed 3 or more test items dealing with finding key facts that support a main idea.

70% of the students not making proficiency in Algebra I were not proficient in math in the 8th grade PSSA.

Females outscore males in English by almost 11 %

The school has had 3 Merit Scholarship winners in the last 5 years. It used to average that many every year.

Okay, if using data is so great, why is it done so little or so poorly?

EXCUSES OFTEN HEARD• We know what the problems are;

we’re here every day.• Listen, we can’t control who gets

sent to the school. We just take who they send us and do the best we can.

• We don’t have the proper technology to do the curriculum well. If I just had ____ I’d be a lot more successful.

• Look, Algebra is Algebra. What’s data going to tell me that I don’t already know?

• If I have to do data analysis on everything I teach, when will I have time for planning?

• It’s not my job.

REASONS OFTEN CITED• The school’s culture doesn’t

value data. The principal never shares it.

• Staff lacks training on how to do data analysis.

• It is perceived as too time consuming.

• It is seen as a weapon to exploit vulnerability.

• Often, the tools for analysis do not accompany the data themselves. Testing coordinators don’t share.

• Because it is seen as painting an incomplete picture, it is dismissed.

• There are not a lot of good, practical models to emulate.

Let’s summarize the importance of data in improving schools and elevating the qualities of the decisions we make

Data are important because…They help us identify the most

important ways to help students.They help us get past guesses and

hunches by making us confront facts.

They give us a reality check on how we are actually doing.

They allow us to demonstrate our successes clearly and without prejudice.

They help us make better decisions by allowing us to focus on priorities and not just preferences.

Data are used most effectively when…

They are correlated with other sources of data, some of which may not be standardized.

They are aligned to the mission of the school and the standards the school is bound to.

They draw from multiple perspectives and multiple sources so a more complete picture of reality can emerge.

They enable school leaders to concentrate organizational attention on things that matter most.

They promote systemic change. We are all a part of the solution

LET’S LOOK AT SOME REAL DATA AND THEIR IMPLICATIONS

What follows are several pages of data on a particular school. This is a real school, and the data are as real as the school is. The only thing that is faked is the name of the school.

Central City Elementary Fifth Grade ReadingPercentage Making Proficiency or Higher

GROUP School District State

All 65.4 69.0 58.0

White 70.2 73.2 66.5

Black 56.7 56.8 28.3

Male 56.6 65.3 54.8

Female 74.6 72.3 61.6

FRL 60.5 53.1 35.6

Disabled 23.8 * 14.0 18.6

* More than 10, fewer than 40

Central City Elementary Fifth Grade MathematicsPercentage Making Proficiency or Higher

GROUP School District State

All 63.0 61.3 56.3

White 67.2 64.6 64.5

Black 55.5 51.5 25.1

Male 57.1 65.8 57.0

Female 68.6 57.7 55.7

ED 54.9 53.1 35.2

Disabled 15.0 * 9.5 21.5

* More than 10, fewer than 40

Anywhere Elementary (Reading and Mathematics)Percent Proficient by Grade Level

GRADE 99-00 00-01 01-02 02-03

READING

THREE 70.0 70.7 77.9 80.6

FOUR 63.7 67.9 70.7 71.9

FIVE 73.3 70.7 83.9 82.0

MATHEMATICS

THREE 66.7 62.8 69.8 79.6

FOUR 85.7 71.7 83.0 93.3

FIVE 82.6 85.9 86.7 86.5


GRADE 99-00 00-01 01-02 02-03

READING

THREE 70.0 70.7 77.9 80.6

FOUR 63.7 67.9 70.7 71.9

FIVE 73.3 70.7 83.9 82.0

MATHEMATICS

THREE 66.7 62.8 69.8 79.6

FOUR 85.7 71.7 83.0 93.3

FIVE 82.6 85.9 86.7 86.5


GRADE 99-00 00-01 01-02 02-03

READING

THREE 70.0 70.7 77.9 80.6

FOUR 63.7 67.9 70.7 71.9

FIVE 73.3 70.7 83.9 82.0

MATHEMATICS

THREE 66.7 62.8 69.8 79.6

FOUR 85.7 71.7 83.0 93.3

FIVE 82.6 85.9 86.7 86.5


GRADE 99-00 00-01 01-02 02-03

READING

THREE 70.0 70.7 77.9 80.6

FOUR 63.7 67.9 70.7 71.9

FIVE 73.3 70.7 83.9 82.0

MATHEMATICS

THREE 66.7 62.8 69.8 79.6

FOUR 85.7 71.7 83.0 93.3

FIVE 82.6 85.9 86.7 86.5


GRADE 99-00 00-01 01-02 02-03

READING

THREE 70.0 70.7 77.9 80.6

FOUR 63.7 67.9 70.7 71.9

FIVE 73.3 70.7 83.9 82.0

MATHEMATICS

THREE 66.7 62.8 69.8 79.6

FOUR 85.7 71.7 83.0 93.3

FIVE 82.6 85.9 86.7 86.5

Looking for other Data Sources to help the School Improvement Processes

• School Data for Students

• Home Data for Students

• Teacher Preparation Data

• Teacher Performance Data

• District Goals and Priorities

• Community Goals and Needs for School

Pulling the D3M Process Together

Step one – Check Your Alignment

Step two – Formulate questions that needs answers

Step three – Gather the data

Step four – Answer your questions

Step five – Communicate your findings

Step six – Plan your school improvement plan


Step one – Check your AlignmentSuccess in any form of accountability depends in large measure on how well the taught curriculum is aligned with the tested curriculum. In Pennsylvania, it is a safe presumption that the PSSA tests the state’s curriculum standards. If that is true, then it makes sense to get a clear and consistent alignment between these standards and what teachers actually teach and what kids are expected to learn.

Your job as the school’s instructional leader is to ensure that such an alignment is in place. In high stakes accountability systems (code for NCLB), you need to (1) make sure teachers know the standards and can articulate their alignment to the standards, (2) monitor both lesson plans and lesson execution to extend the power of your instructional supervision, and, (3) give focused and constructive feedback on continuous improvement, using the teacher’s classroom assessment data to frame the focus for these feedback conversations.

This process not only applies to the tested grades, it applies to those grade levels whose curriculum mastery contributes to success in the tested grade. Make sure the alignments in grades K-4 are as consistent as they are in grade 5, for example.

Pulling the D3M Process TogetherStep two – Formulate questions that needs answers

Examples of Other Data Sources to Help Generate Questions

•School Improvement Plan (e.g.)

•You said you were going to improve math and reading instruction so student learning would improve in those areas. Did you?

•The SIP specifically indicated that the school would integrate thinking maps into the curriculum to improve student thinking. Were they? Did student thinking improve?

•School History(e.g.)

•The school has struggled for a long time with transition difficulties most 9 th graders experienced. The school established a special 9th grade academy to help overcome this problem two years ago. What have been the effects of this academy? Has there been a positive effect on 11th grade achievement scores.

•Recent events (e.g.)

•This past year, the percentage of Hispanic students doubled from the previous year, making them a bona fide sub-group for NCLB consideration. As a defined ethnic group, how did they do in achievement? Attendance? Disciplinary referrals?


Step three – Gather the data

Your primary purpose is to answer your questions. You might find other interesting stuff along the way, but you need to stay focused.

Remember the power of multiple measures to get multiple perspectives.

Gather the data readily at your disposal

Student Data Teacher Data Community Data Leadership Data

Secure the data you don’t have but need to answer your questions (e.g.)

Survey students in 9th grade academy about their experiences

Do parent focus groups to determine their impressions of school safety

Do surveys with teachers on their comfort and use of thinking maps in their curriculum


Step four – Answer your questions

This may be the most straightforward of the steps, but it is sneaky. You will have enough data to answer 200 or more questions, but you can’t work on two hundred separate school improvement initiatives. Stay focused on the questions most likely to produce and let others take your data and mine it for other insights and possibilities. This is where cooperation with a university can be helpful.

If you still have unanswered questions, chances are you don’t have the right assortment of information to answer them. What do you need to know that you don’t have? Can you get it? At what cost? What will happen if this question is carried over until you have the data you need to answer it.

Pulling the D3M Process TogetherStep four – Answer your questions

For example, you wanted to improve instruction in reading and math to promote student learning. What did you do to improve instruction? Did you provide professional development? Was it used? How do you know? How well was it used? What evidence do you have to support this conclusion? Did you assign different teachers to teach in these areas? Why them? Do you have any evidence that this strategic move had the desired effect? Did students improve their learning in these two areas? What are your indicators to support this conclusion (student grades, scores on testlets, improved attendance, fewer referrals, better attention to homework, quality of student work handed in, standardized test scores, etc.)?

You also wanted to integrate thinking maps into the curriculum to improve student thinking. What did you do to make this happen? Was their thinking maps staff development? Who received the training? Did they integrate thinking maps into their curriculum? How do you know? How effectively did they do this integration? What evidence supports your conclusion? Did students use them? What evidence supports that? Did the quality of student thinking improve? How do you know?


Step five – Communicate your findingsInternally

These findings and conclusions need to form the basis for a series of high quality professional conversations with the faculty and with some particular groupings within the faculty. The principal and the entire School Improvement Team should have specific responsibilities to deliver a series of data-based reports that will lead to teacher-led work groups to address one or more of the concerns raised from the data analysis. These presentations must be carefully crafted, since not all members of the faculty will be equally conversant with the language and concepts of data driven decision making.

You also have another “internal” audience. Your superintendent and your immediate supervisor (if different) need to know what you now now about your school and its school improvement efforts. They also need to know what your likely plans for improvement are. You have an opportunity to acquire administrative support and understanding by being proactive with your knowledge. In fact, the only other significant way to get additional help is to fail, and then, the mountain is much higher and harder to climb.

Pulling the D3M Process TogetherStep five – Communicate your findings

Externally

Each school has community and business leaders who are key stakeholders in this business of school improvement. Many are also probably pretty conversant in the importance of data, and will be good customers of the messages of improvement you can deliver. Often, they rely on information sources outside the school’s control to learn about their schools. You can change that by taking what you know to Chamber meetings, Service Club gatherings, education reporters, and well-established social clubs who have educational agendas. If you fail to inform them, they’ll find information elsewhere, and you can then only react, and that’s a position of weakness, not strength. Remember to craft this message as jargon-free as possible. You don’t want to lose them.

Then, there are the parents. They too need to know how their child’s school is doing, but you have to craft this message differently. No jargon, please. The range of language sophistication is likely to be pretty wide, and your objective is to get them to understand complexity in as simple a way as you can communicate. Make information dissemination as short as possible and rely on questions to articulate additional information.

Pulling the D3M Process TogetherStep six – Plan your school improvement plan

Now that you and your School Improvement Team and your faculty have done all this work, and you’ve communicated the results to those who need to know, the important part now kicks in. The decision making part of D3M is now front and center, and the data have helped you identify the areas of greatest accomplishment (Hooray!!) as well as the areas of greatest need. What follows are some guidance for how to manage the decisioning process. First, keep in mind that you probably got to this point by working with key staff, and they need to have a voice in the decisions. You are still the principal, but their insights are important too. Second, recognize that you and the school can’t do it all, so you need to concentrate on those areas most likely to provide the best growth. Focus is important! Keep the main things the main things. For example:

Pulling the D3M Process Together1. Define your areas of need identified by your data analysis. You will probably have more things than you can possibly accomplish in a single year or even two, but they all have to be made visible so everyone can see them.

2. Because you and your folks can’t do all these things and do them well, you have to set priorities. Rank ordering the possibilities is very helpful, so long as everyone understands what the basis of the rank ordering is. If you tell them the critical variable is expense, they will rank on affordability. If you tell them the ranking is to be based on their perception of which will deliver the most student benefit, they will rank on that variable. Assigning numerical values to people’s rankings will yield a prioritized list. However, it is important to get your folks to affirm that the priorities actually represent their collective intentions. If they agree, then pick the top 1-2 for that year. This doesn’t mean that people don’t pay attention to those other things; it only means that the focus for improvement will be on those 1-2 things, because the potential for greatest improvement in student benefit lies there.

Pulling the D3M Process Together3. Now that you have your 1-2 priorities, you need to develop the real planning. What will you and the school actually do to accomplish these priorities? What needs to take place (e.g. training, materials purchase, special approvals, special consultant, etc.) in order for any of these strategies to happen successfully? What timelines are reasonable for each of the strategies to be in place? What leadership assignments have to change to make sure this strategy is on track?

4. Once you know the 1-2 things you’re going to focus on, and what you’re going to do to make these priorities become real, you have to allocate the necessary resources to each priority to ensure its success. This an often neglected step in the school improvement planning process, yet can singly undermine any chance for success. Is new training required? How much will it cost? When will it be done? Where will the money come from? Is that assured? How long will it take to do the training? Are there materials costs? How much? Where will the money come from? Is that assured? Who will be assigned to get the training and implement the strategy? Are they the best persons to do this? Do they work together well? Will this strategy require any additional use of space? What kind? Is it available? Are there space modifications to be made? How much? Where will the money come from? Is that assured?

Pulling the D3M Process Together5. OK! Now, you and your folks need to define, for each of these priorities, what “Acceptable Progress” or “Good Progress” looks like. It is not enough to develop a strategy, create a timeline, and assign resources. Everyone involved has to know what the expectations are. How can anyone know whether things are going as they should be going? What kinds of things should people be able to determine so that they can feel they’re on the right track? Perhaps a rubric would help in an instructional area, or a benchmark from another school doing this successfully in an area of professional development would be useful. People need to know that they are doing the things they need to be doing and that what they’re doing is producing the results they are supposed to produce. Otherwise, people will expend a lot of energy to end up frustrated, and that’s a poor motivation for sustained change and improvement.

6. Finally, someone on the staff has to be accountable for each priority. Yes, the principal is accountable for all of them, but someone else should have both the assignment and the authority to make sure progress is going in the right direction. Someone has to “worry” about a priority’s success, and has to have a sense of accountability for that success. It could be an assistant principal, a lead teacher, a department head, a media specialist, a guidance counselor, etc. But, someone has to be in charge of making sure the good things that are supposed to happen actually happen.

So, why is the use of data so important in our work and in our lives?

• They replace guesses with facts, and allows decisions to be based on those facts.

• It allows us to get past symptoms and try to get at causes.

• They focus on reality, not hyperbole. Do the data support the rhetoric? Data help us answer the “How do I know?” question.

• They help determine if previous decisions are accomplishing what they’re supposed to be accomplishing.

• They allow you to tell your story without sounding like it’s special pleading.

Documents

DATA DRIVEN DECISION MAKING A Leadership Seminar for Pennsylvania Urban Academy June 23-34, 2004