Principles and Strategies of KEYS 2.0 Data Analysis and Interpretation

GAE TrainingJanuary 1009

Jacques Nacson

Senior Policy Analyst

NEA New Products and Programs

jnacson@nea.org

Objectives

To examine how KEYS fits within the context of continuous school improvement

To review key statistical terms To review basic research concepts To consider basic principles of data

analysis To examine and learn to interpret the

KEYS 2.0 school and district reports

Statistical Terms

Mean, Median, Mode 90th Percentile Standard deviation Factor analysis Regression analysis Correlation analysis Statistical significance (.05

level, .01 level)

Data Collection: “How much information do we need?”

at the MOST -- Is the information collected compelling enough to convince any skeptic?

at the LEAST -- Will the information collected at least create cognitive dissonance for the resistors?

Triangulation of Data

Compensates for imperfections of data-gathering instruments.

When multiple measures yield same results, it increases confidence in results.

When multiple measures fail to yield same results, it raises important follow-up questions

Data Collection: “What must we keep in mind?”

1. Do the instruments and methods we plan to use measure what we claim they do?

(Validity)

2. Do the instruments and methods we plan to use accurately measure the phenomena we are studying?

(Reliability)

Data Gathering Techniques

Interviews

Checklists

Diaries

Logs

Questionnaires

Audio or video tapes

Photographs

Consultative advice

General Principles of Data Interpretation Before Looking at Your KEYS 2.0 Results

Recognizing the limits of the bestsurvey instruments

Distinguishing between symptomsand root causes

Are the data surprising orconfirming?

Are you listening to all relevantvoices in your school community?

Moving from data to understandingto knowledge

Without theory, no organizationallearning and improvement canoccur

General Principles to Consider Specifically Related to Your KEYS 2.0 Data

Applying systems thinking: thewhole is greater than the sum of itsparts

Looking for patterns within andacross Keys

Slow and steady: the road tocontinuous improvement has noend (PDSA)

Celebrating your victories andlearning from your strengths

KEYS 2.0 – A diagnostic toolleading to a deliberative process,not a rigid, mechanistic, quick-fixprescription

Horizontal Axis Measure of quality: 5 point scale Left side Disagree (low value) Right side Agree (high value)

Vertical Axis Indicators (groups of questions that measure the same concept)

KEY 1. Shared Understanding and Commitment To High GoalsRespondents Provide Direct Instruction to Students

Understanding the Graph for each KEY

School Average All Schools Average

90th Percentile ScoreData Points

School average (Black) All schools average (Red) 90th percentile score (Yellow) Length of the horizontal bar (Blue/Purple) 1 standard deviation above and one below the school average (measure of agreement or consensus)

Standard Deviation

Understanding the Graphs for each KEY

The Goals for your school in terms of continuous improvement for each indicator The school average moving continuously toward the right side (agree – high value of quality for that indicator) At the same time, reduce the standard deviation (narrow the length of the horizontal bar, meaning greater agreement among respondents)

Hierarchical organization – Keys—Indicators--items

Rating - the degree to which the respondents believe that the indicator accurately describes the school: average of all respondents on the questions that make up each indicator

1 = strong belief that indicator does not describe the school

2 = some belief that indicator does not describe the school

3 = neutral 4 = some belief that the indicator describes the

school 5 = strong belief that the indicator describes the

school

KEYS Vocabulary

KEYS Vocabulary (Con’d)

Consensus - level of agreement among respondents on the rating for each indicator Low consensus = wide band High consensus = narrow band

Demo School Report with Links to Demo School Report with Links to ResourcesResources

Demo School Report with Links to ResourcesDemo School Report with Links to Resources

Demo School Report with Links to Resources Demo School Report with Links to Resources (Drilling Down)(Drilling Down)

Demo School Report with Links to Demo School Report with Links to Resources (Drilling Down)Resources (Drilling Down)

Demo School Report with Links to Demo School Report with Links to Resources (Drilling Down further)Resources (Drilling Down further)

Example of a District Report (Aggregate Example of a District Report (Aggregate Scores for Key 6)Scores for Key 6)

Example of District Report Distribution of School Example of District Report Distribution of School Scores for Indicator 6.5 (InterventionsScores for Indicator 6.5 (Interventions)

Example of a District Report Aggregate Example of a District Report Aggregate Scores for Key 5 (Resources)Scores for Key 5 (Resources)

Example of District Report: Distribution of School Scores for Indicator 5.4 Example of District Report: Distribution of School Scores for Indicator 5.4 (Safe & Healthy Learning Environment)(Safe & Healthy Learning Environment)

PROCESSING THE DATALooking at your groups’ assigned Key Graph

answer these questions:

• On which indicators is there the greatest agreement (the shorter bar)?

• On which indicators is there the least agreement (the longer bar)?

• Why might that variability of perspective exist?

PROCESSING THE DATA – Cont’d

•How might you come to greater consensus on this?•Which of the KEYS indicators have both a high mean score and also have a shorter bar? (Strengths)•Which of the indicators have both a low mean score and a wide degree of variability? (Areas for Improvement)•Which items have the highest correlation with student achievement?

Analyzing and Interpreting KEYS 2.0 Data - The Logic Sequence

1. Are results surprising or do they confirm what you know or believe?

2. If the results are surprising, what might have caused the difference?

3. How might you go about checking further to determine the validity of the findings?

4. If you determine that the findings are valid and the issue is important and relevant to your particular situation, what might be the reasons for such findings?

5. What is the most likely cause? The underlying or root cause? How did you arrive at this conclusion?

Analyzing and Interpreting KEYS 2.0 Data -The Logic Sequence – Cont’d6. What are some possible actions/solutions that you and your colleagues might take to alter or reverse and improve the condition?

7. What is the best possible solution given your particular context or situation?

8. What are the steps that you must plan and take in order to implement the best possible solution?

9. What are the resources and the skills needed to implement the solution successfully?

10. How would you know if your actions/solutions were successful in ameliorating the condition?

Focus group Questions:

• Do the KEYS data support or refute other data previously collected?

• What professional development opportunities are indicated by the data?

• What barriers to achieving previously stated goals surfaced during your discussion?

• What other learning does your focus group wish to share with the larger group?

Steps a School Might Take Once KEYS Preliminary Analyses are Completed

1. GAPS: Decide on one indicator or a group of indicators where gap (s) exist.

2. RELEVANCE: Reflect with the “team” on the relevance, importance and priority of the selection. (Get feedback from stakeholders)

3. DATA COLLECTION-VALIDATION: Consider the need to collect additional data to validate the KEYS findings.

4. DATA COLLECTION-DIAGNOSIS AND REFINEMENT: Examine what is the “root” cause of the problem.

Steps a School Might Take Once KEYS Preliminary Analyses are Completed

5. THEORY OF ACTION: Identify and select the most appropriate solution for your context. (Get Feedback from stakeholders)

6. ACTION PLANNING: Set SMART goals and develop specific action/project plans. (Get feedback and commitments from stakeholders)

7. IMPLEMENTATION: Action plans must be implemented for improvement to occur.

8. DATA COLLECTION-EVALUATION: Both process and product evaluations are necessary for learning to happen.

9. BACK TO STEP 1: Repeat the cycle