Principal Evaluation: The National Story

Joseph MurphyVanderbilt University

CCSSOApril 2011

I. SETTING THE STAGE: EVALUATION IN CONTEXT

Career LineCareer Line

Standards

evaluation

governance

incentives

professional development

Conditions of Work

Standards

licensure

State &

district

policy

District & state policy

Education

District & state policy State & district

policy

induction (residency)

professional development

relicensure

preparation internship

accreditation/program approval

teacher leadership preparation

mentoring

II. PRINCIPAL EVALUATION: THE NATIONAL STORY

PROBLEMS WITHCURRENT

EVALUATION SYSTEMS

Content

• Little evidence that systems evaluate what is important, i.e. not valid

• Insufficient attention to leadership for learning, especially curriculum and instruction

Process

• Limited architecture – focus on one approach (e.g. goals)

• Perfunctory – not a deep process

Impact

• Principals not receiving useful feedback• Not promoting professional growth of

principals• Not promoting organizational improvement

III. BUILDING A NEW PRINCIPAL

EVALUATION SYSTEM

A. Building: Developing Guiding Principles of the Evaluation System

• Foundation– Based on ISLLC Standards

• Process– Be valid, reliable, and equitable– Be transparent– Have objectivity, evidenced through multiple

measures and supported by collection, analysis and application of appropriate data and evidence

– Underscore both formative and summative components

– Be comprehensive yet manageable– Allow for flexibility at local level– Include multiple indicators of academic, social,

and emotional growth for all students– Promote collaboration between supervisor and

evaluatee– Define and prioritize expectations and areas of

emphasis– Provide frequent and specific feedback– Insure due process

• Outcomes– Promote personal accountability– Promote continuous professional growth– Promote school improvement that results in

student achievement– Provide a process for the district to connect

principals to resources and growth opportunities

B. Building a Principal Evaluation System:

An Overview

• Step 1: Anchor the System– Create the system around two elements:• research on principal impact on student learning

– recommendation: ISLLC Standards

• organizational outcomes– recommendation: measures of student learning

• Step 2: Select Components– Choose components that will tap into or shed light

on the two elements noted above– Recommendation: more than one component, no

more than five components– Recommendation: include one component on

student learning and one on 360 degree feedback on performance

• Step 3: Value the Components– Provide weights to the components– Example #1: • organizational goals 15%• student achievement 40%• 360º assessment (VAL ED) 30%• professional developmental goal 15%• total 100%

– Example #2:• 360º assessment (VAL ED) 50%• student achievement 50%• total 100%

• Step 4: Set Performance Levels– Create performance levels for each component– Recommendation: use four levels• Below basic (inadequate)• Basic (satisfactory)• Proficient• Distinguished

• Step 5: Set Parameters– Develop borders for each performance level– Example—student achievement• Below basic ↓.9 years growth• Basic .9 - 1.1 years growth• Proficient 1.2 - 1.3 years growth• Distinguished ↑.1.3 years growth

– Example—360º (VAL ED)• Below basic 1:00 - 3:28 • Basic 3:29 - 3:59• Proficient 3:60 - 3:99• Distinguished 4:00 - 5:00

• Step 6: Assign Scores– Provide an assessment score for each component

and a total score– Example—components and raw total score

• organizational goals (15%) basic (2) 30/60• student achievement (40%) basic (2) 80/160• 360º assessment (VAL ED) (30%) proficient (3) 90/120• professional dev. goal (15%) proficient (3) 45/60

– 245/400 (÷ 4 = 61/100)– example—total score:

» below basic 25-50» basic 51-65» proficient 66-84» distinguished 85-100

• Step 7: Determine Consequences– Ensure that results are used for:• shaping professional development• retention and promotion• contract length• salary/bonuses

– Example—used for contracts:» Distinguished: 3 year contract» Proficient: 2 year contract» Basic: 1 year contract» Below basic: 1 year contract and improvement

program

• Example—used for bonus payments– Rules in example: • Pool of money = $15,000 per principal (20 principals x

15,000 = $300,000 pool); i.e., maximum bonus = $15,000• Set bonus guideline so no bonus money for less than

“proficient” work• Bonus allocated by component• Three component system in use

Component 360 Degree Assessment (VAL ED)

Student Learning

Organizational Goals

Weight 40% 40% 20%

1 Below basic 0 0 0

2 Basic 0 0 0

3 Proficient 4000 4000 2000

4 Distinguished 6000 6000 3000

Principal Williams

Bonus: 8000

Component 360 Degree Assessment (VAL ED)

Student Learning

Organizational Goals

Weight 40% 40% 20%

1 Below basic 0 0 0

2 Basic 0 0 0

3 Proficient 4000 4000 2000

4 Distinguished 6000 6000 3000

Principal Smith

Bonus: 2000

Components States

C. Principal Evaluation Work: Detailed Explanation

PERFORMANCE GOAL

• Source of Goal– National, state, and local assessment– Improvement plans– Standards– Principal self reflection– District goals– School audits– Surveys

• Essential Characteristics of Goals– Be linked to ISLLC Standards– Be organizationally grounded and emphasize the

direct contributions of the leader – organizationally grounded goal(s) may include district, superintendent, and/or community-based goals and priorities

– Be anchored in an analysis of multiple sources around relevant data – sources of data may reflect cognitive or non-cognitive measures

−Be specific, measurable, attainable, realistic, timely, and challenging

−Have a longitudinal focus – evaluation goals should address the hard work that will be required over time to promote change and improvement

−Be mutually determined through collaborative dialogue

−Be collaboratively reviewed with frequent and specific feedback

STUDENT ACHIEVEMENT• Source of Goal

– Norm-referenced (state/district)– Proficiency– End of course– SAT/ACT– AP– Locally developed

• Focus of Goal– Level– Equity– Value-added

306 DEGREE ASSESSMENT

• Anchored on ISLLC standards• Valid and reliable• Helpful for individual and school

PROFESSIONAL GROWTH GOAL

• Source of goal– Previous evaluations– School improvement plans– Demographic information– District goals

• Essential characteristics– Directly linked to Standards and indicators for School

Improvement and ISLLC Standards– The principal should engage in personal and collegial

reflective practice that promotes continuous professional growth

– The principal and supervisor will determine how the acquisition of knowledge and skills will be demonstrated

– The principal’s professional growth goal should be differentiated and based on the need determined by collaborative discussion around formative and/or summative evidence

– Supervisors shall provide specific and timely feedback, mentoring, and coaching through regularly scheduled collaborative meetings and ongoing communication

– The supervisor should provide resources that will aid in this process

– It is the responsibility of the principal to collect and maintain data that evidences the impact of personal growth and school improvement

PARENT SATISFACTION

Drill down on ISLLC Standard

IV. VAL-ED: Assessing Learning-Centered Leadership

Acknowledgments

• The VAL-ED instrument was published and distributed by Discovery Education Assessment starting in July 2008.

• We are authors of the VAL-Ed, and while we have made every effort to be objective and data-based in my statements about this instrument in this presentation, readers should judge the facts and related information materials for themselves and make independent decisions regarding the use of the instrument.

2008

Outline of Presentation

• Background on VAL-ED

• The VAL-ED instrument

• Psychometric properties

• Score reports

• VAL-ED and professional development

2008

Background on VAL-ED

The Case for Leadership Assessment

• Most school leadership improvement focuses on professional development, mentoring, licensing policies, and standards.

• Minimal attention has been paid to assessment, feedback, and subsequent action.

• Leadership assessment and feedback is an important missing link to improving and strengthening school leadership.

2008

Learning-Centered Leadership

• Leaders should be assessed on leadership behaviors associated with student learning.

• Learning-centered leadership is leadership for student performance.

• Learning-centered leadership is the framework for our leadership assessment system.

2008

Our Conceptual Model

2008

Leadership Behavior Framework

2008

Definitions of Core Components• High Standards for Student Learning —There are individual, team,

and school goals for rigorous student academic and social learning.

• Rigorous Curriculum (content) —There is ambitious academic content provided to all students in core academic subjects.

• Quality Instruction (pedagogy) —There are effective instructional practices that maximize student academic and social learning.

• Culture of Learning & Professional Behavior —There are integrated communities of professional practice in the service of student academic and social learning. There is a healthy school environment in which student learning is the central focus.

• Connections to External Communities —There are linkages to family and/or other people and institutions in the community that advance academic and social learning.

• Performance Accountability — Leadership holds itself and others responsible for realizing high standards of performance for student academic and social learning. There is individual and collective responsibility among the professional staff and students.

2008

Definitions of Key Processes• Planning—Articulate shared direction and coherent policies, practices, and

procedures for realizing high standards of student performance.

• Implementing—Engage people, ideas, and resources to put into practice the activities necessary to realize high standards for student performance.

• Supporting—Create enabling conditions; secure and use the financial, political, technological, and human resources necessary to promote academic and social learning.

• Advocating—Promotes the diverse needs of students within and beyond the school.

• Communicating—Develop, utilize, and maintain systems of exchange among members of the school and with its external communities.

• Monitoring—Systematically collect and analyze data to make judgments that guide decisions and actions for continuous improvement.

2008

The Development of VAL-ED

• The development of VAL-ED has been supported by a 3-year grant from The Wallace Foundation.

• Three phases of our work:– Phase 1 – Leadership conceptualization and assessment system

development– Phase 2 – Field testing the behavior rating scale and exploring its

properties– Phase 3 – Dissemination of results and products

2008

The VAL-ED Instrument

The VAL-ED Instrument• The instrument consists of 72 items defining six core component

subscales and six key process subscales. • Principal, Teachers, & Supervisor provide a 360-degree, evidenced-

based assessment of leadership behaviors. • Respondents rate effectiveness of 72 behaviors on scale

1=Ineffective to 5=Outstandingly effective.• Each respondent rates the principal’s effectiveness after indicating

the sources of evidence on which the effectiveness is rated.• Two parallel forms of the assessment facilitate measuring growth

over time.• The instrument will be available in both paper and online versions.

2008

Purpose & Uses • The VAL-ED can be used as part of a comprehensive

assessment of the effectiveness of a leader's behaviors.

• The VAL-ED reports principal performance through– Norm-referenced scores and– Criterion-reference scores.

• VAL-ED can be used annually or more frequently to:– Facilitate a data-based performance evaluation,– Measure performance growth, and– Guide professional development.

2008

Implementation • Identify respondents and invite participation.• Discuss use of results & confidentiality. • Decide paper or online version.• Time and Timing– Average respondent requires 20 to 25 minutes.– Schedule completion after respondents have had a

reasonable time to observe/experience the leader’s work and its effects on the school.

• Designate person(s) to manage collection and submission of response forms, if paper version used.

• Ensure teacher confidentiality.

2008

Directions for Completing Rating Scale Directions: The Vanderbilt Assessment of Leadership in Education (VAL-ED) measures the

effectiveness of a principal’s key leadership behaviors that influence teacher performance and student learning. You will be asked to make effectiveness ratings for each of 72 leadership behaviors based on evidence from the current school year.

1. Read each item describing a leadership behavior. In some cases, the principal may not have actually performed the behavior, but he or she has ensured that it was done by others in the school. Either way the behavior should be rated.

2. Check ( ) the key Sources of Evidence you use for the basis of your assessment. Note, at least one source of evidence must be checked for an item before you make an Effectiveness rating. If you check No Evidence, then Ineffective or Don't Know must be marked in the Effectiveness column.

3. If you check any sources of evidence other than No Evidence, always make an effectiveness rating even if you must estimate the effectiveness of the behavior. The number of Sources of Evidence checked is not necessarily indicative of the effectiveness rating.

4. Circle the 1 to 5 Effectiveness Rating to indicate how effectively the behavior

was performed.

Outstandingly effective means the principal (or the principal's designee) has carried out a particular behavior (e.g., providing necessary support) with a very strong, positive effect on the targeted area of school activity (e.g., rigorous curriculum). Ineffective means the principal (or the principal's designee) has either not done the particular behavior (e.g., not provided necessary support) or has carried out the behavior with very low quality that does not have a positive effect on the targeted area of school activity (e.g., rigorous curriculum).

2008

An Example Set of Responses

2008

Psychometric Properties

Assessing Learning-Centered Leadership:The VAL-ED vision…

A leadership assessment system that has the following properties:

• Works well in a variety of settings and circumstances,• Is construct valid,• Is reliable,• If feasible for widespread use,• Provides accurate and useful reporting of results.• Is unbiased,• Yields a diagnostic profile for summative and formative purposes.• Can be used to measure progress over time in the development of leadership, and• Predicts important outcomes.

2008

Psychometric Evidence• Item and response scale development

– Based on review of learning-centered leadership literature and alignment to ISLLC standards.

– Critiqued by education leaders and leadership researchers.• Item sorting study

– Established content validity by asking education leaders to sort the items into 36 cells.

• Cognitive interviews of paper/pencil version – Two rounds of cognitive interviews in three districts each.– Three respondents evaluated the format and items.

• Nine-school pilot test – Estimated reliability of each of 12 scales.– Established construct validity through factor analysis.– Established face validity through questions to respondents.

2008

Psychometric Evidence• Cognitive interviews of online instrument including revisions based on 9-

school pilot• Bias review

– Submitted to urban districts to evaluate language.• 11-school pilot test

– Confirmed changes made after 9-school pilot test.• 300-school field test

– Conducting differential item functioning to determine biases.– Establishing norms.– Setting performance standards.

• Proficiency standard setting– Using bookmark approach with 24 education leaders.

2008

Cognitive Interviews—Validity • First round

– Sources of evidence cumbersome– Item stem lost– Instructions “wordy”

• Modifications– Instructions bulleted– Stem included in each item– “Not done” added

• Conclusions– VAL-ED captured all the relevant leadership behaviors– Response scale was clear– Respondents able to complete task on their own

• Cognitive interviews of online prototype

2008

• Methods• Confirmatory factor analysis

– High Parsimonious Goodness of Fit Index (.93-.96)– First, second, and third-order factor loadings salient

• Responses to final survey questions– Understood items, sources of evidence

• Teacher and principal effectiveness ratings werecorrelated r =.47

• Cronbach’s Alpha above.92 for 108-item-form scales

• Scores high (generally >4 on the five-point effectivenessscale)

9-School Pilot Study—Validity & Reliability

Scatter of Principal ratings with Teacher Ratings

2

2.5

3

3.5

4

4.5

5

3.5 3.7 3.9 4.1 4.3 4.5 4.7 4.9

Teacher mean ratings by school r=.47

Pri

nci

pal

rat

ing

s b

y sc

ho

ol

2008

11-School Pilot Study—Validity & Reliability• Methods• High agreement between schools– r = .79 for teacher-principal,.51 for principal-

supervisor, .68 forsupervisor-teacher

• Mean effectivenessratings lower, more variable

• Alpha above .89 forall 72-item-form scales

2008

Respondent Feedback

2008

Score Reports

Interpretation of Rating Scale Results• Descriptive Analysis

– Total Score– Core Components Subscale Scores– Key Process Subscale Scores

• Norm-Referenced Profiles– Principal– Teacher– Supervisor– Total respondent composite

• Criterion-Referenced Profiles– Distinguished– Proficient– Basic– Below basic

2008

VAL-ED Results: Performance Descriptors• Distinguished– Leadership behaviors of core components and key processes at levels of effectiveness

that over time are virtually certain to influence teachers to bring the school to a point that results in strong value-added to student achievement and social learning for all students.

• Proficient– Leadership behaviors of core components and key processes of sufficient effectiveness

that over time are likely to influence teachers to bring the school to a point that results in acceptable value-added to student achievement and social learning for all students.

• Basic– Leadership behaviors of core components and key processes of sufficient effectiveness

that over time are likely to influence teachers to bring the school to a point that results in acceptable value-added to student achievement and social learning for some subgroups of students but not all.

• Below basic– Leadership behaviors of core components and key processes of insufficient effectiveness

and consistency that over time are unlikely to influence teachers to bring the school to a point that results in acceptable value added to student achievement and social learning.

2008

Aggregated Effectiveness Ratings

2008

Comparisons Across Respondent Groups

2008

Comparisons Across Respondent Groups

2008

VAL-ED and Professional Development

Cell-by-Cell Feedback

2008

VAL-ED and Professional Growth

• Cell-by-cell feedback highlights up to 6 potential areas of growth.

• Behaviors from these 6 domains are listed.• Areas of growth provide principals with

information about key targets for professional development.

2008

Leadership Behaviors for Possible Improvement

Example of a potential area of growth:

2008

Supporting Research & Publications• Goldring, E., Porter, A.C., Murphy, J., Elliott, S.N., & Cravens, X. (2007, March).

Assessing learning-centered leadership: Connections to research, professional standards, and current practice. New York, N.Y.: Wallace Foundation.

• Murphy, J., Elliott, S.N., Goldring, E., & Porter, A.C. (2007). Leadership for learning: A research-based model and taxonomy of behaviors. School Leadership & Management, 27 (2), 179-201.

• Murphy, J., Elliott, S.N., Goldring, E., & Porter, A.C. (in press). Leaders for productive schools. In M. Brundrett & M. Crawford (Eds.), Developing school leaders: An international perspective, London: Routledge.

• Murphy, J., Elliott, S.N., Goldring, E.B., & Porter, A.C. (2006). Learning-centered leadership: A conceptual foundation. New York, NY: Wallace Foundation.

• Porter, A.C., Goldring, E.B., Murphy, J., Elliott, S.N., & Cravens, X. (2006). A framework for the assessment of learning-centered leadership. New York, NY: Wallace Foundation.

• Murphy, J.F., Goldring, E.B., Cravens, X.C., Elliott, S.N., Porter, A.C. (2007, August). The Vanderbilt Assessment of Leadership in Education: Measuring Learning-Centered Leadership. Journal of East China Normal University.

These and other publications are all available for download at http://www.vanderbilt.edu/lsi/valed/featured.html.

2008

To Learn More …

Visit: http://www.val-ed.com

Contact : Discovery Education2416 21st Avenue, South, Suite 300 Nashville, TN 37212

2008

CONTACT INFORMATION

Joseph MurphyFrank W. Mayborn Chair

Box 404 GPCVanderbilt University230 Appleton Place

Nashville, TN 37203-5721615-322-8038

joseph.f.murphy@vanderbilt.edu