GangiTracydissertation10-20-09

1

SCHOOL CLIMATE AND FACULTY RELATIONSHIPS: CHOOSING AN

EFFECTIVE ASSESSMENT MEASURE

A dissertation submitted in partial

fulfillment of the requirements

for the degree of

DOCTOR OF PSYCHOLOGY

to the faculty of the department of

SCHOOL PSYCHOLOGY

at

St. Johns University

New York

by

Tracy A. Gangi

Date Submitted: _________________ Date Approved: ________________

_______________________________ ______________________________ (Students Signature) (Mentors Signature)

2

Acknowlegements

Id like to thank several people for their role in helping me get from Point A to the

final Point B on this road to attain the doctoral degree in psychology. First and foremost,

Id like to thank my father, Dr. William Gangi; without his positive influence, I would

not have had the desire to also become a Dr. Gangi, nor believe it could be done. My

mother, Patricia DeLeo Gangi, for her unflagging support and desire for me to find

happiness with my life choices. To my sister, Alessandra Gangi Long, and my brother

William P. Gangi, my deepest heartfelt thanks for being my biggest champions

throughout life, regardless of many unconventional decisions I have made along the way.

To my husband, David, whom came on board later in my quest, but my most loyal

fighter, I love you with my whole heart. To my entire extended family, my good

friends and close colleagues, thank you for your patience, your kindness, and your love

and friendship. To Dr. Mary Macedonio, your courage and perseverance have inspired

me from the first day I met you, I doubt I would have made it through this without your

help.

I would like to thank several respected researchers in the field of school climate

whom honored me with their feedback, input, and guidance, especially: Dr. Jonathan

Cohen, Dr. John Shindler, Dr. Megan Tschannen-Moran, Dr. N. Scott Norton, Dr. Wayne

Hoy, Joshua Freedman, Dr. Carina Fiedeldey-Van Dijk, Dr. C. John Tarter, Dr. Martha

Alberg, and Dr. Barry Fraser.

To the members of my committee, both former: Dr. Pauline Magee-Egan, my

original mentor, thank you for your enthusiasm for my original topic (which included the

3

Myers-Briggs Type Inventory), Dr. Jim Curley, Dr. John Hogan, and current: Dr.

Raymond DiGiuseppe, my mentor, and Dr. Marlene Sotelo-Dynega and Dr. Kate Walton,

thank you for agreeing to take on my project and bringing it to fruition. Finally, Id like

to thank past, present, and future researchers of school climate for their assistance in its

evolution, and furthering its benefit to educators and students everywhere.

4

Table of Contents

Chapter I: Introduction1

Chapter II: Literature Review..6

The Evolution of School Climate....6

Impact of Negative School Climate/Low Levels of Trust: Students, Teachers, and

the Community.8

Teachers: Emotional Aspects of School Climate and Student Impact9

Trust Research in Schools..11

School Administrators and School Climate Intervention..12

Systems Level Change and Continuing Professional Development..13

School Psychologists as Effective Consumers of Research and Needs

Assessment.14

Organizational Development, School Climate, and the Use of Team Building15

Team Building and Communication.16

Statement of Problem17

Purpose..17

Research Question18

Chapter III: Methods.19

Procedures.19

Step 1 (Criteria 1): Literature search21

Step 2 (Criteria 2): Faculty relationship factor.22

Step 3 (Criteria 3): Additional school climate variables..22

Step 4 (Criteria 4): Direct measures...23

5

Step 5 (Criteria 5): Viewable test items.23

Step 6 (Criteria 6): Current published instruments23

Step 7 (Criteria 7): Reliability25

Step 8 (Criteria 8): Validity...........26

Step 9 (Criteria 9): School Level..28

Step 10: Expert Panel Procedures.....28

Step 11: Comparative ranking system.31

Reliability..31

Validity.31

Sample size, recency of norms, and representative norms...32

Chapter IV: Results...33

Step 10: Expert Panel Procedures.45

Step 11: Comparative Ranking System.50

Review of Point System50

Reliability..50

Validity.51

Sample size, recency of norms, and representative norms....51

Critical Test Review of the Comprehensive School Climate Inventory (CSCI)...51

Description51

Development.54

Technical...54

Norming....54

Internal consistency reliability..54

6

Test-retest reliability.....55

Inter-rater reliability..55

Construct validity..55.

Convergent validity...55

Divergent/discriminant validity.55

Content validity.55

Criterion-related validity...56

Concurrent validity56

Predictive validity.56

Use of multiple informants56

Critical Test Review of the School Climate Inventory-Revised (SCI-R).56

Description56

Development..58

Technical...58

Norming.58


Test-retest reliability.60

Inter-rater reliability..60.

Construct validity..60

Convergent validity..61

Divergent/discriminant validity61

Content validity61

Criterion-related validity...61

7

Concurrent validity....61

Predictive validity..61

Use of multiple informants....61

Critical Test Review of the Western Alliance for the Study of School Climates

School Climate Assessment Instrument.61

Description.61

Development..64

Technical66

Norming.66


Test-retest reliability.68

Inter-rater reliability..68

Construct validity..68

Convergent validity...69

Divergent/discriminant validity.69

Content validity.69

Criterion-related validity69

Concurrent validity69

Predictive validity..69

Use of multiple informants70

Chapter V: Discussion...72

Expert Panels Confirmation of Criteria72

Comparative Ranking System Outcome73

8

Commentary and Summary of the Comprehensive School Climate Inventory

(CSCI)....74

Strengths74

Weaknesses75

Commentary and Summary of the School Climate Inventory-Revised (SCI-R)..76

Strengths76

Weaknesses77

Commentary and Summary of the Western Alliance for the Study of School

Climate- School Climate Assessment Inventory (WASSC- SCAI).77

Strengths...77

Weaknesses...78

Chapter VI: Implications for Practice...81

Suggestions for School Climate Improvement Efforts.83

Suggestions Regarding Follow-Up Research85

Limitations of This Study..85

A Final Note..86

References..87

Appendix A: Expert Panel Questionnaire105

Appendix B: Additional School Climate Measures109

Appendix C: Expert Panel Biographies..113

9

Chapter 1

Introduction

School climate has been studied for decades. It has been defined variously as the

atmosphere, ethos, tone, ideology, community, personality, or milieu of a school (Hoy,

2008), how one feels about the school and the people involved in the school (Davis &

Peck, 1992), or how one feels about their experiences in a school (Lindelow, Mazzarella,

Scott, Ellis, & Smith, 1989).

Only recently, however, has there been a consolidated, formally agreed upon,

definition of school climate. The definition refers to four areas of school functioning: 1)

physical safety, 2) relationships of those in the school environment, including faculty,

students, and parents, 3) teaching and learning methods, and 4) the actual physical

environment of the school. This definition of school climate was agreed upon during a

consensus-building meeting in April, 2007 by the National Center for Learning and

Citizenship, Education Commission of the States, and the National School Climate

Center at the Center for Social and Emotional Education (Cohen, McCabe, Michelli, &

Pickeral, 2009). The findings from these collaborative organizations determined that

there was an enormous discrepancy between current empirical research on positive

school climate, and what is actually currently being done in the schools, in state and

federal education departments, as well as in the colleges and universities in which

teachers are educated. Their overall consensus deemed that the gap between school

climate research, policy, practice, and teacher education is socially unjust and a violation

of childrens human rights (Cohen, McCabe, Michelli, & Pickeral, 2009).

10

School climate research has indicated that poor school climate is correlated with

absenteeism (Reid, 1983), suspension rates (Wu, Pink, Crain, & Moles, 1982), and higher

school drop-out rates (Anderson, 1982). Research has further confirmed that students

who do not graduate from high school face increased risk of unemployment, poverty,

poor health, and involvement in the criminal justice system (Barton, 2005). Conversely,

research indicates that schools that demonstrate a positive school climate had better

attendance, higher morale (Bryk & Driscoll, 1988) and more academic effectiveness

(Borger et al., 1985, Shindler, et al., 2009). These schools are also a place where self-

worth, pride, respect, and trust are fostered (Kaplan & Geoffroy, 1990; Kelley, Thornton,

& Daugherty, 2005). Cohen, Pickeral, and McCloskey (2009), reiterated the importance

of the learning climate and the various populations that it is comprised of. Specifically,

not only do children thrive academically in a positive school climate but, so too, do

teachers. For example, teachers tend to stay employed longer at schools with positive

climate, and teacher continuity and consistency benefit student academic achievement

(Ingersoll, 2001, Wynn, 2008). Although there has been an abundance of research done

in the area of student/peer relationships and physical safety, especially in regards to

violence and bullying, a growing amount of research has been done in the area of

teacher/colleague relationships and how the quality of their relationships impact students

success (Tschannen-Moran, Parish, and DiPaola, 2006).

Recent school climate research has indicated that teachers and other adults in the

school strongly influence student character, morals, and ethics. The articles urge that

these adults serve as models and moral compasses to students (Lehr & Christenson, 2002;

NASP Blueprint, 2006; Steegman, 2006; Sandberg, 2007; Pantaleno, 2007). They

11

concluded that the way in which adults in schools treat each other is an important

variable in predicting school climate outcomes. Further, they concluded that to build a

positive social-emotional climate within an entire school, it is necessary that all members

are involved (Desrochers, 2007; Ninan, 2006; Sandberg, 2007; Pantaleno, 2007 and

Stewart, 2007). Trust has also been reported to be an important factor in the quality of

teacher relationships and their impact on a positive school climate. Where trust was

higher among teachers, there was a higher trust in students and their parents

(Tschannen-Moran, 2009, p. 240).

As the primary mental health experts in schools, school psychologists are

uniquely suited to facilitate and implement positive school climate changes and invest in

the under-utilized areas of school mental health and social-emotional learning (NASP,

2006; Desrochers, 2007; Lehr & Christenson, 2002; Center for Social-Emotional

Education, 2009; and Steegman, 2007).

In 1987, the Northwest Regional Educational Laboratory (NWREL) released a

report identifying 42 school climate assessments known to exist (as cited in Shindler et

al., 2003). To date, this author has documented more than 100 school climate measures.

A review of the literature indicated that there is a dearth of rigorous, peer-reviewed

studies that instruct educators how to improve the overall school climate specifically by

strengthening faculty relationships. (In the literature, these relationships are also

called: morale/cohesion/affiliation/ collegiality/trust). Surprisingly, no single cohesive

review exists of the more than 100 school climate measures that have been found, nor is

there a review of school climate measures that might specifically help school

psychologists assess the quality of faculty relationships. Finally, there is no known list of

12

critically reviewed school climate measures that have been held to the standards for

educational and psychological tests (American Psychological Association, American

Educational Research Association, and the National Council on Measurement in

Education, 1999). These standards refer to, test construction and development,

specifically: validity, reliability, and norms.

Before embarking on a systems change/ intervention that could assist in

enhancing faculty relationships, as well as other school climate variables, an appropriate

school climate measure needs to be selected. Moving forward, without choosing the

most appropriate assessment tool that will measure critical school climate variables could

waste time, energy, and funds. This concern was also verbalized in Bulach and Malones

(1994) report on school climate and educational reform, [Educational reform] is far too

expensive in terms of money and expended human energy to continue implementation in

the haphazard manner that occurs in many school districts (p. 6). Complicating the

matter of choosing the appropriate measure of school climate are the various emphases

each instrument has and which constructs are desired to be assessed. There is a demand

to locate a school climate measure that not only assesses the academic variables that

impact student success but also assesses the (adult) social-emotional factors, illustrated

here: There is a need for assessment tools that permit schools to monitor, report on, and

improve practice in addressing a wider range of educational outcomes (responsibility,

caring, engagement, and problem solving), as well as linking them to academic

outcomes (CASEL, 2006, p. 5).

Therefore, the focus of this paper is a search for school climate measures that not

only effectively assesses faculty relationships in primary and secondary schools, but also

13

assesses particular school climate predictors that have found consensus in current

research. Presumably, this/these instrument(s) can then be utilized with best practices

currently being applied to school action plans addressing school reform, social-emotional

learning goals, citizen and character education initiatives (Cohen, 2007).

This study proceeded in three stages. First, school climate assessment

instruments were selected according to a set of criteria. These included measures that

could confidently be used as a pre- and post-test in a faculty team-building intervention

or perhaps as part of a larger school climate assessment/intervention. Second, these

selected instruments were ranked using a point system, and third, the chosen instruments

underwent a critical test review, adhering to the standards put forth by the American

Psychological Association, American Educational Research Association, and the

National Council on Measurement in Education (1999).

14

Chapter 2

Literature Review

The Evolution of School Climate

One thing has remained constant throughout the decades of school climate

research: the inability to delineate the school climate literature into specific constructs

that are agreed upon among researchers. Van Horn (2003) reported that due to

measurement difficulties, the selection of variables and controls, and ambiguity

concerning which statistical analyses to use, school climate has been a discouraging area

of research for some. He explained that there are large differences between the various

school climate theories. Anderson (1982) brought up this concern about school climate

researchthat the view of school climate varied significantly among researchers

depending on which theory of school climate was used by the researcher.

Some researchers conceptualized school climate from a school/organizational-

level, one in which all the children in the same school are affected by the same climate

(Tagiuri & Litwin, 1968; James, 1982; Van Horn, 2003). Others conceptualized school

climate from an individual-level, in which a compilation of individual differences would

be expected to influence the overall school climate (Miller & Fredericks, 1990; Dixon,

Johnson, & Toman, 1991; Raudenbush, Rowan & Kang, 1991). Still others viewed

school climate in terms of organizational health. For example, Miles (1969), believed a

healthy organization was one that copes well over a long period of time; it has a good

long-term prognosis for thriving in good times and enduring through bad times.

Organizational health was later described by Hoy and Fedman (1987) as one in which

15

there is harmony in the relationships among, students, teachers and administrators in their

quest to fulfill their schools mission.

Arthur Perry, a principal in a school in Brooklyn, made the first observations

concerning the importance of not only the physical environment of the school but the

esprit (translated from French to mean spirit) of the school (Perry, 1908). In the

following hundred years, school climate has undergone an evolution, as more and more

research is added to our understanding of what it is and its impact on student

achievement.

In 1963, Halpin and Croft developed the Organizational Climate Descriptive

Questionnaire (OCDQ). It focused on teacher-teacher and teacher-administrator

interactions from the teachers point of view. In the 1970s, Moos and Insel (1974)

conceptualized school climate as the interaction of human environments and humans

within the social and physical aspects of the environment, according to three dimensions:

interpersonal relationships, goal orientation/ personal growth, and system

maintenance/system change. Rutter (1979) and his colleagues established the notion of

ethos as an important quality of schools (Rutter et al, 1979). This included caring relationships among faculty members and administrators as positive role models, high

expectations for student success, and an emphasis on positive rewards. In the 1980s school climate was largely conceptualized as the whole of a school

is more than the sum of its parts; the school was viewed as a community. Bryk and

Driscoll (1988) found that schools that demonstrated the importance of school

community had better attendance, higher morale, and better mathematics achievement. It

was further determined that teachers sense of responsibility for student learning, or self-

16

efficacy, was a factor of schools that viewed themselves as a community (e.g., Bandura,

1992, 1997; Tschannon-Moran et al., 1998).

In the 1990s the phrase academic press was utilized as a theory of school

climate. Academic press refers to the extent to which the school community, including

students and teachers, experience a strong emphasis on academic success and conformity

to school values and practices that bolster high standards for student performance (Lee,

Smith, Perry, & Smylie, 1999). This notion has been recognized in the literature as

predictive of relatively high achievement (Evans, 1997; Hoy and Sabo, 1998; Newmann,

1992).

Now, in a seemingly much needed organization of theories and definitions, the

21st century has finally turned out a consensus-determined definition of school climate

highlighting four areas of school functioning (as noted previously): physical safety,

relationships of those in the school environment, including faculty, students, and parents,

teaching and learning methods, and the actual physical environment (Cohen, McCabe,

Michelli, and Pickeral, 2009).

Impact of Negative School Climate/Low Levels of Trust: Students, Teachers, and the

Community

Hoy, Tarter and Kottkamps (1991) research confirmed that a great many schools

across the nation fit the description of an unhealthy school. Schools with a negative

school climate, prevent teachers, support staff and administration from being able to

model positive citizen-forming behaviors for our students (Collaborative for Academic,

Social, and Emotional Learning (CASEL), 2008).

17

Damico, Roth, Fradd, and Hankins, (1991) indicated that schools with a negative

climate send out negative signals to at-risk students, instilling the perception that they are

unworthy and unable to continue in the educational process. Beardon, Spencer, and

Moracco (1989) confirmed that students perception of themselves, along with the school

experience, are paramount to school values and practices, since a negative school climate

could actually decrease a students motivation for learning.

Mowday, Porter, and Steers (1982) and Wynn, Carboni, and Patall (2007)

reported that schools with negative school climates had high teacher absenteeism and

turnover. Also, Haynes, Emmons, and Ben-Avie (1997) state that any setting that has a

negative climate, in which members do not relate well to each other, is a psychologically

unsound environment and contributes to poor mental health for all. In fact, in schools

that were run with an authoritarian, rule-bound manner, teachers were less likely to go

above and beyond contractual expectations for students and have less trust with their

administration (Tschannen-Moran, 2009).

Teachers: Emotional Aspects of School Climate and Student Impact

Echoing Borger (1985), Hoy (2008) stressed the importance of meeting teachers

emotional needs. Hoy posited that if a school has open, healthy and collegial interactions

among the adults in the school along with high academic expectations, that teachers on

the whole will feel empowered and effective. In fact, Murray (2005) suggested that the

ability of new teachers to build quality relationships with other adults in the school

setting (specifically administrators and colleagues) is a significant factor that molds their

decision on whether or not to remain in the teaching profession. Wynn, Carboni, and

18

Patall (2007) corroborated that good principal leadership and positive school climate are

significant factors in teacher retention.

Neither staff cohesion nor staff conflicts exist in a vacuum. Ninan (2006)

illustrated that children learn by example and that the moral fabric of a school is set and

nurtured by the teachers in the environment. Children view teachers values and

(consciously or unconsciously) emulate those behaviors. The manner in which teachers

treat other teachers is also recognized and internalized. Similarly, research has evidenced

that when teachers were more committed to their students, and went beyond typical

contractual expectations, there was a distinct, positive relationship with student academic

success (Tschannen-Moran, Parish, and DiPaola, 2006).

Similarly, McLoughlin, Kubick, and Lewis (2002) believed that to ensure that our

schools are safe, teachers play pivotal parts in communicating equal respect among all

students. They pointed out that teachers send this message to students in their classrooms

by modeling appropriate behavior, as well as teaching social skills and citizenship

directly.

Pantaleno (2007) noted that the Social Emotional Learning (SEL) movement

currently making its way into the public school curriculum (as witnessed by the recent

spate of character-education programs and social-emotion learning initiatives being

considered in state education legislation and supported by federally subsidized grants

[U.S. Department of Education, 2008]), will be the vehicle to teach emotionally

intelligent behavior and that important task will lie on the shoulders of our classroom

teachers. Of note is that results of these pilot programs indicated that successful

19

strategies included modeling character traits by all adults in the school (CASEL, 2008;

Haynes & Thomas, 2007; U.S. Department of Education, 2008).

The importance of having good role models for our children in schools cannot be

over-emphasized. Sandberg (2007), asserted that the most important action for schools is

to shift the focus from teaching just our childrens moral character to creating entire

school environments, (and all who function within that environment), that brim with high

moral character. She suggested that instead of solely teaching character-building lessons

in the classroom, that all adult school members: teachers, administration, and support

staff, build a moral climate within entire schools. In a similar fashion, Cohen (2009)

emphasized that to the extent that students feel safe, cared for, appropriately

supported, and lovingly pushed to learn, [ultimately] academic achievement should

increase (p. 186).

Trust Research in Schools

Several studies indicate that fostering relationships, particularly trust, among

colleagues in schools is not only of utmost importance, but indeed critical to the

development of a successful learning environment (Bulach & Malone, 1994, NWREL,

2003; Homana, Barber, & Torney-Purta, 2005; Hoy, Gage & Tarter, 2006; Tschannen-

Moran & Hoy 1998; Bryk & Schneider, 2002; Norton, 2008). It has also been suggested

that school leaders who demonstrate a strong trust in their faculty, have a reciprocal

trusting relationship with teachers, and parents. As a result, for the greater school

community, teachers will likely evidence greater professionalism, including a

stronger commitment to their students, [and] greater cooperation with colleagues

(Tschannen-Moran, 2009, p. 244).

20

School Administrators and School Climate Intervention

School leaders have a critical amount of influence as to a schools climate,

whether it be clearly expressed or implied (Cohen, et al., 2009). Although research

supports a series of positive consequences of educating students in a school with a

healthy climate, there are few states that have either a climate specialist or have

developed school climate policies (McCabe & Cohen, 2006). In a recent policy scan,

Cohen, et al. (2009) discovered that, of those states that indeed had school climate policy

written into their accountability plans (22), only six of them had done so in a limited

manner, and the other 22 had school climate subscribed to either special education, health

or school safety concerns. This lack of state support to enhance school climate

significantly curtails a school leaders ability to provide for positive changes.

NWREL (2003) offers suggestions to administrators to help foster a positive

school climate in their schools. They include: facilitating and modeling effective

communication, supporting the right and importance to have dissenting views, and

reducing teachers feelings of vulnerability between and among all faculty members.

They also suggest that administrators work hard to make relationship building of

paramount importance and to select a professional development model that encourages

relationship-building (NWREL, 2003). Similarly, Gibbs (2005) and Homana, Barber,

and Torney-Purta (2005), respectively, believe that building the capacity of school

personnel, as well as holding schools and their administrators accountable to teach

students to become more civic-minded, are the best methods to attain a positive school

climate which then encourages student achievement. Tschannen-Moran (2009) believes

that principals with a professional leadership style who vigorously encourages stronger

21

standards of trust throughout the school community will help instill a more professional

demeanor in their teachers. She suggested that, Creating conditions that strengthen

faculty trust in colleagues within the school may in turn allow greater faculty trust in

studentsto emerge (p. 243).

Systems Level Change and Continuing Professional Development

School psychologists are encouraged to be pivotal catalysts/supporters in system

level change in their schools (Curtis & Stoller, 2002; NASP, 2000). Curtis and Stoller

(2002) believe that, due to specialty training in human behavior and collaborative

relationships within the school environment, school psychologists are especially qualified

to work at this level in the school organization. They believe that collaborative

relationships within the schools requires, among other things, mutual respect and trust, in

which school psychologist are qualified to facilitate. Some examples they refer to

include: communication and effective listening skills, as well as team and consensus

building skills, in order to bolster mutual respect and trust among members. In a recent

(June, 2009) article published in the National Association of School Psychologists

(NASP) periodical, the Communiqu, school psychologists were encouraged to learn

about various communication styles among the people they work with and work in teams

with. They believe that doing this can help reduce conflict among team members

because the focus can be on what is being communicated, rather than the way it has come

across. The authors also suggested that being educated about the various communication,

personality, and personal styles, fosters a positive work environment, greater

understanding of others, and collaboration (Insert section, p. 2). In the same issue of the

Communiqu, Zuhkle and Mussman (2009) also describe the many different people from

22

various populations with whom they will need to communicate effectively with-- [I]t is

how these challenges and daily tribulations are handled that determines the value of

school psychologists to our constituents and personal career success (p. 15).

Both NASP (2006) and APA (2003) emphasize that school psychologists are

ethically bound and responsible for their own ongoing professional development. In a

guide provided to NASP leaders responsible for organizing members professional

development activities (Armistead, 2006), school climate was listed as an area in which

school psychologists could help create and maintain positive, supportive environments

for all school members. Additionally, Brown (2002), encouraged school psychologists to

utilize professional experiences to further long-range career goals, e.g., obtain

administration positions or expand their current roles.

School Psychologists as Effective Consumers of Research and Needs Assessment

According to the latest revision of the American Psychological Associations

Ethical Principles of Psychologists and Code of Conduct (2002), psychologists must:

(a) administer, adapt, score, interpret, or use assessment techniques, interviews, tests,

or instruments in a manner and for purposes that are appropriate in light of the research

on or evidence of the usefulness and proper application of the techniques and (b) use

assessment instruments whose validity and reliability have been established for use with

members of the population tested. When such validity or reliability has not been

established, psychologists describe the strengths and limitations of test results and

interpretation (American Psychological Association, 2002). These principles are reiterated by Keith (2002), in Best Practices in School

Psychology IV, all psychologist need to be effective consumers of research; that is,

23

they should be able to critique research that has implications for their practice and

incorporate the findings of that research into their practice (p. 100).

A diagnostic evaluation of the schools strengths and weaknesses in which school

psychologists may involve themselves with, often referred to as a needs assessment-- is

particularly useful for gathering info about the organization (Curtis & Stoller, 2002).

Nagle (2002), defines needs assessment in schools as, a systemic process of

collecting and analyzing data in order to identify needs and problems to be addressed in

program planning, development, and modification. Nagle goes further by emphasizing

important parts of conducting a needs assessment in a school: Determining what the exact

purpose and goal of the assessment is, monitoring how open to change the school is, and

working closely with school faculty and administration to ensure commitment to

proposed and agreed upon changes. He states, as schools become increasingly

involved in reform and restructuring activities (see U. S. Department of Education, 2008),

school psychologists will need system-based skills such as needs assessment.

Organizational Development, School Climate, and the Use of Team Building

There are ways to improve the learning environment at the school building level.

Although experts differ on measuring techniques, many agree that organizational

development and school climate improvement teams are successful interventions to

employ (Lindelow, Mazzarella, Scott, Ellis, & Smith, 1989). Yeager (1994) highlighted

the need for organizational development techniques in the schools:

If it is true, that the major hurdles to high organizational

performance are dysfunctional conflicts, poor communication, structured

rigidity, failure of members to know themselves and how they have an

24

impact on others, then organizational development techniques should be

designed to help to improve a [schools] effectiveness (p. 417).

Organizational development interventions, such as team building, help to

improve or alter the way the organization operates, and has been proven to be a

very effective and highly regarded intervention strategy (Yeager, 1994;

McNamara, 2006). Understanding the multidimensional components of team-

building is important in creating effective teams. Appreciating and nurturing

personal differences and addressing personal diversity, allow for an environment

where creative choices and decisions can occur (Rideout & Richardson, 1989, p.

62).

Team Building and Communication

An educational objective often overlooked in schools is a counseling component

that can include numerous personal growth opportunities for all members. Rome (1990)

noted that school climate becomes more positive when there is effective communication

and understanding among its members. Norton (2008) states that success in todays

schools, is possible if the environment supports: the generation of ideas, collaboration

and team effort from all members. Chandler, Kern, and Durodoye (1996) agree that an

essential element in an integrated team building approach to improving school climate is

communication and understanding, and [t]he future of education depends on innovative

integration and team building in schools (Rome, 1990, p. 42). Tschannen-Moran (2009) suggests, for example, that having teachers collaboratively plan instruction will

aid in their development and commitment towards their students, which may invigorate

them to go above and beyond what is typically expected of them. Organizational

25

development, staff development, and team building have all been successful methods of

increasing effectiveness in schools. Effectiveness in schools is in part, as noted earlier, a

result of a positive school climate. It is essential then, that school psychologists focus

their time and attention on this enhancement. It is universally understood that,

[s]tudents will be the ultimate beneficiaries of enhanced learning environments

(Chandler, Kern, & Durodoye, 1996, p. 44).

Statement of Problem

To this researchers knowledge, no cohesive source of critically reviewed school

climate measures exist that specifically assess the faculty-to-faculty component, or one

that also assesses school climate variables including: safety, teaching and learning, and

external environment. To engage in a comprehensive and time-consuming faculty

intervention without first choosing the most empirically supported and valid tool by

which to measure progress wastes both time and limited financial resources. Or worse, if

an ineffective/ inappropriate tool is used to measure broad-based school climate

constructs for pre-and post-intervention, misleading findings can lead to poor

intervention results. Also, the effort that personnel would expend on inefficient or

inaccurate projects would be crucial time spent on those that are supported and proven by

research.

Purpose

This study identified the most empirically supported broad-based school climate

measures that fit the following criteria: possesses a gauge of relationships (especially

faculty relationships), safety (physical and emotional), teaching and learning, and

external environment; have viewable test items (that this researcher is able to view), and

26

are direct measures (e.g. questionnaires) of primary and secondary education levels

(elementary through high school).

Research Question

Which are the most empirically supported broad-based school climate

assessments that fit the following criteria: possesses a gauge of faculty relationships;

safety; teaching and learning; relationships, external environment components; as well as

have viewable test items and are direct measures of primary and secondary education

levels?

27

Chapter Three

Methods

Procedures

The study included nine criteria review and eleven selection steps. Each

inclusionary criteria was considered a step in the sequence of instrument selection, and

there were two additional steps that were taken to complete this study. Table 1 reviews

the instruments bearing of the following components: Assessment of school climate

(Criteria 1), assessment of faculty relationships (Criteria 2), current research components

that measure school climate (Criteria 3), direct teacher measure (Criteria 4), viewable test

items (Criteria 5), current and representative normative samples (Criteria 6), published

technical characteristics (reliability and validity) (Criteria 7 & 8, respectively), and

school level (Criteria 9). Step 10 describes the expert panel procedure utilized in this

study, and step 11 will introduce and apply a ranking system and critical test review to

those measures that remain after criteria 9 has been applied.

Table 1. Criteria Presented in Sequential Order _____________________________________________________________________

Criteria 1: Literature Search

This will entail a search for measures that specifically assess school climate.

Criteria 2: Faculty relationship factor

This will entail a search for school climate measures that specifically assess faculty-to-

faculty relationships.

Criteria 3: Additional school climate variables.

28

Those that also assess these additional school climate factors: Safety; teaching and

learning; and external environment.

Criteria 4: Direct teacher measures.

Direct measures that include information that is gathered directly from those individuals

that work at the school.

Criteria 5: Viewable test items.

The ability to view individual items of the narrowed list of school climate assessment for

appropriate content.

Criteria 6: Current published instruments.

School climate measures published before 1990 would not adequately represent the

populations in which a measure would be recommended by this study.

Criteria 7: Reliability.

There are different types of reliability estimates that help the test user judge how

confident he/she can be in the amount of measurement error the test has. Adequate

reliability coefficients range from .70 or higher.

Criteria 8: Validity.

There are different types of validity estimates that help the test user judge if the test

assesses what it purports to assess.

Criteria 9: Both Primary and Secondary School Levels.

All assessment tools need their data to have been drawn only from faculty in primary

through secondary schools which, by necessity, excluded colleges/ universities,

preschools, clinics, hospitals, etc.

________________________________________________________________________

29

Step 1 (Criteria 1): Literature Search

First, to ensure that as many school climate measures that were available would

be found, a broad search of the literature was conducted. The following databases were

searched: Education Resources Information Center (ERIC) (1966-2009), PsychLit, the

Buros Institute of Mental Measurements Yearbook, the Mental Measurements Yearbook,

PsycARTICLES (EBSCO & APA), and PsychINFO. Proquest Psychology sites were

also searched, which include APA approved journals, an online search of school climate

tests published by ETS and PsychCorp. Additionally, as a catch-all, the World Wide

Web was searched using search engines (such as Google), using the keywords and

phrases: school climate, affiliation, faculty relationships, cohesion, communication,

collegiality, trust and morale. A search of the names of authors in the field of school

climate was also conducted. Following this strategy, 102 measures were found. In

addition, an informal questionnaire was presented to an expert panel of school climate

researchers via personal email asking them to rate the importance of several of the criteria

applied to the school climate measures in this study (Step 11; see Appendix A).

Step 2 (Criteria 2): Faculty relationship factor

The second step entailed a search for school climate measures that specifically

assessed faculty-to-faculty relationships. The quality of the relationships that teachers

with whom they work have been found to be important climate variables (Rutter et al.,

1979; Phi Delta Kappa study, 1980; Wynne, 1980; Anderson, 1982; Van Horn, 2003).

For this study, instruments were reviewed and retained if they possessed a faculty

(colleague) relationship component, i.e., gave the adult staff an opportunity to report on

30

their perceptions of relations between faculty members in the school environment they

worked in.

Step 3 (Criteria 3): Additional school climate variables

In this step, a table was created for the measures that meet the primary criteria for

review, i.e., those that also assess these additional school climate factors: Safety; teaching

and learning; relationships, and external environment (operationalized below in detail in

Cohen, 2006, p. 245):

[S]afety (physical, social-emotional); teaching and learning (quality of instruction,

expectations for student achievement, leadership, professional development and social-

emotional-ethical education); relationships (respect for diversity among students and

adults, school outreach to parents, school-home partnerships, morale), and external

environment (quality and structure, physical plant, structure of time and space).

These particular factors are supported in the research by Howard, 1987; Knoff,

2002; Lehr & Christenson, 2002; National School Climate Council, 2007; Stichter, 2008.

Likewise, Felner and Felner (1989) also noted the importance of sensitivity to cultural

diversity, and Gotffredson and Gottfredson (1985) emphasized that school is to be an

environment where students felt safe. Uline and Tschannen-Moran (2008) also

emphasized clearly that human comfort, pleasing appearance, adequacy of space,

functional furniture and equipment, a clean and orderly environment, and regular

maintenance effects occupants sense of well being and thus their ability to teach and

learn (p.4).

31

Step 4 (Criteria 4): Direct teacher measures

Lehr and Christenson (2002) and Freiberg (1999) reported that there are two

different kinds of measures of school climate: indirect and direct. Indirect measures are

those in which information is not directly gathered from informants but instead, data is

collected. Such data could include suspensions, discipline referrals, absenteeism, teacher

rate of retention, etcDirect measures, however, include information that is gathered

directly from those individuals that attend, work at, or have children that attend the

school. Questionnaires, in particular, are preferred as a survey method when large

numbers of individual responses are needed (Nagle, 2002). For the purposes of this

study, it was important that the original information was gathered directly from those

individuals associated with the schools. School climate measures in which teacher

questionnaires were utilized were selected for this study.

Step 5 (Criteria 5): Viewable test items

According to the Code of Fair Testing Practices (2004), Test users

shouldevaluate representative samples of test questions or practice tests, directions,

answer sheets, manuals, and score reports before selecting a test ( p. 5), and Rudner

(1994), You should gather the information you need to evaluate a test. ..[g]et a specimen

set from the publisher. It was determined that, for this study, it was important to have

the ability to view individual items of the narrowed list of school climate assessment in

order to evaluate each item.

Step 6 (Criteria 6): Current published instruments

According to the American Psychological Association, the American Educational

Research Association, and the National Council on Measurement in Education (1999),

32

normative samples must be of sufficient size and adequately represent the population it

purports to measure in order to support statements of validity. The instrument also needs

to have adequate norm samples in order for conclusions to be supported regarding its use

for its intended purpose. The normative samples need to include individuals for whom the

test was intended for (e.g., age, experience, and background (Rudner, 1994; Anastasi,

1988). However, Hunsley and Mash (2008) explain that certain psychometric elements

may not have bearing on an assessment (e.g., a program evaluation) and that

psychometrics in regards to assessments used in clinical practice for evidenced-based

assessments (e.g., of symptoms of anxiety or depression) are inherently different than

measuring the constructs of a non-living organization (like a school). Hunsely and Mash

also describe that clinical assessment usually uses both idiographic and nomothetic

instruments. Idiographic instruments track changes in individuals, whereas nomothetic

measures are more global and can track changes across populations. It follows logically,

then, that school climate assessments are nomothetic instruments that are used to track

changes across populations.

Regarding norms, Hunsley and Mash reported that regardless of the population

to which comparisons are to be made, [and] [I]deally, whether conducted at the

national level or the local level (p. 8), .at a minimum, clinicians need to be

provided with an indication of the quality and likely accuracy of the norms for a

measure (p. 9). Hunsley and Mash go on to define adequate norms as those that

consist of measures of central tendency and distribution for the total score (and subscore

if relevant) based on a large, relevant, clinical sample (p. 8), good norms as measures

of central tendency and distribution for the total score (and subscores if relevant) based

33

on several large, relevant samples (must include from both clinical and non-clinical

samples)(p. 8), and excellent as measures of central tendency and distribution for

the total score (and subscores if relevant) based on one or more large, representative

samples (must include data from both clinical and nonclinical samples) (p. 8).

Information from Bond (1996) was used as encouragement to strive for norm

samples within, at most, seven to ten years. In support, the Joint Committee (1999) also

emphasized the importance of renorming tests to be sure that accurate and appropriate

score interpretations (p. 59) can be obtained. For this study it was determined that

school climate measures with norms obtained before 1990 would not adequately

represent the populations in which a measure would be recommended by this study. This

was based on evidence that use of outdated norms in a measure is not best practice

(Reschly & Grimes, 2002). It may be necessary to note however, that psychological

constructs, such as IQ, may be more sensitive to outdated norms than school climate

predictor variables (Beaujean & Gulling, 2006). Norms for school climate instruments,

and their correspondence to current U.S. census statistics, is discussed further in the

Results section of this study.

Step 7 (Criteria 7): Reliability

It is important that test scores are minimally impacted by measurement error

and that they are constant from one administration to another. There are different types of

reliability estimates that help the test user judge how confident he/she can be in the

amount of measurement error the test has. The different types of measurement error are

estimated by:

1. Inter-rater reliability - errors due to differences in judgment between raters

34

2. Alternate-form reliability- errors due to how the individual is expected to rank

on an alternate form of the test

3. Internal consistency reliability - error that is due to content sampling,

4. Test-retest reliability- error due to inconsistent repeat performance after a

specific amount of time has passed (Anastasi, 1988; Joint Committee, 1999;

Rudner, 1994).

In regards to test-retest reliability, Hunsley and Mash (2008) noted, ...not all

constructs or measures are expected to show temporal stability (e.g., measure of state-like

variables, life stress inventories) (p. 10). This reflects the varying interpretations that

some school climate experts have regarding the potential impact (or lack of impact) of

test-retest has on school climate.

As for internal consistency, they reported that adequate internal consistency is

when the preponderance of evidence indicates a values of ...70 - .79, good internal

consistency is when a values [are] ...80-.89, and excellent internal consistency is

when a values [are greater than or equal to] .90 (p. 8). The criteria selection below will

explain how reliability will be rated in this study.

Step 8 (Criteria 8): Validity

There are different types of validity estimates that help test users judge if the

test assesses what it purports to assess. Predictive validity refers to how well a test can

correctly predict something that it is theoretically believed to be able to predict (Anastasi,

1988; Rudner, 1994). Content, construct and criterion validity are widely believed to

measure the same concept (Education Testing Service (ETS) Standards for Quality and

35

Fairness, 2002; Joint Committee, 1999). They refer to how well the test measures specific

content of interest and how well the test items represent the test content.

Hunsley and Mash (2008) report that adequate content validity is when the the

test developers clearly defined the domain of the construct being assessed and ensured

that selected items were representative of the entire set of facets included in the domain,

good content validity is when in addition to the criteria used for an adequate rating,

all elements of the instruments (e.g., instructions, items) were evaluated by judges (e.g.,

by experts or by pilot research participants, and excellent content validity is when in

addition to the criteria used for a good rating, multiple groups of judges were employed

and quantitative ratings were used by the judges (p.9). Hunsley and Mash (2008) also

stated that adequate construct validity is when some independently replicated

evidence of construct validity (e.g., predictive validity, concurrent validity, and

convergent and discriminant validity), that good construct validity is when there is a

preponderance of independently replicated evidence, across multiple types of validity

(e.g., predictive validity, concurrent validity, and convergent and discriminant validity)

and excellent construct validity is when in addition to the criteria for a good rating,

evidence of incremental validity with respect to other clinical data (p. 9).

In addition, multiple informants (considered another form of concurrent or

predictive validity) are utilized when measuring different perceptions of the same

construct(s). Contemporary studies commonly use two or three informants as data

sources. . one anticipates discordant reports. If [there is] no discordance, the

additional reports provide no new information. Multiple sources are used to provide data

either on independent variables (predictors), or dependent variables (outcomes), or both

36

(Horton, 2004). Particularly relevant to this study, multiple informant surveys have been encouraged to be utilized if rigorous and reliable data are to be used in providing evidence-based policy recommendations or reform in public sector

organizations (Kerr, Lunkenheimer, & Olson, 2007, Abstract). The criteria below will

indicate how validity will be rated in this study.

Step 9 (Criteria 9): School Level

All assessment tools needed their data to have been drawn only from faculty in

primary through secondary schools which, by necessity, excluded colleges/ universities,

clinics, hospitals, etc.The target audience for this research study consists of school

psychologists and administrators in primary and secondary school system; other

environments are beyond the scope of this study.

Step 10: Expert Panel Procedure

In addition to a critical review of the three instruments that met this studys

inclusionary criteria, a questionnaire was presented to an expert panel of researchers in

the field of school climate. Several prolific researchers in the area of school climate were

contacted via email and asked if they would participate in a dissertation study on school

climate. These particular researchers were selected according to either the frequency in

which their publications were cited in the school climate literature (at least four or more),

or their affiliation with a school climate research center. Nine out of nine potential panel

members responded favorably. Of the nine expert panel members, eight wished to be

identified in this study: Dr. Jonathan Cohen [Co-founder and Director of The Project for

Social Emotional Learning (PSEL) at Teachers College, Columbia University and the

Center for Social and Emotional Learning (CSEE). The CSEE is the creator of the

37

Comprehensive School Climate Inventory [CSCI]); Dr. John Shindler (Co-director of the

Western Alliance for the Study of School Climate, and the developer of the WASSC

Climate Assessment Instrument, WASSC); Dr. C. John Tarter (Professor, University of

Alabama, Department of Educational Leadership, Policy, and Technology Studies;

Author/co-author of several books and articles regarding school climate); Dr. N. Scott

Norton (Professor Emeritus, Arizona State University; Author of several books and

articles regarding organizational/school climate); Dr. Martha Alberg (Co-author of the

School Climate Inventory-Revised (SCI-R) and co-director of the Center for Research in

Educational Policy (CREP) at the University of Memphis in Tennessee); Dr. Megan

Tschannen-Moran (Associate Professor, The College of William and Mary in Virginia;

Author/co-author of several books and articles regarding school climate); Joshua

Freedman (Chief Operating Officer, Six Seconds, The Emotional Intelligence Network;

Co-Author of the Six Seconds Emotional Intelligence Assessment and the Organizational

Vital Signs Climate Assessment); and Dr. Carina Fiedeldey-Van Dijk (Senior Research

Scientist; a significant contributor to the development, psychometric validation, and

statistical analysis of the Six Seconds Emotional Intelligence Assessment (SEI), the

Organizational Vital Signs Climate Index and the Assessment of School Climate (ASC)).

(Please see Appendix C for a brief biography of each expert panel member).

Although the ninth panel member is also a prolific researcher and educator in the

field of school climate and provided extremely helpful feedback on the questionnaire,

more qualifications of the proposed study were needed in order for this persons

identification on the panel to occur.

38

Each panel members was emailed a questionnaire as an attachment. The

questionnaire contained twelve questions eliciting their perception of the level of

importance of some technical qualities; specifically, the reliability, validity, and norms of

school climate measures (See Appendix A for the questionnaire). The response set

included a five-point Likert-scale: 0-Not important at all, 1-Somewhat important, 2-

Moderately important, 3-Very important, 4-No opinion, 5- Do not know. Six of the nine

panel members answered each of the twelve questions, one member answered five of the

twelve questions with modified levels of importance than were given on the original

Likert-type questionnaire, and two members answered one of the twelve questions with

modified levels of importance than were given on the original Likert-type questionnaire.

In order to find out how much the raters agreed with each other, some modifications to

responses were made by this author (with rater permission). For example, three raters

gave a score of 2.8 (rather than 2-moderately important or 3-very important) to two

questions, this score was reassigned as a 3. These same three raters also gave a score

of 2.5; this score was reassigned as a 2. The rationale was that these score

reassignments would not significantly skew the agreement/non-agreement of the raters

responses. In addition, the following changes were made: Three raters responses were

discarded from the statistical calculation of interclass correlation if they used either 4-No

opinion (or did not answer the question), or 5-Do not know. The rationale was that these

two responses were not part of the ordinal scale: 0-Not important at all, 1-Somewhat

important, 2-Moderately important, or 3-Very important. As a result, only six of the nine

expert rater responses were utilized in the intraclass correlation calculation (ICC).

Step 11: Comparative ranking system

39

As each of the school climate measures reviewed in this paper have various

components and psychometrics obtained, and essentially, only one is needed to serve the

purpose of this study, a ranking system was developed. In determining this ranking

system, a point system was applied to three components: reliability correlation

coefficient; validity information, and current and representative (adequate) normative

samples.

Reliability. In examining the internal consistency, inter-rater, alternate-form, and

test-retest correlations for each measure, the following points were assigned: If the

average of the measures subscales had a correlation below 0.70, then zero points were

assigned; if the average of all subscale correlations was between 0.70 0.79, one point

was assigned; if between 0.80 0.89, two points were assigned; and if 0.90 or above,

three points were assigned. The same point assignment was followed for the faculty

relationship subscale. If there was evidence that test-retest, inter-rater, or alternate-form

reliability gathered for the measure, one point was assigned for each.

Validity. All the measures reviewed in this paper had some form of validity. One

point each was assigned if the following type of validity was applied in review: content,

construct, concurrent, convergent, divergent/discriminative, criterion, and/or predictive.

Anastasi and Urbina (1997) recommend that at least three forms of validity need

to be evidenced in an educational/psychological test: content description [content

validity], criterion prediction [criterion-related validity], and construct identification

[construct validity]. An additional point was added if three forms of validity were

demonstrated in the research of the measure(s). One point (for each informant group)

was added if the instrument utilized multiple informants.

40

Sample size, recency of publication/norms, and representative norms. Lastly,

sample size, recency of publication/norms, and representativeness of norms were

examined in this quantification of assessment tools. The following conversion was

utilized: Zero points for samples under 150, one point for sample sizes between 150-500,

two points for samples between 500 and 1,000, and three points for samples over 1,000

subjects. For recency of publication/norms: Zero points if the measure had norms before

1990, one point if the measure had norms from 1991-1998, and two points if the measure

had norms from 1999-present. If efforts were documented to cull a normative sample

representative of the U.S. Census, an additional point was also added.

41

Chapter Four

Results

The previous chapter focused on the cursory review of each of the climate

instruments culled from several decades of school climate research. A description of the

steps that followed was discussed in detail, as well as specific criteria that the measures

were held to for final critical review. In this chapter the data collected will be presented,

analyzed, and interpreted. Descriptive statistics, specifically reliability coefficients,

claims of validity and norm information, were examined for each of the final three

measures included in the critical test review.

In step one, the original list of school climate measures (see Table 1) were chosen

from the literature and resources if they were simply recognized/considered through title

and/or description to be tests/measures/questionnaires measuring school climate. No

other criterion was needed here. One hundred and two instruments were found.

Table 1 Step1: Literature Search ________________________________________________________________________ 1. Alaska Association of School Boards (AASB) School Climate and Connectedness

Survey (SCCS)

2. Academic Setting Evaluation Questionnaire

3. Affective Work Competencies Inventory

4. Assessment of School Climate, The

5. Assessment of School Needs for Low-Achieving Students: Staff Survey

6. Attitudes and Beliefs on Classroom Control (ABCC) Inventory

42

Table 1 (Continued) Step1: Literature Search ________________________________________________________________________ 7. Barclay Classroom Climate Inventory

8. Building Level Effectiveness Survey

9. California Health Kids Survey

10. California School Climate and Safety Survey

11. Character in Action Survey

12. CharacterPlus, The

13. Charles F. Kettering, Ltd. School Climate Profile

14. Chicago Public Schools Student Connection Survey

15. Civic Mission of Schools Self-Assessment

16. Class Activities Questionnaire

17. Classroom Environment Scale

18. Classroom Level Effectiveness Survey

19. Classroom Practices Inventory

20. Classroom and School Community Inventory

21. College Characteristics Index (CGI)

22. College Student Experiences Questionnaire

23. College and University Classroom Environment Inventory

24. Communication Climate Questionnaire

25. Comprehensive Assessment of School Environment

26. Comprehensive School Climate Inventory

27. Connecticut Effective Schools Questionnaire (CESQ)

43

Table 1 (Continued) Step1: Literature Search ____________________________________________________________________

28. Developmental Studies Center Child Development Project Scales

29. Diagnostic Assessment of School and Principal Effectiveness

30. Dimensions of Excellence Scales

31. Education, Training, Research Character Education Survey

32. Effective School Battery, The

33. Effective School Function Inventory

34. Effective Schools Survey

35. Emotional Climate Inventory

36. Elementary School Environment Survey

37. Elementary Student Opinion Questionnaire

38. Enjoyment of Class Scale

39. Environmental Assessment Technique

40. Glendale School Effectiveness Survey, The

41. High School Survey of Student Engagement

42. High Performance Learning Community Assessments (HiPlaces Assessments)

43. Illinois Quality Schools Index

44. Individualized Classroom Environment Questionnaire

45. Institutional Functioning Inventory

46. Instructional Climate Inventory for Teachers

47. Inventory of School Climate, The

48. K12 School Climate and Diversity Surveys

44

Table 1 (Continued) Step1: Literature Search ________________________________________________________________________

49. Learning Climate Inventory

50. Learning Climate Questionnaires

51. Learning Environment Inventory (LEI)

52. Liking for School Scale

53. Listening to Student Voices Self-Study Toolkit

54. My Class Inventory

55. My School Inventory

56. National Study of School Evaluation: Teacher Opinion Survey

57. Organizational Description Questionnaire-Revised (OCDQ-R)

58. Organizational Health Inventory (OHI)

59. Paces College and University Environment Scales

60. Parent-School Communities Questionnaire

61. Playground and Lunchroom Climate Questionnaire, The

62. Profile of a School, The

63. Psychological Sense of School Membership

64. Pupil Control Ideology/ Pupil Control Behavior

65. Quality of School Life Scale, The

66. Robustness Semantic Differential (RSD) Scale, The

67. Safe and Responsive Schools Survey

68. School Assessment Survey

69. School Attitude Measure

45


70. School as Caring Community Profile (SCCP) II

71. School Characteristics Inventory: Study of Instructional Improvement

72. School Citizenship Education Climate Assessment

73. School Climate Questionnaire

74. School Climate Survey (Yale)

75. School Culture Scale

76. School Description Inventory, the (SDI)

77. School Diversity Inventory, the (SDI)

78. School Discipline Climate Survey

79. School Effectiveness Questionnaire (SEQ)

80. School Environment Preference Survey

81. School Improvement Program Needs Assessment Questionnaires

82. School Level Environment Questionnaire (SLEQ)

83. School Participant Empowerment Scale

84. School Improvement Program Needs Assessment Questionnaires (Elementary

Schools, 1970)

85. School Survey of Interpersonal Relationships (SSIR)

86. School Work Culture Profile

87. Secondary School Moral Atmosphere Questionnaire, The

88. Social Outcomes Survey

89. Staff Development and School Climate Assessment Questionnaire (SDSCAQ)

46


90. Staff morale questionnaire (SMQ)

91. Statements About Schools Inventory

92. Student Satisfaction Inventory

93. Survey of Instructional and Organizational Effectiveness

94. Teacher Involvement and Participation Scale

95. (Tennessee) School Climate Profile, The

96. Thoughts About School Measure

97. Tribes TLV Assessment Questionnaire

98. Vessels School Climate Questionnaire

99. Western Alliance for the Assessment of School Climate (WASSC)

100. What Kids Can Do: Students as Allies in Improving Their Schools

101. WINSS School Climate Survey

102. World Health Organizations Psycho-Social Environmental Profile

_______________________________________________________________________

Second, in step two (Table 2), each of the 102 measures from step one were

reviewed and retained if they possessed a faculty component, i.e., gave the adult staff an

opportunity to report on their perceptions on the school climate of the environment they

worked in/relations between faculty members. Forty-seven measures remained after this

second inclusionary criteria was met.

47

Table 2 Step 2: Faculty Relationship Component ________________________________________________________________________

1. Alaska Association of School Boards (AASB) School Climate and Connectedness

Survey (SCCS)

2. Academic Setting Evaluation Questionnaire

3. Assessment of School Climate, The

4. Assessment of School Needs for Low-Achieving Students: Staff Survey

5. CharacterPlus (The)

6. CFK, Ltd. School Climate Profile

7. Civic Mission of Schools Self-Assessment

8. College Characteristics Index (CGI)



11. Dimensions of Excellence Scales

12. Education, Training, Research Character Education Survey

13. Effective School Battery, The

14. Effective School Function Inventory

15. Emotional Climate Inventory

16. Glendale School Effectiveness Survey, The

17. Illinois Quality Schools Index


19. Instructional Climate Inventory for Teachers

20. Inventory of School Climate, The

48

Table 2 (Continued) Step 2: Faculty Relationship Component ________________________________________________________________________


22. National Study of School Evaluation (NSSE): Teacher Opinion Survey

23. Organizational Description Questionnaire-Revised (OCDQ-R)

24. Organizational Health Inventory (OHI)

25. Profile of a School, The

26. Robustness Semantic Differential (RSD) Scale, The

27. School Assessment Survey

28. School as Caring Community Profile (SCCP) II

29. School Characteristics Inventory: Study of Instructional Improvement

30. School Citizenship Education Climate Assessment

31. The School Diversity Inventory (SDI)

32. School Discipline Climate Survey


34. School Improvement Program Needs Assessment Questionnaires

35. School Level Environment Questionnaire (SLEQ)

36. School Survey

37. School Survey of Interpersonal Relationships (SSIR)

38. School Work Culture Profile

39. Staff Development and School Climate Assessment Questionnaire (SDSCAQ)

40. Staff morale questionnaire (SMQ)

41. Survey of Instructional and Organizational Effectiveness

49

Table 2 (Continued) Step 2: Faculty Relationship Component ________________________________________________________________________

42. Teacher Involvement Participation Scales


44. Tribes TLV Assessment Questionnaire


46. WINSS School Climate Survey

47. World Health Organizations Psycho-Social Environmental Profile

_______________________________________________________________________

Third, in step three (Table 3), each of the 47 measures from step two were

reviewed again and retained if they possessed the additional school climate variables that

have been found in the research to be considerable predictors of positive/negative school,

i.e., 1) safety, 2) teaching and learning, and 3) external environment. Only eight measures

of the 47 possessed these additional school climate variables.

Table 3 Step 3: Additional School Climate Components ________________________________________________________________________ 1. CFK, Ltd. School Climate Profile







50


_______________________________________________________________________

In step four, the eight measures from step three were reviewed again and retained

if they were direct teacher measures of climate. All of the eight measured remaining

possessed at least one teacher questionnaire.

Table 4 Step 4: Direct Teacher Measures ________________________________________________________________________ 1. CFK, Ltd. School Climate Profile








_______________________________________________________________________ In step five (Table 5), the eight measures retained from step four were reviewed

and retained if the actual school climate measure was able to be obtained online or

secured via the publisher. Out of the eight measures, only four measures had individual

items that could be directly observed by this examiner. These included: The Charles F.

Kettering, (CFK) Ltd. School Climate Profile, the Comprehensive School Climate

Inventory (CSCI), The (Tennessee) School Climate Inventory-Revised, and the Western

Alliance for the Assessment of School Climate (WASSC). The primary reason for four

51

of the measures (Comprehensive Assessment of School Environment (CASE),

Institutional Functioning Inventory, Learning Climate Inventory, and School

Effectiveness Questionnaire [SEQ]) to have been dropped from this study for review, was

that they were no longer available/in print. This was determined by contacting the

publishers directly and inquiring.

Table 5 Step 5: Viewable Test Items ________________________________________________________________________

1. CFK, Ltd. School Climate Profile (via Howard book)

2. Comprehensive School Climate Inventory (via publisher directly)

3. (Tennessee) School Climate Inventory, The


_______________________________________________________________________ In step six (Table 6) each of the four measures from step five were reviewed and

retained if the measures were based on normative samples collected in 1990 or later.

This step is similar to step five because much of the reason why the tests were

unavailable/no longer in print was presumably due to the advanced age of the measure.

One measure was dropped at this step, although still available and in print, due to original

norms dating back to 1973 (Charles F. Kettering (CFK), Ltd. School Climate Profile,

1973).

52

Table 6 Step 6: Recently Published (1990 or higher) ________________________________________________________________________ 1. Comprehensive School Climate Inventory (via publisher directly)

2. (Tennessee) School Climate Inventory-Revised (via publisher directly)

3. Western Alliance for the Assessment of School Climate (via publisher directly)

_______________________________________________________________________

In step seven each of the three measures from step six were reviewed and retained

if they possessed statistical information regarding reliability in the climate measures

manual or research. All measures reviewed in step seven possessed this statistical

information and so all were retained for further review.

Table 7 Step 7: Reliability ________________________________________________________________________


2. School Climate Inventory-Revised

3. Western Alliance for the Assessment of School Climate

_______________________________________________________________________

In step eight, each of the three measures from step seven were reviewed and

retained if they possessed statistical information regarding validity in the climate

measures manual or research. All measures reviewed in step eight possessed this

statistical information and so all were retained for further review.

53

Table 8 Step 8: Validity ________________________________________________________________________




_______________________________________________________________________

In step nine (Table 9), the three measures retained after steps 7 and 8 were

reviewed and retained if they measured only primary and secondary education settings,

i.e., no preschool, and university-level measures, as well as business-related/hospital-

related measures. All measures reviewed in steps seven and eight measured only

primary/secondary educational settings and so were retained for further review.

Table 9 Step 9: School-Level ________________________________________________________________________




_______________________________________________________________________ Step 10: Expert Panel Procedures

In step ten (Table 10), an informal questionnaire was presented to an expert panel

of school climate researchers. (See Appendix A for the questionnaire). The response set

included a five-point Likert-scale: 0-Not important at all, 1-Somewhat important, 2-

54

Moderately important, 3-Very important, 4-No opinion, 5- Do not know.

The results of the questionnaire responses are listed in Table 10.

Table 10 Expert Panel Questionnaire Responses __________________________________________________________________

1. How important is the construct of safety in a broad-based assessment of

school climate?

3 out of 9 --- Very Important

4 out of 9 --- Moderately Important (*one rater gave a score of 2.5; this

score was reassigned [with permission from the rater] to a 2 by the

author).

2 out of 9 --- I do not know

2. How important is the construct of teaching and learning in a broad-based

assessment of school climate?

6 out of 9 --- Very Important (*one rater gave a score of 2.8; this


author).

3 out of 9 --- Moderately Important

3. How important is the construct of relationships in a broad-based assessment

of school climate?


4. How important is the construct of external environment in a broad-based

assessment of school climate?


55


2 out of 9 --- I dont know

5. How important is it that a broad-based assessment of school climate possesses

inter-rater reliability?



1 out of 9 --- Not Important At All


test-retest reliability?



1 out of 9 --- Somewhat Important


content validity?

8 out of 9 --- Very important

1 out of 9 --- Did not answer the question.


concurrent validity?


2 out of 9 --- Moderately Important (*one rater gave a score of 2.5; this


author).

9. How important is it that the assessments possess criterion validity?

56


2 out of 9 --- Moderately Important (*one rater gave a score of 2.5;

this score was reassigned [with permission from the rater] to a 2 by the

author).

1 out of 9 --- Somewhat Important

10. How important is it that a broad based assessments of school climate

possess norms within the last 7-10 years?



1 out of 9 --- Answered the question but without choosing a qualifier (and

instead mentioned that local norms seem to be more important than

national norms in the measuring of school climate, and that the nature and

purpose of the work being done would play a role).

11. How important is the sample size in a broad-based assessment of school

climate?



12. How important is it that the standardization sample of a broad-based

assessment of school climate match the U.S. Census?



1 out of 9 --- Not Important At All

1 out of 9 --- Answered the question but without choosing a qualifier (and

57

instead mentioned that it was unrealistic due to the lack of resources)

2 out of 9 --- I dont know * Scores given a 2.5 were rated downward to a 2 (Moderately important) and scores given a 2.8 were rated upward to a 3 (Very important)[with permission from this rater]. This was done to facilitate input and calculations of inter-rater agreement using a computer statistical program (please see next paragraph). __________________________________________________________________

To measure the consistency of raters agreement, an intraclass correlation

coefficient (ICC) was computed using SPSS 11.0.1 (Table 11). Intraclass correlation

coefficients are typically computed when the differences in ratings among judges is an

important consideration. The most common method when using ICC is one in which the

judges are assumed to be a random sample of a larger possible sample of judges, and one

in which each judge only rates the item being rated once (Howell, 2006). This was the

method chosen in reference to the expert raters because it was important to find out how

much the experts agreed on the relevance each item on the questionnaire possessed when

considering a school climate assessment. Responses for three of the nine raters were

discarded from the ICC because they utilized in their responses a non-ordinal rating that

could not be quantified for agreement purposes (either 4-No opinion or 5- I dont know).

Because the remaining six

Documents

GangiTracydissertation10-20-09