Measuring Psychosocial Environments Using IndividualResponses: an Application of Multilevel Factor Analysisto Examining Students in Schools

Erin C. Dunn & Katherine E. Masyn &

Stephanie M. Jones & S. V. Subramanian &

Karestan C. Koenen

# Society for Prevention Research 2014

Abstract Interest in understanding how psychosocial envi-ronments shape youth outcomes has grown considerably.School environments are of particular interest to preventionscientists as many prevention interventions are school-based.Therefore, effective conceptualization and operationalizationof the school environment is critical. This paper presents anillustration of an emerging analytic method called multilevelfactor analysis (MLFA) that provides an alternative strategy toconceptualize, measure, and model environments. MLFA de-composes the total sample variance-covariance matrix for var-iables measured at the individual level into within-cluster (e.g.,

student level) and between-cluster (e.g., school level) matricesand simultaneously models potentially distinct latent factorstructures at each level. Using data from 79,362 students from126 schools in the National Longitudinal Study of Adolescentto Adult Health (formerly known as the National LongitudinalStudy of Adolescent Health), we use MLFA to show how 20items capturing student self-reported behaviors and emotionsprovide information about both students (within level) andtheir school environment (between level). We identified fourlatent factors at the within level: (1) school adjustment, (2)externalizing problems, (3) internalizing problems, and (4)self-esteem. Three factors were identified at the between level:(1) collective school adjustment, (2) psychosocial environ-ment, and (3) collective self-esteem. The finding of differentand substantively distinct latent factor structures at each levelemphasizes the need for prevention theory and practice toseparately consider and measure constructs at each level ofanalysis. The MLFA method can be applied to other nestedrelationships, such as youth in neighborhoods, and extended toa multilevel structural equation model to better understandassociations between environments and individual outcomesand therefore how to best implement preventive interventions.

Keywords Multilevel . Factor analysis . Schoolenvironment . School climate . Latent variable . Ecological

Introduction

Schools are one of the most important social institutions in thelives of youth. Today, schools are no longer solely formaleducational institutions, but instead are the settings wherenumerous health- and development-oriented prevention andintervention activities take place (Greenberg et al. 2003;Rones and Hoagwood 2000) and students acquire knowledge

E. C. Dunn (*)Psychiatric and Neurodevelopmental Genetics Unit, Center forHuman Genetic Research, Massachusetts General Hospital, Boston,MA, USAe-mail: edunn2@mgh.harvard.edu

E. C. DunnDepartment of Psychiatry, Harvard Medical School, Boston, MA,USA

E. C. DunnStanley Center for Psychiatric Research, The Broad Institute ofHarvard and MIT, Cambridge, MA, USA

K. E. MasynDivision of Epidemiology and Biostatistics, School of Public Health,Georgia State University, Atlanta, GA, USA

S. M. JonesHarvard Graduate School of Education, Harvard University,Cambridge, MA, USA

S. V. SubramanianDepartment of Social and Behavioral Sciences, Harvard School ofPublic Health, Boston, MA, USA

K. C. KoenenDepartment of Epidemiology, Mailman School of Public Health,Columbia University, New York, NY, USA

Prev SciDOI 10.1007/s11121-014-0523-x

and learn skills in both cognitive and social-emotional do-mains (Eccles and Roeser 2011; Jones et al. 2011).Manywell-known and successful prevention efforts have been imple-mented in schools, including Positive Behavioral Interven-tions and Support (PBIS) designed to promote school mentalhealth and prevent student disruptive behaviors (Bradshawet al. 2009). School-based interventions have become popularamong prevention scientists as schools offer a unique oppor-tunity to promote, at a population level, the health and well-being of youth. Schools serve more than 95 % of the nation’syouth for 6 hours per day (or upward of 40 % of student’swaking time during the school year) and at least 11 continuousyears of their lives (Aud et al. 2010).

In addition to providing the infrastructure necessary todeliver prevention programs, schools have been increasinglyrecognized as important environments in and of themselvesfor shaping youth health and development. Indeed, a growingnumber of studies has linked characteristics of the schoolenvironment to educational (e.g., academic performance, en-gaged learning, and drop out) and non-educational outcomes(e.g., behavioral problems and psychological well-being) (Co-hen and Geier 2010; Cohen et al. 2009). The school environ-ment (also referred to throughout this manuscript and in theliterature as school climate) can be defined as the overarchingconstruct encompassing both objective and subjective featuresof the school setting, including the following: order, safety,and discipline; peer norms, values, and expectations; the cul-ture of teaching, learning, and academics; quality of the schoolfacilities and other resources; student’s level of connectionand attachment to school; relationships between students,teachers, and staff; and collective student characteristics andbehaviors (Anderson 1982; Cohen et al. 2009; Zullig et al.2010). This paper focuses on these last two elements (socialrelationships within the school and the collective psycholog-ical and emotional characteristics and behaviors of studentswithin the school), which we refer to as the school psychoso-cial environment.

Researchers have used a variety of approaches, includingfocus groups, observations, interviews, and surveys conduct-ed with students, teachers, staff, and/or parents, to measureaspects of the school psychosocial environment. To date,school psychosocial environments have most often been mea-sured through students’ self-report. These student self-reportmeasures are easy to administer, demonstrate good psycho-metric properties, and assess several dimensions of the schoolenvironment (see for example Brand et al. 2003; Haynes et al.2001; National School Climate Center). Thus far, researcherstypically develop one of two types of variables based on thesemeasures: (1) student-level variables that capture student’sperceptions and (2) aggregated scores that capture school-wide experiences (e.g., mean levels of perceived school cli-mate within a school). These variables can be constructed withlittle difficulty and have most often been used in single-level

or multilevel analyses as observed predictors of student-leveland school-level outcomes.

Although the construction of such variables has provided asolid foundation for documenting the role of school psycho-social environments on various youth outcomes, there arechallenges associated with how school psychosocial environ-ments are currently (1) conceptualized, (2) operationalized,and (3) analyzed. These challenges impact prevention scienceresearch, as they potentially restrict the empirical refinementof etiologic theory regarding the role of school settings onstudent health and behavior and may therefore limit the de-velopment, implementation, and evaluation of school-basedinterventions designed to influence such etiological factors.First, the school psychosocial environment is often conceptu-alized solely in terms of the perceptions individuals haveabout their setting. Specifically, researchers frequently defineschool psychosocial environments in terms of the positive ornegative feelings and attitudes students have in relation totheir school. However, school environments, like any organi-zational setting, can be conceptualized not just with respect tothe perceptions of students about their school but also withrespect to the characteristics of individuals that make up theschool (e.g., students’ attitudes, beliefs, behaviors, and demo-graphic features). In other words, the social and psychologicalenvironment of an organization, including a school, can beunderstood in terms of the collection of behaviors, beliefs, andattitudes of people within the environment. Indeed, theoriesfrom organizational climate research illustrate how climatesare defined and influenced by the traits of people within thesetting (James et al. 2008). Moreover, school psychosocialenvironments are often conceptualized as a single-level phe-nomenon, either operating at the level of students or theschool. However, school psychosocial environments could beeven better understood as a two-level phenomenon, operatingat both the level of students and the school (Anderson 1982;Van Horn 2003). Thus, by conceptualizing and measuringschool psychosocial environments as a source of both school-level and student-level (co)variation, researchers will be betterable to identify new dimensions of school psychosocial envi-ronments that may be relevant for targeting and implementingprevention efforts (i.e., to individual students, to whole schools,or both) and studying the role of school settings.

Second, existing research on school psychosocial environ-ments tends to rely on single variables (e.g., single itemscorresponding to a feature of the school psychosocial envi-ronment) or composites (e.g., a single variable denoting theindividual or school-wide average for a set of items capturingthe school psychosocial environment). This is a narrow ap-proach. Instead, prevention scientists can adopt a more com-plex representation of the variables that capture school psy-chosocial environments—at both the level of students andschools—by examining multivariable systems. In this regard,factor analytic approaches are helpful. In factor analysis, a

Prev Sci

measurement model is specified, which characterizes the re-lationship between an unobserved latent factor and a set ofobserved indicator variables that are presumed to be caused bythe unobserved latent factor. Through a measurement model,factor analytic methods enable a more complex understandingof the constructs of interest, as the quality of the indicators aswell as relationships between latent factors can be examined(Brown 2006; Kline 2010). Also, since the model includes aparameter that captures unique variance, factor analyticmodels partially account for measurement error, which regres-sion models do not. Factor analytic methods are also prefera-ble to simple aggregation (e.g., calculating school-level meansdirectly from student data), as aggregation ignores student-level measurement error and within-school variability (Shinn1990) and treats all indicators as exchangeable.

Finally, and related to the second issue, most prior researchon school psychosocial environments has relied on a single-level operationalization and analytic strategy to document theeffects of school settings on a given outcome (Dunn et al.2014). For example, many multilevel studies have been con-ducted using data where the school psychosocial environmentwas assessed using surveys of students. In such instances,researchers have constructed a derived variable (Diez Roux2002) or school-level measure of the psychosocial environ-ment by averaging student responses to items on a given scaleand then subsequently averaging those individual meansacross students in the same school; these school-level meansthen serve as the predictors in subsequent multilevel analyses.Researchers have also used factor analytic approaches to de-termine whether multiple items tapping the school psychoso-cial environment can be grouped together in a common con-struct; these factor analyses are typically conducted such thatlatent factors are constructed only at the student level. Multi-level studies with either derived variables or based on factoranalytic approaches follow directly from the conceptualizationof the school psychosocial environment as a single-level phe-nomenon (i.e., operating primarily on either students or school)(Van Horn 2003). A single-level perspective is limited in that itmixes or conflates what is occurring at each level. In otherwords, a single-level perspective completely dissociates—inthe analytic model—the student behavior from the collectivebehavior in that it does not allow for the explicit modeling ofbehavior covariation at the student-level and collective behav-ior covariation at the school level. The conflation of student-and school-level processes and phenomenon may not onlyhave methodological impacts, such as model misspecificationwith respect to the factor configuration and latent structure, butit can also lead to the generation of inaccurate theories andultimately misidentified intervention targets. In particular,single-level measurement and analytic approaches may inducean ecological fallacy, such that incorrect inferences could beinferred when interpreting a school-level treatment effect onthe aggregate behavior score as evidence of effect on student-

level behavioral outcomes. To address these limitations, meth-odological techniques that allow for the specification of a fullymultilevel measurement model are needed.

This paper uses an analytic method called multilevel factoranalysis (MLFA) (Dedrick and Greenbaum 2011; Dyer et al.2005; Reise et al. 2005; Toland and De Ayala 2005) to addressthe above challenges by providing an alternative approach forconceptualizing, measuring, and modeling environments.MLFA is similar to all factor analytic methods in that it seeksto capture the shared variance among an observed set ofvariables in terms of a potentially smaller number of unob-served constructs or latent factors (Brown 2006; Kline 2010).However, MLFA differs from a traditional factor analysis inone major way: it is multilevel. Unlike a single-level explor-atory or confirmatory factor analysis, which estimates latentfactors at only one level (i.e., the individual or contextuallevel), or a random-effect factor analysis, which decomposesthe variance of the level-one factor into within and betweencomponents (Marsh et al. 2009), MLFA decomposes the totalsample variance-covariance matrix into within-cluster (i.e.,individual level, within an environment) and between-cluster(i.e., environment level) matrices and simultaneously modelsdistinct latent factor structures at each of these levels (Hox2010; Muthen 1991, 1994).

There are many methodological and practical implicationsof MLFA. By allowing for the possibility of two differentlatent factor structures at the two levels, researchers are betterable to understand the variation in structure and meaning thatexists between individuals within an environment, as well asbetween environments, rather than assuming that the factorstructure is the same at both levels. Thus, MLFA may helpresearchers avoid making the erroneous assumption that agiven set of items performs the same at each level of analysisor that a given construct means the same thing at each level ofanalysis; these assumptions would not be detected using hier-archical or multilevel modeling techniques with derived var-iables or with single-level or random-effect factor analyses.Moreover, MLFA can also be useful for generating newtheories regarding the role of environments on youth healthand development outcomes. As many large-scale preventionbased data collection efforts occur through cluster-based sam-pling, MLFA can provide an opportunity to study psychoso-cial environments using data collected at the individual level.

In this paper, we apply the MLFA method using dataascertained from students within schools. As stated previously,we focus on measuring the school psychosocial environmentusing student-level self-reports of attitudes (about themselves),behaviors, and emotions, as this domain of the school environ-ment seems more often relegated to single-level or derivedvariable approaches than some of its companion domains.However, the MLFA approach could be effectively used tomodel other domains of the school environment as well as othersettings, such as neighborhoods, hospitals, and workplaces.

Prev Sci

Methods

Participants Data for this study came from the National Lon-gitudinal Study of Adolescent to Adult Health (Add Health,formerly known as the National Longitudinal Study of Adoles-cent Health), a longitudinal survey focusing on the health andbehaviors of adolescents in grades 7–12 (Harris 2013). AddHealth, although not including instruments specifically de-signed to measure the school psychosocial environment, isuseful for this application of MLFA as it uses a nationallyrepresentative sample of adolescents and includes an array ofstudent-level measures that should directly reflect psychosocialcharacteristics of the school and the students themselves. AddHealth researchers began collecting data for a nationally repre-sentative sample in 1994–1995 (wave 1) using schools as theprimary sampling unit. To ensure that selected schools wererepresentative of US schools, researchers stratified schools bycensus region, urbanicity, size, type, and ethnic background ofthe student body (i.e., percent white) prior to systematic randomsampling. From a sampling frame of 26,666 schools, investi-gators selected a sample of 80 high schools and 52 middleschools for participation. School administrators of these 132schools were asked to administer an in-school survey to theirstudents at wave 1. Of these 132 schools, 128 (96.97 %)participated in the in-school survey, resulting in a sample of83,135 students. Given our concerns that the underlying char-acteristics of the school environment may differ for studentsattending a boarding school compared to those who did not, weeliminated one private boarding school from our analysis,which included responses from 888 students. We also removedone school that did not have demographic data reported on it;this school included responses from 61 students. Our analyticsample therefore consisted of responses from 82,186 studentswho attended 126 schools. Across the 126 schools, an averageof 652.27 students per school (SD=504.41) completed a survey(minimum=29; maximum=2,546). The analytic sample wasbalanced in terms of sex (50.4 % female; 49.6 % male) andgrade level (13.9 % grade 7; 13.3 % grade 8; 20.8 % grade 9;19.6 % grade 10; 17.2 % grade 11; and 15.2 % grade 12), wasracially/ethnically diverse (46.6 % white; 12.6 % black; 15.8 %Hispanic; 19.4 % multiracial), and included mostly native-bornstudents (90.5 % native).

Materials and Procedure

In-School Questionnaire The in-school questionnaire wascompleted by all participating students within each school. Itasked youth to self-report on a variety of topics, includingtheir health status, friendships, household structure, social anddemographic characteristics, expectations for the future, self-esteem, and school-year extracurricular activities. It containedmore than 200 items (most items were focused on relation-ships with parents and friendship network structure) and was

administered during a 45–60-min class period between Sep-tember 1994 and April 1995. Parents were notified prior to thedate the survey was administered and could advise theirchildren not to participate. Questionnaires were opticallyscanned following completion. The questionnaire consistedof predominately individual items rather than groups ofexisting measures. Response options for all items were onLikert scales, ranging from a four-point to a nine-point scale.The Likert scales captured agreement (e.g., ranging from 1=strongly agree to 4=strongly disagree) or frequency (e.g.,ranging from 0=never to 4=everyday).

We selected a subset of 21 items from the in-school ques-tionnaire for the current analyses (refer Table 1). These itemswere selected because they were hypothesized to capture thesocial and psychological characteristics of students and alsoreflect the psychosocial environment of their school. These 21items generally tapped three broad domains: (1) relationships(e.g., student’s ability to get along with teachers and otherstudents), (2) behaviors (e.g., tobacco, alcohol, and drug use;time spent on homework; truancy; and physical fighting), and(3) attitudes/feelings (e.g., beliefs about themselves and feel-ings of sadness). As our purpose was to demonstrate the utilityof the MLFA approach to specifically model school psycho-social environment, we did not consider other measures in thedata set that might reflect other domains of school climate(e.g., physical environment and teaching and learning envi-ronment) although this general technique could certainly beapplied to a larger multidomain item set.

Data Analyses Our primary analyses utilize two variations ofMLFA: multilevel exploratory factor analysis (ML-EFA) andmultilevel confirmatory factor analysis (ML-CFA). Figure 1presents a hypothetical ML-CFA with six observed indicatorvariables, a standard two-factor structure at the within level,and a standard one-factor structure at the between level. In thisfigure, there are two separate measurement models—one atthe within level (e.g., individual students within a school) andthe other at the between level (e.g., between schools). At thewithin level, the individual response for student i in school jon the mth observed indicator variable, represented with arectangle labeled ymij, is a function of one of the twostudent-specific latent factors, represented with circles labeledηW1ij and ηW2ij, a random intercept, represented by a circlelabeled νmj, and a random error term, indicated by a smallunanchored arrow pointing to ymij. The factor loadings,λW1,…,λW6, estimate the direction and size of the associationbetween the within-level latent factors and the observed var-iables. This model can be expressed in matrix notation as:

yi j ¼ ν j þΛWηWi j þ εi j; ð1Þ

where εi j ei:i:d: MVN 0;θεð Þ , ηW ei:i:d: MVN 0;ΨWð Þ , andCov(εij,ηWij)=0.

Prev Sci

At the between level, the intercept for school j correspond-ing to the mth observed indicator variable, represented by acircle labeled νmj, is a function of the school-specific latentfactor, represented with circle labeled ηBj, and a randomerror term, indicated by a small unanchored arrow pointingto νmj. The random intercept refers to the expected value ofthe indicator for school j at the mean of ηBj. The factorloadings, λB1,…,λB6, estimate the direction and size of theassociation between the between-level latent factors andthe random intercepts of the observed variables. Thisbetween-school model can be expressed in matrix notationas follows:

ν j ¼ γþΛBηB j þ ζ j; ð2Þ

where ζ j ei:i:d: MVN 0;θζ

� �, ηB ei:i:d: MVN 0;ΨBð Þ , and

Cov(ζj,ηBj)=0. In a standard ML-CFA, the assumption ofconditional independence is typically imposed such that θε

and θζ are both diagonal matrices. Combing Eqs. 1 and 2, one

can see that in a ML-CFA, responses to items by a student i inschool j are a function of student-level traits, school-leveltraits, and variability unique to student i and to school j:

yi j ¼ γþΛWηWi j þΛBηB j þ ζ j þ εi j: ð3Þ

To showcase the utility of the MLFA approach, we used asplit-sample cross-validation approach, beginning with a mul-tilevel exploratory factor analysis (ML-EFA) on the first ran-dom split of the sample (calibration sample), followed by amultilevel confirmatory factor analysis (ML-CFA), informedby the ML-EFA, on the second random split of the sample(validation sample). Researchers begin by conducting an EFAwhen the goal is to identify the factor structure underlying aset of variables; thus, an EFA is conducted when there are no apriori hypotheses about the number of latent factors or therelationships between each factor and the indicators (Brown2006; Kline 1994). This was our case because we were

Table 1 Items from the National Longitudinal Study of Adolescent Health (Add Health) in-school questionnaire

Item Original response options Revised response option

Since school started this year, how oftenhave you had trouble

1. Getting along with your teachers2. Paying attention in school3. Getting your homework done4. Getting along with other students

0=never; 1=just a fewtimes=2=about once aweek; 3=almost everyday;4=everyday; 9=multipleresponse;

0=never; 1=just a fewtimes; 2=about once aweek or more

5. In general, how hard do you try to do yourschool work well?

1=I try very hard to do mybest; 2=I try hard enough,but not as hard as I could;3=I don’t try very hard; 4=Inever try at all

1=I try very hard to do mybest; 2=I try hard enough,but not as hard as I could;3=I don’t try very hard or Inever try at all

During the past 12 months, how often did you6. Smoke cigarettes7. Get drunk8. Skip school without an excuse

0=never; 1=once or twice;2=once a month or less; 3=2or 3 days a month; 4=onceor twice a week; 5=3 to 5days a week; 6=nearlyeveryday

0=never; 1=once or twice;2=more than twice amonth

In the past month, how often9. did you feel really sick10. did you feel depressed or blue11. did you have trouble relaxing12. were you moody13. did you cry a lot14. were you afraid of things

0=never; 1=rarely;2=occasionally; 3=often;4=everyday

0=never; 1=rarely oroccasionally; 2=often oreveryday

How strongly do you agree or disagree with eachof the following statements:

15. I have a lot of good qualities.16. I have a lot to be proud of.17. I like myself just the way I am.18. I feel like I am doing everything just right.19. I feel socially accepted.20. I feel loved and wanted.

1=strongly agree; 2=agree;3=neither agree nordisagree; 4=disagree;5=strongly disagree

0=strongly agree/agree;1=neither; 2=stronglydisagree/disagree

21. During the past year, how often have yougotten into a physical fight?

0=never; 1=1or 2 times;2=3–5 times; 3=6 or 7 times;4=more than 7

0=never; 1=once or twice;2=more than twice amonth

All of these items were taken from the in-school questionnaire

Prev Sci

seeking to group together items that were not already part ofan existing scale. ML-EFA involves an EFA approach appliedseparately but simultaneously to the within-level item covari-ance matrix and the between-level item covariance matrix. Ina CFA, researchers have a priori hypotheses about the numberof factors and the factor configuration and want to test thevalidity of a hypothesized model by evaluating the model dataconsistency. For both analyses, we used a categorical factoranalysis, a type of analysis designed for ordinal data. Incontrast to a continuous factor analysis, a categorical factoranalysis does not require that indicators are continuous or thatany distributional assumptions such as normality are met(Flora and Curran 2004). Both EFA and CFA models withcategorical indicators use the sample-based polychoric corre-lation matrix for the observed indicators (in essence, thecorrelations that would have been observed between the ordi-nal indicators if their underlying continuous responses wereinstead measured).

To conduct the ML-EFA and ML-CFA, we began byrandomly dividing the sample of students, all of whom haddata on at least one item, into two halves; this split was madeafter stratifying students by school, to ensure that schoolassignment was distributed identically across the two groups.In the first half (calibration sample) we conducted a ML-EFA;in the second half (validation sample), we conducted a ML-CFA. We also used the ML-EFA to trim the item set into asmaller number of indicators and used the ML-CFA to cross-validate theML-EFA results in a second split-half sample. Useof split samples is common practice in factor analysis.

Across all models, we evaluated goodness-of-fit using themodel chi-squared test, normed comparative fit index (CFI;Bentler 1990), root mean square error of approximation(RMSEA; Steiger 1990), and the standardized root meansquare residual (SRMR; Muthén and Muthén 1998). Thesestatistics provide information about the overall fit of the modeland the model data consistency (comparing the model-

Fig. 1 Multilevel factor analysis(MLFA). The components of aMLFA are illustrated above. Thisfigure presents two separatemeasurement models—the withinlevel (i.e., individuals within anenvironment) and the betweenlevel (i.e., betweenenvironments). At the withinlevel, two individual-specificlatent factors, ηW1ij and ηW2ij,influence the individual’sresponses on six items (orobserved variables) (y1ij,y2ij,…,y6ij). At the between level, oneschool-specific latent factor,ηBj,influences the school responsemeans,νmj, that in turn influencethe individual’s responses. Thedirection and size of the factorinfluences at each level aredescribed by the λWs and λBs,respectively. Each item ismeasured imperfectly and thushas a residual indicated by thesmall vertical arrows in thediagram. The residual refers to theunique variance in the item notexplained by or related to thelatent factor; this unique varianceis a combination of measurementerror and other unique sources ofvariability

Prev Sci

estimated within- and between-level correlation matrices tothe within- and between-level sample correlations). Accept-able model fit was determined by a non-significant chi-squared test, CFI values greater than 0.95, and RMSEA andSRMR values below 0.10 (Kline 2010). The CFI, RMSEA,and SRMR values were given more emphasis than the chi-squared test, as the chi-squared test statistic is often significant(implying significant misfit of the model to the data) when thesample size is large. In theMLFA, a SRMR is provided at boththe within and between levels. There are no established guide-lines for interpreting the SRMR at the between level. Thus, weconsidered the guidelines typically applied for single-levelanalyses (≤0.10) and also examined the residuals for thebetween-level correlation matrix, which can signal particularregions of and systematic patterns of misfit.

We began these analyses by collapsing response options toall items. We did this to eliminate response categories thatwere infrequently endorsed and thus provided little informa-tion about individual-level variability. We created three re-sponse options for all items; the collapsed response optionsappear in the right-side column of Table 1, alongside theoriginal response options. Examination of the polychoric cor-relation matrix for collapsed items was nearly identical to theoriginal scaling, confirming that a negligible amount of infor-mation was lost by the category collapsing.

We conducted all analyses using Mplus software version6.1. Mplus handles missing data, under the missing at randomassumption (MAR), using the weighted least squares withmean and variance adjustment (WLSMV) estimator, whichallows missingness to be a function of the observed covari-ates, but not observed outcomes, as is the case for full infor-mation maximum likelihood. When there are no covariates inthe model, this is analogous to pairwise present analysis(Muthén and Muthén 1998). We also calculated intraclasscorrelation coefficients (ICCs) for each item, which indicatethe proportion of variance in each observed indicator variablethat is due to differences between schools.

These analyses were guided by both theory, broadly relatedto child development and specifically to school environments,and empirical findings. In both theML-EFA andML-CFA, weused WLSMV as the estimation method. We also includedweights and stratification variables to adjust for student non-response, the unequal probability of selection of schools, andaccount for the clustering of students in school. In the ML-EFA, we applied a geomin oblique rotation method, allowingthe factors to be correlated. To determine the final model forthe ML-EFA, we followed an iterative process whereby wefirst focused on the within-level results and then proceeded tofocus on the between-level results. Specifically, we looked at avariety of different within-level solutions (e.g., between 1 and7 factors) that were generated with an unstructured betweenlevel; the unstructured between level is a model that is fullysaturated (perfect fit model) at the between level, meaning that

all random item intercepts are allowed to correlate with eachother. After deciding on one or two candidate factor enumer-ations based on overall fit, we then examined several differentbetween-level solutions where the within level was unstruc-tured. Essentially, we considered both the between-level andwithin-level results so that we would not mistakenly excludean item that may have disparate performance across levels(e.g., performing poorly at the within level, but well on thebetween level). For the ML-EFA, we used scree plots, numberof eigenvalues greater than one, and model goodness-of-fitstatistics to help guide us in deciding the final number offactors to retain at both the within and between levels. Afterreaching a small number of candidate within- and between-level factor enumerations, we examined the ML-EFA solu-tions for all possible combinations (e.g., three within-levelfactors with one between-level factor and three within-levelfactors with two between-level factors), evaluating each com-bined within- and between-level factor enumeration accordingto overall fit as well as substantive interpretability and utility.We examined the performance of each individual item at boththe within and between levels because it is at this juncture inconventional single-level EFA that items may be trimmed ifthey fail to load significantly on any of the factors or if theyload significantly on all the factors. We considered trimmingitems based on magnitude of factor loadings, statistical signif-icance, ratio of the smallest to largest factor loading, etc. Usingthe final ML-EFA solution, we fit a ML-CFA to the validationhalf of the split sample, using a model specification whereinany non-statistically significant cross-loadings or cross-loading smaller that 0.32 (standardized) in the ML-EFAwerefixed at zero in the ML-CFA (Tabachnick and Fidell 2001).We trimmed items and cross-loadings to develop a parsimo-nious solution; this is commonplace in factor analysis (Brown2006; Kline 2010). Our decision to exclude items was madeon a similar basis: items that loaded strongly with statisticallysignificant loadings greater than 0.50 on one item at the withinand/or between level or items with statistically significantcross-loadings of at least 0.32 on two items at both the withinand/or between level were retained. Since the ML-EFA solu-tion provides standardized factor loadings by default, wepresented and interpreted standardized factor loadings in theML-CFA.

We found a small degree of missingness across the items,ranging from a low of 5.13% (does not try hard in school) to ahigh of 12.95 % (feeling accepted). We included participantswith data on at least one item in all analyses, resulting in a totalsample of 79,362. The split-half samples (“sample 1,” thecalibration sample used for the ML-EFA; “sample 2,” thevalidation sample used for the ML-CFA) were balanced ondemographics, including sex (50.04 % female sample 1;50.19 % female sample 2), grade level (13.38 % grade 7sample 1; 13.57% grade 7 sample 2; 15.52% grade 12 sample1; 15.40 % grade 12 sample 2), and race/ethnicity (18.83 %

Prev Sci

Hispanic sample 1; 18.80 % Hispanic sample 2). Each samplealso contained a similar number of students in each school(sample 1: n=39,669; sample 2: n=39,693).

Results

Intraclass Correlation Coefficients

Table 2 presents ICC estimates from each split sample. ICCsranged from small to large in magnitude, with 0.9 % of thevariation for feeling afraid being due to differences acrossschools in sample 1, compared to 17.5 % for getting drunk.The remaining variance (1 minus the ICC) was due to differ-ences between students within a school. ICCs were verysimilar across the split samples. Although most of the vari-ability in these items was due to differences within, rather thanbetween schools, there was considerable variability among theindicators as to the proportion of variation explained betweenschools. The discrepancy in ICC values across items suggeststhat school-level sources of variation do not operate uniformlyacross items. These differences in relative student- and school-

level variation also hint at possible differences in the relation-ship between these items at the two levels of analysis.

Correlations

Table 3 presents the within- and between-level correlations, inthe first randomly split sample (the results from the secondrandomly split sample are very similar). Correlations amongindicators were as high as r=0.720 at the within level and r=0.924 at the between level. The average correlations at thewithin and between levels were very similar (mean within-level correlation=0.250, mean between-level correlation=0.243). Most notable, and underscoring the value of theMLFA approach, was the finding that there were differences(in both magnitude and direction) in the correlations betweenitems at the within and between levels. For example, the itemsfight and drunk were correlated 0.39 at the within level, but−0.53 at the between level.

Multilevel Factor Analysis

Multilevel Exploratory Factor Analysis We began byconducting a ML-EFA in the first randomly divided sample

Table 2 Estimated intraclass correlation coefficients (ICCs) for each observed indicator variable in each split sample (n=79,362)

Intraclass correlation coefficient (ICC)

Observed indicator variables Sample one(n=39,669)

Sample two(n=39,693)

Trouble paying attention 0.021 0.034

Trouble getting homework done 0.021 0.029

Trouble getting along with teachers 0.038 0.035

Trouble getting along with other students 0.069 0.052

Skipping school 0.126 0.111

Does not try hard in school 0.071 0.068

Getting into a physical fight 0.036 0.035

Smoking cigarettes 0.092 0.141

Getting drunk 0.175 0.202

Feel blue 0.064 0.040

Cried a lot 0.019 0.014

Were moody 0.043 0.036

Trouble relaxing 0.018 0.019

Afraid of things 0.009 0.012

Not doing everything right 0.032 0.030

Not proud of self 0.023 0.027

Does not like oneself 0.026 0.027

Does not feel socially accepted 0.010 0.009

Feels unloved and unwanted 0.020 0.015

Does not have good qualities 0.017 0.018

Feel really sick 0.011 0.014

ICC refers to the proportion of variance in the observed variable that is due to differences across schools

Prev Sci

Tab

le3

Correlatio

nsam

ongthe21

indicatorvariablesatthewith

inlevel(belowthediagonal)andthebetweenlevel(abovethediagonal)

12

34

56

78

910

1112

1314

1516

1718

1920

21

1TRTEACH

1.00

0.50

0.62

0.91

−0.58

0.83

−0.14

−0.37

−0.44

−0.58

−0.58

−0.59

−0.43

−0.32

0.29

−0.23

−0.03

0.05

−0.34

−0.60

−0.65

2TRPA

YAT

0.63

1.00

0.80

0.34

0.23

0.08

0.39

0.15

0.26

0.0.21

0.37

0.16

−0.18

0.03

0.13

0.46

0.44

−0.10

0.09

0.19

0.21

3TRHOMEW

0.56

0.73

1.00

0.60

−0.02

0.32

0.46

0.01

0.07

−0.04

0.03

−0.10

−0.21

−0.07

0.25

0.29

0.39

0.04

0.11

−0.10

−0.13

4TRALONG

0.57

0.59

0.57

1.00

−0.74

0.90

−0.33

−0.42

−0.63

−0.85

−0.72

−0.78

−0.68

−0.39

−0.07

−0.27

0.14

0.15

−0.43

−0.73

−0.73

5TRYHARD

0.17

0.23

0.24

0.08

1.00

−0.72

0.63

0.76

0.78

0.88

0.82

0.88

0.55

0.27

0.03

0.40

0.35

0.18

0.56

0.85

0.88

6FIG

HT

0.27

0.17

0.16

0.24

0.22

1.00

−0.30

−0.29

−0.53

−0.76

−0.63

−0.68

−0.45

−0.34

0.19

−0.22

0.09

0.19

−0.16

−0.61

−0.70

7TRUANT

0.20

0.22

0.22

0.13

0.39

0.34

1.00

0.65

0.72

0.59

0.57

0.52

0.32

0.03

0.27

0.43

0.48

0.08

0.57

0.52

0.51

8CIG

0.21

0.20

0.18

0.13

0.33

0.33

0.51

1.00

0.91

0.54

0.79

0.77

0.31

0.19

0.29

0.61

0.54

0.17

0.63

0.71

0.74

9DRUNK

0.21

0.19

0.16

0.11

0.36

0.39

0.60

0.68

1.00

0.80

0.79

0.78

0.56

0.27

0.44

0.63

0.37

0.02

0.60

0.71

0.73

10BLUE

0.07

0.15

0.11

0.11

0.14

0.07

0.21

0.25

0.21

1.00

0.82

0.92

0.70

0.26

0.32

0.51

−0.09

−0.09

0.57

0.86

0.84

11RELAX

0.10

0.17

0.13

0.12

0.14

0.11

0.21

0.22

0.19

0.63

1.00

0.87

0.62

0.58

−0.02

0.56

0.45

0.22

0.67

0.84

0.86

12MOODY

0.09

0.16

0.10

0.12

0.14

0.10

0.22

0.24

0.22

0.61

0.56

1.00

0.66

0.27

0.23

0.48

0.13

0.05

0.56

0.87

0.90

13CRY

0.03

0.09

0.07

0.08

0.00

−0.01

0.13

0.19

0.14

0.62

0.48

0.55

1.00

0.40

0.54

0.49

0.02

0.21

0.72

0.62

0.64

14AFR

AID

0.06

0.13

0.11

0.10

0.03

−0.01

0.10

0.12

0.08

0.48

0.45

0.44

0.60

1.00

0.07

0.28

0.23

0.16

0.41

0.43

0.37

15SICK

0.11

0.13

0.11

0.11

0.05

0.11

0.17

0.17

0.15

0.35

0.35

0.36

0.38

0.30

1.00

0.46

−0.03

0.02

0.32

0.14

0.01

16ACCEPT

0.05

0.10

0.08

0.17

0.13

0.07

0.10

0.12

0.05

0.37

0.32

0.27

0.25

0.24

0.20

1.00

0.68

0.38

0.69

0.55

0.61

17WANTED

0.11

0.14

0.11

0.15

0.22

0.17

0.18

0.21

0.16

0.39

0.34

0.28

0.23

0.21

0.20

0.69

1.00

0.68

0.91

0.60

0.43

18GQUAL

0.09

0.14

0.10

0.13

0.22

0.11

.19

0.23

0.17

0.31

0.29

0.23

0.23

0.20

0.19

0.55

0.59

1.00

0.69

0.38

0.28

19PR

OUD

0.13

0.17

0.16

0.14

0.27

0.14

0.24

0.28

0.22

0.39

0.33

0.28

0.26

0.23

0.23

0.58

0.65

0.72

1.00

0.80

0.58

20LIK

ESLF

0.05

0.13

0.10

0.08

0.19

0.04

0.14

0.20

0.14

0.44

0.35

0.35

0.34

0.29

0.24

0.56

0.55

0.56

0.63

1.00

0.90

21RIG

HT

0.06

0.17

0.14

0.08

0.28

0.06

0.19

0.20

0.17

0.43

0.36

0.36

0.31

0.30

0.22

0.55

0.53

0.49

0.56

0.72

1.00

TRTEACHtroublegetting

alongwith

teachers,T

RPA

YATtroublepaying

attention,TR

HOMEW

troublegetting

homew

orkdone,T

RALONGtroublegetting

alongwith

otherstudents,T

RYHARDdoes

not

tryhard,F

IGHTgetting

intophysicalfights,T

RUANTskipping

school,C

IGsm

okingcigarettes,DRUNKgetting

drunk,BLUEfeelblue,R

ELAXtroublerelaxing,M

OODYweremoody,C

RYcriedalot,

AFRAID

afraidof

things,SICKfeelreallysick,A

CCEPTdoes

notfeelsocially

accepted,W

ANTEDfeelsunlovedandunwanted,GQUALdoes

nothavegood

qualities,P

ROUDnotproud

ofself,LIKESL

Fdoes

notlikeoneself,RIG

HTnotd

oing

everything

right

Prev Sci

using the 21 items. As previously noted, we began byconducting a ML-EFA, rather than ML-CFA due to the lackof prior reporting of the number of factors underlying theseitems at the student or school level. To determine the numberof factors, we looked at the scree plot and examined thenumber of eigenvalues greater than one (following Kaiser’scriteria). We found four eigenvalues greater than one at thewithin level, suggesting a four-factor solution; the scree plotsuggested a five-factor solution. We also found four eigen-values greater than one at the between level. The between-level scree plot suggested a four-factor solution.

In examining the results for between a one- and five-factorsolution at the within level, with an unstructured betweenlevel, we found the models with less than four factors hadinadequate model fit. We therefore more closely examined thefour- (χ2=5,293.889; df=132; p<0.00001; CFI=0.979;RMSEA=0.031; SRMRwithin=0.029) and five-factor within-level solutions (χ2=2547.780; df=115; p<0.00001; CFI=0.990; RMSEA=0.023; SRMRwithin=0.021). In evaluatingthese models, we concluded that the four-factor model wasthe best fitting both empirically and conceptually. It had goodfit statistics and four interesting and distinct factors. In con-trast, although the fit of the five-factor solution was good, thefifth factor was not meaningful, as it consisted entirely ofcross-loadings and did not have a sufficient number of itemsper factor to yield a distinct factor. Thus, on the basis ofempirical findings and theoretical insights, we chose thefour-factor solution as our within-level solution.

We next examined the results for between a one- and four-factor solutions at the between level, within an unstructuredwithin level. Here, we found the models with less than threefactors had inadequate model fit. However, the three- (χ2=247.819; df=150; p<0.00001; CFI=1.000; RMSEA=0.004;SRMRbetween=0.082), and four-factor solutions (χ2=162.346; df=132; p=0.0374; CFI=1.000; RMSEA=0.002;SRMRbetween=0.056) did have good fit. Upon closer inspec-tion of the three- and four-factor solutions, we found the three-factor solution provided more meaningful information thanthe four-factor solution. Specifically, the fourth factor provid-ed by the four-factor solution did not provide a unique factor;it consisted entirely of cross-loadings. We therefore chose thethree-factor solution as our between-level solution.

Before proceeding to the ML-CFA, we examined the ML-EFA model with four-factor within and three-factor betweensolution and trimmed any items that lacked convergent valid-ity at both the within and between levels. Specifically, weconsidered the deletion of the item sick (“In the past month,how often did you feel really sick”), as this item had lowloadings at both the within and between levels with severallarge correlation residual values at both levels. We reran theML-EFA excluding the item sick to evaluate whether themodel fit and functioning of other items would change.

Results of the sensitivity analysis revealed that the fit of theoverall model was comparable after removing the item sick(χ2=5,566.936; df=249; p<0.0001; CFI=0.978; RMSEA=0.023; SRMRwithin=0.029; SRMRbetween=0.065). However,the SRMRbetween statistic decreased from 0.082 to 0.065 (adecline of 21 %). The functioning of the remaining itemswas the same across the two models (with and withoutsick). Given that the removal of this item did not affectthe functioning of the remaining items and its lowloading relative to other items on the same factor, wedecided to proceed by removing the item sick from ouranalyses. Although there were some items that per-formed poorly at either the within or between level,these remaining items were not considered for removalas all of them performed well in the factor structure forat least one of the levels. For example, the item “afraidof things” had a relatively weak loading at the betweenlevel, but a relatively high loading at the within level.

Final ML-EFA Solution Table 4 presents the rotated factorloadings for the final ML-EFA solution. At the within level,we named each factor as follows: (1) “school adjustment” torefer to the extent to which students report having difficultyadapting to the role of being a student; (2) “externalizing” tocorrespond to externalizing symptoms; (3) “internalizing” todescribe internalizing symptoms; and (4) “self-esteem” torefer to students’ negative judgments of and attitudes towardthemselves. Each factor consisted of at least three standard-ized loadings above 0.39. High factor loadings indicate a highreliability of that item as an indicator of the correspondingfactor. The factors were modestly correlated with one another,ranging from r=0.14 (for school adjustment with internaliz-ing) to r=0.44 (for internalizing with self-esteem). The com-munalities, which refer to the proportion of an indicator’s totalvariance that is accounted for by the factor solution, rangedfrom a low of 24.1 % (for fighting) to a high of 78.9 % (fordrunk). Simple structure was generally achieved, as mostitems did not cross-load (e.g., the item did not have a signif-icant loading on more than one factor). However, there weresome items that had cross-loadings at the between level; thiswas expected given the noticeably higher correlations amongthe items at the between level compared to the within level.This reinforces the need for an analytic model that does notrequire a simple structure at either level or the same factorenumeration and configuration.

At the between level, we named the three latent factors asfollows: (1) “collective school adjustment” to refer to theshared variation between random intercepts at the school levelfor a nearly identical item set to the school adjustment factor atthe within level, with the addition of truant and feeling un-loved/unwanted at the between level; (2) “psychosocial envi-ronment” to refer to the shared variation between randomintercepts for nearly all the item representing the collective

Prev Sci

behaviors, attitudes, emotions, and relations at the schoollevel; and (3) “collective self-esteem” to refer to the sharedvariation between random intercepts at the school level for anearly identical item set to the self-esteem factor at the withinlevel, with the exclusion of right at the between level. As thevalues of the loadings for the school adjustment and self-esteems factors at the within and between levels weresimilar in sign but not identical in pattern or magnitude,we use the same general labels (i.e., “school adjustment”and “self-esteem”), but modify with the term “collective”rather than “aggregate” in an effort to differentiate thesefactors from a derived variable or aggregate approach. Allthree between-level factors are similarly correlated withone another (r=0.20−0.23). Communalities ranged from49.6 % (crying) to a 98.6 % (feeling blue).

The results of the ML-EFA suggested three findings. First,student reports of more problems related to functioning in

school (e.g., trouble paying attention and trouble getting alongwith teachers) were driven by both a student’s own underlyinglevel of school adjustment and membership in schools withhigher average levels of school adjustment problems acrossthe student population. Similarly, student reports of lowerevaluations of self-worth (e.g., not liking oneself and feelingunloved) were driven by both a student’s own underlying levelof self-esteem and membership in schools with lower averagelevels of self-esteem across the student body. Student negativereports across nearly all the socioemotional and behavioralitems were driven by the students’ own underlying levels ofschool adjustment, externalizing, internalizing, and self-esteem problems and membership in schools with poorerpsychosocial environments. Interestingly, three of the itemsthat load positively on the within-level factors loaded nega-tively on the between level psychosocial environment factor,specifically trteach (trouble getting along with teachers),

Table 4 Factor loadings of items for the multilevel exploratory factor analysis (ML-EFA)

Within-level Between-level

Factor 1 Factor 2 Factor 3 Factor 4 Factor 1 Factor 2 Factor 3Schooladjustment

Externalizing Internalizing Self-esteem Collectiveschooladjustment

Psychosocialenvironment

Collectiveself-esteem

Trouble paying attention 0.861 0.000 0.040 0.012 0.980 0.009 −0.161Trouble getting homework done 0.808 −0.006 0.010 0.002 0.903 −0.289 −0.009Trouble getting along with teachers 0.731 0.077 −0.026 −0.040 0.616 −0.797 −0.010Trouble getting along with other students 0.722 −0.035 0.010 0.024 0.523 −1.013 0.167

Skipping school 0.050 0.670 0.026 0.028 0.488 0.508 0.040

Does not try hard in school 0.089 0.390 −0.082 0.204 0.080 0.890 0.011

Getting into a physical fight 0.179 0.418 −0.079 0.015 0.261 −0.868 0.308

Smoking cigarettes 0.002 0.730 0.089 0.022 0.199 0.664 0.215

Getting drunk −0.044 0.905 0.035 −0.055 0.244 0.817 −0.016Feel blue −0.016 0.089 0.749 0.106 0.064 1.007 −0.273Cried a lot −0.010 −0.065 0.831 −0.051 −0.228 0.679 0.134

Were moody 0.004 0.135 0.702 0.000 0.016 0.950 −0.100Trouble relaxing 0.024 0.102 0.660 0.075 0.118 0.871 0.123

Afraid of things 0.060 −0.103 0.670 −0.005 −0.124 0.363 0.198

Not doing everything right −0.010 −0.022 0.153 0.710 −0.026 0.892 0.151

Not proud of self 0.012 0.113 −0.030 0.802 0.018 0.489 0.769

Does not like oneself −0.044 −0.049 0.150 0.769 −0.051 0.834 0.330

Does not feel socially accepted 0.042 −0.107 0.048 0.739 0.264 0.427 0.379

Feels unloved and unwanted 0.022 0.023 0.006 0.761 0.353 0.011 0.904

Does not have good qualities −0.011 0.089 −0.077 0.790 −0.093 −0.076 0.828

Factor correlations 1.000 1.000

0.277 1.000 0.204 1.000

0.139 0.194 1.000 0.217 0.227 1.000

0.177 0.251 0.444 1.000

All factor loadings in an EFA are standardized. High EFA loadings appear in bold

χ2 =5566.936; df=249; p<0.0001; CFI=0.978; RMSEA=0.023; SRMRwithin=0.029; SRMRbetween=0.065

Prev Sci

tralong (trouble getting along with other students), and fight(getting into a physical fight). This suggests that there may beelements of the school psychosocial environment, such aslevels of control and coercion, that may attenuate overt ag-gression and social discord while also exacerbating engage-ment, internalizing, and self-valuing problems across the stu-dent body.

We reran the final ML-EFA stratified by school type (mid-dle school versus high school) and also stratified by specificgrade levels and found the pattern and direction of loadings atboth the within and between levels to be robust, suggestingthat our results were not confounded by age.

As shown in Table 4, there were six items that cross-loadedon the between level. Additionally, as shown in Table 4, not allitems loaded strongly on factors at both the within and be-tween levels. For example, the item afraid loaded quite highlyon the third within-level factor (loading=0.670), but quite lowon the between-level factors (the highest loading it had was0.363). Conversely, and as noted previously, the item tryhardloaded modestly at within level (loading=0.390), but veryhighly at the between level (loading=0.890). The same wasalso true for the item fight (within loading=0.418; betweenloading=−0.868). Moreover, while the first and third factor onthe between level were nearly the same in loading pattern tothe within level, the values of the loadings were distinct (note:fitting a ML-CFA model constraining the loadings for theschool adjustment items and self-esteem items to be equalacross levels resulted in a significant decrement in fit andoverall poor fit to the data). Given the value and direction ofthe loadings for the psychosocial environment factor, it wasnot merely a simple convergence of within level factors at thebetween level (in other words, fitting a ML-CFAmodel with afour-factor simple structure at the between level matching thewithin level resulted in a significant decrement in fit andoverall poor fit to the data). This emphasizes that not onlycan items function differently when there is a similar factorstructure at the within and between levels, but also that thefactor structure can be distinctly different at each level.

Multilevel Confirmatory Factor Analysis With the 20 vari-ables retained from our ML-EFA, we conducted aML-CFA inthe second randomly divided sample (validation sample). Wespecifically fit a four-factor within and three-factor betweensolution, seeking to validate the ML-EFA results. As shown inTable 5 (and Fig. 2), the fit of the ML-CFA was good (χ2=6,138.098; df=326; p<0.0001; CFI=0.975; RMSEA=0.021;SRMRwithin=0.051; SRMRbetween=0.120). Factor load-ings in the CFAwere similar to the EFA. Although they wereslightly higher in some cases, this was expected given thatCFA estimates tend to be higher as a result of fixing cross-loadings to zero. Only a few indicators showed a notabledifference between the EFA and CFA. For example, at thebetween level, the cross-loadings for not feeling socially

accepted and feeling unwanted were nearly zero and notstatistically significant.

Discussion

The purpose of this paper was to apply a fully multilevelstrategy to model school psychosocial environment usingindividual-level data. Results of this study illustrate thestrengths of MLFA in settings where individual-level datamay capture one or more distinctly different constructs at eachlevel of analysis. This practical illustration also showcases thebroader utility of MLFA as an analytic tool that could beuseful to prevention scientists when measuring and modelingindividual and environmental-level features on the basis ofdata collected from individuals.

We found evidence to suggest the factor structure underly-ing observed variables may differ at the level of students andschools. That is, we found four latent factors at the within level(school adjustment, externalizing symptoms, internalizingsymptoms, and self-esteem). Three factors emerged at theschool level, two of which represented school-level collectiveanalogues of the individual-level (school adjustment and self-esteem) factors and one of which represented a distinct con-struct of psychosocial environment. The finding that therewere different latent factor structures at the student and schoollevels highlights the need for prevention theory and practicerelated to schools (and other contexts targeted by preventionscientists, such as neighborhoods) to separately consider andmeasure phenomenon at each level of analysis. In otherwords, our results underscore the need to consider a fullymultilevel measurement model. Had we assumed the indica-tors of student attitudes and behaviors covaried in the sameway at the school level as they do at the student level, wewould have simply specified a four-factor model at the withinlevel with random factor intercepts at the between level. In sodoing, we would have produced a poorly fitting measurementmodel.

Moreover, our finding that there were differences in thesign of the relationships between the school-level psychoso-cial environment factor and the school-level social relation-ship indictors compared to the factor loadings at the studentlevel is also important. These results suggest that if we found aschool-based intervention that positively affected the school-level psychosocial environment factor but ignored the differ-ent factor structure at the student level, we might incorrectlyinfer that the positive effect would translate to more positiveobserved outcomes for all the psychosocial indicators at thestudent level.

In terms of prevention science practice, these findings sug-gest that interventions may need to be tailored to the specificlevel where change is intended. Thus, it may not be reasonable

Prev Sci

to assume an intervention operating at one level will have“trickle-up” or “trickle-down” effects on the other, non-directly intervened upon level. For example, targeting the self-esteem of individual students could influence the levels ofschool self-esteem, as these factors share common items. Incontrast, targeting the psychosocial environment of the schoolcould impact a broad range of student attitudes and behaviors,although some changes that may appear favorable with respectto certain problems (e.g., overt aggression and relational aggres-sion) could negatively impact other psychosocial outcomes.

Beyond its conceptual and methodological advantages,there are several applied benefits of MLFA (Dunn et al.2014). Most notable is the fact that MLFA can be easilyincorporated into research studies where individuals are sam-pled through clustering methods and items are collected fromstudents that could provide information about environment-level phenomenon. Many prevention and intervention studies

focused on children and adolescents already collect data fromyouth nested in school and neighborhood environments.However, very few explicitly study how the school environ-ment, in particular, is linked to youth outcomes. UsingMLFA,researchers can model the effects of contexts on youth out-comes, even when data were not designed for such purposes.Thus, MLFA provides an opportunity for prevention re-searchers to study psychosocial environments by using datacollected from individuals. This may allow researchers to usefewer resources, while making the most out of their datacollection efforts. This is particularly salient in today’s eco-nomic and political context, where data collection efforts inschools have been constrained by shrinking educational bud-gets and a culture of high stakes testing.

Given the limited number of studies published usingMLFA, we offer a few recommendations. First, as notedpreviously, researchers using the MLFA should trim items

Table 5 Standardized factor loadings of items for the multilevel confirmatory factor analysis (ML-CFA)

Within level Between level

Factor 1 Factor 2 Factor 3 Factor 4 Factor 1 Factor 2 Factor 3School adjustment Externalizing Internalizing Self-esteem Collective

schooladjustment

Psychosocialenvironment

Collectiveself-esteem

Trouble paying attention 0.859 1.000

Trouble getting homework done 0.806 0.756

Trouble getting along with teachers 0.718 0.664 −0.743Trouble getting along with other students 0.726 0.533 −0.883Skipping school 0.700 0.261 0.618

Does not try hard in school 0.566 0.936

Getting into a physical fight 0.442 −0.616Smoking cigarettes 0.784 0.776

Getting drunk 0.820 0.825

Feel blue 0.857 0.945

Cried a lot 0.735 0.701

Were moody 0.718 0.935

Trouble relaxing 0.720 0.913

Afraid of things 0.621 0.630

Not doing everything right 0.785 0.777

Not proud of self 0.825 0.367 0.766

Does not like oneself 0.816 0.711 0.250

Does not feel socially accepted 0.732 ns 0.819

Feels unloved and unwanted 0.761 ns 0.992

Does not have good qualities 0.741 0.755

Factor correlations 1.000 1.000

0.404 1.000 0.119 1.000

0.184 0.266 1.000 0.559 0.251 1.000

0.209 0.326 0.541 1.000

Notes. χ2 =6138.098; df=326; p<0.0001; CFI=0.975; RMSEA=0.021; SRMRwithin=0.051; SRMRbetween=0.120. Results shown fix two Heywoodcases (non-significant negative residuals) at zero

Prev Sci

only after examining how each item performs on both levelsso that items are only trimmed if it does not function at bothlevels. It would be problematic to eliminate an item on thebasis of its loading at only one level; both levels must beexamined simultaneously. Otherwise, there is a risk of remov-ing an item that has a weak loading on one level, but a strongloading on another. Second, although it may be tempting touse techniques to account for the lack of independence in theobservations (e.g., the type=complex command in Mplus),rather than conducting a MLFA, using such a model undercircumstances where there are distinct measurement structuresat each level may lead to serious model misspecification.

This study has several strengths. Data come from a large,nationally representative survey of youth. We therefore had avery large sample size from which to conduct these analysesand can generalize results to diverse populations. However,there were some limitations. For example, although the sam-ple of students was large, the number of schools they weredrawn from was moderate. In addition, we intentionally lim-ited our item set to those specifically hypothesized to reflectthe school psychosocial environment. However, a more ex-pansive model including items spanning the other domains ofschool climate could be specified. By examining more tradi-tional school climate constructs, in combination with the

Fig. 2 Results of multilevelconfirmatory factor analysis (ML-CFA). Latent factors are denotedby circles, observed indicators byrectangles. TRPAYAT troublepaying attention; TRHOMEWtrouble getting homework done,TRTEACH trouble getting alongwith teachers, TRALONG troublegetting along with other students,TRUANT skipping school, TRYHARD does not try hard, FIGHTgetting into physical fights, CIGsmoking cigarettes, DRUNKgetting drunk, BLUE feel blue,CRY cried a lot, MOODY weremoody, RELAX trouble relaxing,AFRAID afraid of things, RIGHTnot doing everything right,PROUD not proud of self,LIKESLF does not like oneself,ACCEPT does not feel sociallyaccepted,WANTED feels unlovedand unwanted, GQUAL does nothave good qualities

Prev Sci

constructs identified here, researchers could develop newinsights that expand our understanding of school climate.Given the subjective nature of factor analysis, there may alsobe limitations to how we labeled each factor. Of course, aswith any factor analysis, our final model is by no means theonly latent variable model that would be consistent with thisdata. Arriving at and cross-validating the MLFAwith this dataneither proves the existence of these particular factors norvalidates our labeling and substantive interpretation of thefactors. Future research is needed to identify whether theseare the best labels for these factors.

In summary, this study contributes to the literature byshowing how data collected from individuals can be used toprovide information about the settings to which they belong.The MLFA method provides researchers with a unique tool toguide the development of theory, research, and practice onschool and other environments.

Acknowledgments This research uses data from Add Health, a pro-gram project directed by Kathleen Mullan Harris and designed by J.Richard Udry, Peter S. Bearman, and Kathleen Mullan Harris at theUniversity of North Carolina at Chapel Hill, and funded by grant P01-HD31921 from the Eunice Kennedy Shriver National Institute of ChildHealth andHuman Development, with cooperative funding from 23 otherfederal agencies and foundations. Special acknowledgment is due RonaldR. Rindfuss and Barbara Entwisle for assistance in the original design.Information on how to obtain the Add Health data files is available on theAdd Health website (http://www.cpc.unc.edu/addhealth). No directsupport was received from grant P01-HD31921 for this analysis. VisitAdd Health online: https://www.cpc.unc.edu/projects/addhealth.Research reported in this publication was supported by the NationalInstitute of Mental Health under Award Number F31 MH088074 toErin C. Dunn, ScD, MPH. The content is solely the responsibility of theauthors and does not necessarily represent the official views of theNational Institutes of Health.

Conflict of Interest The authors declare that they have no conflict ofinterest.

References

Anderson, C. (1982). The search for school climate: A review of theresearch. Review of Educational Research, 52(3), 368–420.

Aud, S., Hussar, W., Planty, M., Snyder, T., Bianco, K., Fox, M., &Drake, L. (2010). The condition of education 2010 (NCES 2010–028). Washington, DC: National Center for Education Statistics,Institute of Education Sciences, U.S. Department of Education.

Bentler, P. M. (1990). Comparative fit indexes in structural models.Psychological Bulletin, 107, 238–246.

Bradshaw, C. P., Koth, C. Q., Thornton, L. A., & Leaf, P. J. (2009).Altering school climate through school-wide positive behavioralinterventions and supports: Findings from a group-randomized ef-fectiveness trial. Prevention Science, 10, 100–115.

Brand, S., Felner, R. D., Shim, M., Seitsinger, A., & Dumas, T. (2003).Middle school improvement and reform: Development and valida-tion of a school-level assessment of climate, cultural pluralism, andschool safety. Journal of Educational Psychology, 95, 570–588.

Brown, T. A. (2006). Confirmatory factor analysis for applied research.New York, NY: Guilford Press.

Cohen, J., & Geier, V. K. (2010). School climate research summary. NewYork, NY: Center for Social and Emotional Education.

Cohen, J., McCabe, L., Michelli, N. M., & Pickeral, T. (2009). Schoolclimate: Research, policy, teacher education, and practice. TeachersCollege Record, 111(1), 180–213.

Dedrick, R. F., & Greenbaum, P. E. (2011). Multilevel confirmatoryfactor analysis of a scale measuring interagency collaboration ofchildren's mental health agencies. Journal of Emotional andBehavioral Disorders, 19, 27–40.

Diez Roux, A. V. (2002). A glossary for multilevel analysis. Journal ofEpidemiology and Community Health, 56, 588–594.

Dunn, E. C., Masyn, K. E., Yudron, M., Jones, S. M., &Subramanian, S. V. (2014). Translating multilevel theory intomultilevel research: Challenge and opportunities for under-standing the social determinants of psychiatric disorders.Social Psychiatry and Psychiatric Epidemiology, 49, 859–872.

Dyer, N. G., Hanges, P. J., & Hall, R. J. (2005). Applying multilevelconfirmatory factor analysis techniques to the study of leadership.The Leadership Quarterly, 16, 149–167.

Eccles, J. S., & Roeser, R. W. (2011). School and community influenceson human development. In M. H. Bornstein & M. E. Lamb (Eds.),Developmental science: An advanced textbook (6th ed., pp. 571–643). New York, NY: Psychology Press.

Flora, D. B., &Curran, P. J. (2004). An empirical evaluation of alternativemethods of estimation for confirmatory factor analysis with ordinaldata. Psychological Methods, 9(4), 466–491.

Greenberg, M. T., Weissberg, R. P., O'Brien, M. U., Zins, J. E.,Fredericks, L., Resnik, H., & Elias, M. J. (2003). Enhancingschool-based prevention and youth development through coordinat-ed social, emotional, and academic learning. AmericanPsychologist, 58(6–7), 466–474.

Harris, K. M. (2013). The add health study: design and accomplishments:Carolina Population Center, University of North Carolina at ChapelHill.

Haynes, N. M., Emmons, C. L., & Ben-Avie, M. (2001). The schooldevelopment program student, staff, and parent school climatesurveys. New Haven, CT: Yale Child Study Center.

Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nded.). New York, NY: Routledge.

James, L. R., Choi, C. C., Ko, C. H. E., McNeil, P. K., Minton, M. K.,Wright, M. A., & Kim, K. (2008). Organizational and psychologicalclimate: A review of theory and research. The European Work andOrganizational Psychologist, 17(1), 5–32.

Jones, S. M., Brown, J. L., & Aber, J. L. (2011). Two-year impacts of auniversal school-based social-emotional and literacy intervention:An experiment in translational developmental research. ChildDevelopment, 82(2), 533–554.

Kline, P. (1994). An easy guide to factor analysis. London, England:Routledge.

Kline, R. B. (2010). Principles and practice of structural equationmodeling (3rd ed.). New York, NY: Guilford Press.

Marsh, H. W., Ludtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T.,Muthen, B., & Nagengast, B. (2009). Doubly-latent models ofschool contextual effects: Integrating multilevel and structural equa-tion approaches to control measurement and sampling error.Multivariate Behavioral Research, 44, 764–802.

Muthen, B. O. (1991). Multilevel factor analysis of class and studentachievement components. Journal of Educational Measurement,28(4), 338–354.

Muthen, B. O. (1994). Multilevel covariance structure analysis.Sociological Methods & Research, 22, 376–398.

Muthén, L. K., & Muthén, B. O. (1998–2010). Mplus user's guide (6thedition ed.). Los Angeles, CA: Muthén & Muthén.

Prev Sci

National School Climate Center. The Comprehensive School ClimateInventory, from http://www.schoolclimate.org. Accessed 9 June2013.

Reise, S. P., Ventura, J., Neuchterlein, K. H., & Kim, K. H. (2005). Anillustration of multilevel factor analysis. Journal of PersonalityAssessment, 84(2), 126–136.

Rones, M., & Hoagwood, K. (2000). School-based mental health ser-vices: A research review. Clinical Child and Family PsychologyReview, 3(4), 223–241.

Shinn, M. (1990). Mixing and matching: Levels of conceptualization,measurement, and statistical analysis in community research.In P. Tolan, C. Keys, F. Chertok, & L. A. Jason (Eds.),Researching community psychology: Issues of theory and methods(pp. 111–126). Washington, DC: American PsychologicalAssociation.

Steiger, J. H. (1990). Structural model evaluation and modification: Aninterval estimation approach.Multivariate Behavioral Research, 25,173–180.

Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics.Boston, MA: Allyn and Bacon.

Toland, M. D., & De Ayala, R. J. (2005). A multilevel factor analysis ofstudents’ evaluations of teaching. Educational and PsychologicalMeasurement, 65(2), 272–296.

Van Horn, M. L. (2003). Assessing the unit of measurement for schoolclimate through psychometric and outcome analyses of the schoolclimate survey. Educational and Psychological Measurement,63(6), 1002–1019.

Zullig, K. J., Koopman, T.M., Patton, J.M., &Ubbes, V. A. (2010). Schoolclimate: Historical review, instrument development, and school as-sessment. Journal of Psychoeducational Assessment, 28(2), 139–152.

Prev Sci