18

Click here to load reader

CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

  • Upload
    hanhi

  • View
    212

  • Download
    0

Embed Size (px)

Citation preview

Page 1: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1226

CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR SURVEYS OF JUVENILE FACILITIESCONCEPTS, CONSTRUCTS AND THE IMPACT OF EMPIRICAL RESEARCH

Joseph Moone, U.S. Department of JusticeOffice of Juvenile Justice and Delinquency Prevention, 810 Seventh Street NW , Washington, D.C. 20531

[email protected]

In developing new surveys for juvenile offenders in custody and for the facilities that hold these juveniles, the Office of JuvenileJustice and Delinquency Prevention (OJJDP) in the U.S. Department of Justice faced a broad range of conceptual and operationalissues posed by the wide variety of facilities used by States and local governments. The specific issues such as functional unitswithin facilities, facility crowding, and security constitute some of the more important concepts for measurement and analysis. However, they are also some of the most complicated issues to measure in a survey environment. This paper describes someof the obstacles faced in conceptualizing and operationalizing the definitions of these issues in new surveys covering these specialpopulations of juveniles and establishments.

Key Words: Residential Facility, Offenders, Crowding, Security, Functional Units

1. INTRODUCTION

Federal statistical agencies use establishment surveys to measure the actions, commerce and affairs of organizations orbusinesses. In this somewhat restricted sense, the establishments themselves act as individuals in the economy or insociety. The monitoring or surveillance of establishments constitutes an important activity for many Federal agencies.When the interest in the establishment is economic, monitoring of the establishment=s activity benefits from a fairlycommon unit of measurement, namely money. However, when the establishment (such as a government agency) providespublic services or provides services to individuals on behalf of the government, the nature of the survey becomes moredifficult and complex. Financial activities and measurement in terms of dollars no longer become the main goals. Inthe case of service providing establishments, often the statistical agency will want information on the establishment=scharacteristics but also on the services provided and the individuals served. This paper considers one such example: thatof juvenile offenders held in an institution or detention center by the justice system. In particular, it considers theconceptual and operational issues involved in measuring important non-monetary issues.

The Office of Juvenile Justice and Delinquency Prevention (OJJDP) has developed two particular surveys of interest,the Census of Juveniles in Residential Placement (CJRP) and the Juvenile Residential Facility Census (JRFC). Theuniverse for these two collections is the set of residential juvenile facilities used by the juvenile justice system to holdoffenders (either accused or convicted). However, the salient characteristics needed from these statistical programs arethe types of youth housed in the facilities and the relationship between the facilities and youth such as the level of controlover the youth, the services provided, and the legal relationship between the facility and the youth. Several conceptualand operational issues involved in these two surveys have proven uniquely difficult to overcome. OJJDP did not settleupon specific solutions until the survey instrument and the survey procedures were finalized. Even in the final draft ofthe questionnaire, the solutions were not considered optimal but rather patches that work in the given context. In oneexample consider here (crowding), the ultimate operationalization of the issue required changing the underlyingconceptual definition. In another (functional units), OJJDP has yet to arrive at a suitable set of questions.

2. BACKGROUND

The Office of Juvenile Justice and Delinquency Prevention (OJJDP) was established in 1974 and charged with collectinginformation on offending youth (juvenile delinquents) and the juvenile justice system. Since that year, OJJDP hassponsored statistical to describe and monitor the population of youth in custody, i.e., those held by the justice system,because of a specific offense. These statistical efforts cover juvenile courts and juvenile corrections. Other agenciessuch as the FBI and BJS cover arrests and juveniles in the criminal justice system. The following describes the majordata efforts covering the juvenile custody population.

Children in Custody. In 1971, the Department of Justice fielded the first Census of Juveniles in Detention andCorrectional Facilities. The scope of the survey expanded over the years, and by 1995 it was known as the Census ofPublic and Private Juvenile Detention, Shelter, and Correctional Facilities. It was more commonly known as the Childrenin Custody Census or CIC. In 1974, when OJJDP was established, the survey was transferred to that office. From 1971to 1995 it served as the main source of information on youth in the custody of the juvenile justice system in the UnitedStates. It was conducted every two years until 1995.The CIC forms were mailed to individual facilities for completion with questions covering the characteristics of thefacilities, the characteristics of the population in the facilities (offense, sex, age, race) and to an extent the servicesavailable in the facilities. The population data was aggregated in that facilities completed a grid that classified juveniles

Page 2: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1227

into exclusive categories by type of offense by sex, age by sex, and race by sex. This scheme limited the usefulness ofthe individual youth data. Analysts could not analyze the age or race of the youth by the specific offense.In 1991, OJJDP began a long-term evaluation of the statistical series dealing with the juvenile justice system. Initially,discussion centered on methods of improving or refining the data elements, standardizing the frame and revising the datacollection methodology. The examination brought to light several discrepancies between the CIC data collectioninstrument and the concepts it was intended to measure. Further, an analysis of data needs showed the inadequacies ofthe CIC to address current policy needs such as the over-representation of minorities in custody and the problems ofyounger offenders. These problems led OJJDP to discontinue the CIC and develop a new data collection methodologyand data collection instrument.

The effort to develop a new instrument focused on collecting individual information on each youth in custody. The newcollection would mimic the decennial census in which individual information is collected on each household member. OJJDP also examined how best to capture data on the services provided to the youth in these facilities. CIC includedsuch questions; however, it was clear that the burden of including both population and service questions on one censuswould prove too burdensome for respondents. OJJDP=s efforts, then, resulted in two new data collection efforts thatuse the juvenile facility as the collection unit and that attempt to describe the facilities and the youth separately.

Census of Juveniles in Residential Placement. The Census of Juveniles in Residential Placement (CJRP) begandevelopment in 1993. OJJDP contracted with the Center for Survey Methods Research (CSMR) within the CensusBureau to conduct the background research for the questionnaire development, assist with writing the survey questions,and to test the questionnaire. After a multi-phase development effort (described below), OJJDP fielded the first CJRPin 1997. These data have been published in various documents and on the web (Gallagher 1999; Snyder and Sickmund1999; Sickmund 2000). The second CJRP went into the field in October 1999. Data are expected in late summer 2000.

Juvenile Residential Facility Census. The CIC contained an assortment of questions on the education, health, treatmentand staffing in the facilities. OJJDP considers this information as important as information on the individual offenderslocated in the facilities. However, OJJDP decided that the burden on respondents would be too great if these questionswere also included in the CJRP. In 1996, then, OJJDP began developing the Juvenile Residential Facility Census (JRFC)which uses the same frame as the CJRP. This new survey covers security in the facility and other basic facilitycharacteristics, education, physical health services, mental health treatment and drug abuse counseling. It also includesquestions on deaths in custody as required of OJJDP by Congress since 1988. The conceptual and operational issuesof facility staffing posed significant problems (much like the other examples here) which could not be adequatelyaddressed in this census environment. OJJDP again contracted with CSMR for the development and testing of the newinstrument. A feasibility test was conducted in 1988. The survey will be fielded for the first time in October 2000.

3. PHASES OF DEVELOPMENT FOR CJRP

Following the decision to revise the CIC, The Census of Juveniles in Residential Placement underwent four differentdevelopment phases. Originally, both the Census Bureau and OJJDP had anticipated a three-phase development effort.

Phase I included an initial serious of structured interviews in 40 facilities. These interviews were designed to explorethe facilities= characteristics and their ability to respond to particular types of questions. The interview protocol wasbased on the CIC questionnaire and specifications for the new survey design. OJJDP required that the revised censuswould include a roster in which respondents would report on each youth in residence in that facility on a specific day. The interview covered facility classifications (e.g., detention center, training school, halfway house, etc.), security levels,and key terms for juvenile offenders such as delinquent, delinquent offender, status offender, committed, or detained. Based on the report of these interviews (Schwede and Ott 1995), the Census Bureau and OJJDP developed an initialdraft questionnaire.

Phase II of the project used cognitive methods to test the draft questionnaire with facility respondents. Census Bureaustaff visited 18 facilities and used a think-aloud method to follow the respondent=s thoughts through the task ofcompleting the questionnaire. The think-aloud method allowed the Census Bureau staff to determine how therespondents understood the question and what steps they took to answer the questions appropriately and accurately. Based on a report of these tests (Schwede and Moyer 1996), additional modifications were made to the instrument.

The questionnaire was field tested in an actual census environment with a random sample of 400 facilities in October1996. Up to the completion of this field test, the actual workability of the CJRP as a data collection effort was not clear. The ability and willingness of the respondents to complete the form was always in question. Ultimately, though, theresults of the test proved successful enough to go ahead with the first full fielding of the CJRP in 1997. (See Schwedeand Ellis 1997.)

Page 3: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1228

In 1996, OJJDP contracted once again with CSMR for the development of the JRFC. This data collection underwentthree testing phases much like the CJRP. The phases included open-ended interviews with facility visits, cognitive testingof a draft instrument, and feasibility testing of a subsequent draft in an actual census environment with a sample of 500facilities. The testing was completed in 1998 and the instrument was finalized in June 2000. (See Gallagher andSchwede 1997; Birch et al. 1998 and 1999.)

4. CONSTRUCT AND OPERATIONAL CHALLENGES

OJJDP had set a number of requirements for the CJRP: First it would need to identify the security arrangements ofindividual youth, and second it would need to identify which facilities were crowded. As a basic point of departure,OJJDP also expected the CJRP to provide some idea of the environment of the youth. It was originally believed that allyouth in a particular facility shared a common environment in terms of length-of-stay, staff supervision, education,counseling, and control of movement, etc. It was thought that indicating these issues for each facility would capture thecharacteristics for all resident youth. However, each proved more challenging to operationalize than anticipated.

Crowding. OJJDP considers crowding in juvenile facilities a serious issue both from the point of view of rehabilitationof the youth and the security of staff and the community. Crowded facilities can overwhelm the resources of the facilityto deal with internal problems such as maintaining order, preventing escapes, and controlling the entrance into anddissemination of contraband within the facility. Similarly, as facilities and organizations must divert more and moreresources to the maintenance of order, other services such as education and counseling suffer both quantitatively andsubstantively. In short, the entire purpose and goals of the facility are subverted.

Initially, OJJDP had a succinct and immutable definition of crowding: A facility is crowded when its daily populationexceeds its capacity. The definition here hinged on Acapacity.@ By measuring the total population on a given day andcomparing that to the capacity, OJJDP could determine if a particular facility was crowded. Originally, it was believedthat capacity would be a relatively standard measure of a facility. However, the development of the CJRP proved thatthe facilities were unable to respond to questions of crowding in a clear and consistent manner. If based on clearstandards from the American Correctional Association, each facility had a single level of capacity. However, thisdefinition was not standard. In fact, the measurement of a facility=s capacity was more fluid and changeable than OJJDPor the Census Bureau originally anticipated.

Census Bureau researchers visited many facilities where respondents did not know either the original design capacity(i.e., the number of residents originally intended under the layout and plans) or the capacity based on some standard. In some cases, facilities had closed particular sections of the building reducing the capacity from a standardized squarefootage measurement or had increased the capacity by changing the use of some rooms such as day rooms, cafeterias oreven gymnasiums. These facilities used alternate measures of capacity including: (1) budget capacity wherein thefacilities could only take so many youth as budgets would permit, (2) legal capacity where judicially mandated populationceilings determined the number of residents the facility could hold, (3) regulatory capacity where, for example, fire codeswould restrict the number of residents in the facility as a whole or in a particular part of the facility, or (4) some arbitrarycapacity where the facility might increase the number of juveniles it could hold simply by double bunking roomsoriginally intended for one resident. Because of these different definitions, the concept of crowding (if left to therespondent) would become unworkable.

Census Bureau staff and OJJDP were unable to develop standard questions on capacity that would address all of thesediverse definitions yet arrive at a single, uniform measure. Because the questions were clearly burdensome, OJJDPagreed to remove them from the CJRP and make them a part of the JRFC. During the development of the JRFC, CSMRdesigned new questions for testing. After each phase, the Census Bureau and OJJDP determined that the changedquestions that asked about Acapacity@ could not cover the differing definitions and complexities of that term. As astopgap measure, the Census Bureau recommended the JRFC use counts of beds used (both standard beds and makeshiftbeds such as couches, mattresses or costs) to determine crowding (Birch et al. 1998). Under this new concept, wheneverthe number of youth in a facility exceeded the number of standard beds, the facility would be considered crowded. Crowding would also be present when makeshift beds were used, although such use might indicate that only parts of thefacility were crowded or that circumstances prevented the use of all standard beds. This operational definition has itsown problems. It does not address the issue of living space, an important concept used when defining a facility=scapacity. However, the straightforward questions concerning the number of beds used on a given night appeared topresent the best common measurement across all facilities.

Security. The juvenile facilities included in the frame must maintain the physical security of the grounds. First, thefacility must ensure that juveniles do not leave the facility itself without appropriate permission and supervision. Equallyimportant, the facility must maintain the safety and security of the juveniles present in the facility. Finally, not all

Page 4: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1229

facilities maintain the same level of security; not all facilities need to. The lowest level of security might include onlystaff supervision whereby the individuals with the youth prevent unauthorized departure from the facility. In facilitiesthat house the more violent offenders, youth may be subject high security measures including locked doors, fences withrazor wire, and constant supervision by staff or guards.

OJJDP considers the levels of security important characteristics of the facility. The level of security reflects the sanctionimposed on the youth. The more serious the offense (e.g., serious personal assault, drug trafficking, etc.) the moreserious and restrictive the residential environment ought to be. The level of security also describes the relative dangerposed by the youth to the community and other residents. Youth in a higher security facility are viewed as moredangerous to themselves and potentially to other residents or staff.

The original operational definition of security in CIC used three levels of security: minimum, medium and maximum. As with other questions from this particular census, the phrasing was inadequate for distinguishing various securitylevels. No specific definitions were provided on the form to direct the respondents to the most appropriate responsecategory. Without such definitions, it became difficult to interpret responses. The simple three categories also maskedthe significant variations in security procedures from locked doors and gates to intense supervision during the day. Moving to a more rigorous instrument required consideration of the various components of security and the measurementof these components.

One aspect of security became clear during the development of CJRP: not all residents in a facility shared the same levelof security. For various reasons, one group of youth might be subject to a higher level of security than others. For somefacilities, as youth progressed through the program, they would receive greater benefits including greater freedom ofmovement. In other facilities, if a youth proved to be a disciplinary problem, he would be subject to greater securityincluding being placed in a higher security wing. OJJDP hoped to be able to make individual security distinctions basedon security.

Following Phase I, OJJDP and the Census Bureau included security questions in the initial screener form. This screenerwould have established exact security procedures for the facility and its subunits (functional units). This form waseliminated during Phase II (see below) requiring a new set of security questions be added in the roster. The questionultimately decided upon was: AHow many locked doors and/or gates confined THIS young person within the facilityand its grounds during the after-school, daytime hours on October 27?@ Because the full battery of security questionscould not be included in the CJRP, they were included in the JRFC.

Functional Units. In the Children in Custody census, each facility was asked to classify itself into one of six differenttypes. These types include detention center, reception center, shelter, training school, ranch/camp, and halfway house. At the initial inception of the CIC, these groupings appeared to exhaustively and exclusively describe the facility types. However, information from the development of CJRP and JRFC proved that facilities did not fit into these categories. On the one hand, the limited number of classifications did not fully encompass the differing functions of the facilities. Second, the facilities often performed different functions for different residents. One facility might provide detentionservices for a county and at the same time provide a long-term sex-offender treatment program. Another facility mightprovide long-term and short-term shelter for a number of youth in a fairly non-secure environment. In the first instance,the two groups of youth would be separated as a matter of course, possibly in two different security levels. In the second,the youth might be mingled during all times of the day. In both cases, the youth would be in the facilities for quitedifferent lengths-of-stay. Thus, using a single characterization of the facility might or might not adequately capture thenature of that facility.

During the development phases of the CJRP, Census Bureau staff noted several facilities which had significantdistinctions between different units in the facility. Such distinctions brought to question the basic concept of Afacility@for the entire series of surveys. For both subject matter specialists and the survey methodologists, the idea of Aonefacility, one type@ proved strong and convincing. Under this particular view, facilities may hold different types ofjuveniles, but the overall characteristics of the facility remained uniform. However, as with any strongly held, non-empirically-based concept, the truth proved much more complicated than the original concepts.

In a separate paper, Schwede (1998) described in detail the particular issues facing the development of the CJRP andother surveys concerned with this particular universe of facilities. This paper presents results of the Census Bureauresearch undertaken as part of the CJRP development. The point raised in that paper concerns the target statistical unitof the survey. The author describes the wide variety of arrangements that can make up a single facility and the problemscaused when attempting to identify the basic statistical unit of count in a survey. As has been the case in businesssurveys, the identification of this basic unit is complex. A differentiation of the functional units for the purposes of framedevelopment and survey administration would not present a problem had the surveys not been concerned with the

Page 5: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1230

combined description of the facilities and the youth within the facilities. Census Bureau and OJJDP staff havedetermined that the most salient distinctions for functional units include security, location, services, length-of-stay ofyouth, and separation of populations (among other more detailed distinctions). In a more general survey of the facilitiesthemselves as establishments, descriptions of such differences within the facility might suffice. However, in the currentsurvey environment where establishments themselves become variables for the individuals, a clear distinction must bedrawn. This distinction would serve to describe the treatment of youth based on the demographic and offensecharacteristics of these youth.

The Census Bureau and OJJDP included functional unit questions as defining variables in the initial CJRP developedas a result of the Phase I interviews. The facilities were to be sent an initial screener questionnaire that would determinethe existence of separate functional units in the facility. The screener would also attempt to determine some fundamentalcharacteristics of the unit including type of youth held, daily population, and level of security. In Phase II of the CJRPdevelopment, this screener was tested along with the roster questionnaire. Unfortunately, a workable screenerquestionnaire could not be devised and the distinction of functional units was ultimately dropped. Although the CensusBureau and OJJDP continued examining this issue with the development of the JRFC, to date no adequate questions onfunctional units have been included in either census.

5. DISCUSSION

The development of the CJRP and JRFC questionnaires was successful. While some important concepts could not beincluded in them (e.g., staffing or functional units), the Census Bureau was able to develop two censuses that collectindividual level information on all juvenile offenders and provides facility classifications for all facilities. Given thatwithin the justice field no similar censuses have ever been conducted, this accomplishment is worthy of note. The CJRPwas first implemented in 1997 with no significant problems. The resulting data have proved a rich source of policyrelevant information previously unobtainable (Gallagher, 1999, Snyder and Sickmund, 1999).

Although survey methodologists have devoted considerable discussion to survey error, a complete and theoretically baseddiscussion of concept and construct definition development has not taken place. In the development of juvenile custodydata collections, as illustrated in the examples presented here, the overarching concepts (crowding, facility, security)often cannot be adequately constructed into survey questions. Refined definitions of the concepts required an iterativeprocess during the development of the questionnaire. Ideally, a sufficient base of research would exist (or be developed)that would provide the necessary information concerning the real world validity and reliability of these concepts and theresulting constructs or questions. The development process, then, is circular. The concepts determine what questionsto test and the tests indicate what works and is valid. The tests inform researchers how the original conceptual definitionsdo not match the real world situations encountered. The development team then reconsiders the conceptual definitionsand the operationalization of those definitions.

In the case of juvenile custody, the statistical systems had existed for a considerable period of time, creating their ownculture. The data collectors were lead to believe (to an extent) that the definitions used in the old CIC were still currentand valid. However, in the intervening 20 years, considerable changes had taken place in the field of juvenilecorrections. These definitions no longer fit the conditions at the turn of the century. The old survey was developed inan era where the use of rigorous methods of development was not the norm (due to funding levels and the general lackof a solid knowledge base.) When starting from a state of ignorance of the universe (here the juvenile residentialfacilities), it becomes important to allow for a series of research phases that will inform the concept development. Aseach phase is completed, the new information should be used to determine the veracity and acceptability of the currentparadigm. More often than not, the information collected will pose new questions and require refinement of the conceptsand questions.

The study of establishment surveys has begun expanding beyond the initial specifications that were limited (rightly so)to incorporated or semi-incorporated organizations that acted as autonomous individuals within the economy. This areaof study is beginning to examine the peculiarities and needs of surveys of service providing establishments and what isknown in the Census Bureau as group quarters. The experiences from the development of the CJRP and JRFC provideimportant lessons for future survey efforts in this area. First, the definition of specific concepts must be changeable. Whereas at the outset, the subject matter specialists in OJJDP had what were considered national definitions of capacity,security and the like, as the survey research progressed, the definitions had to change to accommodate unanticipatedcircumstances. At times, the concepts had to be abandoned (as with functional units and to an extent crowding) until amore workable operational definition could be developed. Second, the idea of a particular item=s importance must bestated in terms of its ability to be effectively communicated in the questionnaire. Crowding is a critical concept forOJJDP. Although efforts were made to include it in the CJRP, ultimately it could not be sufficiently and effectively

Page 6: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1231

operationalized. Time and burden constraints forced OJJDP to remove the questions on crowding from this survey andexplore alternate measures with the JRFC.

OJJDP has not yet given up efforts to effectively capture information on a function unit level or to measure crowdingbased on a national standard of living space. In the short-term, OJJDP had to make decisions so that the entire projectcould move forward. However, these issues require more research. Already, OJJDP has adopted a policy of routineresponse analysis surveys with each CJRP or JRFC. These simple surveys will aid in addressing these and otherimportant issues. The development of the more difficult questions will always be a long-term, iterative process.

REFERENCES

Birch, S., L. Schwede, and C. Gallagher (1998), AJuvenile Residential Facility Census Questionnaire Redesign Project:Results from Phase 2 Cognitive Interview Testing,@ unpublished report, Washington, D.C., U.S. Census Bureau.

Birch, S., L. Schwede, and C. Gallagher (1999), AJuvenile Residential Facility Census Questionnaire Redesign Project:Results from Phase 3 Mail-out Test Analysis,@ unpublished report, Washington, D.C., U.S. Census Bureau.

Gallagher, C. and L. Schwede (1997), AFacility Census Questionnaire Redesign Project: Results from Phase 1Unstructured Interviews and Recommendations for Facility-Level Questionnaire,@ unpublished report, Washington,D.C., U.S. Census Bureau.

Gallagher, C. (1999), AJuvenile Offenders in Residential Placement,@ Washington, D.C.: Office of Juvenile Justice andDelinquency Prevention.

Schwede, L. and K. Ott (1995), AChildren in Custody Questionnaire Redesign Project: Results from Phase 1,@unpublished report, Washington, D.C., U.S. Census Bureau.

Schwede, L. and C. Gallagher (1996), AChildren in Custody Questionnaire Redesign Project: Results from Phase 3Cognitive Testing of the Roster Questionnaire,@ unpublished report, Washington, D.C., U.S. Census Bureau.

Schwede, L. and L. Moyer (1996), AChildren in Custody Questionnaire Redesign Project: Results from Phase 2Questionnaire Development and Testing,@ unpublished report, Washington, D.C., U.S. Census Bureau.

Schwede, L. and Y. Ellis (1997), AChildren in Custody Questionnaire Redesign Project Final Report: Results of theSplit-Panel and Feasibility Tests,@ unpublished report, Washington, D.C., U.S. Census Bureau.

Schwede, L. (1998), AConceptual and Methodological Issues in Defining the Basic Statistical Units in a FacilityCensus,@ unpublished report, Washington, D.C., U.S. Census Bureau.

Sickmund, M. (2000), ACensus of Juveniles in Residential Placement Databook,@ OJJDP Fact Sheet, Washington, D.C.:Office of Juvenile Justice and Delinquency Prevention.

Snyder, H., and M. Sickmund (1999), AJuvenile Offenders and Victims: 1999 National Report, Washington, D.C.:Office of Juvenile Justice and Delinquency Prevention.

Page 7: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1232

STRUGGLING TO ANSWER: THE ROLE OF FACILITY CHARACTERISTICS IN RESPONSEPROBLEMS IN A CENSUS OF JUVENILE FACILITIES1

Sharon Birch, U.S. Census BureauSharon Birch, Statistical Research Division, U.S. Census Bureau, Washington D.C. 20233

[email protected]

ABSTRACT

Response problems to questionnaires sent to juvenile correctional facilities may be the result of drastic variations betweenfacilities and their ability to respond to a one-size-fits-all questionnaire rather than specific questionnaire and itemcharacteristics. This study examines response data from the 1998 mail-out field test of the Juvenile Residential FacilityCensus (JRFC) to help explore these issues. Item non-response is compared between different types of facilities andbetween different kinds of items in the form. Item non-response did vary by facility characteristics and by itemcharacteristics. While the nature of the relationship between item characteristics and item non-response was the sameregardless of facility characteristic, the strength of the relationship varied.

Key Words: establishment, item non-response

1. INTRODUCTION

Juvenile correctional facilities often straddle a line between establishment and household. While even the smallesthouse-based facility is still an establishment and not a household, the simplicity of its organization may make therespondent less amenable to questionnaires designed for establishments that are more complex. On the other hand,large, complex, and possibly dispersed juvenile facilities may find the task of aggregating information on individualsor providing individual level data (as is typically found in household surveys) too difficult. Thus, response problemsto questionnaires sent to juvenile facilities may be the result of drastic variations between facilities and their abilityto respond to a one-size-fits-all questionnaire rather than specific questionnaire and item characteristics. Small,simple facilities and large, complex facilities may both have response problems, albeit for different reasons. Thegoal of this paper is to compare item non-response (INR) between different types of facilities and between differentkinds of items in the form using response data from the 1998 mail-out field test of the Juvenile Residential FacilityCensus (JRFC).

2. BACKGROUND AND METHODOLOGY

The Juvenile Residential Facility Census (JRFC) project is being conducted for the Office of Juvenile Justice andDelinquency Prevention in the Department of Justice by Census Bureau staff in the Center for Survey MethodsResearch and the Criminal Justice Statistics Branch of the Governments Division of the Census Bureau. Theobjective of this project is to develop a new questionnaire to collect facility-level data on facility characteristics andjuvenile services to replace the collection of data on these topics in the now discontinued Census of JuvenileDetention, Correctional and Shelter Facilities (also know as the Children in Custody -- CIC-- census).

2.1 Frame and Sample

There were 3,464 total juvenile residential facilities in the universe for the JRFC test. Of these, 1,657 are single-reported facilities and 1,807 are centrally-reported facilities. Data from the 1997 Census of Juveniles in ResidentialPlacement (CJRP) was used to develop and draw the sample for the mail-out test of the JRFC. Facilities weresampled within central vs. single reporting, and stratified based on public vs. private status and size (population ofyoung persons).

For the item non-response and other analyses that required a returned form from an in-scope and currently operatingfacility, only 309 of the originally sampled cases (497 facilities) were used. Facilities not returning a form, anyfacility that closed or became out-of-scope between the time the frame was obtained and the mail-out test wasfielded, and any facility that self-identified as out-of-scope in the JRFC form itself was not included. The resultingdata (309 cases -- a 74% response rate from open and in-scope facilities) consisted of responses from only thosecases returning a form, currently operating, and in scope.

1 This paper reports the results of research and analysis undertaken by Census Bureau staff. It has undergone a morelimited review by the Census Bureau than its official publications. This report is released to inform interestedparties and to encourage discussion.

Page 8: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1233

2.2 Methodology

2.21 Data

Two data sets were used for the analyses reported in the paper – a facility-level data set with 309 cases and an item-level data set with 187 cases. The first set consists of the responses to the JRFC form and some additional variablesrelated to the facility (sampling characteristics, when the form was returned, etc.). In this set, the facility was theunit of analysis (n=309). These data were used to make direct comparison of item non-response between facilitiesand to derive the summaries of item non-response used in the item-characteristic data set. The second data setconsisted of item characteristic codes and summary measures of item non-response. Analyses of facilitycharacteristics and INR used the first data set and analyses of item characteristics and INR used the second data set.

2.22 Variables

A variable was created in the primary data set (n=309) that used an algorithm based on skip patterns to identify, foreach item and within each case, whether or not a question should have been answered. This was compared to therespondents’ actual behavior (answered vs. blank). The resulting variable indicates whether each item was correctlyanswered, correctly left blank, incorrectly answered, or incorrectly left blank. Throughout this paper, the generalterm “item non-response” (INR) is used to refer to items left blank when they should have been answered, althoughin analyses of facility characteristics, INR is the number of items left blank incorrectly and in analyses of itemcharacteristics, INR is the percent of forms with the given item left blank incorrectly. These variables werecomputed for the analysis.

A new facility variable (“complexity variable”) was computed that classified facilities as either “simple” or “notsimple” using facility layout items in the form (from the first data set, where each case is a facility and each item isan item in the form). Assumptions about facility complexity were based on extensive experience and backgroundresearch into juvenile facilities conducted by members of the JRFC and CJRP research team (see bibliography).Basically, simple facilities are facilities that have a simple layout (a single facility in one building or on one campus,with no buildings elsewhere) and have a single residential component (no separate living/sleeping units designed tokeep residents separate in housing and activities). Non-simple facilities would be any facilities that do not meetthese assumptions.

In the second data set – the item-characteristic data set – the unit of analysis was item in the form (the JRFCcontains a total of 187 items). The variables included item characteristics such as type of item (multiple choice,yes/no, etc.) and item topic. The mean INR (expressed as the percent of forms with this item left blank incorrectly)across all 309 returned and usable forms was calculated and included and separate mean INRs for forms fromfacility subgroups (centrally-reported facilities, single-reported facilities, etc.) are also included.

2.23 Analysis

Comparisons of item non-response will be made between facilities based on their size, ownership/operation (publicvs. private), reporting type (central vs. single reporter), and complexity to determine if INR varies by any of thesecharacteristics. Analysis in this paper is limited to descriptive figures and comparisons of means using analysis ofvariance.

Item non-response will also be compared by item, since item wording or form design may lead to item non-response. The percentage is used rather than the raw number since the number of facilities at risk to leave an itemblank varies from item to item (not all items are required for all facilities). As opposed to the analysis on facilitycharacteristics, in these analyses, item is the unit of analysis. Average item non-response for a given itemcharacteristic is the mean percent of facilities leaving the item blank of all items with that given characteristic.These means are compared to determine if the item characteristics are related to item non-response.

The first item characteristics of interest is topic. The Juvenile Residential Facility Census covers many topics,which could easily be broken into fine detail when trying to categorize. Fortunately, the items do tend to fall intogeneral categories that can be used for this analysis. Item type/question type may also influence item non-response.The items in this form fall into six categories.

As in the comparisons of INR by facility characteristics, all analyses of INR by item characteristics are descriptiveor means comparisons using ANOVA.

Page 9: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1234

3. RESULTS

3.1 Overall Item Non-response

The mean number of items left blank incorrectly per form (in the facility-level data set) was 6.3 items (median = 5items, mode = 3 items). This translates into 3.4% of all the items in the JRFC form. Almost every item in the JRFCwas left blank incorrectly on at least one form (164 out of 187 items). Thus, item non-response was not exclusivelyfound among certain items, but the extent of this non-response did vary, with some items exhibiting higher INR thanothers.

3.2 Facility Characteristics and INR

Table 1 presents the mean item non-response in forms from different subcategories of facilities and Table 2 indicateswhich differences in mean INR were significant and the strength of the relationship between facility characteristicand item non-response.

Table 1. Mean Item Non-response by Facility Characteristics

Characteristic Category Mean s.d. Median Mode nSingle 7.34 9.00 5.00 3.00 162ReportingCentral 5.22 4.63 4.00 3.00 147Public 5.78 6.2 5.00 2.00 109OwnershipPrivate 6.64 7.87 4.50 4.00 200Small 5.67 5.39 4.00 4.00 155Medium 5.86 6.13 4.00 3.00 76

Size Category

Large 8.12 10.76 5.00 3.00 78Simple 4.96 3.24 4.00 4.00 138ComplexityNot Simple 7.43 9.27 5.00 3.00 171

All Facilities 6.33 7.33 5.00 3.00 309

Table 2. Results of Analysis of Variance of Item Non-response by Facility Characteristics

Characteristic F Significance (p=) Hays’ Omega- squaredReporting* 6.539 .001 .018Ownership .961 .328 ---Size Category* 3.146 .044 .017Complexity* 8.94 .003 .025

* p< .05

Single reporters have significantly higher item non-response than central reporters. There is no significantdifference between public and private facilities. Size category was marginally significant. Large facilities hadsignificantly higher item non-response than small or medium facilities. There was no significant difference betweensmall and medium facilities. Simple facilities had significantly lower INR than non-simple facilities. The highestmean item non-response is among large, single reported, private facilities (mean = 11.5 items). This is followed bysmall, single reported, public facilities (mean = 9.56 items). The lowest item non-response is among medium sized,single reported public facilities (mean = 4.68 items) followed by small, centrally reported public facilities (mean =4.78 items). The relationship between item complexity and INR is stronger than the relationship between reportingstyle and INR or size and INR.

3.3 Item Characteristics and INR

Factors related to the item wording or form design could potentially result in item non-response. The results ofcomparisons of INR by item characteristics are as follows:

Page 10: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1235

Table 3. Mean Item Non-response (percent facilities leaving item blank) by Item Topic

Topic Mean s.d. Median Mode nInformation pertaining to beds 3.1 1.8 3.2 1.3 3Population 5.8 3.8 6.8 .7 8Conditions of confinement 2.9 1.9 3.2 0 9General services 3.3 1.3 3.9 4.2 5Educational services 3.0 3.8 2.3 0 30Mental health services 2.0 2.3 1.3 0 35Physical health services 3.1 3.4 1.9 1.6 34Substance abuse services 2.0 2.0 1.8 .6 24Ownership/operation 2.6 2.7 2.3 0 8Security 8.1 16.7 2.6 .3 9Facility characteristics 2.2 1.7 2.5 0 14Items pertaining to the questionnaire itself 17.7 17.6 11 4.2 7All items 3.5 6.2 2.6 0 187

Overall, the difference in INR between item topics is statistically significant (p< .000). The strength of theassociation between item topic and INR is .2 (measured using Hays’ Omega – squared, which produces a figure thatcan be interpreted similarly to a correlation coefficient (see Kerlinger, 1986)). A post-hoc comparison of mean INRfor each item topic indicates that the high INR for items pertaining to the form itself drives the overall significanceof topic. The somewhat conservative Bonferroni procedure only indicates a significant difference between itemsrelated to the form and the other topics. The population and security items, while not statistically significantlydifferent from other items, were more troublesome for respondents than some of the other topics were.

Table 4. Mean Item Non-response (percent facilities leaving item blank) by Item Type

Type of Item Mean s.d. Median Mode nMark-all-that-apply 3.5 7.2 1.9 .6 54Mark-only-one 3.6 1.3 3.8 4.2 8Fill-in-the-blank 9.6 11.9 6.3 3.2 20Yes/No 3.4 1.7 3.2 2.9 50Grid 6.4 4.7 4.5 6.5 7“Specify” items .9 2.8 .3 0 48All items 3.5 6.2 2.6 0 187

Item non-response did vary significantly by item type (p<.000). Hays Omega2 was .13, which is somewhat weakerthan the association between topic and INR. The post-hoc comparisons of the different item types shows that thehigh INR for fill-in-the-blank items was responsible for the impact of item type (in the Bonferroni comparisonresults, fill-in-the-blank items had significantly higher INR than mark-all-that-apply items, specify items, and yes/noitems).

3.4 Combination of Effects of Facility Characteristics and Item characteristics

The original impetus for this paper was to determine if item characteristics play a larger role in item non-responsefor some kinds of facilities than they do for others. One way to test this is by using the same data used to determinethe impact of item characteristics (discussed in the previous section of this paper), in which the item is the unit ofanalysis, but instead of using overall INR for all facilities use only INR for certain facilities. As in the previoussection, the INR measure is the percent of facilities leaving the item blank incorrectly. Thus, we can performseparate analyses of variance that determine if a given item characteristic leads to significantly different INR foreach facility characteristic. The results have indicated that for all facilities, INR varies significantly by the topic ofthe item. This remains the case when facilities are divided into separate categories.

Page 11: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1236

Table 5. Significance and Strength of Relationship between Item Topic and Item Non-Response for DifferentCategories of Facilities

Facility Category Significance (p=) Hay’s Omega -squaredAll Facilities .000 .20Centrally-Reported .000 .17Single-Reported .000 .24Public .000 .24Private .001 .11Small .000 .19Medium .000 .31Large .000 .16Simple .000 .25Not Simple .000 .24

As a whole, there is very little difference between the impact of the item topic on each category of facility’s INR.Each of the categories has a significant relationship between INR and topic. This is not surprising given the strongrelationship for all facilities. The only differences are found in the strength of the relationships – the proportion ofthe variance in INR that can be explained by topic. These results show that INR on forms from simple facilities areno more or less influenced by topic than for non-simple facilities. Medium sized facilities are more influenced bytopic than small or large facilities.

Table 6. Significance and Strength of Relationship between Item Type and Item Non-Response for DifferentCategories of Facilities

Facility Category Significance (p=) Hays’ Omega -squaredAll Facilities .000 .13Centrally-Reported .000 .10Single-Reported .000 .22Public .000 .16Private .000 .11Small .000 .14Medium .000 .19Large .000 .18Simple .001 .08Not Simple .000 .21Detention Center .000 .22Not Detention Center .000 .14Group Home .001 .09Not Group Home .000 .19

Item non-response for all facility categories is influenced by item type. Again, this is unsurprising given the stringrelationship overall for all facilities. As in item topic, item type is significantly related to INR for all facilitycategories, but the strength of the association does vary somewhat (and varies differently from item topic). In thiscase, simple facilities are clearly less influenced by item type than are non-simple facilities. Given how this variableis defined, it is no surprise that INR for group homes are less influenced by item type than other facility types andthat detention centers are more influenced than others. The interesting difference is that INR for single-reportedfacilities appear to be more influenced by item type than centrally-reported facilities.

4. DISCUSSION AND CONCLUSIONS

While item non-response is clearly related to both facility characteristics and to item characteristics, as well as to thecombination of these factors, interpretation of these results is difficult. Size of facilities played a weak role in itemnon-response. Item non-response did vary by complexity. The results indicate that simple facilities had lower itemnon-response than non-simple facilities (an average of about 2.5 fewer items left blank incorrectly per form). Smallfacilities had lower INR than larger facilities. Among these facilities, it appears that the smaller, simpler facilitieshave less trouble with the form than larger, more complex facilities. This is not an unusual finding, and is consistent

Page 12: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1237

with some other establishment survey research into unit non-response (for example, Tomaskovic-Devey, Leiter, andThompson, 1994).

As expected, item characteristics did play a role in item non-response. Item topic, type, and location were allsignificant factors. Basically, respondents had an easier time with some types of items than with others. Items thatasked questions about the form itself (such as items asking about preprinted addresses or how many forms did therespondent receive) were routinely left blank. Items pertaining to security and population were also left blank moreoften than the other topics. This can be partially explained by the nature of these items – asking for specific,detailed information. The results by item type were as expected -- the easier the format of the question (multiplechoice, yes/no) the less likely item non-response).

Topic and item type are significant indicators of INR for all facilities, regardless of the category the facility fallsinto. However, the strength of the relationship between the item characteristic and item non-response varies byfacility category. So, the items themselves are more strongly related to INR on forms returned by single reportersthan those returned by central reporters. One possible explanation is that central reporters fill out multiple forms andthus grow more accustomed to the layout, required information, kinds of questions, etc.

While the strength of the relationship between topic and INR is about the same for both simple and non-simplefacilities, simple facilities were less affected by item type than non-simple facilities. Detention centers were morestrongly affected by both item topic and item type. Group homes were less affected by item type than non-grouphome facilities. There is no consistent pattern here – smaller, simpler facilities are not always less affected by itemcharacteristics, and nor are larger more complex facilities. However, these results do support the idea that differentkinds of facilities will experience the questionnaire in different ways.

When developing an establishment survey, these issues – that both characteristics of the establishment as well ascharacteristics of the form itself/items within the form can be related to item non-response – must be kept in mind.While the results of this analysis can not necessarily be generalized to all juvenile residential facilities, much less allestablishments, they do point to a possible interrelationship between facility complexity, item complexity, and itemnon-response.

5. REFERENCES

Birch, Sharon, Schwede, Laurie, and Gallagher, Catherine (1998) Juvenile Residential Facility CensusQuestionnaire Redesign Project: Results from Phase 2 Cognitive Interview Testing. Center for SurveyMethods Research, U.S. Bureau of the Census.

Birch Sharon and Schwede, L. Juvenile Facility Questionnaire Redesign Project: Report of Findings Pertaining toSection 1. Center for Survey Methods Research, U.S. Bureau of the Census. 1999

Birch, Sharon and Schwede, L. Juvenile Residential Facility Census Questionnaire Redesign Project: Results fromPhase 3 Mail-out Test Analysis. Center for Survey Methods Research, U.S. Bureau of the Census, 1999.

Gallagher, Catherine and Schwede, Laurie (1997) Facility Questionnaire Redesign Project: Results from Phase 1Unstructured Interviews and Recommendations for Facility-Level Questionnaire. Center for SurveyMethods Research, U.S. Bureau of the Census.

Tomaskovic-Devey, Donald, Leiter, Jeffrey, and Thompson, Shealy (1994) Organizational Survey Nonresponse.Administrative Science Quarterly 39(3); 439-457.

Schwede, Laurie and Ellis, Yukiko (1997) Children in Custody Redesign Project Final Report: Results of the Split-Panel and Feasibility Tests. Center for Survey Methods Research, U.S. Bureau of the Census.

Schwede, Laurie and Gallagher, Catherine (1996) Children in Custody Redesign Project: Results from Phase 3Cognitive testing of the Roster Questionnaire. Center for Survey Methods Research, U.S. Bureau of theCensus.

Schwede, Laurie and Moyer, Laurie (1996) Children in Custody Redesign Project: Results from Phase 2Questionnaire Development and Testing. Center for Survey Methods Research, U.S. Bureau of the Census.

Schwede, Laurie and Ott, Kathleen (1995) Children in Custody Redesign Project: Results from Phase 1. Center forSurvey Methods Research, U.S. Bureau of the Census.

Page 13: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1238

Minimizing Biases in the Survey of Youth in Residential Placement (SYRP)Andrea J. Sedlak

Westat, Inc.

This paper reports on the planning activities for the Survey of Youth in Residential Placement (SYRP),sponsored by the U.S. Department of Justice’s Office of Juvenile Justice and Delinquency Prevention (OJJDP). TheSYRP will be the newest addition to surveys of the juvenile justice system that are conducted by the Office ofJuvenile Justice and Delinquency Prevention (OJJDP). These currently include the Census of Juveniles inResidential Placement (CJRP) and the Juvenile Residential Facility Census (JRFC). The SYRP will be conducted ina sample of juvenile residential facilities that hold youths in juvenile justice placement and will involve interviewswith youths sampled from those in residence at the time of the survey. The SYRP interviews will ask youths abouttheir offense histories and sanctions received; the conditions of their confinement; their needs for various services;services received while in the facility; and expectations for the future.

OJJDP is planning the SYRP through a cooperative agreement with Westat and Westat’s subcontractor, theNational Council on Crime and Delinquency (NCCD). Planning the SYRP has included designing the sample, theinterview instrument, the data collection procedures and the analysis plan. The plans have benefitted the expertfeedback and recommendations of a study advisory board, a roundtable of facility administrators in the metropolitanWashington DC area, and telephone discussions with administrators of 26 facilities in 10 large states. In addition, alarge Field Test provided an in situ test of the feasibility and adequacy of all aspects of the SYRP procedures andinstrumentation. This paper reports on ways the SYRP has been designed to reduce bias and on some potentials forbias uncovered during the Field Test, and the resulting recommendations for improving the national SYRP.

Facility SampleFacilities surveyed in the Census of Juveniles in Residential Placement (CJRP) will constitute the universe

for the SYRP. The CJRP is conducted biennially, and has been implemented twice to date, in 1997 and 1999. The1999 CJRP data were not available at the time SYRP Field Test facilities were sampled, so the 1997 CJRP frameand size data were used.

The CJRP indicates the number of offender youths who were residents on the last Wednesday in Octoberduring the survey year, together with some key characteristics such as the youths’ gender and race/ethnicity. Thedraft SYRP sampling plan calls for oversampling females at 3 times and Hispanic males at 1.5 times the rate of othermale offenders, so these weighting factors were used in computing the facility-level measure of size. Also, thesampling plan was designed to undersample small facilities and to provide larger facilities with higher probabilitiesof selection. For the Field Test, this approach was used to select a sample of 49 facilities in three contiguous states inthe mid-Atlantic region. Subsequent comparisons between the 1997 and 1999 CJRP listings of the universes offacilities in this tri-state area showed that one facility had ceased to exist and 10 new ones were established over the2-year interval. This finding suggests that some method of incorporating new facility listings from the most recentCJRP into the sampling frame for the national SYRP should be considered. Since CJRP size data may not yet beavailable on these facilities, a method of providing them with a likelihood of selection will need to be devised.

The Field Test recruitment uncovered other changes beyond those observable in the CJRP framesthemselves. Two of the sampled facilities had closed. Others had considerably larger populations in residence thansuggested by their 1997 CJRP responses. For instance, one facility had reported 85 youths on the 1997 CJRP, butclaimed to have 106 youths when contacted just one week prior to data collection and actually had 128 youths at thetime of our Field Test visit. Still other sampled facilities were organizationally reconfigured. Two had “traded”youths to become single-gendered institutions—one facility taking all the males and the other all the females sincethe 1997 CJRP. Another complication had to do with the fact that a number of facilities in the CJRP are multiple-unit institutions, and not all of their functional units housed juvenile offenders. Moreover, the facility may not havereported the youth in all units to the CJRP. For example, one facility reported that it submitted CJRP data on youthsin its shelter program but not those in its substance abuse unit. Discussions with this facility and with the CJRPdesigners raised the question as to whether the SYRP should include all units that could house juvenile offendersplaced in the facility because they were charged with or court-adjudicated for an offense (the underlying intent ofthe CJRP), or only target the youth in those units that the facility actually reports on the CJRP.

Facility RecruitmentRecruiting the sampled facilities in Field Test involved getting clearances from a number of administrative

offices, and often multiple administrative offices in connection with a single sampled facility. Recruitment beganwith the three state-level juvenile justice agencies. In two of the states, the state juvenile justice office grantedclearance for the team to contact the sampled facilities and endorsed SYRP participation for all sampled facilities

Page 14: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1239

whose participation was approved by the state. In the third state, the state juvenile justice office had directadministrative authority over only a subset of the sampled facilities; was responsible only for monitoring anothersubset of the remaining facilities for compliance with the state manual of standards; and had no authority over theremaining sampled facilities in the state. Interestingly, this juvenile justice office sent a letter endorsing SYRPparticipation to both the facilities it administers and the facilities that it monitors, but entered into intensenegotiations about the parent/guardian consent procedures that would be allowable in its administered facilities.

The SYRP Field Test interviews are administered to all youths residing a sampled facility, regardless of whythey are placed there. Because of the broad scope of the facility universe, there are a number of facilities that houseboth offenders and nonoffenders. Some of the nonoffender youths are placed there by their families or by childwelfare authorities (child protective services or foster care agencies). In the SYRP field Test, not all administratorsof facilities that housed nonoffenders under child welfare placement required clearance or permission from childwelfare agencies before granting access to these youths. Some decided to grant in loco parentis approval for theparticipation of all youth in residence, regardless of their placement circumstances. While 16 facilities wereexpected to house nonoffender residents, the administrators of only seven of these facilities actually required childwelfare clearance before allowing their nonoffender youth to participate.

However, more than one local or county child welfare agency was typically involved in placing youths in agiven facility, which meant that multiple clearances often had to be negotiated. In fact, in attempting to recruit the 7facilities that involved child welfare clearances, discussions were held with 12 different child welfare offices.Private facilities are often run by larger corporations, which means that these facilities also frequently require otheroutside clearances. For 16 facilities, there were other non-governmental administrative authorities above the level ofthe individual facility administrator that had to approve the study. At the final level, participation of all agencies hadto be cleared with the local facility administrator. A total of 77 administrative offices had to be contacted forclearance in recruiting the 49 sampled facilities in the Field Test. Moreover, this does not reflect the full scope ofclearances that would have been applicable had no facilities refused or been identified as ineligible.

Facility recruitment was fairly successful, with 87 percent (n=41) of the 47 eligible facilities agreeing toparticipate, 8.5 percent (n=4) refusing, and the remainder (n=2) still in negotiations when the Field Test recruitmentwas terminated. Refusals occurred at both the state level (2 facilities) and at the level of the individual facility (2facilities). Interestingly, state administrators specifically denied approval for two facilities while granting approvalfor other sampled facilities in their state. They argued that these facilities were in upheaval, and could notaccomodate the demands of study participation at the time. One facility that refused was in the process of closingseveral of its programs due to a financial crisis, but did plan to continue operating its residential services for thejuvenile justice youths. The two facilities that were classified as out-of-scope reported that they were closed orwould be serving no youths at the time of the Field Test. Both had been emptied of youths in the wake of seriouscrises involving charges that staff had abused juvenile residents. Note that their situations are not very different fromthe situations of refusing facilities who were in upheaval or in the process of gradually closing down theirprograms—the difference may have simply been one of timing, with refusing facility less far along in their closingprocess than the out-of-scope institutions. Thus, the boundary between refusals and out-of-scopes may hinge on thetiming of recruitment and data collection relative to disintegration.

Overall, the Field Test experience suggested that facility refusals should not notably bias the SYRP findings.Although the potential for bias is present (facilities with the worst conditions or poorest services may be more likelyto refuse), the refusal rate is fairly low. Moreover, there is encouraging evidence from the fact that a number offacilities with relatively serious problems nonetheless did agree to participate. For example, participating facilitiesincluded one that had been emptied of youths in the recent past after a staff abuse crisis, another that was in lock-down after a riot just before Field Test, and a third that was recently severely criticized for problematic conditions.

Note that the multiple and overlapping administrative jurisdictions encountered during the Field Testrecruitment indicates that facility refusals in the national SYRP are likely to be clustered. For example, an entirestate could refuse to participate, or a private corporation that runs a group of facilities might refuse. Such clusteredlosses could contribute other biases to the study estimates. While there is no obvious remedy for these losses, theyshould be carefully documented in the national SYRP, so that any potential bias in the findings can be recognized.

In planning the SYRP sample design, it was difficult to decide whether to include or exclude offender youthsfrom the within-facility youth samples. Eventually, it was decided to include them in the Field Test, and to base thefinal decision on the Field Test experiences. At the time of this writing, we are recommending that the nationalSYRP exclude nonoffender youths, considering that the effort to include these youths brought another bureaucracy(the child welfare system) into the recruitment process, that efforts to get clearance from these agencies provedextremely time-consuming; and that most of these recruitment efforts failed. The possibility of excluding thenonoffenders will be revisited below in the discussion on within-facility sampling of youths.

Page 15: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1240

Interview Format and ProceduresOne source of bias that has been a central concern in this study is the measurement error that might affect

respondents’ interview answers concerning sensitive and private matters, such as their offense histories,victimization experiences, and needs for treatment and services. Two key SYRP design features were craftedspecifically to reduce this type of error—the format of the interview as an Audio Computer-Assisted Self-Interview(ACASI) and the use of strong anonymity procedures through all phases of the data collection process.

The SYRP ACASI interview is designed to be self-administered on a laptop computer with a touchscreen. Apre-recorded interviewer’s voice is included in the interview program. The participating youth dons a set ofheadphones and the computer “reads” the questions and answers to a youth who responds by touching the selectedanswer alternative on the screen display.

ACASI was chosen for several reasons. By having the computer read the questions, even juveniles with poorliteracy skills can self-administer the interview. This affords respondents greater privacy in answering questions thanstandard interviews and has been shown to increase the reporting of sensitive behavior, like drug use and offending,as well as mental health (Tourangeau and Smith, 1998; Moum, 1998). Use of this technology also increases dataquality because it reduces respondent error, restricting respondents’ answers within allowable ranges and ensuringthat skip instructions are correctly followed since they are automatically specified by the program. ACASI alsoprovides for greater standardization of the interview than can be achieved through the person-to-person format andallows for crucial checks during the course of the interview that can trigger interventions by study field staff whennecessary. As will be noted later, the potential of this feature has been exploited during the SYRP Field Test toensure that offender youths answer all offense-related questions.

Group administration of ACASI is a recent methodological expansion of this technology, and there wasconcern about its potential for undermining participants’ perceptions about the privacy of their responses. There issome evidence that group ACASI administration does reduce the reporting of sensitive information, but that thisdisadvantage can be overcome if respondents are seated more than 5 feet from each other (Beebe, Harrison, McRae,Anderson & Fulkerson, 1998). Preliminary pilot tests of the ACASI with groups in detention centers and sheltersindicated that seating could be arranged in these contexts so that youths were assured privacy during theirinterviews. Accordingly, the SYRP Field Test followed a specific protocol regarding seating distances andorientations that maintain privacy, and debriefings were conducted in selected facilities to determine the success ofthis protocol. We also mapped the seating arrangements and measured the seating distances achieved in the pretestfacilities. The Field Test indicated that is possible to seat the youths so they are not side-by-side at the same table inthe majority of facilities and that sufficient spacing can generally be arranged within the accomodations available, ifonly by expanding the number of sessions so that there are fewer youths in any given group.

Strict anonymity procedures were also specifically designed to enhance the validity of respondents’ answers.The study protocol dictates that the SYRP project staff never possess information that would allow them topersonally identify individual youths. Facilities are asked to provide listings of their youths to the study by usingfacility identification codes. The facility staff retain their master listing that links the facility codes with youths’names, and they are asked to destroy this listing after all the study data are collected. Thus, SYRP staff haverespondents’ interview answers but no names, whereas facility staff have respondents’ names but no interviewanswers. Maintaining this strict anonymity has meant that the facility staff had to assume responsibility for obtainingconsent from parents or guardians—an arrangement that risks reduced response rates in favor of enhancingrespondents’ privacy.

Finally, in administering the SYRP interview, youth are introduced to the study on the ACASI system, andthey register their assent by touching an “I agree” response on the screen. This was the first electronic “signature”approved by Westat’s IRB. It was necessitated in the SYRP by the strict anonymity procedures, which rendered themore traditional signed consent forms unfeasible.

Sampling Youth Within FacilitiesThe SYRP sample design is based on the assumption that a cluster size of about 60 youths will be optimal.

The final results of the Field Test experience will permit a more informed decision about cluster size, taking bothcosts and operational issues into account. The draft design also assumed that most of the nonresponse would occur atthe facility level and that a 94% within-facility participation rate could be achieved—an assumption that certainlymust be altered on the basis of the Field Test experience, as noted below. In small facilities, every youth is selectedinto the sample, whereas in larger facilities, only a subset of residents is selected. Oversampling is again applied atthis stage of sample selection, with females sampled at 3 times the ordinary rate, Hispanic males at 1.5 times theordinary rate, and nonoffenders at 0.75 the ordinary rate.

In practice, all youths in residence at the time of the data collection visit are rostered, together withinformation about the youths’ gender, Hispanicity, and offender status, and a sample is selected to participate. Indeciding who should be included on the roster and how they should be classified, it was necessary to standardize themeanings of the terms “resident” and “offender.” Youths occasionally are temporarily sent to another facility (e.g.,

Page 16: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1241

psychiatric hospital). In many cases, those youths would still be “on the facility roster.” Youths for whom bed spaceis saved would always be “on the roster.” This means that if the roster listings are used as the sampling frames, itmay happen that a sampled youth is not physically in the facility, but the facility is holding bed space for him/her.Many facilities provide weekend passes to their residents, who are out of the facility on home visits at that time. Stillother youths may be gone from the facility because they ran away. While some of these might return, others maynot. During the extensive telephone and roundtable discussions, facility administrators reported that they keep thebed space for a runaway youth until the court order or custody order is signed that releases the facility fromresponsibility for that youth. For the Field Test, the procedure was to include on the sampling roster all youths withan assigned bed at the time the SYRP field team entered the facility to begin data collection. When sampled youthscould not participate because they were not physically present in the facility at the time of testing, the field teamrecorded the reason for the youth’s absence.

In the Field Test, the facilities were asked to indicate which of the rostered youths were “offenders” andwhich are not, since nonoffenders were slightly undersampled. One of the difficulties faced by the CJRP designerswas how to ensure that facility administrators would correctly classify youths as offenders, since status offendersconstitute a “gray zone” the boundaries of which vary across facilities and states. Status offenders are youths whohave done something that would not be a crime if they were an adult (e.g., running away, being truant from school,violating curfew, etc.). Status offenders are considered offenders in the CJRP and will be in the SYRP as well, aslong as their offense is considered illegal in their jurisdiction. Status offenders are sometimes placed in thesefacilities by non-justice agents and agencies (social services, schools, parents), another factor that might confusetheir classification from the facility’s perspective. If nonoffenders are excluded in the national SYRP, there will bepotential for bias if facility staff do not list some offender youth who should be included. However, the SYRP FieldTest found extremely few offender youth who were not classified as such on facilities’ sampling rosters. It waspossible to examine this type of classification disagreement in the Field Test, because the facility’s classificationwas not the only source of information about the youths’ offender status—both the ACASI interview and the CJRPPart II data provided information on this as well.

Examination of the records for the 811 youths who completed the ACASI interviews showed that there werea total of only 22 cases where the indicators about the youth’s offender status did not agree across all three of themeasures. For 15 of these cases, the facility identified the youths as offenders but the youth identified themselves asnonoffenders during the ACASI interview. In six additional cases, the facility indicators disagreed with each other(the sampling frame had identified the youths as nonoffenders but the CJRP Part II had identified them asoffenders). In the remaining case, the youth claimed to be an offender but had been listed as a nonoffender on thesampling roster. This suggests that the loss of youth stemming from facilities’ omissions in listing offenders wouldgenerally be quite low (less than two percent). Nevertheless, it would be well to include some additionalprecautions to keep it negligible in the national SYRP. Toward this end, it would be helpful to provide clearer andmore comprehensive instructions to facilities defining the criteria for classifying offenders.

The ACASI interview was designed to uncover the type of discrepancy where the facility roster lists theyouth as an offender but the youth denies being an offender during the interview. In this situation, the ACASIprovides an alert message that signals the respondent to call the SYRP field staff to intervene at that point. Directintervention by the SYRP field staff was arranged because it is important to ensure that the youth not skip over theseries of offense-related questions if they are applicable.

Because the Field Test revealed that the effort to include the nonoffender youth added a major burden to therecruitment effort, our recommendation is that these youth be excluded in the national SYRP sample design. Giventhat nonoffenders are not the central focus of the study, the implication is that nonoffender youth should not beincluded in the national SYRP.

Youth-level NonresponseIn the Field Test, 69 percent of the 1,179 eligible sampled youths completed ACASI interviews. Eleven

percent of the eligible sampled youths were not interviewed because they were unavailable (out of the facility) at thetime of data collection. Sixteen percent of the eligible sampled youths were lost due to consent issues. There werethree consent issues that affected participation for individuals within facilities: individualized facility consent,parent/guardian consent, and youth consent. Although the Field Test did not encounter other individual-consentbarriers, the roundtable and telephone discussions did indicate that the national SYRP might expect to contend withconsent by individual caseworkers, probation officers, juvenile court judges, or even the youths’ lawyers.

Several Field Test facilities denied access to some of the individual youths who were sampled to participate,typically because of disciplinary or security reasons. One percent of the eligible sampled youths did not participatein the Field Test because the facility refused access to these specific individuals.

Because of the broad range of facility types, jurisdictions, and placement situations represented in the SYRP,Westat’s Human Subjects IRB (Internal Review Board) determined that the study would comply with federalrequirements for human subjects’ protection by following whatever parent/guardian consent arrangements were

Page 17: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1242

stipulated by the the administrators of the participating facilities. Accordingly, the SYRP Field Test negotiated arange of consent procedures.

The preferred option, from the perspective of the SYRP project team, was in loco parentis, whereby thefacility administrator assumed the authority in the place of the parents/guardians to grant consent for all sampledyouths. Ten of the 41 Field Test facilities with consent procedures finalized had offered in loco parentis consent forsome or all of their youths. In eight facilities, the administrator granted in loco parentis consent for all the youths. Inone facility, written consent was required for the child welfare youths but in loco parentis consent was granted forthe youths who were placed by juvenile justice. In another facility, passive consent was required for youths whowere in the detention unit, but in loco parentis consent was granted for the youths in all other units.

Next in order of preference from the study’s perspective was passive consent. Under this arrangement,parents/guardians of sampled youths were sent written notices describing the study, explaining that the facility wasparticipating in this survey, and instructing them to contact the facility if they had any objection to their child beinginterviewed. Fifteen of the 41 recruited facilities accepted passive consent arrangements for youths—13 accepted itfor some youths but required written consent for others. and two facilities combined passive consent with otherconsent arrangements for different sectors of their residents.

Thirty two facilities required active consent for some or all of their youths: In 13 of these facilities, onlywritten consent was acceptable, while in the remaining 19 facilities, written consent arrangements for some sector ofthe youths were combined with other consent arrangements for other youths.

When facilities required active consent and at least a sector of residents was expected to enter the facilitybetween the time of recruitment and the date of the data collection visit, arrangements were made to have the facilityinclude the request for written consent in the intake packet that parents/guardians completed when the youth wasadmitted into residence. Another strategy for obtaining written consent was to have the study request conveyed toparents/guardians when they came to the facility for visiting hours.

As discussed above, the strict anonymity procedures used in the study precluded SYRP project staff frompossessing identifying information about study participants. This has meant that facility staff had to bear theresponsibility for pursuing any parent/guardian consents they required. The Field Test revealed that 11 percent of theeligible sampled youths could not participate in the ACASI interview because of nonresponse by parents/guardians.No parents/guardians ever overtly refused to permit their children to participate—in all cases these were failures tosubmit the required written consent form in advance of the SYRP data collection at the facility. Although the FieldTest results have not yet been explored in detail, it is reasonable to suppose that such losses were more prevalentwhen written consent procedures were required in contexts where the intake packet or visiting hours strategies werenot useful (because turnover rates were low or visits by relatives infrequent). Note that these circumstances are morelikely in the higher security facilities with the more serious offenders. Higher rates of nonresponse in these contextswould bias the study findings by underrepresenting this sector of offenders unless the impact of the nonresponse canbe countered by well-designed nonresponse adjustments or the nonresponse can itself be reduced by enhancedefforts to obtain parent/guardian consents. In the Field Test, the timeframe did not afford much lead-time for thelow-turnover facilities to pursue parent/guardian consent—and it was found that parent/guardian consents were stillreceived substantially after the field team had completed their ACASI data collection visits. Based on this, a keyrecommendation for the national SYRP is to substantially increase the lead-time for written parent/guardian consent.

Scheduling ConstraintsIn practice, a number of scheduling constraints affected youth-level response rates in the SYRP Field Test.

The need to provide adequate time to pursue parent/guardian consent had to be balanced against the need to ensurean up-to-date sampling frame of facility residents. The SYRP sampling procedures were designed to accomodateboth of these needs. When parent/guardian consent was required, an initial rostering of the youths in residence wasobtained in advance of the data collection visit, with an eye toward allowing enough time to obtain the necessaryconsent. The decision as to when this initial sampling frame should be developed was made in cooperation with thefacility liaisons, based on their judgments as to how quickly the frame would become outdated, and as to how longthey would require to accomplish the required consent procedures. Whenever a roster was developed in advance ofthe data collection visit, the sampling frame was updated at the time of that visit. Youths who were currentlyassigned a bed but who were not in the initial roster were added and a final sample was drawn from these recententrants. For some of the youths in this final sample, consent was already in hand if it had been obtained via theintake packet strategy for all newly-entering youths, or if the facility decided to grant in loco parentis consent forthis sector of youths.

Even when it was not necessary to draw a sample in advance because the facility had granted in loco parentisconsent, it was still necessary to predict the expected sample size at a facility so that sufficient time could bescheduled for the data collection visit. Actual sample size depended on the number of youths in residence at the timeof data collection, and some facilities experienced quite a bit of fluctuation in the size of their resident populations.In the Field Test, this circumstance led to an unfortunate situation in one facility, where the expected sample size

Page 18: CONCEPTUALIZATION AND MEASUREMENT ISSUES FOR … · conceptualization and measurement issues for surveys of juvenile facilities concepts, constructs and the impact of empirical research

1243

was considerably lower than the size of the sample that was actually selected on the day of the visit and only two-thirds of the selected youths could be tested in the time available. On arriving at the facility, the field staff enteredthe necessary roster information into the field management software and were surprised to learn that the sample was50 percent greater than anticipated. At the same time, the facility could not adjust the data collection visit to provideadditional time. Many facilities have had to arrange for extra staff to be on duty during the days and times of testingso that the youths could be brought to and from the testing sessions as necessary. Some facilities indicated that theywould need as much as two weeks notice if there was to be any change in the data collection schedule. To avoidlosing participants because of unexpected sample fluctuations when available data collection time is tightlyconstrained, the national SYRP should provide field teams with the ability, if necessary, to specify a maximumfeasible sample size in a facility.

In the Field Test, 4 percent of the eligible sampled youths refused to participate, and it was found thatschedule considerations often influenced their decision. Facility liaisons sometimes suggested data collectionarrangements that proved to conflict with other activities that were considerably more attractive to the youths. In oneinstance, the field staff had conducted several interview sessions uneventfully, but then suddenly began toexperience a rash of refusals—with many youths not even agreeing to sit down at a laptop in the ACASI testingsession. Upon questioning the facility staff, it was discovered that the youths had begun to watch the movie, “TheSixth Sense.” After that episode, SYRP recruiters were careful to ask facility staff what other activities would bescheduled at the facility during the data collection visits. Only by explicitly asking this question was it learned that atime window the facility liaison offered would overlap with a Mother’s Day celebration!

In summary, a number of features of the SYRP design have been tailored to minimize bias and to enhanceresponse rates and the validity of measurements. As indicated at several points above, the Field Test experienceshave shaped several recommendations for revising the design and procedures for the national SYRP.

ReferencesBeebe, T.J., Harrison, P.A., McRae, J.A., Anderson, R.E., & Fulkerson, J.A. (1998). An evaluation of computer-

assisted self-interviews in a school setting. Public Opinion Quarterly, 62, 623-632.Moum, T. (1998). Mode of administration and interviewer effects in self-reported symptoms of anxiety and

depression. Social Indicators Research, 45, 279-318.Tourangeau, R., & Smith, T.W. (1998). Collecting sensitive information with different modes of data collection. In

M. Couper, R.P. Baker, J. Bethlehem, C. Clark, J. Martin, W.L. Nicholls II, and J.M. O'Reilly (Eds), ComputerAssisted Survey Information Collection (pp. 431-454). John Wiley & Sons: New York.