2003 Populating an International Web-Based Randomized Trials Register in the Social, Behavioral, Criminological, And Education Sciences

7/28/2019 2003 Populating an International Web-Based Randomized Trials Register in the Social, Behavioral, Criminological, …

http://slidepdf.com/reader/full/2003-populating-an-international-web-based-randomized-trials-register-in-the 1/22

American Academy of Political and Social Science

Populating an International Web-Based Randomized Trials Register in the Social, Behavioral,Criminological, and Education SciencesAuthor(s): Herbert Turner, Robert Boruch, Anthony Petrosino, Julia Lavenberg, Dorothy deMoya and Hannah RothsteinReviewed work(s):Source: Annals of the American Academy of Political and Social Science, Vol. 589, MisleadingEvidence and Evidence-Led Policy: Making Social Science More Experimental (Sep., 2003), pp.203-223Published by: Sage Publications, Inc. in association with the American Academy of Political and Social

Science

Stable URL: http://www.jstor.org/stable/3658567 .

Accessed: 13/03/2013 22:03

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of

content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms

of scholarship. For more information about JSTOR, please contact [email protected].

.

Sage Publications, Inc. and American Academy of Political and Social Science are collaborating with JSTOR

to digitize, preserve and extend access to Annals of the American Academy of Political and Social Science.

http://www.jstor.org

This content downloaded on Wed, 13 Mar 2013 22:03:01 PMAll use subject to JSTOR Terms and Conditions

http://www.jstor.org/action/showPublisher?publisherCode=sage

http://www.jstor.org/action/showPublisher?publisherCode=aapss


http://www.jstor.org/stable/3658567?origin=JSTOR-pdf

http://www.jstor.org/page/info/about/policies/terms.jsp





http://www.jstor.org/stable/3658567?origin=JSTOR-pdf



http://www.jstor.org/action/showPublisher?publisherCode=sage



Populatingan

International

Web-BasedRandomized

TrialsRegisterin the Social,

Behavioral,

Criminological,and Education

Sciences

By

HERBERT TURNER,ROBERT BORUCH,

ANTHONY PETROSINO,

JULIALAVENBERG,DOROTHY DE MOYA,

andHANNAH ROTHSTEIN

Underlying the work of the Campbell Collaboration

(C2) is the Sociological,Psychological,Educational,and

CriminologicalTrialsRegister (C2-SPECTR).A Web-accessibledatabase,C2-SPECTRisuniqueintheworld.

With more than 11,600 citations, it is an internationalregisteron randomizedcontrolledtrials(RCTs)orpossi-ble trials.This article describes the framework orpopu-latingC2-SPECTR,otherregistersthatareprospective,and the practical ssues of implementation.The authorsdiscuss the growing mportanceof RCTsand the recenthistories of organizations that have influenced this

growth-the Cochrane Collaboration,the C2, and theWhatWorksClearinghouse.Next, the authorsdescribethe originsof C2-SPECTRandplanstopopulateit anda

prospective register. The authors conclude with plans

forimplementingthe surveillancesystemsandthe antic-ipated challenges in actualizingthese plans.

Keywords: randomized trials; randomized experi-ments; Campbell Collaboration;system-atic reviews; electronic databases;search

engines; Cochrane Collaboration; WhatWorksClearinghouse

Populatinga Web-BasedRegisterof Randomized

ControlledTrials

Randomized controlled trials (RCTs) arestudies in which individuals or groups of individ-

uals, including organizations, are randomly

assigned to different interventions to evaluatethe relative effectiveness of the interventions.

When implemented well, RCTs produce unbi-ased estimates of these relative effects (Boruch1997; Rossi, Freeman, and Lipsey 1999). The

Herbert Turner s Managerof ResearchProjectsand amemberof the InformationRetrieval Methods GroupsSteeringCommittee,bothwith the InternationalCamp-bell Collaboration.He is also a postdoctoralstudent inthe Graduate Schoolof Education at the UniversityofPennsylvania.Robert Boruch is UniversityTrusteeChairProfessoratthe Graduate School of Education, Statistics Depart-ment of the Wharton School, and the Fels Centerfor

DOI: 10.1177/0002716203256840

ANNALS, AAPSS, 589, September2003 203







THE ANNALS OF THE AMERICAN ACADEMY

freedom from bias is ensured by the random assignment of individualsor groups of

individuals to the different regimens. There is no systematic difference between

the groups so composed (Shavelson and Towne 2002; Mosteller and Boruch 2002).This in turnproduces a faircomparison. Alternativesto RCTs do not carrythe same

assurancebecause they arenot randomized. In particular,assumptions areinvoked

and implemented through statistical adjustments that make the groups appear

equivalent. The assumptions usually cannot be tested.Some recent data show that researchers are using RCTs more frequently in the

social, behavioral, and education sciences to generate scientific evidence on "what

works" (Petrosino et al. 2000). This bodes well for policy makers in the United

States and abroadwho rely on scientific evidence to develop the most effective pol-icies,

programs,and

practices.However, researchers interested in

identifying

and

synthesizing RCTsface the formidable challenge of efficiently locating the highest

possible number of RCTs to include in their syntheses. To meet this challenge,these researchers need access to a Web-based register of RCTs that is routinely

populated through retrospective and prospective surveillance systems.This article describes the framework for populating a Web-based register of

RCTs and practical issues of implementation. The primaryfocus is on the Camp-bell Collaboration's Sociological, Psychological, Educational, and CriminologicalTrials Register (C2-SPECTR), which is available through the virtual Campbell

Library at www.campbellcollaboration.org. Also described is the proposed con-

nection between C2-SPECTR and the U.S. Department of Education's WhatWorks Clearinghouse (WWC).

In what follows, we discuss the growing importance of RCTs and the recent his-

tories of organizationsthat have influenced this growth-the Campbell Collabora-

Government,ll at the UniversityfPennsylvania. e is co-chair f theCampbellCollabora-tion'sSteeringGroup.

JuliaLavenbergs a doctoraltudentandUniversityChairTrustee ellowship ecipientn theGraduate chool fEducationttheUniversityfPennsylvania.he erves ntheSteeringCom-

mitteeof TheCampbellCollaboration'snformation etrievalMethodsGroup.She hasbeenactive in the development f hospitalpoliciesandprocedures, ocumentationrotocolsornurses, ndpatienteducationmaterialsnherpastroleas a registeredurse.Herresearchen-tersonidentifyingandomizedrials n socialwelfare, ducation,ndcriminologyndexploringthe specializederminology sedby different ocialscientists o discussanddocumentheirresearch.

Dorothyde Moya s ExecutiveOfficer fTheCampbellCollaborationndserves heInterna-tionalCampbellCollaborationSteeringGroup.She s anadjunct ssistant rofessor t theUni-

versity fPennsylvania,chool fNursing.Dr.DeMoyao-convenedhe C2RockefellerellagioConferencen PlaceRandomizedrials ndcontributeso theDepartmentfEducation'sWhatWorksClearinghouse.heearned er doctoraldegreefrom heCatholicUniversityfAmerica

School fNursing,Washington, .C.HannahRothsteinsprofessor fmanagementt BaruchCollege ndcoordinatorfthedoctoral

specializationn Organizationalehavior ndHumanResourcesManagementf thePh.D. nBusinessProgramt theGraduateCenter ftheCityUniversityfNewYork. he s theconvener

oftheInformation etrievalMethodsGroup ftheCampbellCollaborationnda member fthe

Reporting iasMethodsGroupofthe CochraneCollaboration.

204







POPULATING A WEB-BASED RANDOMIZED TRIALSREGISTER

tion (C2), the Cochrane Collaboration, and the What WorksClearinghouse. Next,we describe the originsof C2-SPECTR andplans for retrospective andprospective

surveillance systems to populate it. We conclude with plans for implementing thesurveillance systems and the anticipated challenges in actualizing these plans.

The GrowingImportanceof RCTs

Indicatorsof the growing importance of RCTs

Several indicators point to the increasing importance of RCTs in the social,behavioral, and education sciences. These indicators include

special

conferences

on RCTsand resulting publications, references in law to RCTs,and the prevalenceof articles and reports on RCTs in various social science domains.

Specialconferencesand publications

One indicator is the surge in recent international conferences on RCTs and

resulting publications. For example, a recent international conference was orga-nized in Germany on the importance of RCTs in research in civil, criminal, and

juvenile justice (Krisbergand Schumann 2000). The conference contributorswere

from Germany, Switzerland, the United Kingdom, the United States, and Israel.Similarly,international meetings have been convened by Sweden's new NationalCenter for Evaluation of Social Services. These meetings involved researchersfrom Sweden, Denmark, Norway, Finland, Britain, Scotland, Canada, and theUnited States and focused on how to enhance the qualityof evidence on the effec-tiveness of social services using RCTs and other approaches. A more recent inter-nationalexample is the conference that focused on place-randomized trialsto eval-uate the effectiveness of interventions in health, education, crime, welfare, and

employment held in Bellagio, Italy.Plans areunder way to publish the proceedings

of this conference as an edited volume.In the United States, political scientists recently hosted a special conferencefocused partlyon the use of RCTs to evaluate the relative effectiveness of strategiesto increase voter participation (Green and Gerber 2002). Efforts to increase our

understanding of RCTs in the education arena, undertaken under the auspices ofthe American Academy of Arts and Sciences, have led to both national meetingsand an edited volume on the topic (Mosteller and Boruch 2002).

Legislation

At times, the interest in RCTs is reflected in legislation. In the United States, for

instance, the 2001 No Child Left Behind Act (NCLBA) uses the term "scientificevidence" more than one hundred times, making clear congressional interest inbetter evidence.1 RCTs are identified explicitly as a device for estimating the rela-tive effectiveness of innovative education policies, programs,andpractices. In gen-

205







THE ANNALS OF THE AMERICANACADEMY

FIGURE 1PREVALENCE OF TRIALS OR POSSIBLE TRIALS IN C2-SPECTR (1925-1999)

12000

10000

8000

6000

4000

2000

0 * , ?I + , ,- # I

1920s 1930s 1940s 1950s 1960s 1970s 1980s 1990s

Publication Years

NOTE: Chartplots entries on trialsorpossible trials n the CampbellCollaboration'sSociologi-cal, Psychological,Educational,andCriminologicalTrialsRegister(C2-SPECTR).The numberof entries to trialspriorto 1960 is low but not zero.

eral, the use of research-proven effective policies, programs, and practices is a

guiding principle of the NCLBA.

Through the Education Sciences Reform Act of 2002, the U.S. Congress autho-rized the formation of the National Institute for Education Sciences (NIES) (H.R.

3801).The NIES is

chargedwith

providing leadershipin the

productionand use of

scientific evidence to evaluate the efficacy of education policies, programs, and

practices. Both the NIES and WWC (described below) depend on their organiza-tional capacityto identify scientificallyvalid studies of interventions that purportto

improve educational outcomes such as school achievement.

Prevalence of articles and reports

Yetanother indicatorof interest in RCTsis the number of articlesand reportsonthem (Petrosino et al. 2000). Since Campbell and Stanley published their seminal

work in 1966, Experimental and Quasi-Experimental Designs for Research, theprevalence of articles on RCTs (or possible RCTs) has increased steadily in the

social, behavioral, and education sciences (see Figure 1). The number of articles or

reports on RCTs is monitored by C2 through its Web-based trials register, C2-

SPECTR.

206







POPULATINGA WEB-BASED RANDOMIZED TRIALSREGISTER

Choppinessin the growing importance of RCTs

AlthoughRCTs have become

increasingly important,their use has not

increaseduniformly rsimultaneouslycross ocialsciencedomains.Forinstance,incriminology,GarnerandVisher 2002)madethepointthatthe numberof trials

sponsoredby the NationalInstitute for Justicefrom 1990 to 2000 had actuallydecreased.Similarly,GreenandGerber 2002)notedthat n the U.S.political ci-ence arenafrom1990 to 1999,not a singleRCTwas conducted.

U.S. educationpolicyhasonly recentlyemphasized he use of RCTs o answerscientificquestionsaboutdiscernibleeffectsof programs. n contrast, he earlier

emphasis during he 1980sand1990s)was onansweringquestionsregarding ro-gramprocessandimplementation.With this recentemphasis, he expectations

thatin education he numberof RCTswillgrowmoreuniformly see theWWC'sWeb site athttp://w-w-c.org).

Implications or systematicreviews

Systematicreviews assemble and synthesizeresearchto addressquestionsabout effects of a particular nterventionor class of interventions(Chalmers,Hedges,andCooper2002).Whenconductedwithexplicitandtransparentmeth-odsthatareopento criticismbyothers n the field,systematic eviewshelpbuild

scientificknowledge.Inanyscientificdiscipline,making decisionabout"whatworks" asedon asin-

gleRCT srisky, egardless fthetrial'squality.Forexample, heTennessee rialonclasssize demonstrated hatreducedclass size hadpositiveeffects on measuredoutcomessuch asstudentachievement Ritterand Boruch1999).However, heseresultscouldnot be replicatednanonrandomizedrial n California. hedifferingresultsof these trialshighlight he importanceof examiningmultiplestudieson a

particularopicbeforedrawing onclusionsabout"whatworks"-especially ndif-ferentsettings(Traub 002).

Makinga decisionabout"whatworks"basedon

multiple

RCTs s becomingas

critical n the social,behavioral,andeducationsciencesas it is in medicine.Onecanimaginehow in the past,a systematicreviewmightcontainbut a few RCTs.This low numberwas acceptablebecause RCTswere difficultto locate throughelectronicsearches.Handsearcheswere too laboriousandtime-consuming.Fur-

thermore,reviewers ackedaccessto trialsregisters hatwouldempowerthem tolocate the maximumnumberof trials n the leastamountof time.

Makingteasier orresearchers oidentifyrelevantRCTs,whetherpublishedn

peer-reviewed ciencejournalsor inreportsby organizationshathavegoodinter-nalandexternalpeerreview, eemsimportant.2Onewayto makethe compilation

taskeasier is to develop registersof completedor newlyinitiatedRCTs.A modelfordeveloping ucharegisterexists nthefieldof health.TheCochraneCollabora-tion's rialsregister,CENTRAL,containsmore than350,000citationsof RCTs nhealthcare andrelated fields.With access to this registerandothers,CochraneCollaborationparticipantshaveproducedmore than 1,000 systematicreviews.3

207








With the founding of the international C2 in 2000, the goal is to establish a similar

trials register in the educational, social, crime, and behavioral sciences. The U.S.

Department of Education'sWWC also will develop a register of studies, includingRCTs, but with a focus on U.S. education.

With the emergence of the Cochrane Collaboration, the C2, and the WWC,research synthesizers can and should expect to have access to trials registers to

accumulate the best evidence in their review portfolios. While these organizationsareworking to find common ground on a number of issues, they agree that some of

the best evidence to include in the research synthesis comes from RCTs.They also

agree that an important vehicle for providing this evidence is a trials register.

The C2, the WWC, and C2-SPECTR

The C2

The C2's mission (see http://www.campbellcollaboration.org) is to prepare,maintain, and promote access to systematic evidence on the effects of interven-tions implemented in the United States and abroad (Schuerman et al. 2002). In

particular,the C2 is concerned with evidence on "whatworks" n crime andjustice,social welfare, education, and other social and behavioral sectors. This evidence is

not based on popular opinion, political expedience, hidden agendas, and personaltestimonies. Rather, it is based on systematic reviews of research findings gener-ated primarily by experiments and, to a lesser extent, on high-quality quasi-

experiments.

TheWWC

Established in August 2002 by the U.S. Department of Education, the WWC

aspires to prepare, maintain, and promote access to systematic reviews of studies

on the effects of U.S. education policies, programs,andpractices (see http://w-w-c.org). The WWC is developing four Web-based registries including one on educa-tion interventions that will provide reviews of potentially replicable programs,products, and practices intended to enhance student outcomes. A linked registerwill provide information on people and organizations that do scientific evaluation

studies. C2-SPECTR is a resource for the WWC and others.

C2-SPECTR

The C2-SPECTR underlies some of the common work of the C2 and the WWC.

As defined by Dickersin (1994), a trials register is a database of research studiesthat are either planned, under way, or completed (or some combination of all

three) that areorganized around a common set of characteristics.In the case of C2-

SPECTR, studies are included if they can be classified as an RCT;acluster-, place-,

208








or group-randomized control trial (CRT); or a possible randomized control trial

(CCT). The CCT classification is used for studies in which random assignment

could neither be determined nor ruled out. C2-SPECTR serves as an importantpotential resource for reviewers to identify trialsfor inclusion in C2 and WWC sys-tematic reviews and for reviews conducted by other organizations.

The initial effort to populate C2-SPECTR was led by the U.K.'sCochrane Cen-

tre (see http://www.cochrane.org/cochrane/centres.htm#1700) and the U.S.'s C2

Crime and Justice Group (see http://www.aic.gov.au/campbellcj).Trainedby U.K.

Cochrane Centre staff, hand searchers examined the contents of all issues of forty-

eight journals in sociology, psychology, education, criminology, and other related

fields. Steve MacDonald in the U.K. Cochrane Centre also directed the search of

Makinga decisionabout "whatworks"based

onmultipleRCTs s becomingas critical

in thesocial,behavioral, ndeducation

sciencesas it is in medicine.

three electronic bibliographic databases: Educational Research Information

Clearinghouse (ERIC), Sociological Abstracts, and Crime and Justice Abstracts.

Published bibliographies and books on RCTswere also searched to populate C2-

SPECTR.4In the end, the electronic search yielded 7,249 records that met the inclusion

criteria. The hand search yielded 3,200 records. In all, 10,449 records were used to

initially populate C2-SPECTR. For more information on this search process, seePetrosino et al. (2000). In its early form, the trials register was released in CD-

ROM and public form as SPECTR.In December 2000, the U.K. Cochrane Centre transferred SPECTR to the C2,

which renamed it C2-SPECTR. The C2 Secretariat immediately began augment-

ing it with additional trials,orpossible trials.For instance, approximatelytwo hun-

dred new citations on place-randomized trials have been included based on a spe-cial C2 project sponsored by the Rockefeller Foundation. In addition, more than

one thousand citations to RCTsor possible RCTshave been added to C2-SPECTR

through periodic hand searches and spontaneous referrals.C2-SPECTR is unique in the world. With more than 11,600 citations, it is the

most comprehensive register on RCTs or possible RCTs,covering interventions in

the sociological, psychological, educational, and criminological spheres.5 Despitethe ambitious initial effort to populate C2-SPECTR, Petrosino et al. (2000) esti-

209








mated that C2-SPECTR contained only a fraction of relevant published and

unpublished studies in the C2's ambit. For C2-SPECTR's coverage of trials to

increase substantially,it is critical that the C2 develops a strategy for routine sur-veillance and augmentation. Doing this requires targeting sources of completedand newly initiated trials and determining the feasibility of identifying and retriev-

ing these trials from the targeted sources.6Hereafter, the main question addressed

is this: Howwill the C2 substantiallyincrease the number of trialsin C2-SPECTR?

Surveillanceof Sourcesof Completed Trials

At least sevenmajor

intellectual resourcesmight

beexploited

toaugment

C2-

SPECTR with completed trialsand make it useful to the C2, the WWC, and others.

These resources include

1. handsearches f fulltextofjournals;2. sophisticatedlectronic earches fjournals nddatabases;3. systematiceviews rom heC2,theWWC,andsimilarources;4. non-C2reviews;5. C2 secretariatetectionandspontaneouseferrals;6. CochraneCollaboration;nd7. theC2andemerging echnologies.

Directed hand searches offull text ofjournals

Directed hand searches of peer-reviewed research journals involve the use of

supervised and trained individuals to systematically search the full text of all arti-cles in each journal issue to identify trials and possible trials. This type of search

usually directs attentions to the methods section for each article in ajournal issue.The C2 began directed hand searches of journals in 2002. Prior to that time, theC2'ssearch for trials-in whatjournals and when-rested on inclination and moti-

vation ofgraduate

students at theUniversity

ofPennsylvania

and researchers in the

Campbell and Cochrane Collaborations. For a directed hand search of journals to

become routine in the C2, one can argue that the C2 needs a strategyfor systematicrecruitment of hand searchers, a manual for systematic hand search procedures,and a strategy for raising funds to pay hand searchers.'

Directed hand searches routinely yield many more randomized trials than elec-

tronic searches on the same journals. In the health care arena, for example, in a

study of trialson the efficacy of treatments to improve patient vision, an electronicsearch of MEDLINE was compared to a corresponding hand search of sixtop-tiermedical journals. Only 41 percent of studies identified through the hand search

were identified by the electronic search of the same journals in MEDLINE(Dickersin 1994). In a more recent comparison, Hopewell et al. (2002) hand

searched twenty-two specialized health care journals for all reports of trials and

possible trials and compared this with an electronic search of MEDLINE. Only 48

210







POPULATINGA WEB-BASED RANDOMIZED TRIALS REGISTER

percent of studies identified through the hand search were identified by the elec-tronic search.

In education, Leow and Boruch (2000) also found that directed hand searchesyielded more RCTsthan machine searches in their hand search for K-12 math and

science education trials reported between 1963 and 1998 in the American Educa-tion ResearchJournal. They identified thirty-five trials. In contrast, an electronicsearch of PsychINFO yielded ten of the thirty-five trials (29 percent). A search ofERIC yielded only one of the thirty-five trials (3 percent). Similarly, n apilot study,Boruch conducted an online hand search of the Journal of Education Psychologyfrom 1997 to 2002, while Turner did an electronic search for the same journal andtime period in ERIC, PsychINFO, and Sociological Abstracts.8 Boruch's handsearch

yielded ninety-four

articles whereas Turner'selectronic search

yielded onlythirty-one (33 percent). In other words, the hand search identified 67 percentmore citations than were identified by the electronic search.

These studies underscore the need for trials searchers to conduct hand searches

complemented by electronic searches to obtain the maximum number of trials,

especially in the social, behavioral, and education sciences.9 Studies of this kindalso illustrate that hand searches, though labor intensive, routinely yield a highernumber of trials.The WWC presents another opportunity to study the discrepan-cies in yields between hand and electronic searches. In particular,a senior scientistcan hand search published reports and brisklyjudge whether the studies are trials,

possible trials, and so on. Sophisticated younger scientists can try to do at least aswell by exploiting existing and emerging technologies for electronic searches. Theresults of the hand and machine searches could then be compared. The WWC'sefforts permit John Henry comparisons of this sort.10

Sophisticatedelectronicsearchesofjournals and databases

Sophisticated electronic searches use Web-based search engines (such as OVIDor WebSPIRS) and thesauri to search bibliographic databases such as ERIC or

International Political Science Abstracts. Searches of this kind are strategicallyconstructed to identify studies on a particular topic or question. To get a sense ofthe sophistication necessary to conduct a reliable and valid electronic search, con-sider this. Searching for RCTs on a topic, such as the efficacy of school-based vio-lence reduction programs, requires one set of search strategies for ERIC, anotherfor PsychINFO, andyet another for Crime andJusticeAbstracts. One reasonis thateach database has a "controlledvocabulary" o which the search strategymust con-form. For example, trials retrieved using keywords such as randomized trials andschool-based andviolence mayyield a different number of citations for ERIC thanfor PsychINFO or for Crime and Justice Abstracts. This is because synonyms for

the keywords used are constructed differently for each database. The searchermust also be cognizant of the Web-based search engine used, such as OVID or

WebSPIRS, because each engine requires a different syntax for searching thedatabase. For example, the truncation search term character for OVID is differ-

211








ent (a dollar sign, $) from the truncation search term character in WebSPIRS (an

asterisk, *).11

Sophisticated electronic searches retrieved a lower number of RCTs than thehand searches. However, those electronic searches require far less investment of

resources, especially time. The hit rate for electronic searches may improve over

time, but improvements depend on understanding the discrepancy between the

two types of searches. If the source of the discrepancy is the ineffective use of syn-

onyms in the electronic search strategy,then effective use of such synonyms-thatis, using athesaurus to identify synonyms-will improve the search (Hertzberg and

Rudner 1999). However, if the source of the discrepancy is that the RCT can onlybe identified by reviewing the methods section of the journal articleor report, then

even the mostsophisticated

electronic searchwill notimprove

because the method

sections of an article or report is not included in the bibliographic database.12

As of January2003, neither the C2 nor the WWC nor any other such organiza-tion had developed an instructional manual for conducting electronic searches

using different vernacular for different search engines and different databases.

Such a manual would articulate principles and provide examples that can be tai-

lored to retrieve trials from various bibliographic databases such as PsychINFO,ERIC, Sociological Abstracts,and Crime andJustice Abstracts.The C2 is develop-

ing such a manual, as is the WWC.

Similarly, C2 and WWC staffs are documenting search strategies and terms

used in electronic searches for RCTsin education. These RCTs will be entered intoC2-SPECTR. In refining their search strategies, C2 and WWC staffs are consult-

ing with library scientists. For the C2, these search strategies will be expanded

beyond education to include other domains of inquiryconsistent with the C2'smis-

sion. The C2 plans to develop an electronic search manual that will serve as a com-

panion guide to the hand search manual. Government sponsored efforts such as

the WWC can capitalize on each.To populate C2-SPECTR with completed trials using sophisticated electronic

searches, the C2 is capitalizing on the search strategies used initially to populate

C2-SPECTR (Petrosino et al. 2000). These search strategies are being modified toreflect updated "controlled vocabulary"across the bibliographic databases to be

searched such as ERIC and PsychINFO.

C2 and WWCsystematicreviews,

syntheses,and meta-analyses

The C2 and the WWC aim to produce systematic reviews that include the high-est possible number of RCTsin the body of evidence. This goal is demanding. Both

the C2 and the WWC havehigh

standardsfor reviews setby

theirsteering

commit-

tees. Reviews must include the highest number of high-quality studies. Some stud-ies areeasy to retrieve, whereas others are more difficult. The latter studies include

those injournalsnot subscribed to by reviewers' academic libraries,reports by gov-

212







POPULATING A WEB-BASED RANDOMIZED TRIALS REGISTER

ernment agencies and research organizations, dissertations and master's theses,conference proceedings, and studies that reside in the file cabinets of members of

the invisible college (an informalnetwork of colleagues with shared research inter-ests). Studies that are difficult to identify and retrieve, and fall into the aforemen-

tioned categories, are referred to among U.S. informational retrieval specialists as

the "fugitive literature."They are what researchers in other countries refer to as

the "greyliterature"(Eldredge 2000).For obvious reasons, individuals who undertake a C2 or WWC systematic

review are a valuable resource for providing information about RCTs to populateC2-SPECTR. So too are individuals and organizations that produce reviews based

on standards that are similar to those of the C2 or the WWC. C2 reviewers, for

example,

are asked to

expand

their searchbeyond

RCTs that are relatively easy to

Withtheemergence f the Cochrane

Collaboration,he C2,andtheWWC,research

synthesizers anand shouldexpect o have

access o trialsregisterso accumulatehebest evidence n their reviewportfolios.

retrieve. C2 Review Teams areexpected to retrieve RCTsfrom and supply RCTsto

C2-SPECTR. In addition, the C2 Information Retrieval Methods Group (IRMG)and Managerof Research Projects areworkingwith the C2 Steering Committee to

ensure that all candidate RCT citations, identified by systematic reviewers, areentered into C2-SPECTR.

Non-C2 reviews

Non-C2 reviews that are high quality and contain candidate entries for C2-

SPECTR will be used to populate C2-SPECTR. To assure the detection of these

candidate entries, the C2 IRMG Chair and Manager of Research Projects will

work with the C2 Steering Committee to develop policies and procedures. In prin-

ciple,then, the IRMG will

proposethat

anynon-C2 review be examined to detect

trials that were identified in the review.Twopromising sources for non-C2 reviews

arethe WWC andthe Cochrane Collaboration'sDevelopmental, PsychoSocial and

Learning Problems Group (DPLP).

213








The C2 Secretariat and Spontaneous Referrals to RCTs

The C2 Secretariatoften receives information about RCTsthrough

the informalnetwork of colleagues with a shared interest in the design and implementation ofRCTs in the United States and abroad. During 2002, for instance, references tomore than 120 RCTs were entered into C2-SPECTR based on referrals fromresearch colleagues, from papers submitted by scholars attending special confer-ences on place-randomized trials, and by civil servantswho are positioned well tounderstand RCTs that are mounted. These spontaneous referrals from collabora-tive networks inside and outside of the C2 are a valuable source of information ontrials and are referred to as the "invisible college."

As part of their work, the C2's IRMG and Manager of Research Projects are

working with the C2 Secretariat and Steering Committee to develop policies andprocedures to manage these referrals. The C2 also plan to periodically surveymembers of the "invisiblecollege."

The Cochrane Collaboration

Like the C2 in the social sector, the Cochrane Collaboration in the health sectoralso has a keen interest in "whatworks,"based on systematic reviews of accumu-lated high-quality evidence. Founded in 1993, Cochrane is more experienced than

Campbell in identifying RCTs. As mentioned earlier,the U.K. Cochrane Center'sDPLP Coordinating Group led the ambitious initial hand search of forty-eightjournals that yielded thirty-two hundred records for entry into C2-SPECTR. TheC2 will continue to strengthen the collaborative relationship between the two

organizationsto furthercapitalize on the Cochrane Collaboration'srich experiencewith identifying RCTs for use in reviews. To this end, the C2 is considering the

following:

1. aCochrane-Campbelliaison hat ostersheexchange f informationnd deasbetweenmembers f bothorganizations;nd

2. the C2's RMGandManagerf ResearchProjects, longwiththeCochrane-Campbellliaisonand he C2CommunicationsndInternationalizationroup,willdevelopafor-malprocedureorsharingrialsbetweenC2-SPECTRndCENTRAL.

The C2 and emerging technologies

A number of emerging technologies hold some promise for detecting RCTs. For

instance, some commercial firms such as the Institute for Scientific Information

(ISI) advertise their ability to electronically scan the full text of research journalsand use optical characterrecognition to convert this text into searchable words and

phrases at moderate costs. ISI purports that its TermTrackersoftware can performthese searches in more than two thousand journals. However, as of 2003, few ofthese journals were in the C2's domains of inquiry

Another emerging technology is exemplified by the highly advertised handscanner. The line-by-line models can be used instead of alibrarycopy machine, for

214








example, to scan the lines that document the study as an RCT,including the meth-

ods, title and author,and so on. C2 hand searchers experienced little success using

line-by-line scanners partly because they are not yet capable of accurate and fastscans. The emergence of the digital scanner is promising because it digitizesscanned articles into machine searchable text, but we have no empirical evidenceon this yet.

Most important, as more journals become available online in electronic format

(i.e., as E Journals)such that the full text of each reported study can be scrutinized,hand searches of journal articles can be conducted without a trip to the library.With a personal computer, Internet Service Provider (ISP), and an E Journalsub-

scription, hand searches can be conducted online and relevant sections of thetext-which identifies the

study

as an RCT-can be

printed.For instance, the C2 is conducting studies to compare the yield of RCTs fromhand searches of E Journals and machine searches of ERIC and PsychINFO.Because of resource constraints, the hand searches depend heavily on access toonline E Journals with full-text presentation. The full-text access is essential for

examining the methods section to determine whether a study is a trial or possibletrial. The online access also is essential in that it allows the examination to take

place wherever there is a personal computer with Internet connection, ISP soft-

ware, and a user ID and password to access a libraryWeb portal.Universities, such as the University of Pennsylvania, and organizations, such as

the C2, have access to high-end libraryWeb portals that provide access to the fulltext of a wide arrayof E Journals.This access enables researchers to conduct handsearches for RCTs efficiently and reliably. One of our studies showed that eighthours of searching an E Journal, the Journal of Educational Psychology, yieldedthree times more RCTs than a sophisticated machine search of ERIC and

PscyhINFO. This study also showed that searching an E Journal is limited by the

journal issue years that are available online. In the case of the Journal of Educa-tional Psychology, the years available online began with 1988.

Surveillanceof Potential Sourcesfor Newly Initiated Trials

In addition to C2-SPECTR, the C2 is developing aregister of newly initiated tri-als. This C2 Prospective Register of Trials (C2-PROT) will permit the C2, the

WWC, and others to trackthe progress of these trials. Reviewers can note these tri-als for inclusion in their future systematic reviews or to update completed ones.C2-PROT will be accessible through the C2 Library under the C2 Web site

homepage: http://www.campbellcollaboration.org.In

addition,C2-PROT will be

coupled to the work of the WWC (http://w-w-c.org), Nordic Campbell Center

(http://www.nc2.net), and other organizations with missions similar to the C2's.As one would expect, the majorpotential sources of surveillance for newly initi-

ated RCTs can differ from those for completed RCTs. Information on newly initi-

215








ated RCTs, for instance, usually does not appear in peer-reviewed journals, publicgovernment documents, or organizational reports. The majorpotential sources for

uncovering newly initiated RCTs, discussed seriatim, are as follows:

1. theinvisiblecollege,2. researchorganizationsndsponsors,3. PaperWorkReductionAct,4. TheFoundationGrants Index (database),5. institutionalreview boards,and6. trialistsas informants.

The nvisiblecollege

The invisible college here refers to a network of colleagues with shared researchinterests. Like Cochrane, the C2 will rely on the invisible college to help identifynewly initiated RCTs. The sharing of information on RCTs within this network

depends ntrust.AstheC2becomesmore mportanto the invisibleollege, histrustwill ncrease nd tsmemberswillassist heC2 with dentifyingewlyniti-atedRCTs.Thereare severalways hat he C2 cantap nto thisnetwork f col-

leagues. First, the C2 Secretariat can survey-through its mailing lists-partici-pants romprevious olloquia. econd,hroughts databasef Websitevisitors,the C2cansurveyndividuals hohave ignedupto receiveC2mailings.13hird,

theC2Secretariatansurveyhe authors freports n trials hatarecited ntheC2-SPECTR.nsum, he C2plans o routinelyurveymembers f the invisible

collegeonnewlynitiatedrials.

Research rganizationsndsponsors

IntheUnitedStates ndabroad,varietyf social olicy esearchrganizationsconductarge-scale,igh-qualityCTs hatare undedbyfederalagencies rpri-vatefoundations.he results f these RCTs ftenarereportednorganizational

reportsubmittedo sponsorsather han opeer-reviewedournals,orreasonsunrelatedo thequality fthe RCT.Often, heincentive tructures ithin ocial

policyresearchorganizations,t leastin the UnitedStates,aremoreheavilyweighted o producing report or the client,rather hanproducing journalarticle.

Organizationsuchasthe SocialDevelopmentResearchCorporationn Can-ada,heManpoweremonstrationesearch orporationntheUnitedStates ndtheUnitedKingdom,MathematicaolicyResearchUnited tates),heAmericanInstitutes f Research ttimes,and othershavea sturdyhistory f conductinglarge-scale,igh-qualityCTs. hesponsorsfsuch rialsncludediverse overn-mentagenciesuchastheU.S.DepartmentfEducation,he U.S.DepartmentfHealth ndHuman ervices,nd heNationalnstitutefJustice ndprivatenti-tiesincludingheRockefeller oundation,mithRichardsonoundation, aanFoundation,ndW.T.GrantFoundation.

216








To identify newly initiated RCTs and ensure that they are entered into C2-PROT routinely is achallenge. To meet this challenge, the C2 is working on a set of

procedures to:

1. create and maintain a populationlist of sponsorsof RCTs;2. create andmaintaina list of researchorganizations hatdesign, implement,oranalyze he

resultsof RCTs;3. periodicallysurveythese sponsorsand researchorganizations;and4. create a software nterfacebetween C2-PROT andC2-SPECTRto ensure that a report

on each RCT in C2-PROT s augmentedand transferred o C2-SPECTRwhen the trial s

completed.

PaperworkReductionAct ReportsEnacted in 1995, the U.S. Paperwork Reduction Act (PRA) was designed to

reduce response burden on participants in studies funded through federal con-tracts and to reduce redundancy in research. The U.S. Office of Management and

Budget (OMB) is responsible for ensuring that federal agencies comply with PRA.PRA has been integrated into the federal government'srequest for proposal (RFP)

process, which is used to award federal contracts to research organizations.Whenthe federal contract is awarded, and if there are plans to collect data on at least ten

participants, the successful bidder is required to complete a PRA Submission

Form (OMB 83-I). Using this form, the bidder provides information about thestudy, including whether statistical methods will be used to produce and analyzedata. If statistical methods will be used, the bidder provides more detailed docu-mentation on the methods and research design.

The OMB Paperwork Requirements Web site contains only a subset of theinformation on the PRA Submission Form.14This subset does not include informa-tion on the methods and research design and therefore is not especially useful in

identifying RCTs.In principle, this Web site, or one like it, could routinely include existing infor-

mation that isrequired by

law and add the information on thestudy's

method andresearch design, including whether it is a sample survey or an RCT.15As apracticalmatter,additions to the Web site depend on OMB policy, resources, andpriorities.

Posting information about the study method and research design to the site wouldhave anumber of advantages.First, the researchers andpublic could identify RCTs

sponsored by avariety of federal agencies that awardcontracts to research organi-zations. Second, asthese RCTsget underway, C2 staff (oranyone else) can monitorthe trial'sprogress so that the citation can be entered into C2-SPECTR.

The Foundation Grants Index

Foundation-sponsored research is archived in The Foundation Grants Indexavailablethrough CD-ROM or through the Web by subscription from DIALOG at

http://www.dialogl.com. The Index catalogs the grantsof more than one thousand

217








of the largest independent, corporate, and community foundations in the United

States. Subscribers can search various fields including an abstract, which briefly

describes the grant-sponsored activityin a few sentences. C2 staff acquired a tem-poraryID andpassword from a DIALOG account executive and searched the title,

abstract, and descriptor fields of the Index to identify two known foundation-

sponsored RCTs under way.The C2 could not identify these RCTs because the search fields lacked aword or

phrase such astrial, experiment, or randomly assigned that typicallydenotes RCTs.

C2 staff verified this deficiency in the Index with a DIALOG Information

Specialist.

Sinceit became Web accessiblein May2001,

C2-SPECTRhas receivedmorethan

nineteenthousandhits.

The Index is a potential source for electronically identifying trialssponsored byU.S. foundations. Fulfilling this potential depends in part on the willingness of

both Foundation and DIALOG staffs to populate the title, abstract,and descriptorfields to include words or phrases that denote a trial and also on their ability to

accurately catalog all research activities sponsored by each U.S. foundation.

Institutional review boards

Dickersin (1994) argued that with the increasing use of technology to recordand retrieve information electronically, connecting the human subject protectionactivities of institutional review boards (IRBs) through information systems is long

overdue-especially in the social sciences. Dickersin also suggested that lessons

could be learned, on this connection, from other industries such as health care and

from other countries such as Spain, where cooperation between local and central-

ized IRBs has been mandated.In the United States, information on IRBs is not centralized.16That is, it is

impossible to identify newly initiated RCTs by searching an electronic databasethat unifies and catalogs the work of IRBs.17This finding applies to research con-

ducted in the federal, philanthropic, nonprofit, and corporate sectors. Littleresearch has been done on IRB workload in the social, behavioral, and educationsciences that could shed insights on this lack of centralization. However, a study byMarwick(2002) in the biomedical arenaprovides some insight that maybe applica-

218








ble here. Marwick found that IRBs are overburdened and some of their practicesoutdated.

Trialistsas informants

Trialistshere means people who have authored or contributed to reports on tri-als.They constitute a target population of individuals who may report on new trials.A surveillance system that surveys them routinely seems sensible. Their contact

information, typically provided in journal articles, is aby-product of a hand search.The feasibility of conducting periodic surveys of trialists hinges on the ease inwhich their contact information can be extracted and put into electronic addressbooks. The C2 is unaware of any empirical research on the feasibility of this

approach, its costs, or its yield. The approach seems worth testing in the context of

government-sponsored efforts such as the WWC'sregister of evaluators and regis-ter of studies of interventions.

The C2 is positioned to surveyfinalist partlybecause of C2-SPECTR. This reg-ister of trials includes the identity of trialists who are potential informants aboutnew trials and trials done by others. To put this bluntly, good trialists know aboutone another and about other trialists. Understanding how the Campbell Secretar-iat can surveythem routinely,and to generate the in-kind effort or the resources todo so, is a fine challenge.

Handlingand ProcessingRetrieved Information

Each study that is identified for possible entry into C2-SPECTR or C2-PROT

requires handling and processing. For instance, retrieved studies are examined todetermine whether they meet criteria for inclusion:

1. the reportclearlystatesthat ndividuals,groups,orentitieswere randomlyassignedto theintervention(s)ndcontrol(s);r

2. thereportsambiguousstowhether andomassignment asused,but tsusecannotberuledout.

Since 2000, efforts to identify and process entries for C2-SPECTR have been

choppy.That is, the efforts have depended on advanced graduate students workingunder supervision at the University of Pennsylvania who work on related topicssuch as dissertation research or are enrolled in particularcourses such as advanced

experimental design. These efforts have also depended on specialized research byCampbell Colleagues in Sweden such as Haluk Soydan and Marie Nystrom.

Of the more than11,600

such entries inC2-SPECTR,

some haveabstracts,some have uniform synopses, and some have neither (i.e., have a citation only).

Uniform synopses are written by advanced graduate students. They are designedto present an orderly and informative description of the study, using the followingcategory structure:

219








1. backgroundand studyobjectives,2. intervention,3. targetpopulation,4. researchdesign,5. variablesmeasured,and6. results.

In C2-SPECTR, studies with uniform synopses are a small percentage of all

entries. C2 staff are working toward increasing this percentage.

C2-SPECTRImplementationand Challenges

Since it became Web accessible in May 2001, C2-SPECTR has received more

than nineteen thousand hits. Clearly, there is interest in C2-SPECTR's content.

However, for C2-SPECTR to reach the level of usefulness to C2 participants that

CENTRAL has with Cochrane participants, the C2 is implementing a strategic

plan for a surveillance system to identify completed and newly initiated RCTs.This

implementation depends on resources, of course. The goal in populating C2-

SPECTR is to leverage the work of the C2, the WWC, Cochrane, and others to

increase the trial coverage substantially.The effort to generate uniform synopses is related to Mosteller, Miech, and

Nave's (2002) independent effort to encourage journaleditors to require authors togenerate abstracts of their own articles that are uniform in structure. Entities such

as the C2, the WWC, professional societies, andjournal editorial boards arepoten-tial assets in advancing the idea of uniform abstracts or synopses for public access

databases on trials.To populate C2-SPECTR with entries on completed RCTs, C2 staff are updat-

ing the hand search of peer-reviewed journals and the electronic search of biblio-

graphicdatabases initiallydone by the U.K. Cochrane Centre. In addition, C2 staff

are expanding their electronic search to include eleven electronic databases rec-

ommended by Petrosino et al. (2000) (see the appendix). From these searches, theC2 expects to add a substantial number of citations to C2-SPECTR.

Plans for identifying newly initiated RCTs for entry into C2-PROT are in the

research and development phase. At present, the C2 relies on the C2-Secretariat,the invisible college, andtrialistsfor spontaneous referralsof newly initiated RCTs.

This strategywill continue until detection of RCTsfrom all other sources outlined

earlier can be routinized in a practical and cost-effective way. Routinely surveyingtrialists is one promising approach, but there are others.

Among these potential sources, the OMB Web site with PRA compliance infor-

mation and The Foundation Grants Index are promising for detecting newly initi-

ated RCTs.Thoughtful negotiation with civil servants could result in the use of theOMB Web site, or one like it, to identify prospective RCTs.The usefulness of The

Foundation Grants Index for trial detection depends on the willingness of Founda-

tion Center and DIALOG staffs to include words or phrases that denote a trial in

the Index'sabstract and descriptor fields. It also depends on the abilityof these par-

220








ties to accurately catalog all research activities conducted by each foundation.More challenging among the options is depending on IRBs in the United States.

Identifying newly initiated trials through the review of IRBs' administrativerecords is presently unfeasible in the absence of a centralized database.

Appendix

Petrosino et al. (2000) identified the followingeleven electronicbibliographicdata-bases for augmentingand updatingC2-SPECTR:

1. National CriminalJustice

References Service(NCJRS)2. PsychINFO (or PsycINFO)

3. EducationIndex (orWilson Education Index Full Text)4. GPO (U.S. GovernmentPrintingOffice) Monthly5. MEDLINE6. PAISInternational(PAIS)7. PoliticalScience Abstracts8. DissertationAbstracts9. NationalClearinghouseon Child Abuse and Neglect (NCCAN)

10. Current Contents11. Social Sciences CitationIndex

Crime and Justice Abstracts, Sociofile, and ERIC were excluded from this list because

these databases were already targeted for searches by the C2.

Notes1.Authorsof the legislationdefined scientificallyvalidevidence as use of a randomizedcontrolledtrial

when feasible or a researchdesign that allows for the highest qualitycausal inference from the generatedevidence.

2. See Farringtonet al. (2001) and Petrosino (1995) on how to compile the most complete data set

possible.3. Cochraneprovidesaccess to CENTRAL either throughits Web portalor on CD-ROM.4. A retrieved recordwasincludedin the CampbellCollaboration'sSociological,Psychological,Educa-

tional,andCriminologicalTrialsRegister(C2-SPECTR) f the abstractor methodssection of the full articlestated that random or quasi-randomassignmentmethods were used, or it could not be ruled out that such

methods were used. Randomassignmentmethods included the use of random number tables, computerrandomizationmethods, or a lottery.Quasi-randommethods included the use of alternatingcases, date of

birth,or administrative ecordnumber.5. This count also includes more than six hundred systematicreviews coded WPH in C2-SPECTR's

notes field.

6. Newly initiated trialsare funded trials that are planned or underway.7. The C2 InformationRetrievalMethods Groupis workingon waysto fulfill these needs.8. Turner used the searchstrategiesdocumented in AppendixB of Petrosino et al. (2000).9. Emphasisonthe social,behavioral,andeducation sciences havebeen addedhere because unlikewith

MEDLINE, the retaggingof citationsto distinguishtrials andpossible trialsfrom othertypesof studieshas

not been done.

221








10. The storyof John Henry is about a laborergiantwho competes againsta steam drillingmachine to

drive steel througha mountain o layrailroad racks.He narrowlydefeatsthe machine andsubsequentlydies

of a heart attack.

11. Unlimited truncationof a search termis the use of thistermwiththe searchengine'struncationchar-acter to retrieveallpossible suffixesfor a root term. Using Ovid,to retrieveallpossible suffixesfor the rootterm random,the searchterm would be random$.

12. An increasingnumberof journalarticlesare availableas full text in databases;however,most are in

journalsfor the biologicaland medical sciences.13. The numberof individualson this list is growing.14. This Web site maybe viewed at http://www.whitehouse.gov/omb/inforeg/infocoll.html.15. The Web site currentlyposts informationincludingthe study's itle, author,number of participants,

and workload.

16. Ourinitialreconnaissancewasa reviewof federalanduniversityWeb sites, e-mails to a smallsampleof university nstitutionalreview boardstaffs,and conversationswith an informed federal bureaucrat.

17. One organizationalexception is the Health and Human Services Office of Human ResearchProtections.

References

Boruch, R. F. 1997. Randomizedexperiments or planning and evaluation:A practical guide. Thousand

Oaks,CA:Sage.

Campbell, D. T., andJ. C. Stanley.1966. Experimentaland quasi-experimentaldesignsfor research.Chi-

cago: Rand McNally.Chalmers, I., L. V.Hedges, and H. Cooper.2002. A brief historyof researchsynthesis.Evaluation & the

HealthProfessions

25 (1): 12-37.

Dickersin,K. 1994. ResearchRegisters.In Thehandbookof researchsynthesis,edited by H. Cooper & L.V.

Hedges, pp. 71-83.. New York:Russell Sage Foundation.

Eldredge,J.D. 2000. Evidence-basedlibrarianship: earching or the needed EBL evidence. MedicalRefer-ence ServicesQuarterly19 (3): 1-18.

Farrington,D. P.,and B. C.Welsh,eds. 2001. Whatworks npreventingcrime?Systematicreviews of experi-mental andquasi-experimental esearch[special ssue].AnnalsoftheAmericanAcademyof Politicaland

Social Science 578.

Garer, J.H., and C. A.Visher.2002. Theco-productionofcriminologicalexperiments.Washington,DC: The

UrbanInstitute.

Green,D. P.,andA. S.Gerber.2002.Theunderprovisionofexperimentsnpoliticalscience. New Haven,CT:

YaleUniversity,Institute for Social and Policy Studies.

Hertzberg,S.,and L. Rudner.1999.Qualityof researchers' earches of the ERICdatabase.OnlineJournalofEducationPolicyAnalysisArchives7 (25): 14.

Hopewell, S., M. Clarke,A. Lusher,C. Lefebvre,andM.Westby.2002. Acomparisonof handsearchingver-

sus MEDLINE searchingto identify reportsof randomized controlledtrials. Statistics in Medicine21

(11): 1625-34.

Krisberg,B., and K. E. Schumann.2000. Introduction.Crime& Delinquency46 (2): 147-55.

Leow, C., and R. F.Boruch.2000. Locatingrandomizedexperimentsonmathand science education:A hand

searchand machinebasedsearchesof the AmericanEducation ResearchJournal.Philadelphia:Univer-

sity of Pennsylvania,Graduate School of Education.

Marwick,C. 2002. Controls on human researchparticipantsare "outdated."British MedicalJournal 325

(7365): 617.

Mosteller,F, and R. Boruch,eds. 2002. Evidence matters:Randomized rials in educationresearch. Wash-ington, DC: BrookingsInstitution.

Mosteller, F., E. Miech,and W.Nave. 2002. Uniformabstracts orreportingon studiesin education. Manu-

scriptsubmitted for publication.Petrosino,A. 1995.The hunt for randomizedexperimentalreports:Document searchand effortsfor a what

worksmeta-analysis. ournalof CrimeandJustice 18 (2):63-80.

222







POPULATINGA WEB-BASED RANDOMIZED TRIALS REGISTER 223

Petrosino,A., R. F. Boruch,C. Rounding,S. McDonald, and I. Chalmers.2000. "TheCampbellCollabora-

tion Social, Psychological,Educational and CriminologicalTrialsRegister (C2-SPECTR)" o facilitate

the preparationandmaintenanceof systematicreviewsof social andeducationalinterventions.Evalua-

tion & Research n Education 14 (3/4): 206-19.Ritter,G.W.,&R. F.Boruch. 1999.The politicalandinstitutionaloriginsof a randomizedcontrolled trial on

elementaryschoolclass size:Tennessee'sProjectSTAR.EducationalEvaluation&PolicyAnalysis21 (2):111-25.

Rossi,H. R., H. E. Freeman,and M.W.Lipsey.1999.Evaluation:Asystematicapproach.6th edition.Thou-

sand Oaks,CA:Sage.Schuerman,J., H. Soydan,G. Macdonald,M. Forslund,D. DeMoya, and R. Boruch. 2002. The Campbell

Collaboration.Research on SocialWork Practice 12 (2):309-17.

Shavelson,R.J.E., and L. E. Towne.2002. Scientificresearch n education.Washington,DC: NationalAcad-

emy Press.

Traub,J. 2002. Does it work? New YorkTimes,10 November,p. 24.

Documents

2003 Populating an International Web-Based Randomized Trials Register in the Social, Behavioral, Criminological, And Education Sciences