Designs, Participants, and Measurement Methods in Psychological Research

AbstractCritical reviews of psychological scholarship suggest thatself-report questionnaires, experimental designs, and col-lege students dominate psychological research. Althoughresearchers within specific subfields of psychology have therequisite knowledge to assess the generality of these con-cerns, novices of psychological research do not. To providesuch knowledge, we surveyed a random sample of 200journal articles in the PsycINFO database coding for psy-chological content area, research design, measurementmethod, and participant type. Results indicate self-reportquestionnaires, experimental designs, and college studentseach appeared in a minority of studies and these studycharacteristics often varied significantly by content area.Results also suggest no single dominant “typical” study inpsychological research but rather that the characteristicsof such research exhibit remarkable diversity.

What features characterize a “typical” empiricalstudy in psychological research? How “typical” is thatkind of study? Answers to these motivating questionswill vary depending on one’s schema of psychologicalresearch. Schemata are mental structures that orga-nize prior knowledge, guide current perceptions, andprovide future expectations (Fiske & Taylor, 1991).Research suggests that experts have more complex(and presumably more accurate) schemata in theirareas of expertise compared to novices (Newell &Simon, 1972). Thus, researchers in psychology, stu-dents in college, and individuals outside of academiaare likely to answer these questions about psychologi-cal research differently. Indeed, past research suggeststhat people hold many misconceptions about psychol-ogy and psychologists (Friedrich, 1996; Nauta, 2000;Rosenthal, McKnight, & Price, 2001; Rosenthal, Soper,Rachal, McKnight, & Price, 2004; Webb & Speer,1985; Wood, Jones, & Benjamin, 1986), that correcting

these misconceptions is difficult (Gutman, 1979;Lamal, 1979; Vaughn, 1977), but that these miscon-ceptions decrease with greater levels of education andexpertise (Gardner & Hund, 1983).

The purpose of this article is to answer the twomotivating questions empirically to correct potentiallyinaccurate schematic representations of the character-istics of psychological research. We view this researchas important because perceptions of psychology canaffect the level of available funding by private andpublic agencies (Shaffer, 1977; Wood et al., 1986), thechoice of undergraduate major and career by collegestudents (Flowers & Schandler, 2006; Nauta, 2000),and the perceived usefulness of psychological researchby the general public (Webb & Speer, 1985; Wood etal., 1986). We focus this investigation on the types ofresearch designs, participants, and measurementmethods used in psychological research because infor-mation sources from inside and outside of psychologymay lead novices to inaccurate schematic representa-tions about these particular features.

We are unaware of any formal research on portray-als of psychological research outside of psychology (e.g.,in the entertainment industry, the news and printmedia, and everyday conversations). However,research suggests that portrayals of psychologists in film,radio, and television are often narrow and stereotypicin nature with particular emphasis given to the thera-peutic and health services side of psychology (Fishman& Neigher, 1982; Flowers & Frizler, 2004; Flowers &Schandler, 2006). We conjecture that portrayals of psy-chological research (by novices to novices) would alsobe highly stereotypical in nature and not capture therange of features in actual psychological research. Forexample, stereotypical portrayals of psychologicalresearch in cartoons often involve a researcher in awhite laboratory coat and rodents. Such stereotypicportrayals encourage a narrow view and potentiallyinaccurate schemata of psychological research.

From within psychology, several sources encouragethe perception (particularly to a novice) that a typicalstudy in psychology is an experiment on college stu-dents or rodents using self-report questionnaires (i.e.,for human samples). Research methods textbooks inpsychology (e.g., Cozby, 2004; Elmes, Kantowitz, &Roediger, 2003; Gravetter & Forzano, 2003) generallygive much more coverage to experimental methods

TODD E. BODNERPortland State University

Canadian Psychology/Psychologie canadienne, 2006, 47:4, 263-272

Canadian Psychology Copyright 2006 by the Canadian Psychological Association2006, Vol. 47, No. 4, 263-272 DOI: 10.1037/cp2006017

CP 47-4 11/9/06 12:50 PM Page 263

264 Bodner

than nonexperimental methods. Furthermore, manycritical reviews of psychological scholarship lamentthe overuse of college students and laboratory rats inartificial environments (e.g., Neisser, 1978; Schultz,1969; Sears, 1986; Smart, 1966) and self-report ques-tionnaires as the primary bases for knowledge accu-mulation (e.g., Campbell, 1982). The empirical basesfor these concerns arise primarily from publishedresearch in leading social, personality, and experimen-tal psychology journals. Within social and personalitypsychology, reports of the percentage of studies in topjournals using college student samples range from42% (Endler & Speer, 1998) to 90% (Adelson, 1969).Wintre, North, and Sugar (2001) reported that about80% of studies within experimental psychology used acollege sample and concluded, “Articles reviewing thepsychological literature agree that psychologicalresearch is dominated by and overdependent onundergraduate participants” (p. 222). Within the topsocial and personality psychology journals, Endler andSpeer (1998) report that about 88% of articles pub-lished from 1993 to 1995 used self-report question-naires and Sherman, Buddie, Dragan, End, andFinney (1999) report that about 60% of studies pub-lished in 1996 used experimental designs.

We emphasize that this research was based on stud-ies published in particular journals and subfields with-in psychology. Researchers (i.e., experts) within specif-ic subfields of psychology have the requisite knowl-edge to assess the generality of these concerns toother subfields within psychology. However, to thenovice of psychological research, a critical question iswhether and to what degree these results (and theperceptions of psychological research they imply) gen-eralize to the broader domain of psychologicalresearch. Our goal is to provide such information tothose less familiar with the breadth of characteristicsdescribing psychological research. Therefore in thepresent study, we broadened the intended populationof studies to include published empirical researchacross the domain of psychological research and ran-domly selected journal articles from this population.From these articles and the studies they contain, wesought to assess the prevalence of various measure-ment methods, participant sample types, and researchdesigns and to explore how these study features varyby subfield within psychology.

MethodStudy Design

We used the PsycINFO database (AmericanPsychological Association, n.d.) to define the sam-pling frame for the population of published empiricalresearch studies in psychology. PsycINFO is an elec-

tronic bibliographic database containing scholarlycontributions to psychology and related fields in theform of journal articles, books and book chapters, andmaster’s theses and doctoral dissertations. The data-base covers over 1,900 journals and is updated weekly.Our use of this database for the present purpose is notwithout complications. The database contains entriesfrom a variety of fields, not just psychology (e.g., psy-chiatry, business, education, neuroscience) and thefield of origin is not included in the entry descriptionsbut can be inferred at times from the journal titles.1

The fact that psychologists conduct research in manyof these other fields further complicates a cleardemarcation of psychology and nonpsychology list-ings. Upon review, we were unable to reject with confi-dence any of the selected studies described below asclearly “not enough psychology” using as a criterionwhether a description of the study could in principleappear in a basic or advanced psychology textbook. Toreflect this complication, we refer to these studies asreflecting psychological research rather than researchin psychology and we refer to specific contents areasof psychological research rather than to specific sub-fields within psychology.

This study used the publication year of 1999 to rep-resent psychological research near the end of the 20thcentury, an important historical benchmark in time.Using PsycINFO, we identified 67,626 entries pub-lished in 1999 and further limited the search to “jour-nal articles,” “empirical study,” and “published inEnglish,” yielding 35,878 articles. To form a represen-tative sample, we drew a simple random sample of 200of these articles using the following procedure.PsycINFO orders responses to queries by assigning aninteger to each record matching the query. The soft-ware program MATLAB provides a function thatreturns a random permutation of an array of integersbased on a uniform distribution of such integers.Using this function on the integers between 1 and35,878, we used the first 200 numbers in the randomlypermuted integer array to select the articles for therandom sample. Selection of a sample size of 200 per-mits the 95% confidence interval margins of error forthe percentages of study characteristics based on theentire sample to be less than 7% for percentagesaround 50% and smaller for percentages that deviatefrom 50%. The list of articles in the sample is availablefrom the author on request.

1 PsycINFO offers classification codes to describe the subject mat-ter of the database entries; however, those classification codeswere not designed to categorize subdisciplines within psychologyor to distinguish psychology from nonpsychology sources.

CP 47-4 11/9/06 12:50 PM Page 264

Psychological Research 265

Sample DescriptionThe 200 selected articles appeared in 167 journals.

Nine (5%) of the journals were published by theAmerican Psychological Association (APA) and 164(98%) were peer-reviewed. Nine of the 200 articleswere published in these APA journals. As can beexpected given the focus on articles written inEnglish, study authors resided primarily in North andSouth America, Europe, the Middle and Far East, andthe South Pacific. The following are the percentage ofarticles with at least one author from that area: UnitedStates (57%), European countries (24%), Canada(7%), Australia/New Zealand (6%), Israel (3%), FarEast Asian countries (6%), Central and SouthAmerica countries (2%).2 The number of studiesreported in each article varied; 176 (87%) articlesreported one study, 14 (7%) articles reported twostudies, and 12 (6%) articles reported three or morestudies. For the purpose of this article, one study wasselected randomly from articles reporting more thanone study using a random sampling procedure analo-gous to the procedure described above for selectingarticles. Thus, analysis of these 200 articles focused atthe individual study level (i.e., the single or a random-ly selected study reported within each article).

Procedures and MeasuresStudy coding occurred in two phases. In the first

phase, five advanced undergraduate raters, all havingcompleted two courses in statistics and one course onresearch methods, coded the studies with at least tworaters coding each study. As discussed below, theraters coded the studies on several dimensions,including the research design, the sample characteris-tics, the measurement methods, and the content areaof psychological research. Generally, the ratings exhib-ited adequate average pairwise interrater agreementwith sample characteristics and measurement methodabove 96%, research design at 86%, and content areaat 79%. The author also coded each study andresolved all disagreements. In the second phase, agraduate student coded the studies on these dimen-sions and these ratings were compared to the author’scoding. Again, the ratings exhibited adequate inter-rater agreement with sample characteristics at 99%,measurement method at 93%, research design at84%, and content area at 84%. The author and thegraduate student resolved disagreements through dis-cussion until consensus was reached.

Content area of psychological research. Ratings of con-tent area involved six mutually exclusive categoriesconstructed in part from categories used in priorresearch (e.g., Feingold, 1989; Sears, 1986): 1)applied, 2) clinical/health, 3) developmental, 4)social/personality, 5) methods, and 6) cognition,brain, and behaviour. Several studies spanned morethan one category (e.g., brain function in patientswith schizophrenia). The author and graduate studentrater resolved disagreements by considering the con-tent area to which the study made the largest contri-bution using as a guide the nature of the journal thatpublished the study.

The applied content area involves the applicationof psychological theories and methods to address orto build theory to address nonclinically related indi-vidual, group, organizational, and societal concerns(e.g., industrial/organizational psychology, appliedhuman factors research, educational psychology, pub-lic opinion, advertising and consumer behaviour, andpolitics). The clinical/health content area involves theapplication of psychological theories and methods toaddress or to build theory to address clinical- andhealth-related concerns (e.g., psychopathology, men-tal, and physical health). The developmental contentarea explores psychological change over the lifecourse (e.g., in cognitions, behaviours, and abilities).The social/personality content area explores interper-sonal influences on cognition and behaviour, cogni-tive processes in social perception, and nonclinicalindividual differences such as personality. Themethodology content area involves the developmentor evaluation of quantitative or qualitative researchmethods with general applicability beyond the studyof a particular phenomenon. For example, the devel-opment of a new method to evaluate measurementinstruments would be included in the methodologycontent area; in contrast, the development of a mea-surement instrument for a particular phenomenon(e.g., depression) would be included in the contentarea most relevant to that phenomenon. The cogni-tion, brain, and behaviour content area involves basicresearch on learning, memory, perception, cognition,neuroscience, biopsychology, and psycholinguistics aswell as behavioural studies with animals.

Participant type. Category ratings of participant char-acteristics were less elaborate than the other codedstudy features as prior criticisms internal to psycholo-gy have focused primarily on the relative prevalenceof college student samples. Furthermore, given vary-ing degrees of reported information across studieswhen describing participant characteristics, we did notcode for other characteristics in the human samples

2 The sum of percentages exceeds 100% as a few multiauthorstudies had authors from different areas.

CP 47-4 11/9/06 12:50 PM Page 265

266 Bodner

other than student status. Thus, we employed fourparticipant type categories: college students, noncol-lege students, rodents, and other nonrodent/nonhu-man species. Studies were coded as using college stu-dents if any of the participants in the study were col-lege students. After analyses of the relative prevalenceof the rodent and other nonrodent/nonhumanspecies, we combined these categories into a singlecategory representing nonhuman participants.

Research design. Seven research design categorieswere constructed based on chapter and section head-ings in introductory and advanced research methodstextbooks in psychology (e.g., Cozby, 2004; Reis &Judd, 2000): 1) experimental, 2) cross-sectional, 3)qualitative (e.g., focus groups, ethnographies, casestudies), 4) longitudinal, 5) original sample survey, 6)archival, and 7) commentary (e.g., comments on pre-viously published empirical research) designs.3 A fewstudies involved more than one of these design cate-gories (e.g., experiment with a longitudinal compo-nent). For the purpose of this article (to resolve over-lapping design features), we gave coding priority firstto experiments and second to original random samplesurveys as both designs permit cleaner statistical infer-ence due to design-based randomness (i.e., randomassignment and random selection, respectively; Rubin,1990). Thus, our results will slightly underestimate theprevalence of longitudinal research designs.

We defined the research design categories accord-ing to the following rules. In an experiment, theresearcher manipulates at least one of the variablesunder investigation. In an original sample survey, theresearcher measures all variables on a random samplefrom a well-defined population. In a cross-sectionalstudy, the researcher measures all variables at a singlemeasurement occasion and does not use a randomsampling strategy from a well-defined population. In alongitudinal study, the researcher measures all vari-ables, at least one of which is measured over time, anddoes not use a random sampling strategy from a well-defined population. In a qualitative study, theresearcher uses verbal descriptions of events ratherthan quantitative summaries of data. In an archivalstudy, the researcher uses pre-existing data fromanother source for a novel use. In a commentary, theresearcher comments and perhaps provides additionaldata in response to a prior investigation.

Measurement method. Ratings of measurementmethod involved a set of five categories: self-reportquestionnaire, interview, observer ratings of behav-iour, behavioural measurement (e.g., task perfor-mance, reaction time), and physiological measure-ment. Studies could contribute to more than onemeasurement method when appropriate (e.g., if astudy used both self-report questionnaires and observ-er ratings). As with the research design coding, weconstructed measurement method categories basedon chapter and section headings in introductory andadvanced research methods textbooks in psychology(e.g., Cozby, 2004; Reis & Judd, 2000).

Analysis StrategyAnalyses focus on the marginal relative frequencies

(i.e., percentages) of the content area, participanttype, research design, and measurement method cate-gories. To facilitate inference for these estimated per-centages to the population of studies of interest, 95%confidence intervals are provided. Next, chi-squaretests of independence are used to assess whether thepresence of the coded study features vary by contentarea. Following suggested reporting standards (APA,2001), the effect size Cramér’s Phi (φc; Cramer, 1946)is reported along with each chi-square test.4 As thetables analyzed for these tests are all of the form “pre-sent/absent by content area,” the chi-square tests ofindependence in these cases are also tests of theequivalence across content areas in the proportion ofstudies with that feature (see e.g., Wickens, 1989).Therefore, we used post-hoc comparisons of propor-tions as outlined in Rosenthal and Rosnow (1991, Ch.23) following statistically significant omnibus (i.e.,many degree of freedom) chi-square tests to discernwhich particular proportions differed from the others.A difference in proportions (percentages) is an effectsize corresponding to these post-hoc comparison tests(Rosenthal, 1991). To save space these differences willnot be reported; however, the compared percentagesare reported individually and one can easily computetheir difference.

ResultsOf the 200 studies, we classified 66 [33%: CI(26%,

40%)] studies into the clinical/health category, 53[27%: CI(20%, 33%)] studies into the cognition,brain, and behaviour category, 35 [18%: CI(12%,

3 The commentary category was added to the research designcoding upon an initial review of the studies; as discussed in theResults section, these studies were not included in the analyses asthey did not tend to provide sufficient information on the othercoded study characteristics.

4 Cramér’s Phi is a generalization of the correlation coefficientbetween two dichotomous variables (i.e., φ) for use with tableslarger than 2 x 2 and φc equals φ in a 2 x 2 table; values rangefrom 0 to 1 with larger values indicating a stronger relationshipbetween variables (Howell, 2002).

CP 47-4 11/9/06 12:50 PM Page 266

Psychological Research 267

23%)] studies into the applied category, 32 [16%:CI(11%, 21%)] studies into the social/personality cat-egory, 11 [6%: CI(2%, 9%)] studies into the develop-mental category, and three [2%: CI(NA)5] studies intothe methodology content area. The results that followreport the relative frequencies of each coded studyfeature (i.e., participant type, study design, and mea-surement method) followed by an investigation ofhow these features varied by content area of psycho-logical research. In these results, we excluded six stud-ies from all analyses: a meta-analysis, one methodolog-ical study using simulated data, and the four commen-taries as these did not provide sufficient informationon the coded variables. The two other methodologicalstudies are included in the presentation of marginalpercentages, but are not included in analyses by con-tent area due to the small number of studies in thatarea.

Participant TypeThree studies were excluded from the following

analyses due to insufficient description of the humanparticipants as students or nonstudents. Humans werethe most commonly studied sample type appearing in170 [89%: CI(85%, 93%)] studies. Rodents and othernonrodent/nonhuman species appeared in 12 [6%:CI(3%, 10%)] and 9 [5%: CI(2%, 8%)] of the studies,respectively. Of the studies using human participants,36 [25%: CI(18%, 31%)] employed a college sample.As displayed in panel one of Table 1, use of collegestudents for human samples varied by content area,χ2(4, N = 169) = 28.01, p < .001, φc = .41. Post-hoc com-parisons of proportions indicate the following:Researchers within the social/personality area (50%)were significantly more likely to use college studentsamples than in the applied (26%, Z = 5.49, p < .001)and clinical/health (5%, Z = 4.91, p < .001) areas;researchers within the clinical/health area were signif-icantly less likely to use college student samples com-pared to those in the applied (Z = 2.55, p = .01) andcognition, brain, and behaviour (39%, Z = 3.72, p <.001) areas; no other differences in percentages werestatistically significant.

Research DesignIn descending order of occurrence, researchers

most frequently used an experimental design appear-

TABLE 1Frequencies of College Student Samples, Research Designs, and Measurement Methods by Content Area in Psychological Research Studies

––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––Psychological Content Area

––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––Applied CH Develop. CBB SP Totals

––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––Human Samples

College 8 (26%) 3 (5%) 3 (27%) 12 (39%) 16 (50%) 42 (25%)Noncollege 23 (74%) 61 (95%) 8 (73%) 19 (61%) 16 (50%) 127 (75%)

Totals 31 (18%) 64 (38%) 11 (7%) 31 (18%) 32 (19%) 169––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––Research Design

Experiment 5 (15%) 14 (22%) 4 (36%) 41 (79%) 15 (47%) 79 (41%)Cross-Sec. 7 (21%) 27 (42%) 4 (36%) 11 (21%) 14 (44%) 63 (33%)Longitudinal 1 (3%) 10 (16%) 2 (18%) 0 (0%) 1 (3%) 14 (7%)Qualitative 12 (36%) 3 (5%) 1 (9%) 0 (0%) 0 (0%) 16 (8%)Survey 4 (12%) 5 (8%) 0 (0%) 0 (0%) 2 (6%) 11 (6%)Archival 4 (12%) 5 (8%) 0 (0%) 0 (0%) 0 (0%) 9 (5%)

Totals 33 (17%) 74 (33%) 11 (6%) 52 (27%) 32 (17%) 192––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––Measurement

SR Question. 14 (45%) 33 (52%) 3 (27%) 7 (13%) 28 (88%) 85 (45%)Interview 15 (48%) 38 (59%) 4 (36%) 2 (4%) 3 (9%) 62 (33%)Observ. 6 (19%) 4 (6%) 3 (27%) 5 (10%) 2 (6%) 20 (11%)Behavioural 1 (3%) 0 (0%) 5 (45%) 30 (58%) 8 (25%) 44 (23%)Physio. 1 (3%) 8 (13%) 1 (9%) 26 (50%) 0 (0%) 36 (19%)

No. of Studies 31 (16%) 64 (34%) 11 (6%) 52 (27%) 32 (17%) 190––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––Notes. Numbers in parentheses are percentages rounded to whole units. Percentages in the Totals rows and columns represent percentageof total sample; percentages in the interior represent percentage of the studies within a given content area. Column sums of frequencies(percentages) in the third panel on measurement exceed the number of studies (100%) as some studies used more than a single measure-ment method.CH = Clinical/Health, Develop. = Developmental, CBB = Cognition, Brain, and Behaviour, SP = Social/Personality, Cross-Sec. = Cross-sec-tional, SR Question = Self-report questionnaire, Observ. = Observation, Physio. = Physiological measurement.

5 The sample percentage is too small, given this sample size, forreasonable normal approximation to the sampling distributionand therefore this confidence interval is not computed or report-ed.

CP 47-4 11/9/06 12:50 PM Page 267

268 Bodner

ing in 79 [41%: CI(34%, 48%)] studies, followed bycross-sectional [33%: CI(26%, 39%)], qualitative [8%:CI(4%, 12%)], longitudinal [7%: CI(4%, 11%)], sur-vey [6%: CI(2%, 9%)], and archival [5%: CI(2%,8%)] research designs. As displayed in panel two ofTable 1, the use of various research designs varieddescriptively by content area.6 Descriptively,researchers within the applied and clinical/healthcontent areas used a larger variety of research designs(i.e., all six categories of research designs) thanresearchers within the developmental, cognition,brain, and behaviour, and social/personality areasusing no more than four of the research designs.

Collapsing the nonexperimental designs into a sin-gle category, the use of experimental designs varied bycontent area, χ2(4, N = 192) = 50.08, p < .001, φc = .51.Post-hoc comparisons of proportions indicate the fol-lowing: Researchers in the cognition, brain, andbehaviour area (79%) were significantly more likely toemploy an experimental methodology thanresearchers in the social/personality (47%, Z = 2.79, p= .005), developmental (36%, Z = 2.65, p = .008), clini-cal/health (22%, Z = 6.36, p < .001), and applied(15%, Z = 6.58, p < .001) areas; researchers in thesocial/personality area were significantly more likelyto employ an experimental methodology thanresearchers in the clinical/health (Z = 2.44, p = .01)and applied (Z = 2.93, p = .003) areas; no other differ-ences in percentages were statistically significant.

Measurement MethodWe omitted two studies from these analyses due to

an insufficient articulation of measurement proce-dures. The most frequent measurement method wasthe self-report questionnaire appearing in 45%

[CI(38%, 52%)] of the studies followed by interviews[33%: CI(26%, 39%)], behavioural measures [23%:CI(17%, 29%)], physiological measures [19%:CI(13%, 25%)], and observer ratings [11%: CI(6%,15%)]. Panel three of Table 1 displays the number ofstudies using the five different categories of measure-ment method by psychology content area.7

Descriptively, studies within each content area used atleast four of the six measurement methods.

Most studies [76%: CI(69%, 82%)] involved only asingle measurement method. The use of single versusmultiple measurement methods did not vary signifi-cantly across the content areas, χ2(4, N = 190) = 1.26, p= .87, φc = .08. Of the 168 studies on human sampleswith adequate articulation of measurement proce-dures, 56 [33%: CI(26%, 40%)] involved self-reportquestionnaires as the sole measurement method.However, exclusive use of self-report questionnaireson human samples varied by content area, χ2(4, N =167) = 26.18, p < .001, φc = .40. Post-hoc comparisonsof proportions indicate the following: Researchers inthe social/personality area (66%) were significantlymore likely to use self-report questionnaire measure-ment methods exclusively compared to those in theapplied (34%, Z = 2.56, p = .01), clinical/health (33%,Z = 3.20 , p = .001), developmental (18%, Z = 3.81, p <.001), and cognition, brain, and behaviour (7%, Z =6.23, p < .001) areas; researchers in the cognition,brain, and behaviour area were significantly less likelyto use self-report questionnaire measurement meth-ods exclusively compared to those in the applied (Z =2.84, p = .004) and clinical/health (Z = 3.59, p < .001)areas; no other differences in proportions were statis-tically significant.

TABLE 2The 10 Most Frequent Combinations of Participant Type, Research Design, and Measurement Method in 189 Psychological Research Studies

––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––Rank % Frequency Description––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––1 12% 23 Cross-sectional nonexperimental study on nonstudent participants using self-report questionnaires exclusively2 5% 10 Cross-sectional nonexperimental study on nonstudent participants using interviews exclusively2 5% 10 Experiment on college students using behaviour measures exclusively 4 5% 9 Experiment on college students using self-report questionnaires exclusively 5 4% 8 Experiment on nonstudent participants using behavioural measures exclusively 6 4% 7 Experiment on animals using physiological measures exclusively6 4% 7 Cross-sectional nonexperimental study on college students using self-report questionnaires exclusively6 4% 7 Qualitative study on nonstudent participants using interviews exclusively6 4% 7 Experiment on nonstudent participants using physiological measures exclusively6 4% 7 Longitudinal study on nonstudent participants using interviews exclusively––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––Notes. Percentages rounded to whole units.

6 We do not report a chi-square test of independence given alarge number of cells (70%) with expected frequencies less than 5.

7 We do not report a chi-square test of independence given thatmany studies used multiple measurement methods violating theindependence assumption underlying the test.

CP 47-4 11/9/06 12:50 PM Page 268

Psychological Research 269

“Typical” StudiesIgnoring content area, we cross-classified studies by

participant type, research design, and measurementmethod and then rank ordered the relative frequen-cies of the possible feature combinations. Table 2 dis-plays the 10 most frequently occurring combinations.The modal study, occurring 12% [CI(8%, 17%)] ofthe time, was a cross-sectional design on a noncollegestudent sample with a self-report questionnaire as theonly measurement method. Of note, an experimenton college students using self-report questionnairesexclusively was ranked fourth most frequent at 5%[CI(2%, 8%)].

DiscussionThe results of the present study challenge the per-

ception that experiments and research involving col-lege student participants, rodents and self-report ques-tionnaires dominate psychological research. Less thanone-half of study designs were experimental in nature,about one-fifth of studies on human populationsinvolved college students, less than two-thirds of ani-mal studies focused on rodents, and only about half ofthe studies on human populations included self-reportquestionnaires. The modal study (occurring 12% ofthe time) was a cross-sectional design on a noncollegesample and involved self-report questionnaires. Thus,it is hard to argue that there is a single “dominant” or“typical” study in psychological research.

For those more and less familiar with psychologicalresearch, these results provide a more completedescription of the prevalence of the studied researchcharacteristics. For novices and communicators of psy-chological research (e.g., textbook authors), thesedescriptions may help rectify inaccurate perceptionsregarding psychological research; for psychologicalresearchers, these descriptions may facilitate furthercritical inquiry regarding research practices. Forexample, many scholars have voiced construct validityconcerns over researchers relying on a single measure-ment method (e.g., Campbell & Fiske, 1959; Cote &Buckley, 1987; Podsakoff, MacKenzie, Lee, &Podsakoff, 2003; Spector, 1992). When a single mea-surement method is used, one cannot distinguish vari-ance due to the construct of interest from variancedue to the method used to measure that construct;furthermore, correlations among variables that sharea measurement method will in part reflect theirdependence on that measurement method. Howeverin our sample, over three-fourths of studies involved asingle measurement method for all measured vari-ables in a given study. Thus, these results suggest thatpsychological researchers should pay more attentionto measurement and construct validity issues.

Using a broader pool of journals than in priorreviews, our results largely replicated research withinthe social/personality content area on the use of col-lege samples and self-report questionnaires (e.g.,Endler & Speer, 1998, Schultz, 1969; Sears, 1986;Smart, 1966). However, the percentages of studieswithin the social/personality content area usingexperimental designs and studies within the cogni-tion, brain, and behaviour content area on humanparticipants using college students were smaller thanobserved in prior research (cf. Sherman et al., 1999;Wintre et al., 2001). We attribute these differences tothe broader definition of studies included in the cog-nition, brain, and behaviour content area (e.g., neuro-science) and to the inclusion of studies from a largernumber of journals.

A second important result from this study is thefinding that study design, sample type, and measure-ment method use varied by psychological contentarea. Researchers within the cognition, brain, andbehaviour as well as the social/personality areas weremore likely to employ college student samples thanresearchers in other areas. Thus, the “overuse” of col-lege samples appears to be limited largely to theseareas. Furthermore, studies in cognition, brain, andbehaviour were more likely to involve experimentaldesigns and researchers in social/personality weremore likely to use self-report questionnaires as thesole means of data collection. These results suggestthat conclusions about the characteristics of psycho-logical research drawn from specific areas of psycholo-gy do not necessarily generalize to other areas. Takenout of context, statements like the quote from Wintreet al. (2000) in the introduction convey an inaccuratedescription of psychological research beyond the topjournals in social/personality and experimental psy-chology.

Limitations and Future Research NeedsWe note four limitations to the present study that

qualify our results and motivate need for furtherresearch. The first limitation involves our use of thePsycINFO database, which contains contributions froma variety of fields of interest to psychologists (e.g., psy-chiatry, neuroscience). As any IntroductoryPsychology instructor can attest, the scope of psycho-logical inquiry is extremely broad. These realitiesmake difficult distinguishing what listings in the data-base are and are not from the field of psychologyproper. Although we were unable to disqualify any ofour studies with confidence as “not including enoughpsychology,” we acknowledge that others with moreprescribed views of psychological research might havebeen able to do so. Efforts to classify the contents of

CP 47-4 11/9/06 12:50 PM Page 269

270 Bodner

PsycINFO by discipline could facilitate future workrelated to the present study.

The second limitation involves our inclusion crite-ria for this study. We confined attention to journalarticles written in English for the publication year1999. Thus, these results may not accurately reflectthe broader domain of psychological research beyondthis frame (e.g., other publication years, other lan-guages, and nonjournal sources such as books and dis-sertations). For example, the advent of new measure-ment technologies like functional magnetic resonanceimaging, implicit measurement methods (e.g.,Greenwald, Nosek, & Banaji, 2003), and an increasingfocus on positive health issues in psychology (Fowler,Seligman, & Koocher, 1999) could easily impact thecontent and perhaps the methodological characteris-tics of psychological research. Future research includ-ing more recent or several publication years (e.g.,Sherman et al., 1999; West, Newsom, & Fenaughty,1992) will help to assess the generality of the presentresults.

The third limitation involves the quality of the con-sidered research studies. Prior reviews of this typehave focused on the top journals within psychologysubfields (e.g., Wintre et al., 2001). This was a reason-able decision because these journals are thought tocontain the best and most carefully conductedresearch and to have the most impact and visibility.Given that so few of our sampled studies came fromAPA journals, often considered the top journals in thefield, our results may not reflect the best of psycholog-ical research. However, our results do represent psy-chological research more broadly and our results with-in content areas are largely consistent with resultsfrom prior reviews that focused exclusively on top psy-chology journals.

The fourth limitation of this study involves the sta-tistical power to detect content areas differences onthe studied characteristics. Our use of a simple ran-dom sampling method left some content areas withfew studies (e.g., only eleven and three studies werecoded as representing the developmental andmethodology areas, respectively). Thus, we had lessthan adequate statistical power to detect differencesamong some areas. However, our results do provideinformation on the relative prevalence of the contri-butions that each content area under study made tothe PsycINFO database.

In addition to the stated limitations, our investiga-tion leaves several questions for future research. Wemention two here. First, the features of psychologicalresearch that we considered were narrow in scope.Studies also vary on dimensions other than those con-sidered. For example, studies may vary across content

areas in the use of various statistical techniques or thedemographic characteristics of participants; studycharacteristics may further vary by country of origin.For brevity and clarity, we chose to focus on thosestudy features garnering past criticism from withinpsychology, criticism that might convey potentiallymisleading portrayals of psychological research.Second, we framed this research to correct potentiallyinaccurate representations of psychological research,particularly among those less familiar with psychologi-cal research. Although research suggests that peoplehold inaccurate general perceptions of psychologyand psychologists (see introduction for references),the accuracy of representations of psychologicalresearch is an open question both for the public andfor psychological researchers. Research on this topicwould help assess how both psychologists and nonpsy-chologists view this aspect of the field. Furthermore,investigations are needed to explore how dominantcultural institutions such as the news media and theentertainment industry portray psychologicalresearch. Such investigations would complement theexisting research on portrayals of psychology and psy-chologists. Given the power of these institutions toshape public perceptions, it is important to assess theaccuracy of these portrayals.

ConclusionsFurther development of psychology as a science

depends critically on its epistemological foundations(Sears, 1986). Such development would also be influ-enced by views of our discipline held by the nonpsy-chologists who fund our work and use psychologicalresearch results. Although thoughtful criticism is use-ful in improving the practice of psychologicalresearch, such criticisms taken out of context bynovices can imply inaccurate perceptions of what con-stitutes “typical” research. The present findings sug-gest that experimental designs, studies of college stu-dents, studies of rodents, and use of self-report ques-tionnaires do not dominate psychological research asa whole. Though some content areas appear tooveruse some of these features, other areas do not.Overall, psychological research exhibits remarkablediversity with respect to designs, samples, and mea-surement methods, which partly mitigates the general-ity of prior concerns and suggests that more complexschemata of psychological research are appropriate.

Portions of this study were presented at the 2004Annual Convention of the American Psychological Society,Chicago, Illinois. I thank Ann Derry, Jacob Haugen, MarkMcDonald, Angela McLeod, Diana Rempe, and DianeWille for assistance in retrieving and coding the articles. I

CP 47-4 11/9/06 12:50 PM Page 270

Psychological Research 271

also thank Keith Kaufman, Robert Sinclair, DonaldTruxillo, the editor, and three anonymous reviewers forhelpful comments on earlier drafts of this article.

Correspondence concerning this article should beaddressed to Todd Bodner, Department of Psychology,Portland State University, P.O. Box 751, Portland, Oregon97207 (E-mail: tbodner@pdx.edu).

RésuméDes comptes rendus critiques d’études en psychologie sug-gèrent que les questionnaires à autoévaluation, les concep-tions expérimentales et les étudiants collégiaux dominentla recherche en psychologie. Même si les chercheurs ausein de sous-domaines particuliers de la psychologie possè-dent les connaissances requises pour évaluer la portéegénérale de ces préoccupations, les débutants dans ledomaine de la recherche psychologique ne les ont pas.Pour acquérir de telles connaissances, nous avons étudiéun échantillon aléatoire de 200 articles de revue dont lecontenu, la conception de la recherche, la méthode demesure et le type de participant en psychologie avaient étécodés dans la base de données PsycINFO. Les résultatsindiquent que les questionnaires autoévalués, les concep-tions expérimentales et les étudiants collégiaux figuraienttous dans un très petit nombre d’études et que les carac-téristiques de ces études variaient souvent de façon signi-ficative dans leur contenu. Les résultats suggèrent égale-ment qu’il n’y a aucune étude dominante « typique » enpsychologie, mais plutôt que les caractéristiques desétudes manifestent une diversité remarquable.

ReferencesAdelson, J. (1969). Personality. Annual Review of Psychology,

20, 217-252.American Psychological Association. (2001). Publication

manual of the American Psychological Association (5th ed.).Washington, DC.

American Psychological Association. (n.d.). [Descriptionof PsycINFO database.] Retrieved July 16, 2004, fromhttp://www.apa.org/psycinfo/products/psycinfo.html

Campbell, D., & Fiske, D. (1959). Convergent and discrim-inant validity by the multitrait-multimethod matrix.Psychological Bulletin, 56, 81-105.

Campbell, J. P. (1982). Editorial: Some remarks from theoutgoing editor. Journal of Applied Psychology, 67, 691-700.

Cote, J., & Buckley, R. (1987). Estimating trait, method,and error variance: Generalizing across 70 constructvalidation studies. Journal of Marketing Research, 24, 315-318.

Cozby, P. C. (2004). Methods in behavioral research (8th ed.).New York: McGraw-Hill.

Cramér, H. (1946). Mathematical methods of statistics.

Princeton, NJ: Princeton University Press.Elmes, D. G., Kantowitz, B. H., & Roediger III, H. L.

(2003). Research methods in psychology (7th ed.).Belmont, CA: Thomson-Wadsworth.

Endler, N., & Speer, R. (1998). Personality psychology:Research trends from 1993-1995. Journal of Personality,66, 621-669.

Feingold, A. (1989). Assessment of journals in social sci-ence psychology. American Psychologist, 44, 961-964.

Fishman, D., & Neigher, W. (1982). American psychologyin the eighties: Who will buy? American Psychologist, 37,533-546.

Fiske, S., & Taylor, S. (1991). Social cognition (2nd ed.).New York: McGraw-Hill.

Flowers, J., & Frizler, P. (2004). Psychotherapists on film,1899-1999: A worldwide guide to over 5000 films. Jefferson,NC: McFarland Publishers.

Flowers, J., & Schandler, S. (2006, April) Forensic psychologyin cinema: Dramatic, positive portrayals of psychology,usually. Paper presented at the annual conference ofthe Western Psychological Association, Palm Springs,CA.

Fowler, R. D., Seligman, M. E. P., & Koocher, G. P. (1999).The APA 1998 annual report. American Psychologist, 54,537-568.

Friedrich, J. (1996). Assessing students’ perceptions of psy-chology as a science: Validation or a self-report mea-sure. Teaching of Psychology, 23, 6-13.

Gardner, R., & Hund, R. (1983). Misconceptions of psy-chology among academicians. Teaching of Psychology, 10,20-22.

Gravetter, F. J., & Forzano, L. B. (2003). Research methods forthe behavioral sciences . Belmont, CA: Thomson-Wadsworth.

Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003).Understanding and using the Implicit Association Test:I. An improved scoring algorithm. Journal of Personalityand Social Psychology, 85, 197-216.

Gutman, A. (1979). Misconceptions of psychology andperformance in the introductory course. Teaching ofPsychology, 6, 159-161.

Howell, D. (2002). Statistical methods for psychology (5th ed.).Pacific Grove, CA: Duxbury.

Lamal, P. (1979). College students’ common beliefs aboutpsychology. Teaching of Psychology, 6, 155-158.

Nauta, M. (2000). Assessing the accuracy of psychologyundergraduates’ perceptions of graduate admission cri-teria. Teaching of Psychology, 27, 277-280.

Neisser, U. (1978). Memory: What are the important ques-tions? In M. M. Gruneberg, P. E. Morris, & R. N. Sykes(Eds.), Practical aspects of memory (pp. 3-24). London:Academic Press.

Newell, A., & Simon, H. (1972). Human problem solving.Englewood Cliffs, NJ: Prentice-Hall.

CP 47-4 11/9/06 12:50 PM Page 271

272 Bodner

Podsakoff, P., MacKenzie, S., Lee, J., & Podsakoff, N.(2003). Common method bias in behavioral research:A critical review of the literature and recommendedremedies. Journal of Applied Psychology, 88, 879-903.

Reis, H., & Judd, C. (Eds.). (2000). Handbook of researchmethods in social and personality psychology. New York:Cambridge University Press.

Rosenthal, G., McKnight, R., & Price, A. (2001). Who,what, how, and where the typical psychologists is… The Profession of Psychology Scale. Journal ofInstructional Psychology, 28, 220-224.

Rosenthal, G., Soper, B., Rachal, C., McKnight, R., & Price,A. (2004). The Profession of Psychology Scale:Sophisticated and naïve students responses. Journal ofInstructional Psychology, 31, 202-205.

Rosenthal, R. (1991). Meta-analytic procedures for socialresearch (Rev. ed.). Newbury Park, CA: Sage.

Rosenthal, R., & Rosnow, R. (1991). Essentials of behav-ioral research: Methods and data analysis (2nd ed.).New York: McGraw-Hill.

Rubin, D. (1990). Formal modes of statistical inference forcausal effects. Journal of Statistical Planning and Inference,25, 279-292.

Schultz, D. P. (1969). The human subject in psychologicalresearch. Psychological Bulletin, 72, 214-228.

Sears, D. O. (1986). College sophomores in the laborato-ry: Influence of a narrow data base on social psycholo-gy’s view of human nature. Journal of Personality andSocial Psychology, 51, 515-530.

Shaffer, L. (1977). The golden fleece: Anti-intellectualism

and social science. American Psychologist, 32, 814-823.Sherman, R., Buddie, A., Dragan, K., End, C., & Finney, L.

(1999). Twenty years of PSPB: Trends in content,design, and analysis. Personality and Social PsychologyBulletin, 25, 177-187.

Smart, R. (1966). Subject selection bias in psychologicalresearch. The Canadian Psychologist, 7, 115-121.

Spector, P. (1992). A consideration of the validity andmeaning of self-report measures of job conditions. InC. Cooper & I. Robertson (Eds.), International review ofindustrial and organizational psychology (pp. 123-151).New York: Wiley.

Vaughan, E. (1977). Misconceptions about psychologyamong introductory psychology students. Teaching ofPsychology, 4, 138-141.

Webb, A., & Speer, J. (1985). The public image of psychol-ogists. American Psychologist,40, 1063-1064.

West, S., Newsom, J., & Fenaughty, A. (1992). Publicationtrends in JPSP: Stability and change in topics, methods,and theories across two decades. Personality and SocialPsychology Bulletin, 18, 473-484.

Wickens, T. (1989). Multiway contingency table analysis for thesocial sciences. Mahwah, NJ: Lawrence Erlbaum.

Wintre, M., North, C., & Sugar, L. (2001). Psychologists’response to criticisms about research based on under-graduate participants: A developmental perspective.Canadian Psychology, 42, 216-225.

Wood, W., Jones, M., & Benjamin, L. (1986). Surveyingpsychology’s public image. American Psychologist, 41,947-953.

CP 47-4 11/9/06 12:50 PM Page 272