Assessing Literacy in Science:Evaluation of ScientificNews Briefs

CONNIE A. KORPAN, GAY L. BISANZ, JEFFREY BISANZCentre for Research in Child Development, Department of Psychology, University of Alberta,Edmonton, AB T6G 2E9, Canada

JOHN M. HENDERSONDepartment of Psychology, Michigan State University, East Lansing, MI 48824, USA

Received 15 May 1996; revised 5 March 1997; accepted 5 March 1997

ABSTRACT: Media reports of scientific research are a pervasive and important source of newscientific knowledge. Evaluating conclusions found in those reports is an important form ofscientific literacy. We examined the types of requests for information made by university stu-dents as they evaluated scientific news briefs. Students generated a variety of requests for in-formation, focusing most often on how the research was conducted and why the results mighthave occurred. Fewer requests were made for information about what was found, who con-ducted the research, and where it was conducted. Least frequent were requests about relatedresearch. This pattern of findings may reflect emphasis of instruction in science classrooms.Requests appeared to be influenced by three dimensions of news briefs: plausibility of theconclusions; typicality (the degree to which the phenomena described are typical in biology,chemistry, and physics); and personal familiarity with the phenomena. Individual differencesin age, number of science courses completed in university, and degree of belief in paranormalevents were correlated with the types of requests made. Models of the evaluative process areneeded to gain additional insights into how students’ knowledge, dimensions of text, and per-sonal characteristics interact in the course of learning. © 1997 John Wiley & Sons, Inc. Sci Ed81:515–532, 1997.

INTRODUCTION

The need for a scientifically literate public has been an enduring theme in science educationin Canada and the United States for much of this century. Public awareness of this need, how-ever, tends to rise and fall with major social and historical events (Aikenhead, 1990; DeBoar,

© 1997 John Wiley & Sons, Inc. CCC 0036-8326/97/050515-18

Correspondence to: C. A. Korpan or G. L. Bisanz

Contract grant sponsors: Social Sciences and Humanities Research Council of Canada; Natural Sciences and En-gineering Research of Canada

1991). Examples of such events include World War II and the launch of Sputnik, both of whichcreated a need for people with talent in science and technical fields. Today, movement toward aglobal economy and mounting environmental problems are among the changes contributing togrowing public recognition of inadequate levels of scientific literacy (e.g., Rutherford &Ahlgren, 1989; Science Council of Canada, 1984). Definitions of scientific literacy vary, butthey may include knowledge about facts, terms, and concepts in various scientific disciplines,as well as knowledge about scientific processes and the social context of science (e.g., DeBoar,1991; Eylon & Linn, 1988; Ryan & Aikenhead, 1992). Scientific literacy can be manifested ina variety of skills, including identifying problems for investigation, formulating hypotheses,designing and conducting research, and evaluating evidence and conclusions (e.g., Alberta Ed-ucation, 1995; Roberts, 1995). In this article, we explore how individuals evaluate conclusionsbased on scientific research and, in particular, conclusions that appear in brief news reports.This topic has received relatively little study but nevertheless is an essential aspect of scientificliteracy in contemporary life (Jacobs, 1982; Millar & Wynne, 1988; Wellington, 1991).

EXTERNAL EVALUATION OF RESEARCH

To acquire the information necessary for scientific literacy, individuals must comprehend,interpret, and evaluate information and conclusions based on scientific research. Evaluationentails judgments about the value, credibility, or validity of conclusions arising from research.Presumably these judgments are influenced by inferences about the extent to which conclu-sions are supported by evidence which in turn require inferences about the extent to which theevidence is valid or compelling.

Aspects of the evaluation process have been studied in children, adults, and scientists (e.g.,Dunbar, 1995; Inhelder & Piaget, 1958; Klahr, Fay, & Dunbar, 1993; Kuhn, Amsel, &O’Loughlin, 1988; Schauble, 1996). A common characteristic in most previous studies is thatevaluation is internal in the sense that it is performed by research participants who collect evi-dence, determine whether the evidence supports or contradicts hypotheses, and arrive at aconclusion. Not all evaluation is of this type, however: Teachers evaluate science fair projectsconducted by their students, scientists evaluate the research of their peers, journalists evaluateand write about research conducted by scientists, and members of the lay public evaluate re-search reported by journalists. In each of these cases, the process of evaluation is external inthe sense that the evaluator considers a communicated, and necessarily incomplete, descrip-tion of the research. The communication can be prepared by the original researchers (e.g., thescientist who conducts the research, or the student who conducts the science fair project) orby people with various levels of expertise. These reports of scientific research may take manyforms (e.g., disciplinary journal articles, media reports). Furthermore, the target audience mayvary in expertise, from disciplinary experts to nonspecialists. We are particularly interested inhow nonspecialists critically evaluate reports about scientific research that are prepared bymedia writers who themselves may or may not have discipline-related expertise.

EVALUATING CONCLUSIONS IN MEDIA REPORTS

Reports of scientific research are pervasive in the media and are a major source of continu-ing science education for lay people (Grandy, 1995; Wellington, 1991). These reports are of-ten brief, sometimes preliminary (as when the focus is a conference report or an interviewwith a researcher about work in progress), and necessarily incomplete. For example, the newsbrief presented in Figure 1 is only 100 words in length. The first paragraph contains a generalconclusion (“people’s susceptibility to infection may be genetically linked”). In the body of

516 KORPAN ET AL.

the brief only four other features of the study are provided: the investigators are identified asscientists; the study was published in Nature; the sample size was 1000; and a specific findingis offered in support of the general conclusion (the worst cases of cerebral malaria are linkedto production of large amounts of tumor necrosis factor). The news brief ends with a commenton the study by an independent group, Britain’s Medical Research Council. Notably absent inthis brief report is any methodological detail.

Evaluation of news reports about scientific research can influence personal or professionaldecisions (e.g., product safety, medical treatment) and participation in public policy debatesabout pressing societal issues (e.g., acid rain, day care) (Collins & Bodmer, 1986; Jacobs,1982; Layton, Davey, & Jenkins, 1986; Peterson & Markle, 1979). In an age of information,lifelong learning is essential and the task of formal education becomes one of teaching foun-dational skills that enable citizens to understand classic and cutting-edge science (Bauer,1992). Given the prevalence of news reports about science and the importance of decisionsbased on scientific research, skill at analyzing these reports critically should be a valued out-come of contemporary science education (Glynn & Muth, 1994).

A related reason to be concerned is that curriculum designers and test builders are beginning tosee the potential value of using news reports of scientific research as a means for teaching and as-sessing scientific literacy. For example, Thier and Nagle (1996) have described an issue-orientedscience course in which media reports are used to help junior high students evaluate and use sci-entific evidence to make decisions about personal and social issues. Also, proposals for revisionsto major, standardized tests include evaluations of reports in the popular press as instances ofnew, more open-ended items (e.g., Michigan Educational Assessment Program Science AdvisoryCommittee, 1993). Optimal use of these items in instruction and assessment will depend, to alarge extent, on an understanding of how students read and think about science-related materials.

EVALUATION OF SCIENTIFIC NEWS BRIEFS 517

Figure 1. A sample news brief from The Globe and Mail, October 6, 1994, p. A13.

INVESTIGATING EXTERNAL EVALUATION

Despite the educational significance of media reports, little attention has been paid to un-derstanding the skills and knowledge students need to critically evaluate articles about sciencein the popular science press, in newspapers, and in news magazines (Mallow, 1991). Findingsfrom the few studies that have been conducted are provocative because they suggest that criti-cal thinking about scientific research may not be well developed in a wide range of students(Dreyfus & Jungwirth, 1980a, b; Gray, 1992; Gray & Mill, 1990; Jungwirth, 1987; Norris &Phillips, 1994). These studies highlight the importance of examining students’ evaluations ofscientific passages. They fail, however, to provide a comprehensive account of the broadrange of issues and questions individuals could raise, or fail to raise, as they think criticallyabout reports of scientific research. Also lacking is information about how external evaluationmight be affected by differences in informational content or text structure, or by individualdifferences among readers.

We conducted an exploratory study of external evaluation in which university students wereasked to read a news brief about a fictitious finding in each of four domains. Similar to the ar-ticle in Figure 1, each news brief contained relatively few details and a conclusion. The newsbriefs were designed to examine students’ knowledge about treatment studies in which a con-clusion is made about effects of an intervention. Treatment studies are common in a variety ofscientific disciplines and are often reported in the media. Students’ evaluation of each newsbrief was assessed by instructing them to request any information they would need to deter-mine whether each conclusion was true. Inherent in this approach is an assertion that a hall-mark of scientific literacy is the ability to make effective requests for information or to askgood questions about scientific research.

Our study was focused on three types of evidence that are essential for formulating a com-prehensive account of the evaluation process: (a) the requests for information students makewhen evaluating scientific news briefs; (b) the influence of text characteristics on the evalua-tion process; and (c) the extent to which requests vary systematically with personal character-istics. Concerning the first type of evidence, we examined the range and frequency of requeststhat students made when evaluating scientific news briefs. Skilled evaluation requires, amongother things, knowledge about features of research that are correlated with the quality of theinvestigation (e.g., Aikenhead & Ryan, 1992; Dreyfus & Jungwirth, 1980a, b; Gray, 1992;Lehman, Lempert, & Nisbett, 1988; Norris, 1995). One set of features corresponds to goodscientific methodology (e.g., use of control groups, adequate sample sizes). Another set offeatures corresponds to the ways in which evidence is linked to theoretical explanations (e.g.,identification of causal agents). To the extent that science education is focused on process-oriented approaches and on helping students to understand causal relations, we would antici-pate that students would request information about the design and conduct of research, thekinds of data collected, and possible causes and theories. Failure to find this outcome mightoccur, however, if knowledge and styles of thinking emphasized in the classroom remain in-ert, and do not transfer to relevant tasks in everyday life (Bransford, Vye, Kinzer, & Risko,1990; Whitehead, 1929).

Yet another set of features corresponds to elements in the social context of research (e.g.,sources of funding for the research, qualifications or prestige of the researchers, quality of thejournal in which the research is reported). Norris (1995) has argued persuasively that knowl-edge about the social context of research is used extensively by practicing scientists to evalu-ate the work of their peers. He also suggests that knowledge about the social context ofscience should be emphasized in science education because it enables individuals to judge thecredibility of research in ways that instruction focused on facts, methodology, and theory can-not. If students understand how communities of scientists function, they should request infor-

518 KORPAN ET AL.

mation about the social context in which the research is conducted. Most undergraduates,however, do not have much exposure to this type of information. In contrast, graduate stu-dents acquire knowledge about the social context of research through immersion in discipli-nary training and participation in the process of research.

The frequency with which students request certain types of information provides insightsinto the salience of methodology, explanation, social context, and other features in students’evaluation of news briefs. Failure to generate requests corresponding to a wide variety of suchfeatures raises the question of whether science education is preparing students adequately tobe knowledgeable consumers of scientific information.

The second type of evidence we examined is the extent to which critical dimensions ofnews briefs influence the evaluation process. Short, fictitious reports were chosen for severalreasons. News briefs have an advantage over longer articles in that critical characteristics ofthe text can be identified and manipulated readily to examine their effects on different readers.News briefs also provide less information than longer articles and thus are less likely to con-strain or cue the readers’ use of knowledge in arbitrary ways. Use of fictitious findings ispreferable to use of real findings at this early stage of research because the latter inevitablyelicit individual differences in knowledge about specific scientific facts that could mask moregeneral knowledge important to the evaluation of a wide range of news stories. Also, use offictitious events makes it possible to ensure that the structure of the briefs is comparableacross domains. We expected that news briefs with relatively implausible conclusions wouldelicit requests for different types of information than news briefs with more plausible conclu-sions (cf. Koslowski, 1996). Consequently, we included a news brief about a paranormalevent. We thought, a priori, that many students would find this news brief less plausible, bothfrom the perspective of everyday experience and normal science, than news briefs with moreplausible conclusions regarding medicine, environment, and education. We also were open tothe possibility that other critical dimensions might emerge from our data.

Finally, we explored whether the types of requests that university students make vary sys-tematically with personal characteristics, such as age, gender, educational background, anddegree of belief in paranormal phenomena. University students are of interest because theyrepresent a segment of society with a relatively strong background in science. Their perform-ance provides one indication of the success, or failure, of secondary science education inpreparing graduates to evaluate new scientific information critically. They also are relativelylikely to become involved in occupations in which they use or disseminate reports of scientificresearch. Age, gender, and educational background within this sample were examined becausethey have been linked to interest in science or to other reasoning skills important to scienceeducation (e.g., Erikson & Erikson, 1984; see also Nisbett, 1993). The degree of belief inparanormal phenomena was assessed because of the suggestion that such beliefs may interferewith critical analyses of information about scientific research (Tobacyk & Milford, 1983). Al-ternatively, Gray (1985, 1987) has suggested that ignorance of what constitutes good and badevidence may contribute to beliefs in scientifically unsubstantiated phenomena. Indeed, de-gree of belief in paranormal phenomena has been found, in some cases, to correlate negativelywith tests of critical thinking (Gray, 1987; Gray & Mill, 1990; Tobacyk & Milford, 1983).

METHOD

Participants

Sixty university students, including 32 females and 28 males, from introductory psychologycourses, participated to fulfill a course requirement. The median age was 21 years (range17–38 years).

EVALUATION OF SCIENTIFIC NEWS BRIEFS 519

Materials and Procedures

A fictitious news brief was created in each of four domains: medicine (“Diet”); paranormalphenomena (“Crystal”); environmental issues (“Insecticide”); and education (“Textbook”).Each news brief had the following structure: (a) researchers report a finding; (b) a generalconcern or issue is described; and (c) an independent group promotes the importance of theresearch finding and arrives at a conclusion. The text of each news brief is included in the Ap-pendix. Four separate orders were created, using a Latin square, and the order in which thenews briefs were presented was counterbalanced almost completely across students.

Students were given booklets and tested in groups. On the first page of the booklet, studentswere asked to read four questions, each similar to the conclusion of a news brief. For exam-ple, the question corresponding to the Insecticide news brief was “How likely do you think itis that the use of a certain insecticide could be important for causing a decline in mating be-havior in robins?” They rated the plausibility of the conclusion embedded in the question on aseven-point scale (1 5 not very likely; 7 = very likely).

For each news brief, students were given 7 minutes to: (a) read the news brief; (b) rate theplausibility of the underlined conclusion, again using a seven-point scale; and (c) generate re-quests for information they would need to determine whether the conclusion was true. Using theInsecticide news brief as an example, the instructions for the generation task were as follows:

Suppose that this conclusion is very important to you and that you must determine whether itis true. What additional pieces of information, if any, would you like to have about the re-searchers’ report to decide whether the conclusion made by members of Nature Unlimited iscorrect. Please list each point you make separately.

Nine numbered places for requests were provided for each news brief, with additional spaceavailable if needed.

Finally, students completed two additional questionnaires. The first was Tobacyk andMilford’s (1983) Paranormal Scale, consisting of 25 statements designed to assess degree ofbelief in seven types of paranormal phenomena: Traditional Religious Belief; Psi Belief;Witchcraft; Superstition; Spiritualism; Extraordinary Life Forms; and Precognition. Studentsresponded to each statement using a five-point scale (1 5 “strongly disagree”; 3 5 “undecidedor don’t know”; 5 5 “strongly agree”). Examples of these statements include, “Black catsbring bad luck” (Superstition) and “A person’s thoughts can influence the movement of aphysical object” (Psi Belief). The full set of items is available in the original paper. Tobacykand Milford found that test– retest reliabilities ranged from 0.60 to 0.87 for each of the sevensubscales. In a second questionnaire, students were asked to indicate their age and gender, aswell as the number of semester-long university courses they had taken in arts (fine arts, hu-manities, and social sciences), engineering, and science (including mathematics).

Classification of Requests

Requests were classified using a taxonomy and procedures developed to categorize ques-tions about reports of scientific research (Korpan et al., 1994). This taxonomy is hierarchical,encompassing rudimentary requests as well as very specific and sophisticated requests. Thetaxonomy is intended to represent a superset of features that individuals could generate whenevaluating conclusions from research. The coder’s task was to read each request and assignthe topic code that most clearly represented the person’s request. The nine major categoriesused to characterize students’ responses are described in Table 1, along with typical requests.Two coders independently scored responses for 12 students who did not participate in the pre-sent study. The reliability of their judgments (number of agreements divided by total number

520 KORPAN ET AL.

TABLE 1Categories of Requests

Category Requests for information about . . . Examples

Social context Social factors that could influence Are the researchers qualified to dojudgments about the quality of this research?the research, the interpretation Who paid for this research?of the data, or the validity of the Where was this research conclusions. Included are the conducted?credentials and motivations of What does Greenpeace think aboutpeople associated with the Permaldrin?research, publication outlets, and Was this research ever published sources of funding. in a peer-reviewed journal? Did it

come from the National Enquirer?

Agent /theory The agent or treatment involved in What is Permaldrin?the research, and about possible How does Permaldrin enter the causes or explanations for the body of the robin?observed results. What other pesticides have this

effect?

Methods How the research was conducted, What methods did they use?including research design, Was a control group used?subjects, procedures, and What kind of robins did they study?replicability. How many robins did they study?

How did they measure decline inmating behaviors?

Data/statistics The data collected or about the What kind of proof do they have?statistical methods used to By how much did mating behaviorsanalyze the data. decline?

What kind of statistical tests did they do?

Related research Relations between the reported Do other researchers get the samefindings and other scientific result?studies.

Relevance The importance, applicability, or Are other birds affected bygeneralizability of the findings, or Permaldrin in the same way?about the impact of the research. Have manufacturers responded by

not making Permaldrin?

Other Other kinds of information that are Is the theory behind this researchrelevant but not covered in the falsifiable?categories above.

Ambiguous/relevant Topics that fall into two or more How did they come to this categories. conclusion?

Off-task Topics that are inappropriate, often Does Permaldrin really exist?because they reflect rejection ofthe premises.

EVALUATION OF SCIENTIFIC NEWS BRIEFS 521

of judgments) across the first six topics in Table 1 was 0.93, ranging from 0.73 (Data/Statis-tics) to 1.00 (Social Context). Frequencies in the remaining topics were too low to estimatereliabilities. One of these coders scored the data for the present study.

RESULTS

Personal Characteristics

Characteristics of the students are described in Table 2. On average, students had completedthe equivalent of almost 2 years of university education. Scores on all subscales of the Para-normal Scale, except Religious Belief, were highly interrelated: None of the six correlationsinvolving Religious Belief were significant, but 14 of the other 15 correlations ranged from0.36 to 0.62 ( ps , 0.01). Consequently, scores for the subscales other than Religious Beliefwere averaged to create a score reflecting Paranormal Belief. As indicated in Table 2, studentstended to agree with statements about Religious Belief, but to disagree with statements aboutParanormal Belief.

Plausibility Judgments

Correlations between ratings before and after reading each news brief were positive andsignificant, means were similar, and the pattern among means was identical. Because readingthe news briefs had little or no effect on plausibility judgments, we focus on the second set ofratings. As expected, mean plausibility ratings were high for the Textbook (5.60), Insecticide(5.37), and Diet (5.30) news briefs, and low for the Crystal news brief about paranormalevents (2.20). A 2(Gender) 3 4(News Brief) analysis of variance with repeated measures onthe last variable revealed an effect of news brief, F(3, 174) 5 94.00, p , 0.01, and an interac-tion, F(3, 174) 5 6.69, p 5 0.01. Tests of simple effects confirmed that the mean rating for theCrystal news brief was lower than the means for the other three news briefs ( p , 0.01), thatthe latter did not differ from each other, and that these effects held for both genders. Malesand females had similar ratings for all news briefs except Insecticides, where males rated theplausibility of the conclusion somewhat higher than did females (5.92 vs. 4.88, p 5 0.02).

To determine whether plausibility ratings varied as a function of personal characteristics,scores for age, gender, and the variables in Table 2 were correlated with plausibility ratings.The strongest correlations involved Paranormal Beliefs. Not surprisingly, students who exhib-

522 KORPAN ET AL.

TABLE 2Personal Characteristics

Characteristic N Mean Standard deviation

University arts courses 57 5.16 9.05University science courses 58 4.00 5.68University engineering courses 59 1.08 3.99Other university courses 59 2.42 5.57Paranormal belief 60 2.42 0.67Religious belief 59 3.86 0.99

Note. N refers to the number of students from whom usable data were obtained. Numbers ofuniversity courses are given in semester-equivalent units. The scale for Paranormal belief andReligious belief ranged from 1 (strongly disagree with statement of belief in a phenomenon) to 5(strongly agree).

ited a greater belief in paranormal phenomena judged the conclusion in the Crystal news briefto be more plausible (r 5 0.49, p , 0.01). A similar relation was found for the Diet news brief(r 5 0.32, p 5 0.04), but not for the other news briefs. The reasons for obtaining a positivecorrelation with the Diet news brief are not entirely clear. One possibility is that people whotend to believe more in paranormal events are more likely to believe in other phenomena thatoften lack sound scientific support. Links between diet and health are well established scien-tifically, but reports of incredible dietary cures are common and sometimes appear in super-market tabloids alongside reports about paranormal events.

Requests for Information

We examined the range and frequencies of requests about various topics (e.g., methods, so-cial context) and the relations among requests, text dimensions, and personal characteristics.If a student requests information on a particular topic for any of the news briefs, then clearlyhe or she is able to consider that topic spontaneously. Requesting the same type of informa-tion for all four news briefs is evidence that the topic is very salient to the student. Failure tomake the request about a topic at all suggests that it is either less salient or unfamiliar. Per-centages of students making at least one request as a function of topic and news brief are pro-vided in Table 3.

Generally, students were more likely to make a request for information on the first four top-ics in Table 3 than for the remaining two topics. Percentages tended to be moderately stableacross news briefs for some topics (e.g., Methods) but less so for others (e.g., Social Context).Two striking patterns of data are evident in the two rightmost columns in Table 3. First, a ma-jority of students request information at least once about Social Context, Agent/Theory, Meth-ods, and Data/Statistics (see the column labeled “Any” in Table 3). In contrast, only 33% everrequested information about related research. Nearly half requested information about the rele-vance of the research, a topic that is unrelated to evaluating the causal connections in the con-clusions. Second, a slim majority of students made requests for information about methods onall four news briefs (see the column labeled “All” in Table 3), a level of consistency that wasnot matched for other topics. Thus many students appeared to consider information about thefirst four topics, but often did not do so consistently across all four news briefs.

EVALUATION OF SCIENTIFIC NEWS BRIEFS 523

TABLE 3Percentage of Students Making at Least One Request as a Function of Topic Category and News Brief

News brief

Topic Textbook Insecticides Diet Crystal Anya Allb

Social context 30 27 33 52 60 10Agent /theory 53 72 73 45 87 28Methods 78 80 68 82 93 52Data /statistics 63 43 45 47 87 13Related research 2 18 13 7 33 0Relevance 7 27 17 12 42 0

aPercentage of students making at least one request for a given topic on any of the four newsbriefs.

bPercentage of students making at least one request for a given topic on all four news briefs.

More specific comparisons can be made by examining the numbers of requests as a func-tion of topic and news briefs. Students generated an average of nearly five requests per newsbrief, and the number of requests did not differ significantly across news briefs. For each stu-dent, we counted the number of responses in each topic category for each news brief. Meansare presented in Table 4. Requests classified as Other, Ambiguous, and Off Task were rela-tively infrequent and are combined in a category labeled Other Responses in the table. Re-quests classified as Related Research, Relevance, and Other Responses were too infrequent towarrant statistical analysis.

To identify patterns in these means, a 2(Gender) 3 4(Topics: Social Context, Agent/Theory,Methods, Data/Statistics) 3 4(News Brief) analysis of variance (ANOVA) was performedwith repeated measures on the last two variables. This analysis revealed an effect of Topic,F(3, 174) 5 22.16, p , 0.01. According to Newman–Keuls comparison (a 50.05), requestsfor Methods were more frequent than for Agent/Theory, Data/Statistics, and Social Context,and the means for these latter three topics did not differ significantly.

This pattern among topics differed somewhat across news briefs, as indicated by the interac-tion among these variables, F(9, 522) 5 6.25, p , 0.01. Using separate 2(Gender) 3 4(Topic)ANOVAs on each news brief and subsequent Newman–Keuls tests (a 50.05), we found thatthe overall pattern held for the Textbook and Crystal news briefs. The pattern for the other twonews briefs, Diet and Insecticide, was similar, except that Agent/Theory requests were morefrequent than Social Context and Data/Statistics requests. In fact, Agent/Theory requests didnot differ significantly from methods requests in the Diet news brief. The Diet and Insecticidenews briefs involve phenomena that are closely related to sciences typically taught in elemen-tary and secondary schools (i.e., biology, chemistry, and physics). In contrast, the content in theTextbook and Crystal news briefs would be uncommon in most school science courses. Theseresults raise the possibility that students are more likely to ask questions about agent, theory, orexplanation when news briefs involve phenomena related to science as it is taught in school.

In the following sections we consider the data for each topic in turn. In each case, we reportthe results of: (a) a 2(Gender) 3 4(News Brief) analysis of variance, with repeated measureson the second variable; and (b) subsequent pairwise comparisons using Newman–Keuls tests(with a 50.05). These analyses were used to identify characteristics of the news briefs thatmight influence the number or types of requests. Finally, correlations were computed betweenrequests and the personal characteristics in Table 2 to determine whether characteristics of thestudents were associated with the number or types of requests generated.

524 KORPAN ET AL.

TABLE 4Mean Number of Requests as a Function of Topic Category and News Brief

News brief

Topic Textbook Insecticides Diet Crystal Mean

Social context 0.53 0.47 0.58 0.92 0.63Agent /theory 0.77 1.15 1.52 0.68 1.03Methods 2.00 1.70 1.63 2.17 1.88Data /statistics 1.03 0.58 0.75 0.57 0.73Related research 0.02 0.20 0.13 0.07 0.11Relevance 0.07 0.35 0.22 0.13 0.19Other responsesa 0.18 0.21 0.23 0.24 0.22Total number of requests 4.60 4.66 5.06 4.78 4.78

aThis category includes requests classified as Other, Ambiguous/Relevant, and Off-Task.

Social Context. Requests for information about social context varied across news briefs,F(3, 174) 5 5.73, p , 0.01. These requests were more frequent for Crystal than for the othernews briefs, which in turn did not differ. Note that Crystal is distinguished by its relativelylow plausibility rating. Nearly all requests for information about social context were aboutpeople associated with the study (see Table 1), including researchers (33%), promoters (38%),creators of the agent used as a treatment (12%), and other experts (12%). Requests werelargely for information about the identity and credentials of the first three types of people andthe opinions of the latter. Requests for information about associated research institutions, pub-lication outlets, and funding sources were rare, perhaps reflecting a lack of emphasis in curric-ula on the social institutions associated with quality of research. No significant correlationswere found between personal characteristics and number of requests for any news brief.

Agent/Theory. Requests for information about the agent and about possible causes or expla-nations for findings were more common for some news briefs than for others, F(3,174) 5 11.81, p , 0.01. The mean number of requests was greater for the Diet than the Insecti-cide news brief, which in turn was greater than for the Textbook and Crystal news briefs. Asnoted earlier, these differences may be a function of students’ perceptions about how relatedthese news briefs are to sciences typically taught in elementary and secondary schools. MostAgent/Theory requests involved the nature and properties of the treatment agent (36%) or atheory of how the agent actually caused the reported effects or side effects (43%). Requestsalso occurred for information about other agents that may cause similar effects (17%).

Agent/Theory requests were associated with only one personal characteristic: People whoare more skeptical about paranormal phenomena are more likely to inquire about properties ofthe agent or about an explanation for reported findings, but this relation is evident only insome news briefs. Specifically, people with less belief in paranormal phenomena were morelikely to make requests for information about the agent or a theory when reading the Textbooknews brief (r 5 20.36, p , 0.01) and the Insecticide news brief (r 5 20.29, p 5 0.026).

Methods. Requests for information about methods differed somewhat as a function of newsbrief, F(3, 174) 5 2.89, p 5 0.037. No significant pairwise comparisons were found usingNewman–Keuls analyses, but a test of simple effects revealed that the combined means forthe Textbook and Crystal news briefs were greater than for the Insecticide and Diet newsbriefs, F(1, 174) 5 8.02, p , 0.01. This recurring difference may reflect a greater belief in theadequacy of the methods used to study phenomena closely related to science as taught inschool (as in the Insecticide and Diet news briefs) than to study other phenomena (as in theTextbook and Crystal news briefs).

Of all the Methods requests, the most frequent were for information about features of re-search design (34%), and especially for information about control of confounding variablesand control groups, which accounted for 10% of the total number of all requests in the study.Requests for information about subjects also were frequent (28%), except for the Insecticidenews brief in which the subjects were birds rather than humans. Finally, students occasionallyasked for further information about measures (16%). These requests were most frequent afterreading the Crystal news brief, perhaps because students are likely to anticipate that the de-pendent variable in this news brief, accuracy of dreams in predicting the future, would be dif-ficult to measure. Requests for information about procedures were rare.

The frequency of methods requests correlated with two types of personal characteristics,but this association was limited primarily to the least plausible news brief. For the Crystalnews brief, requests for information about methods were more frequent for older than

EVALUATION OF SCIENTIFIC NEWS BRIEFS 525

younger students (r 5 0.26, p 5 0.05). Students with more university science courses re-quested more information about methods when requests were averaged across all four newsbriefs (r 5 0.28, p 5 0.04), but when data were analyzed for separate news briefs this relationwas significant only for the Crystal news brief (r 5 0.33, p 5 0.01). Age and number of sci-ence courses were uncorrelated (r 5 0.07).

Data/Statistics. Requests concerning data and/or statistics differed across news briefs, F(3,174) 5 4.48, p , 0.01, and were made most often for the Textbook news brief. This differencedoes not appear to be due to plausibility but may be due to students’ personal experienceswith educational assessment. That is, students’ familiarity with academic tests and grades mayhave provided them with the insights necessary for questioning whether the differences in thereported study were meaningful. Across all news briefs, most of the students’ responses wererequests for further information about the data (85%) rather than about statistical tests (11%),perhaps indicating a lack of sophisticated knowledge about statistical tests.

Students who were more likely to hold traditional religious beliefs were less likely to makerequests for information about data and statistics when averaged across all four news briefs(r 5 20.26, p 5 0.05), although this relation was neither reliable for any individual news briefnor readily explainable.

DISCUSSION

Reading media reports of scientific research is a common means of acquiring informationabout science, and evaluating conclusions found in those reports is an important form of sci-entific literacy. As an initial attempt to understand this type of literacy, we examined the re-quests for information made by university students as they evaluated conclusions in scientificnews briefs. We also identified some potentially critical dimensions of news briefs that influ-ence types and frequencies of requests, and we assessed relations between various personalcharacteristics (age, gender, education, and degree of belief in paranormal phenomena) andthe types of requests that students made. We discuss each of these three topics in turn.

Requests for Information

As a group, university students generated such a diverse set of requests that an extensivescoring taxonomy had to be developed to categorize them (Korpan et al., 1994). A majority ofstudents asked questions about how the research was conducted (Methods), why the resultsmight have occurred (Agent/Theory), what was found (Data/Statistics), and who conductedthe research or where it was conducted (Social Context). Most frequent (see Table 4) and mostconsistent (see the column labeled “All” in Table 3) were requests for information aboutMethods, including queries about research design, possible confounding variables, subjects,and measures. Less frequent, but more consistent than other types of responses, were requestsfor information about Agent/Theory. Included were requests for information about treatmentsand underlying mechanisms that could be useful for developing a theory to explain the re-ported outcomes. In science, conclusions are closely linked to method and theory. Histori-cally, concerns that students might not recognize these links have resulted in advocacy forprocess-oriented approaches to science education (DeBoar, 1991). The prevalence of requestsfor information about how and why may reflect these approaches in science classrooms.

Information about Data/Statistics includes details about the nature, size, and duration of ef-fects, as well as about the statistical tests used. Almost all students requested this type of in-

526 KORPAN ET AL.

formation, but not as often as Methods or as consistently as Methods and Agent/Theory.Many students in this sample probably would not have taken a course in statistics, and so theirknowledge about statistics would not have been sophisticated. Nisbett and colleagues (e.g.,Fong, Krantz, & Nisbett, 1986) have found that even a single course in statistics can improvestatistical reasoning and thus might affect the types of requests for information made in a tasklike ours.

Requests for information about Relevance, including requests for information about thevalue or applicability of the research, were more frequent than expected. These requests werenot critical for evaluating the validity of conclusions in our task, but they could be importantfor understanding the value of conclusions in a broader context. Concern that students be ableto relate scientific research to societal issues has led curriculum planners in Canada and theU.S. to include units that emphasize the relations between science, technology, and society(STS) (Fleming, 1988; Yager & Tamir, 1993). Requests for information about Relevance mayreflect the effects of exploring STS themes in the secondary curriculum.

Disappointing, but not unexpected, was the relatively low frequency and inconsistency ofrequests about Social Context. Only marginally more than half of the students demonstratedany awareness of the credentials and biases of people associated with research, and virtuallyno one requested information about sources of funding, publication outlets, and institutionalsettings. These findings may reflect a lack of sensitivity to the fact that scientific researchtakes place within a social community that can influence the selection of research questions,the interpretation of results, and the acceptance of research findings and theory. A relatedtrend was the low frequency and inconsistency of requests for information about Related Re-search. This finding may reflect a lack of sensitivity to the fact that judgments about the valid-ity of findings and conclusions depend to a large extent on consensual agreement amongresearchers, which in turn depends on whether a study has been replicated and on the degreeto which results and conclusions fit with extant data and theory. Recognizing the importanceof consensual processes in science reflects an understanding of the intersection between goodscientific practice and the social context in which research is conducted.

In a provocative paper, Norris (1995) discussed science as communal practice, pointing outthat even scientists are dependent upon the word of other scientists for much of what theyknow. The dependence of nonscientists is even greater. This inherent epistemic dependencehas led Norris to argue that instruction and practice in judging expertise is an important com-ponent of science education. In fact Norris has gone so far as to argue that “students need tobe taught first that the object of their skepticism should be the believability of experts, not theevidence supporting scientific knowledge claims” (p. 216). In our view, an understanding ofthe social context of science (e.g., the credibility of practitioners, journals, funding sources)needs to be balanced with an understanding of good scientific practices and conventions (i.e.,Methods, Theory, Data/Statistics) as established by the research community. The optimal bal-ance may vary: Instruction about social context may be particularly useful for individuals whodo not pursue a career in science.

Given this view, a broad framework is necessary to assess the types of knowledge individu-als use when evaluating scientific research. Studies of the type that we have conducted haveimportant implications for science education because they provide information about the de-gree to which graduates of our educational system fully appreciate the multifaceted nature ofscience as a human activity. In particular, competent and thorough evaluation of research re-quires not only knowledge about scientific methods and theories, but also knowledge aboutthe role of statistics in evaluation, the social context of science, and consensual processes inscience. The low frequency of requests for information related to these latter three categoriesis cause for concern.

EVALUATION OF SCIENTIFIC NEWS BRIEFS 527

Influential Dimensions of News Briefs

Given their prevalence, scientific news briefs are natural vehicles for preparing studentsfor lifelong learning and for assessing levels of scientific literacy. Before media reports canbe used effectively for instructional and assessment purposes, however, text dimensions mustbe identified that affect how students learn and think about scientific research. We found sys-tematic differences in requests as a function of news brief (compare the columns in Table 3labeled “Any” and “All’). We initially speculated that one dimension, the plausibility of con-clusions, may affect the features that students consider (cf. Klahr, Fay, & Dunbar, 1993). In-deed, students’ perceptions about plausibility were related to the frequency with which theyasked about Social Context: Students requested information about Social Context most oftenfor the least plausible news brief, Crystal. Although requests for information about SocialContext were generally infrequent, the least plausible news brief elicited these requests froma majority of students.

A second, potentially influential dimension may be the degree to which the phenomena de-scribed in news briefs are related to the scientific disciplines emphasized in school curricula.For the Insecticide and Diet news briefs, requests for information about Agent/Theory wererelatively frequent and requests about Methods were relatively infrequent. For the Textbookand Crystal news briefs, in contrast, requests for information about Agent/Theory were rela-tively infrequent and requests about Methods were frequent. These two pairs of news briefsdiffer along a dimension we refer to as typicality. The Insecticide and Diet news briefs refer tophenomena that are typical in biology, chemistry, and/or physics, disciplines that are the focusof science education in elementary and secondary school. In contrast, the phenomena de-scribed in the other two news briefs are less typical. We speculate that typicality influencesthe relation between the strong tendency to ask questions about method and the somewhatweaker tendency to seek information about causes (Agent/Theory). More specifically, whenthe phenomena described are more typical, students are more likely to presume that the meth-ods used by researchers are adequate. Consequently, students are more likely to request infor-mation they need to understand the causes of the phenomena. When the phenomena are lesstypical, students have less confidence in research methodology and focus their requests ac-cordingly, at the expense of seeking information about causes.

A third dimension may be the personal familiarity of the phenomena described in the newsbrief. Requests for information about Data and Statistics were most common for the newsbrief, Textbook, likely to be most closely related to students’ experiences. We speculate thatfamiliarity with the phenomena, and particularly with the treatments and measuring deviceslikely to be used, increases the meaningfulness of numerical information. Students may havemade more requests of this type because they were confident that, given their personal experi-ence with textbooks and educational assessment, they would be able to evaluate the meaningof the data. If news briefs are to be used to promote discussion of numerical information, or toassess the degree to which students refer to numerical information in evaluation, then choos-ing news briefs in domains familiar to students could be important. The relation between fa-miliarity and meaningfulness may have other effects. For example, Alexander, Kulikowich,and Schulze (1994) found that familiarity with physics is related to both interest in and recallof information in media reports about physics.

Our investigation of the dimensions that influence requests for information is exploratory,and clearly our interpretations are tentative. If these results are replicated and extended,then researchers, teachers, journalists, and others who disseminate information aboutscientific research would do well to consider how dimensions of text influence the processof evaluation.

528 KORPAN ET AL.

Personal Characteristics

Individual differences in gender were unrelated to the frequency of requests for informa-tion, but some correlations were found for age, number of university science courses, and de-gree of belief in paranormal phenomena. For the Crystal news brief, requests for informationabout methodology increased as a function of age and number of science courses. Thus, stu-dents appear to adopt an increasingly empirical orientation as they become older and acquiremore science education at the university level, but this relation may be limited primarily toimplausible news briefs. Students who were more skeptical about paranormal phenomenamade more requests about causes, but only for two of the four news briefs. In general, how-ever, correlations were neither pervasive nor particularly systematic. The evaluation skills ofuniversity students are of interest, but the range of variation in personal characteristics mayhave been restricted, thereby reducing correlations between personal characteristics and re-quests. We are currently studying individuals who differ in educational backgrounds, from ju-nior high school to graduate students.

FINAL REMARKS

Various definitions of scientific literacy can be debated, but certainly one practical and im-portant index is the extent to which people can make effective requests for information aboutscientific research reported in the media. Students’ requests can be telling reflections of whatwe, as science educators, have taught them or failed to teach them about the nature of science.Moreover, requesting information effectively and formulating a series of good questions abouta topic of importance can be the opening moves in students’ dialogs about science both in andout of formal educational settings. These evaluative processes, and the knowledge about sci-ence on which they draw, can limit or broaden the scope of what students learn as they inter-act with their world. Considering the importance of these processes, further research isneeded that leads to psychological models of how people evaluate scientific evidence and gen-erate requests for information. An important index of the utility of these models will be theirability to provide insights into how students’ knowledge, dimensions of texts, and personalcharacteristics interact in the course of learning. These insights should be useful to scienceeducators who seek to optimize the growth of students’ evaluation skills in ways that fosterlifelong learning in formal and informal settings.

APPENDIX: NEWS BRIEFS

Textbook

Researchers have reported that high school students in schools where the Science for Lifetextbook is used have higher scores than the provincial norms on science tests. The alarminglypoor performance of Canadian youth in science has become a source of great concern amongeducators, scientists, engineers, and parents. Members of the educational reform group Learn-ing Now have hailed this new finding and have concluded that this textbook is important forimproving high school students’ understanding of science.

Insecticide

Researchers have reported that robins that have been exposed to the insecticide Permaldrinare much less likely to mate than usual. Robins have declined drastically in numbers over the

EVALUATION OF SCIENTIFIC NEWS BRIEFS 529

past several years. Members of Nature Unlimited, an environmental group, have hailed thisnew finding and have concluded that the use of this insecticide is important for causing a de-cline in the mating behavior in robins.

Diet

Researchers have reported that colon-cancer patients who have a diet high in mono-casanates show remarkable recovery from chemotherapy. Recovery from chemotherapy hasbeen an especially serious problem for many cancer patients. Members of the Cancer SupportGroup have hailed this finding and have concluded that this diet is important for counteringthe negative effects of chemotherapy in colon-cancer patients.

Crystal

Researchers have reported that people who wear ollinite crystals during sleep are morelikely to dream accurately about a future event. Public interest in this type of phenomenon hasgrown in recent years. Members of a human potential group, Mind Matters, have hailed thisnew finding and have concluded that wearing this type of crystal is important for helping aperson to dream accurately about future events.

We thank Tammy Dukewich, Mark Gierl, Karen Hubbard, Jacqueline Leighton, Donald Mabbott, Daniel Pilon, and Greg Sadesky for their valuable contributions to the study.

REFERENCES

Aikenhead, G. S. (1990). Scientific/technological literacy, critical reasoning, and classroom practice. InS. P. Norris & L. M. Phillips (Eds.), Foundations of literacy policy in Canada (pp. 127–145). Cal-gary, AB, Canada: Detselig Enterprises.

Aikenhead, G. S., & Ryan, A. G. (1992). The development of a new instrument: “Views onScience–Technology–Society” (VOSTS). Science Education, 76, 477–491.

Alberta Education (1995). Program of studies, elementary schools: Science. Edmonton, AB, Canada:Curriculum Standards Branch, Alberta Education.

Alexander, P. A., Kulikowich, J. M., & Schulze, S. K. (1994). How subject-matter knowledge affects re-call and interest. American Educational Research Journal, 31, 313–337.

Bauer, H. H. (1992). Scientific literacy and the myth of the scientific method. Urbana, IL: University ofIllinois Press.

Bransford, J. D., Vye, N., Kinzer, C., & Risko, V. (1990). Teaching thinking in content knowledge: To-ward an integrated approach. In B. F. Jones & L. Idol (Eds.), Dimensions of thinking and cognitive in-struction (pp. 381–413). Hillsdale, NJ: Lawrence Erlbaum.

Collins, P. M. D., & Bodmer, W. F. (1986). The public understanding of science. Studies in Science Ed-ucation, 13, 96–104.

DeBoar, G. E. (1991). A history of ideas in science education: Implications for practice. New York:Teacher’s College Press.

Dreyfus, A., & Jungwirth, E. (1980a). A comparison of the ‘prompting effect’ of out-of-school with thatof in-school contexts on certain aspects of critical thinking. European Journal of Science Education,2, 301–310.

Dreyfus, A., & Jungwirth, E. (1980b). Students’ perception of the logical structure of curricular as com-pared with everyday contexts—study of critical thinking skills. Science Education, 64, 309–321.

Dunbar, K. (1995). How scientists really reason: Scientific reasoning in real-world laboratories. In R. J.Sternberg & J. E. Davidson (Eds.), The nature of insight (pp. 365–395). Cambridge, MA: MIT Press.

Erickson, G. L., & Erickson, L. J. (1984). Females and science achievement: Evidence, explanation,and implications. Science Education, 68, 63–89.

530 KORPAN ET AL.

Eylon, B., & Linn, M. C. (1988). Learning and instruction: An examination of four research perspec-tives in science education. Review of Educational Research, 58, 251–301.

Fleming, R. (1988). Undergraduate science students’ views on the relationship between science, tech-nology and society. International Journal of Science Education, 10, 449–463.

Fong, G. T., Krantz, D. H., & Nisbett, R. E. (1986). The effects of statistical training on thinking abouteveryday problems. Cognitive Psychology, 18, 253–292.

Glynn, S. M., & Muth, K. D. (1994). Reading and writing to learn science: Achieving scientific literacy.Journal of Research in Science Teaching, 9, 1057–1069.

Grandy, R. E. (1995). Expertise and the critical nonexpert (comment on Norris). Science Education, 79,219–221.

Gray, T. (1985). Changing unsubstantiated belief: Testing the ignorance hypothesis. Canadian Journalof Behavioral Science, 17, 263–270.

Gray, T. (1987). Educational experience and belief in paranormal phenomena. In F. Harrold & R. Eve(Eds.), Cult archaeology and creationism: Understanding pseudoscientific beliefs about the past(pp. 21–33). Iowa City, IA: University of Iowa Press.

Gray, T. (1992). Short, supplementary report: Critical abilities, graduate education (psychology), andbelief in unsubstantiated phenomena. Canadian Journal of Behavioral Science, 24, 399–403.

Gray, T., & Mill, D. (1990). Critical abilities, graduate education (Biology vs. English), and belief in un-substantiated and substantiated phenomena. Canadian Journal of Behavioural Science, 22, 162–172.

Inhelder, B., & Piaget, J. (1958). The growth of logical thinking from childhood to adolescence. NewYork: Basic Books.

Jacobs, J. A. (1982). News from health research: Toward a better understanding. The Journal of SchoolHealth, December, 614–618.

Jungwirth, E. (1987). Avoidance of logical fallacies: A neglected aspect of science-education and sci-ence-teacher education. Research in Science & Technological Education, 5, 43–58.

Klahr, D., Fay, A. L., & Dunbar, K. (1993). Heuristics for scientific experimentation: A developmentalstudy. Cognitive Psychology, 25, 111–146.

Korpan, C. A., Bisanz, G. L., Dukewich, T., Robinson, K. M., Bisanz, J., Thibodeau, M. H., Hubbard,K. E., & Leighton, J. P. (1994). Assessing scientific literacy: A taxonomy for classifying questions andknowledge about scientific research (Tech. Rep. No. 94-1). Edmonton, AB, Canada: University of Al-berta, Centre for Research in Child Development (ERIC Document Reproduction Service No. EDSE054604).

Koslowski, B. (1996). Theory and evidence: The development of scientific reasoning. Cambridge, MA:MIT Press.

Kuhn, D., Amsel, E., & O’Loughlin, M. (1988). The development of scientific thinking skills. SanDiego, CA: Academic Press.

Layton, D., Davey, A., & Jenkins, E. (1986). SSSP: Perspectives on adult science literacy. Studies inScience Education, 13, 27–52.

Lehman, D. R., Lempert, R. O., & Nisbett, R. E. (1988). The effects of graduate training on reasoning:Formal discipline and thinking about everyday-life events. American Psychology, 43, 431–442.

Mallow, J. V. (1991). Reading science. Journal of Reading, 34, 324–338.Michigan Educational Assessment Program Science Advisory Committee (1993). Assessing new direc-

tions in science: Recommendations for the Michigan Educational Assessment Program in Science.Draft document being prepared for the Michigan Department of Education.

Millar, R., & Wynne, B. (1988). Public understanding of science: From contents to processes. Interna-tional Journal of Science Education, 10, 388–398.

Nisbett, R. E. (1993). Rules for reasoning. Hillsdale, NJ: Lawrence Erlbaum.Norris, S. P. (1995). Learning to live with scientific expertise: Towards a theory of intellectual commu-

nalism for guiding science teaching. Science Education, 79, 201–217.Norris, S. P., & Phillips, L. M. (1994). Interpreting pragmatic meaning when reading popular reports of

science. Journal of Research in Science Teaching, 31, 947–967.Peterson, J. C., & Markle, G. E. (1979). The Laetrile controversy. In D. Nelkin (Ed.), Controversy: Poli-

tics of technical decisions. Beverly Hills, CA: Sage.

EVALUATION OF SCIENTIFIC NEWS BRIEFS 531

Roberts, D. A. (1995). Junior high school science transformed: Analyzing a science curriculum policychange. International Journal of Science Education, 17, 493–504.

Rutherford, F. J., & Ahlgren, A. (1989). Science for All Americans. New York: Oxford University Press.Ryan, A. G., & Aikenhead, G. S. (1992). Students’ perceptions about the epistemology of science. Sci-

ence Education, 76, 559–580.Science Council of Canada (1984). Science for every student: Educating Canadians for tomorrow’s

world (Report 36). Hull, Quebec, Canada: Canadian Government Publishing.Schauble, L. (1996). The development of scientific reasoning in knowledge-rich contexts. Developmen-

tal Psychology, 32, 102–119.Thier, H. D., & Nagle, B. W. (1996, August). Issues, evidence, and you—obtain your own evidence. Pa-

per presented at the eighth symposium of the International Organization for Science and TechnologyEducation, Edmonton, AB, Canada.

Tobacyk, J., & Milford, G. (1983). Belief in paranormal phenomena: Assessment instrument develop-ment and implications for personality functioning. Journal of Personality and Social Psychology, 44,1029–1037.

Wellington, J. (1991). Newspaper science, school science: Friends or enemies. International Journal ofScience Education, 13, 363–372.

Whitehead, A. N. (1929). The aims of education. New York: Macmillan.Yager, R. E., & Tamir, P. (1993). STS approach: Reasons, intentions, accomplishments, and outcomes.

Science Education, 77, 637–658.

532 KORPAN ET AL.