LiteratureReview

I&MC QiiarierliiVal77,No.lSprin^lOOO80-98

THE MICROSCOPE AND THE

MOVING TARGET: THE CHALLENGE

OF APPLYING CONTENT ANALYSIS

TO THE WORLD WIDE WEB

By Sally } . McMillan

Analysis of nineteen studies t}tat apply content analysis technicfues to theWorld Wide Web found that this stable research technique can be appliedto a dynmnic environment. However, the rapid growth and change ofWeb-based content present some unique challenges. Nevertheless,researchers are now using content analysis to examine themes such asdiversity, commercialization, and utilization of technology on the WorldWide Web. Suggestions are offered for how researchers can apply con-tent analysis to the Web with primary focus on formulating researchquest ions/hypotheses, sampling, data collection and coding, training/reliability of coders, and analyzing/interpreting data.

Content analysis has been used for decadesas a microscope that bringscommunication messages into focus. The World Wide Web has grownrapidly since the technology that made it possible was introduced in 1991 andthe first Web browsers became available in 1993.' The Commerce Depart-ment estimates that one billion users may be online by 2005, and the growthin users has mirrored growth in content.' More than 320 million home pagescan be accessed on the Web^ and some Web sites are updated almostconstantly.* This growth and change makes the Web a moving target forcommunication research. Caji the microscope of content analysis be appliedto this moving target?

The purpose of this article is to examine ways that researchers havebegun to apply content analysis to the World Wide Web. How are theyadapting the principles of content analysis to this evolving form of computer-mediated communication? What are the unique challenges of contentanalysis in this environment? This review of pioneering work in Web-basedcontent analysis may provide researchers with insights into ways to adapt astable research technique to a dynamic communication environment.

Krippendorff found empirical inquiry into communication contentdates at least to the late 1600s when newspapers were examined by theChurch because of its concern of the spread of nonreligious matters. The firstwell-documented case of quantitative analysis of content occurred in eigh-teenth-century Sweden. That study also involved conflict between thechurch and scholars. With the advent of popular newspaper publishing atthe turn of the twentieth century, content analysis came to be more widely

Sfl//i/ /. McMillan is an assistant professor in the Advertising Department at theUniversity ofTennessee-Knoxvitk. The author acknowledges M. Mark Miller'6 valuableinput oil ail earlier draft of this manuscript and Emmanuela Marcoglou's researchassistance.

80 louRfJMJSM & MASS CoMMUNioinofJ QUAXTEKLI

used.^ it was witb the work of Berelson and Lazarsfeld'' in the 1940s andBerelson'' in the 1950s that content analysis came to be a widely recognizedresearch tool that was used in a variety of research disciplines.

Berelson's definition of content analysis as "a research technique forthe objective, systematic and quantitative description of the manifest contentof communication"** underlies much content analysis work. Budd, Thorp,and Donohew built on Berelson's definition defining content analysis as "asystematic technique for analyzing message content and message han-dling."^ Krippendorff defined content analysis as "a research technique formaking replicable and valid inferences from data to their context.""'

Krippendorff identified four primary advantages of contentanalysis: it is unobtrusive, it accepts unstructured material, it is contextsensitive and thereby able to process symbolic forms, and it can cope withlarge volumes of data." All of these advantages seem to apply equally to theWeb as to media such as newspapers and television. The capability of copingwith large volumes of data is a distinct advantage in terms of Web analysis.Holsti identified three primary purposes for content analysis: to describe thecharacteristics of communication, to make inferences as to the antecedents ofcommunication, and to make inferences as to the effects of communication.'^Both descriptive and inferential research focused on Web-based contentcould add value to our understanding of this evolving communicationenvironment.

How-to guides for conducting content analysis suggest five primarysteps that are involved in the process of conducting content analysis re-search.'•* Those steps are briefly enumerated below with consideration forapplication to the Web.

First, the investigator formulates a research question and/or hypoth-eses. The advent of the Web has opened the door for many new researchquestions. The challenge for researchers who apply content analysis to theWeb is not to identify questions, but rather to narrow those questions andseek to find a context for them either in existing or emerging communicationtheory. The temptation for researchers who are examining a "new" form ofcommunication is to simply describe the content rather than to place it in thecontext of theory and/or to test hypotheses.

Second, the researcher selects a sample. Thesizeofthe sample dependson factors such as the goals of the study. Multiple methods can be used fordrawing the sample. But Krippendorff noted that a sampling plan must"assure that, within the constraints imposed by available knowledge aboutthe phenomena, each unit has the same chance of being represented in tbecollection of sampling units."'"' This requirement for rigor in drawing asample may be one of the most difficult aspects of content analysis on theWeb. Bates and Lu pointed out tbat "with the number of available Web sitesgrowing explosively, and available directories always incomplete and over-lapping, selecting a true random sample may be next to impossible."'-'' Notonly do new Web sites come online frequently, but old ones are also removedor changed, McMillan found that in one year 15.4 percent of a sample ofhealth-related Web sites had ceased to exist.'" Koehler found that 25.3percent of a randomly selected group of Web sites had gone off-line in oneyear. However, be also found that among those sites that remained function-ing after one year, the average size of the Web site (in bytes) more thandoubled,''

The third step in the content analysis process is defining categories.Budd, Thorp, and Donohew identified two primary units of measurements:

THR MICRO'COPFANDTHE MOVING TARGFJ

coding units and context units. Coding units are the smallest segment ofcontent counted and scored in the content analysis. The context unit is thebodyofmaterialsurrounding thecodingunit. Forexample, if thecodingunitis a word, the context unit might be the sentence in which the word appearsor the paragraph or the entire article. Many researchers use the term "unit ofanalysis" to refer to tbe context from which coding units are drawn."^

Defining the context unit/unitof analysisisa unique challenge on theWorld Wide Web. Ha and James argued that the home page is an icfeai unitof analysis because many visitors to a Web site decide whether they willcontinue to browse the site on tbe basis of their first impression of the homepage. They also argued tbat coding an entire site could be extremely time-consuming and introduce biases based on Web-site size.'* '̂ Web Techniquesestimated tbat Web sites range from one page to 50,000 pages.'" Otherresearchers have attempted to take a more comprehensive approach todefining the unit of analysis. For example, in a study of library Web sitesClyde coded all pages mounted by a library on its server.^'

Fourth, coders are trained, they code the content,and reliability of theircoding is checked. Each step of this coding process is impacted by Wehtechnology. The Web can facilitate training and coding. For example,McMillan trained twocoders who were living in different parts of the countryto assist in coding Web sites. Instructions and coding sheets were deliveredby e-mail and coders were given URLs for sites to be coded. The codingprocess was carried out remotely and without paper,^^ Wassmutb andThompson noted potential problems in checking intercoder reliability forchanging content. They controlled for possible changes in content by havingsites that were being coded by multiple coders viewed at the same day andtime by all coders.-^ Massey and Levy addressed the problem of changingcontentby having all sites evaluated twicewith the second visitmade twenty-four hours after the initial visit to the site.^''

Fifth, the data collected during the coding process is analyzed andinterpreted. The statistical tools used to analyze the data will depend on thetype of data collected and on the questions that are addressed by the research.Similarly, the interpretation will grow out of the data analysis as well as theresearcher's perspective. Tbe fact that the content that has been analyzed isWeb-based is not expected to change the basic procedures of analysis andinterpretation used in content analysis.

Methodology The author took several steps to identify both published and unpub-lished studies that have applied content analysis to the Web. First, a searchof the electronic Social Sciences Citation Index was conducted searching forkey words "Web" and "content analysis." Tbe SSCI was also searched for"Internet" and "content analysis." Selected communication journals werealso reviewed for any articles that applied content analysis to the Web (seeAppendix ]). Tbe author also sought out papers from communicationconferences (e.g.. Association for Education injournalism and MassCommu-nication and Internationa! Communication Association) that applied contentanalysis to the Web. Finally, bibliographies were checked for all studies thatwere identified using the above techniques. Any cited study tbat seemed toapply content analysis to the Web was examined.

Eleven studies were identified that applied content analysis to com-puter-mediated communication technologies other than the Web. Threestudies analyzed content of Hstservs designed for academicians and practi-

o2 JOURNALISM & MASS CuMMUNicAriDN

tioners of public relations^^ and advertising.-^* Two studies analyzednewsgroups tbat served as online support/information groups on the topicsof abortion^'' and eating disorders,̂ ** Two studies analyzed pornographicmaterials found on Usenet.^^ Two studies focused on e-mail and electronicbulletin board postings related to public affairs: Newhagen, Cordes, andLevy examined e-mail sent to NBC Nightly News^'^ and Ogan examinedcontent of an electronic bulletin board during the Gulf War.-'' The final twostudies focused on organizational communication issues with analysis of e-mail messages related to an online training program^^ and an Internetdiscussion group established in conjunction with a conference.^''

In addition to the studies that examined computer-mediated commu-nication forms such as e-mail and listservs, nineteen studies were found thatfocused specifically on content analysis of the World Wide Web, Thesenineteen studies are listed in Appendix 2 and examined in tbe Findingssection of this article. Tbe five steps in content analysis identified in theliterature are the basis for examination of how these nineteen studies appliedtbe microscope of content analysis to the moving target of tbe World WideWeb.

Tbe first step in content analysis is to formulate research questionsand /or hypotheses. Table 1 summarizes the primary purpose and questionsaddressed by each of the nineteen studies tbat applied content analysis to theWorld Wide Web. The majority of these studies were descriptive in nature.However, a few of these studies do seem to be moving more towardhypothesis testing and theory development. For example, McMillan pro-posed four models of funding for Web content based in part on level ofinteractivity that was determined by analysis of Web site features.-*^ Wassmu thand Thompson suggested that their study might be a first step in developinga theory that explains relationships between specificity of banner ad mes-sages and the goal of information tasks,- '̂'

The second step in conducting a content analysis is to select a sample.Table 2 summarizes information about samples for the nineteen studies thatanalyzed content of Web sites. The sampling frame from which sites weredrawn varied widely. One common way of defining a sampling frame wasto use an online list of sites in a given category. A second popular techniquewas to use search engine(s) to identify sites that met criteria related to thepurpose of the study. Two studies used off-line sources to identify Websites^'' and two studies combined online and off-line listings of Web sites,^'Finally, one site analyzed three newspaper Web sites with no discussion ofthe sampling frame.-***

After definingfhe sampling frame, researchers typically draw a samplefor analysis. Nine of tbe nineteen studies did not sample; instead theyanalyzed all sites in the sampling frame. The goals of these studies weretypically to describe and/or set a benchmark for analysis of a given type ofWeb sites.

Among studies that used sampling, use of a table of random numberswas tbe most commonsamplijig technique, McMillan pointed outchallengesof applying a table of random numbers to an online listing or search engineresults. Most lists generated in this way are not numbered leaving tberesearcher a tedious hand-counting task. Search engine listings may bepresented in multiple hypertext sub-menus that further complicate the taskof assigning random numbers.^" However, few of the studies cited hereprovided much, if any discussion, of how to address sampling difficulties

THE MiCFOSCOPt: AND THE MOVING TARGET 83

TABLE 1

Study Overview

Primary Puq^ose and Questions Addressed By Study

Aikat

Alloroet al.

Bar-Iian

Bates and Lu

Clyde

Elliott

Esrock and Leichty

Frazer and McMillan

Gibson and Ward

Ha and James

Ho

U

Liu et al.

Massey and Levy

McMillan

McMillan and Campbell

Peng, Tham, and Xiaoming

Tannery and Wessel

Wassmuth and Thompson

To analyze academic, government, and commercial Web sites to deter-mine their information content.

Compare digital journals to printed journals in order to understand ifthe traditional paper journal is becoming obsolete.

Examine what the authors of Web pages say about mathematician PaulErdos.

To develop a preliminary profile of some personal home pages andthus provide an initial characterization of this new social form.

To identify purposes for which a library might create a home page onthe Web and of the information that might be provided.

Classify sites on the Internet as family life education and compile anextensive list of sites that could be used as a reference.

Examine how corporations are making use of the Web to presentthemselves as socially responsible citizens and to advance theirown policy positions.

Examine the structure, functional interactivity, commercial goals,marketing communication analogies, and types of businesses thathave established a commercial presence on the Web.

Addresses the question of whether the Internet is affecting the domi-nance of the two major parties in U.K. politics.

Explore the nature of interactivity in cyberspace by examining asample of business Web sites.

Propose a framework to evaluate Web sites from a customer's perspec-tive of value-added products and services.

Examine how Internet newspapers demonstrated a change from theconvention of newspaper publishing to the new media age.

Identify ways in which large U.S. firms have used their Web sites andhome pages to conduct business.

Investigate online journalism's practical state through a more cohesiveconception of interactivity and to contribute toward building ageographically broad perspective of how journalism is being doneon the Net.

Examine relationships between interactivity, perceptions of propertyvalues, audience size, and funding source of Web sites.

Evaluate current status of networked communication as a tool forbuilding a forum for community-based participatory democracy.

Explore current trends in Web newspaper publishing bylooking at various aspects of such operations as advertising,readership, content, and services,

To demonstrate bow medical libraries utilize the Web to developdigital libraries for access to information.

To examine banner ads in online editions of daily, general interestnewspapers in the United States to determine if the banner ads arespecific, general, or ambiguous.

that arise from using online sources to identify the sampling frame. Thesample size for these nineteen studies varied dramatically from three to 2,865.The majority of the studies (fourteen) analyzed between fifty and 500 sites.

The third step in content analysis is defining categories. Table 3summarizes the categories and other details of data collection. In contentanalysis of traditional media, one of the first steps is to define the time periodof the study (e.g., a constructed week of newspaper issues). But, in analysis

84 JfjuKNAUSM Er MASS CoMMumymoN QuARTEOy

TABLE 2Sampling

Sampling Frame Sample Selection Method SampleSize

Aikat

Alloroet a!,

Bar-Ilan

Bates and Lu

Clyde

Elliott

Esrock and Leichty

Frazer and McMillan

Gibson and Ward

Ha and James

Ho

Li

Liu et al.

Massey and Levy

McMillan

McMillan and Campbell

Peng, Tham, and Xiaoming

Tannery and Wessel

Wassmuth and Thompson

Master list of 12,687WWW addresses

Online biomedical journals

6,681 documents retrievedusing seven search engines

The People Page Directoryhttp;//wvi'w.peoplepage.com

413 library Web sites frommultiple sources

AU World Wide Web sitesthat met four criteria

Fortune 500 list of companieson the Pathfinder Web site.

Web sites mentioned in firstyear of Web coverage inAdvertising Age

U.K. political party Web sites

Web Digest for Marketers(online)

Representative sites foundusing Yahoo and AltaVista

Online editions of threemajor newspapers

Fortune 500 Companies

Daily, general-circulation,English-language news-papers in Asia that publishcompanion Web editions.

Yahoo listing of health-relatedWeb sites

City.net list of Web sites

Web newspapers publishedin the United States

Association of AcademicHealth Science Libraries

Editor and Publisher listingof online U.S. newspapers

Random selection of 1,140about 10 percent of sites

All found using multiple 54search tools

Examined all that made 2,865reference to Paul Erdos

The first five entries under 114each letter

Randomly selected 50

All sites found using and 356links search engines

Every fifth site after 90a randomstart

AU sites mentioned that 156were still functioningat time of study

All listed in one or more 28of four Indexes

Census of all business Web 110sites during given time

Stratified random sample 1,400of business categories

Arbitrary 3

All that could be found using 322multiple search tools

All that could be found using 44multiple search tools

Random-number table 395selection of sites

Random-number table 500selection of sites

Stratified sample of national 80newspapers, metropolitandailies, and lwal dailies

All medical libraries listed 104in online list of libraries

Every ninth site after a 75random start

of Web sites, study authors placed emphasis on the time frame (e.g., April1997) of the study. As noted earlier, changes in the content of Web sitesnecessitates rapid collection of data. The most rapid data collection reportedwas two days and the longest was five months. Most studies collected datain one to two months.

After identifying the time frame of the study, researchers need toidentify context units for coding. The most common context unit used for

THE MJCRoxore. AND THI. MOVINC. TAR

TABLE 3Data Coltectiou and Coding

Time Frame Context Unit Coding Unit

Aikat A two-week period Web site

Alloroet al.

Bar-Ilan

Bates and Lu

Clyde

Elliott

Esrockand Leichty

April 1997

21 Nov. 199(1-7 Jan.1997

June 1996

]6-17Sept. 1996

Januaryto April 1997

November 1997

Web site

Web site

Homepage

All pageson server

Web site

Web site

Frazerand McMillan

Gibsonand Ward

30 hoursin one weekendin April 1996

Marchand April 1997

Web site

Web site

Ha and James

Ho

U

October 1995to January 1996

May throughSeptember 1996

9-18 Sept, 1996

Home page

Portionsof Web Site

Articleson homepage

Content categories: news, commentary, pub-lic relations, advertising, bulletin board,service/product support information, en-tertainment, exhibits/picture archive, databank/general information, miscellaneous/other.

Content categories: aim/scope, author in-structions, table of contents, abstracts ortables, full text, searchable index.

Content categories; mathematical work,Erdos number, in honor/memory, jokes/quotations, math education.Home page internal structure and physicalfeatures. Content categories: informationabout the site as a whole, purposes of homepage, personal information content ofpages, misc. content/elements/features.

A list of 57 features. Country of Web siteorigin. Education level of Web site.

Content categories: human developmentand sexuality; interpersonal relationships;family interaction; family resource man-agement; education about parenthood, fa-mily, and society.

Messages relating to 13 social responsibilitycontentareas.Otherattributes: annual sales,industry sector, ranking on the Fortune list,and rank within industry. Interactive fea-tures (e.g., e-mail links). Potential use ofthe Web sites for agenda-setting activity.

Marketing type (Hoffman/Novak typol-ogy), business type (e.g., entertainment,media), content features (e.g., text, graph-ics), interactive fimctions (e.g., online or-dering), selling message (price, brand,none).

E-mail links to: party headquarters, mem-bers of parliament, regional/local parties.Also coded feedback requested as: policy-based, non-substantive. Web site designfeatures: graphics, split screen, flashing/moving icon, links to other sites, frequencyof updating, web design, targeted youthpages.Nature of business, dimensions of inter-activity, information collection.

Business purpose, types of value creation.

Pictures, graphs, news articles, news links.

Table 3 cont. next page

86 &• M A S S CoMMUMCAni

Table 3 cont.

Liu et a).

Masseyand Levy

McMillan

McMillanand Campbell

Time Frame

20 Sept. 1995-1 Nov. 1995,updated )uly 1996

23 March -10 April 1998

January toFebruary 1997

Spring 1996

Context Unit

Web site

Web site

Web site

Web site

Peng, Tham, January 1997and Xiaoming

Tannery January -March 1997and Wessel

Wassmuth One weekand Thompson in September 1998

Web site

Web site

Banner adsfound insearch task

Coding Unit

Content categories: description, ov'en-'iew,feedback, "what's new," financial, cus-tomer service, search, employment, guestbook, index/directory, online business,other sites, CEO message, FAQ.

Content features: complexity of choice avail-able, responsiveness to the user, ease ofadding information, facilitation of inter-personal interaction

Number ofhot links, banner advertising, hitcounter, indicates last update, organiza-tional tools (e.g., search engine, site map),interactive features (e.g., bulletin boards,chat rooms).

Content categories: contact information,city functions, time/place of public meet-ings, agendas/ minutes of public meet-ings, rules and regulations, budget/fi-nances, e-mail contact, voting information,"hot topics," community "forum."Not explicitly reported in the study.

Level I: basic information such as hours,l(x:ation and policy. Level II: adds interac-tive services such as databases and Internetresources. Level III: adds unique informa-tion such as original publishing, online tu-torials, and digital applications.Banner content (specific, general, ambigu-ous), banner on home page, banner in clas-sified, "trick" banner, IPT tools {e.g., Java},animation.

these studies was the "Web site." Many of the studies did not specify whatwas meant by the Web site. However some did limit analysis to the "homepage" or initial screen seen upon entering the site while others specifiedexamination of all pages that could be found at a given site and othersindicated that they had searched sites in "sufficient detail" to identify keyfeatures.

Given the magnitude and changing nature of sites, some creativitywould seem to be needed in defining the context unit. Wassmuth andThompson devised one creative approach in their analysis of banner ads thatwere encountered when the coder performed a specific task at a newspaperWeb site (find a classified ad for a BMW).'"' Another approach was to havecoders analyze all sites for a specific length of time—about ten minutes.^'This helps to reduce bias that might be involved in coding sites of varyinglengths. However, it might introduce "error" in that different coders maychoose to examine different parts of a Web site during the ten-minuteexamination period.

These nineteen studies varied widely in terms of coding units used bythe researchers. The most common coding unit was "content categories."However, no stanciard list of categories emerges from these studies. Nor dothey rely on established lists such as the fifty categories identified by Bush."Instead, content categories seem to be specifically related to the goals of the

THI: MICROSCOPE AND THF MOVING TARGFT

given study. For example, Bar-Ilan developed content categories related toa specific historical figure,''^ and McMillan and Campbell developed contentcategories related to tbe community-building features of city-based Websites.'"

Another common coding unit is structural features of the Web site(e.g., links, animation, video, sound, etc.). Two studies used Heeter's''^conceptual definition of interactivity as a frame for organizing analysis ofWeb site features.""" One study conducted an in-depth analysis of e-maillinks.'''' Many of the other sites also coded e-mail links as a structuralelement. Tannery and Wessel developed a three-level categorization systemfor evaluating overall sophistication of Web sites based on both contentcategories and structural features.'"* Some studies also reported on "demo-graphic" characteristics of sites such as country of origin and type ofinstitution that created the site while others explored the nature and/orpurpose of the sponsoring organization in more detail. The Wassmuth andThompson study is unique in itsanalysisof banner advertisements in Websites.'*''

The fourth step in content analysis is training coders and checking thereliability of their coding skills. One of the primary reasons for this attentionto training and checking coders is that, as Budd, Thorp, and Donohew noted,an important requirement for any social-science based research is tbat it becarried out in such a way that its results can be verified by other investigatorswho follow the same steps as the original researcher."^' Krippendorff wrotethat at least two coders must be used in content analysis to determinereliability of the coding scheme by independently analyzing content.^'Table 4 summarizes information about bow many coders were used for eachstudy, how they were trained, how mucb of the data was cross-coded(independently coded by two or more individuals), and the method used forcalculating reliability of the coding.

Eight of the nineteen studies did not report any information aboutcoders. Of those that did report on coders, thenumber of coders used rangedfrom two to twelve with an average of about five coders. Only seven studiesreported on training of coders and most said little about how training wasdone.

Eleven studies reported cross-coding techniques. Most cross-coded apercentage {10 to 20%) of all sampled sites. Elliott had all sites cross-checkedby two coders^^ while Li had two coders cross-check all the data collected inthe first three days of a ten-day sample."^^ Ha and James used a pre-test/post-test design that enabled them to correct coding problems prior to the start ofdata collection as well as to check reliability of collected data.^ Ten studiesreported reliability of the coding. Formulas used for testing reliabilityincluded Holsti's reliability formula, Perreault and Leigh's reliability index.Spearman-Brown Prophesy Formula, and Scott's Pi. Bar-Ilan did not indi-cate what type of test was used to generate a 92% reliability score for herstudy.̂ -"̂ Reliability scores ranged from .62 to 1.00.

The fifth stage of content analysis is to analyze and interpret data.Table 5 summarizes key findings. As noted earlier, the purpose of most ofthese studies was descriptive in nature. Therefore, it is not surprising thatkey findings are also descriptive in nature.

One theme that emerged from several of these studies was the diver-sity found at Web sites. For example. Bates and Lu found that the "personalhome page" is an evolving communication form,^ Clyde found variety inthe structure and function of library Web sites,'^^and McMillan concluded

Aikat

AUoro et al.Bar-IlanBates and LuClydeElliott

Esrockand Leichty

Frazerand McMillan

Gibson and WardHa and James

HoU

Liu et al.Massey and Levy

McMillan

McMillanand Campbell

Peng, Tham,and Xiaoming

Tanneryand Wessel

Wassmuthand Thompson

How Many

2

Not stated2Not statedNot stated9

Not stated

7

Not stated7

Not stated2

Not stated12

3

2

Not stated

Not stated

2

TABLE 4Coders and Coding

How Trained

Coded sites notin the sample

Not statedNot statedNot statedNot statedTrained to usesearch enginesand recognizesearch criteria

Not stated

Trained in codingprocedures andvariables

Not statedTrained in using thecoding instrument.

Not statedNot stated

Not statedNot stated

Trained using non-sampled sites

Trained using non-sampled sites

Not stated

Not stated

Trained in mean-ing of primaryvariable

Cross-Coding

16 percent codedby both coders

Not stated287 sites (10%)Not statedNot statedAll sites were cross-checked by twocoders

20% of all sites

lQTo of all sites werecoded by two coders

Not statedAll coders codedtwo pre-test andthree post-test sites

Not statedAll data collectedin first three daysof ten-day sample

Not stated11% of all sites

10% of all sites

10% of all sites

Not stated

Not stated

About 107ocoded cross-coded;tallied by indepen-dent researcher

Reliability

.98 using HolsH'sreliability formulaNot stated92%Not statedNot statedNot stated

Scott's Pi indexranging from .62to 1.0

.88 using Holsti'sintercoderreliability formula

Not statedFrom .64 to .96 us-ing Perreauit andLeigh's reliabilityindex

Not statedSpearman - Brown

Prophecy For-mula Standard-ized item scoreof .92

Not stated.80 to 1-0 usingHolsti's Inter-coder reSiabilityformula

.92 using Holsti'sIntercoder relia-bility formula

.97 using Holsti'sIntercoder relia-bility formula

Not stated

Not stated

Rangingfrom.70to1.0 using Scott's Pifor each variable

that the Web has diversity of content, funding sources, and communicationmodels.''"

A second theme found in these studies is commerciali7,ation of theWeb. Some studies view commercialization as positive for marketers and/or consumers while others express concern about impacts of commercializa-

TffE MtcsascoPi. AND THF. MOVING T.iRGtT 89

TABLE 5Summary of Findings

Primary Findings of the Study

Aikat Public relations and advertising were the dominant information categories for the threetypes of WWW sites studied.

Alloro The Internet seems useful only for publishers seeking to market their products or foret al. authors needing information for submission of papers. The replacement of printed

journals by their electronic counterparts in the field of biomedicine does not seempossible in the short term.

Bar-llan At this time, the Web cannot be considered a good quality data base for learning aboutPaul Erdos, but it may serve as an adequate reference tool once the noise introduced byduplicates, structural links, and superficial information is removed.

Bates The home pages reviewed displayed a great variety of content and of specific types ofand Lu formatting within broader formatting approaches. The home page as a social institution

is still under development.Clyde Although there is some agreement about some of the features that are necessary on a

library's home page there is nevertheless great diversity, too, and not all library Webpages contained even basic information to identify the library.

Elliott Of the key concepts analyzed in this study, \7% were covered by only one site, and 18%were not covered at all. There are many concepts that family life educators couldfocus on in on-line education.

Esrock 82'̂ i of the Web sites addressed at least one corporate social responsibility issue. Moreand Leichty than half of the Web sites had items addressing community involvement, environmental

concerns, and education. Few corporations used their Web pages to monitor publicopinion. The number of social responsibility messages was positively correlated withthe size of an organization.

Frailer and Thus far, itdoesnotseem that many marketers are taking full advantageoftheinteractiveMcMillan capabilities of the World Wide Web.Gibson The Internet allows minor parties in the United Kingdom to mount more of a challengeand Ward to their major counterparts than in other media.Ha and The generally low use of interactive devices reveals a discrepancy between the interactiveJames capability of the medium and the actual implementation of interactivity in a business

setting.Ho Sites are primarily promotional and the marketing approach taken is conventional:

product news, catalogs and portfolios, previews and samples, special offers anddiscounts, contests and sweepstakes.

Liu et al. Web sites can be used to support pre-sales, sales, and after-sales activities. Companiesthat have higher market performances will more likely use Web sites to reach theircustomers. Using Web sites to reach potential customers is not limited to certainindustries.

Massey A fuller portrait of online journalism can be developed by applying a more unifiedand Levy conception of interactivity to news-making on the Web.

McMillan The computer-mediated communication environment is currently robust enough fordiversity of content, funding sources, and communication models to co-exist.

McMillan It is possible to use the Web to implement activities that have been designed to increaseand citizen participation in public life; it is also possible to use this technology to further theCampbell commercial interests that define consumption communities.

Peng, Tham, New technologies can bring new opportunities as well as threats to existing media,and Xiaoming

Tannery Most libraries are at Level U (basic information plus interactive resources), but the futureand Wessel lies with Level III sites that use digital technology as a distinct medium with online

tutorials, locally created databases, and digital archives.As a user navigates deeper into the site and closer to the goal of the information locationtask, banner ads become progressively more specific.

WassmuthandThompson

90

tion on public life and/or academic expression. As a counterpoint to theseconcerns, Gibson and Ward found the Web offered a forum for minorpolitical parties to have a voice.^''

A third major theme was the fact that many site developers are notusing the Web to its full potential as a multi-media interactive environment.Researchers found many sites made limited use of interactivity, graphicsandmotion video, and search functions. And Elliott's study suggested that thereis still significant room for development of new content in important areassuch as family life education.''"

The studies reviewed above found that the stable research technique ofcontent analysis can be applied in the dynamic communication environmentof the Web. But using content analysis in this environment does raisepotential problems for the researcher. Some of these studies exhibit creativeways of addressing these problems. Unfortunately, others seem to havefailed to build rigor into their research designs in their haste to analyze a newmedium. Future studies that apply content analysis techniques to the Webneed to consider carefully each of the primary research steps identified in theliterature: formulating research questions and/or hypotheses, sampling,data collection and coding, training coders and checking the reliability oftheir work, and analyzing and interpreting data.

For the first step, formulating the research questions and/or hypoth-eses, content analysis of the Web is both similar to and different fromtraditional media. Content analysis of traditional media, such as newspapersand broadcast, assume some linearity or at least commonly accepted se-quencing of messages. Hypertext, a defining characteristic of the Web, defiesthis assumption. Each individual may interact with content of a Website indifferent ways. Furthermore, the Web is both "like" and "unlike" print andbroadcast as it combines text, audio, still images, animation, and video. Theseunique characteristics of the medium may suggest unique research ques-tions. However, in some fundamental ways, the first step in the researchprocess remains similar. Researchers should build on earlier theoretical andempirical work in defining their Web-based research. Approaches taken bythe authors cited above tend to be consistent with Holsti's first purpose ofcontent analysis: describing the characteristics of communication. This isappropriate forearly studies ofan emerging medium. However, researchersalso need to move on to the other two purposes identified by Holsti: makinginferences as to antecedents of communication and making inferences as tothe effects of communication.'''

The second step in content analysis research, sampling, presents someunique challenges for Web-based content analysis. As noted earlier, a keyconcern in sampling is that each unit must have the same chance as all otherunits of being represented. The first challenge for the researcher is to identifythe imits to be sampled. This will be driven by the research question. Forexample, if the researcher wants to examine a sample of Web sites for Fortune500 companies, it may be fairly simple to obtain a list of these sites and applya traditional sampling method (e.g., a table of random numbers, every nthitem on the list, etc.) But, if one seeks a different kind of sample (e.g., allbusiness Web sites) the task may become more difficult. Essentially, theresearcher has two primary sources from which to develop a sampling frame:offline sources and online sources.

Recommend-ationsfor FutureResearchers

THE MICROSCOPE AND THE MOVING TARC 91

If the researcher draws a sample from offline sources (e.g., directories,lists maintained by industry groups, etc.), then drawing a random sampleis fairly nonproblematic. Because the sample universe exists in a "set"medium, traditional methods of sampling can be used. However, a majorproblem with offline sources is that they are almost certainly out of date.Given the rapid growth and change of the Web, new sites have probably beenadded since the printed list was created while others have moved or beenremoved.

Online sources can be updated more frequently, and thus help toeliminate some oftheproblemsassociated with offline sources. And, in manycases (e.g., directories, lists maintained by industry groups, etc.), the list canappear in the same kind of format that is found in an offline source makingsampling fairly straightforward. But, human list-compilers are less efficientthan search engines for identifying all potential Web sites that meet specificcriteria. And in some cases no compiled list may exist that directly reflects thecriteria of a study. In these cases, search engines may be the best way ofgenerating a sample frame. However, care must be taken in using searchengine results. First, authors must make sure that they have defined appro-priate key words and that they understand search techniques for the searchengine that they are using. If they do not, they run the risk of either overregistration or under registration. Human intervention may be needed toinsure that the computer-generated list does not include any spurious and/or duplicate items. Once a list has been generated, the researcher needs todetermine the best way to draw a random sample. One way to ease theproblem of random assignment may be to generate a hardcopy of the list.Then the researcher can assign random numbers to the list and/or moreeasily select every nth item. However, if a hierarchical search engine (suchas Yahoo!) is used, special care must be taken to ensure that all items in sub-categories have equal chance of being represented in the sample. In somecases, stratified sampling may be required.

Research needs to be done to test the validity of multiple samplingmethods. For example, does the use of different search engines result inempirically different findings? What sample size is adequate? How cansampling techniques from traditional media (e.g., selection of "representa-tive" newspapers or broadcast stations) be applied to the Web? A keyconcern is that sampling methods for the Web not be held to either higher orlower standards than have been accepted for traditional media.

The rapid growth and change of the Web also leads to potentialproblems in tbe third stage of content analysis: data collection and coding.The fast-paced Web almost demands that data be collected in a short timeframe so that all coders are analyzing the same content. Koehler hassuggested some tools that researchers can use to download Web sites tocapture a "snapshot" of content.''- However, as Clyde noted, copyright lawsof some countries prohibit, among other things, the storage of copyrightedtext in a database system.''-' Whether or not the site was downloaded priorto analysis, researchers must specify when they examined a site. Just asnewspaper-based research must indicate the date of publication, and broad-cast-based research must indicate which newscast is analyzed, so Web-basedcontent analysis must specify tbe timeframe of the analysis. For sites thatchange rapidly, exact timing may become important.

Researchers must also take care in defining units of analysis. Thecoding unit can be expected to vary depending on the theory upon which thestudy is based, the research questions explored, and the hypotheses tested.

92 louKMAijUM & MASS QiMMiiMomoN QuAfnr.Kiy

However, some standardization is needed for context units. For example,analyses of traditional media have developed traditional context units (e.g.,the column-inch and/or a word count for newspapers, time measured inseconds for broadcast), but no such standard seems to have yet emerged forthe Web. The fact that multiple media forms are combined on the Web maybe one of the reasons for this lack of a clear unit of measurement. The majorityof the studies reviewed above simply defined the context unit as the "Website." Future studies .should specify how much of the Web site was reviewed(e.g., "home page" only, first three levels in the site hierarchy, etc.). But asanalysis of the Web matures, entirely new context units may need to bedeveloped to address phenomena that are completely nonexistent in tradi-tional media. For example, Wassmuth and Thompson measured the ways inwhich the content of the Web was "adapted" in real-time to become moreresponsive to site visitors' needs.*^

The fourth step in content analysis, training coders and checking thereliability of their work, involves both old and new challenges on the Web.Given the evolving nature of data coding described above, training mayinvolve "teaming together" with coders how to develop appropriate contextand coding units. But the rapid change that characterizes the Web mayintroduce some new problems in checking intercoder reliability.'''' This doesnot mean that well-developed tools for checking reliability (e.g., Holsti'sindex, Scott's Pi, etc.) should be abandoned. Rather, the primary challengeis to make sure that coders are actually cross-coding identical data. If Websites are checked at different times by different coders and/or if the contextunit is not clearly defined, false error could be introduced. The coders mightnot look at the same segment of the site, or data coded by the first coder mighthe changed or removed before the second coder examines the site. Only oneof the studies reviewed above directly addressed this issue. Wassmuth andThompson carefully defined a task to be preformed at a site and had twocoders perform that identical task at exactly the same date and time.** Thisis one viable alternative. However, if the content of the site is changingrapidly, this control may not be sufficient. Another alternative is to havecoders evaluate sites that have been downloaded. These downloaded sitesare "frozen in time" and will not change between the coding times. However,as noted earlier, there may be some legal problems with such downloadedsites. Furthermore, depending on the number of sites being examined, thetime and disk-space requirements for downloading all of the studied sitesmay be prohibitive.

The Web does not seem to pose any truly new challenges in the finalstep in content analysis: analyzing and interpreting the data. Rather, re-searchers must simply remember that rigor in analyzing and interpreting thefindings are needed as much in this environment as in others. For example,some of the studies reported above used statistics that assume a randomsample to analyze data sets that were not randomly generated. Findings andconclusions of such studies must be viewed with caution. Such inappropriateuse of analytical tools would probably not be tolerated by reviewers had notthe Web-based subject mater of the studies been perceived as "new" and"innovative." But new communication tools are not an excuse for ignoringestablished communication research techniques.

In conclusion, the microscope of content analysis can be applied to themoving target of the Web. But researchers must use rigor and creativity tomake sure that they don't lose focus before they take aim.

THF MKROSCOK AND THF MOVINC. TARCFT 93

APPENDIX 1

The following journals were searched for articles that reference content analysis of the WorldWide Web. The search began with January 1994 issues and continued through all issues that wereavailable as of early July 1999.

Coiiinninicatioii ResearchHuman Coinmiinicatiati Researchlournal of Advertisiii;^fournal of Advertising ResearchJournal of Broadcasting & Electronic MediaJournal of CommunicationJournal of Computer Mediated CommunicationJournal of Public Relations ResearchJournalism & Mass Communication Quarter!}/Journalism &• Mass Communication EducatorNewspaper Research JournalPublic Relations Review

NOTES

1. Barry M. Leiner, Vinton G. Cerf, David D. Clark, Robert E. Kahn,Leonard Kleinrock, Daniel C. Lynch, Jon Postel, Larry G. Roberts, andStephen Wolff, A Brief History of the Internet (Available online at: h t tp: / /www.isoc.org/intemet-history/brief.htmt).

2. Commerce Department, The Emerging Digital Economy (availableonline at: http://www.ecommerce.govdanintro.htm).

3. Internet Index (available online at: http://www.openmarket.com/intindex/content.html).

4. Wallace Koehler, "An Analysis of Web Page and Web Site Constancyand Permanence," Journal of the American Society for Information Science 50(February 1999): 162-80.

5. Klaus Krippendorff, Content Analysis: An Introduction to Its Method-ology (Beverly Hills, CA, 1980), 13-14.

6. Bernard Berelsonand Paul F. Lazarsield, The Analysis of Conimuuica-tion Content (Chicago and New York: University of Chicago and ColumbiaUniversity, 1948).

7. Bernard Berelson, Content Analysis in Communication Research (NewYork: Free Press, 1952).

8. Bereison, Content Analysis, 18.9. Richard W. Budd, Robert K. Thorp, and Lewis Donohew, Content

Analysis of Communications (New York: The Macmillan Company, 1967), 2.10. Krippendorff, Content Analysis, 21.lL Krippendorff, Content Analysis, 29-31.12. Ole R. Holsti, Content Analysis for the Social Sciences and Humanities

(Reading, MA: Addison-Wesley, 1969).13. See for example Budd, Thorp, and Donohew, Content Analysis, and

Krippendorff, Content Analysis.14. Krippendorff, Content Analysis, 66.15. Marcia J. Bates and Shaojun Lu, "An Exploratory Profile of Personal

Home Pages: Content, Design, Metaphors," Online and CDROM Rei'iew 21Qune 1997): 332.

94 loURNAUSM & M^SS COMMUNICAnLVJ Ql

APPENDIX 2

Following is a list of the nineteen studies analyzed in this article:

Aikat, Debashis. 1995. Adventures in Cyberspace: Exploring the Information Content of theWorld Wide Web Pages on the Internet. Ph.D. diss.. University of Ohio.

AUoro, G. C. Casilli, M. Taningher, and D. Ugolini. 1998. Electronic biomedical journals: Howthey appear and what they offer. European journal of Cancer 34, no. 3: 290-95.

Bar-Ilan, Judit. 1998. The mathematician, Paul Erdos (1913-1996) in the eyes of the Internet.Scientometrics 43, no 2: 257-67.

Bates, MarciaJ.and Shaojun Lu. 1997. An exploratory profileofpersonal homepages: Content,design, metaphors. Online and CDROM Rci>ieu> 21, no. 6: 331-40.

Clyde, Lauren A. 1996. The library as information provider: The home page. The ElectronicLibrary 14, no. 6: 549-58.

Elliott, Mark. 1999. Classifyingfamily life education on the World Wide Web. Family Relations48, no. 1: 7-13.

Esrock, Stuart L. 1998. Social responsibility and corporate Web pages: Self-presentation oragenda-setting? Public Relations Review, 24, no. 3: 305-319.

Frazer, Charles and McMillan, Sally J. (1999). Sophistication on the World Wide Web: Evaluatingstructure, function, and commercial goals of Web sites. ]n Advertising and the World Wide Web, ed. EstherThorson and David Schumann, 119-34. Hillsdale, NJ: Lawrence Erlbaum.

Gibson, Rachel K. and Stephen J. Ward. 1998. U.K. political parties and the Internet: "Politicsas usual" in the new media? PressjPolitia 3, no. 3: 14-38.

Ha, Louisa and E. Lincoln James. 1998. Interactivity Re-examined: A Baseline Analysis of EarlyBusiness Web Sites. Journal of Broadcasting & Electronic Media 42 (fall): 457-74.

Ho, James. 1997. Evaluating the World Wide Web: A global study of commercial sites, journalof Computer Mediated Couimunication 3, no. 1: Available online at: http:/www.ascusc.org/jcmc/vol3/issuel/ho.html.

Li, Xigen. 1998. Web page design and graphic use of three U.S. newspapers. Journalism & MassCommunication Quarterly 75, no. 2: 353-65.

Liu, Chang, Kirk P. Amett, Louis M.Capella, and Robert C.Beatty. 1997. Web sites of the Fortune500 companies: Facing customers through home pages. Information & Management 31: 335-45.

Massey, Brian L. and Mark R. Levy, 1999. Interactivity, online journalism, and English-languageWeb newspapers in Asia, journalism & Mass Communication Quarterly 76, no. 1: 138-51.

McMillan, Sally J. 1998. Who pays for content? Funding in interactive media. Journal ofComputer-Mediated Communication 4, no. 1: Available online at: http:/www.ascusc.org/icmc/vol4/issuel/mcmillan.html.

McMillan, Sally J. and Katherine B. Campbell. 1996. Online cities: Are they building a virtualpublic sphere or expanding consumption communities? Paper presented at the AEJMC annualconference, Anaheim, CA.

Peng, Foo Yeuh, Naphtali Irene Tham, and Hao Xiaoming. 1999. Trends in online newspapers:A look at the US Web. Newspaper Research Journal 20, no. 2: 52-63.

Tannery, Nancy and Charles B. Wessel. 1998. Academic medical center libraries on the Web.Bulletin of the Medical Library Association 86, no 4: 541-44.

Wassmuth, Birgit and David R. Thompson. 1999. Banner ads in online newspapers: Are theyspecific? Paper presented at the American Academy of Advertising annual conference, Albuquerque,New Mexico.

16. Sally J. McMillan, "Virtual Community: The Blending of Mass andInterpersonal Communication at Health-Related Web Sites" (paper pre-sented at the International Communication Association Annual Conference,Israel, 1998).

17. Wallace Koehler, "An Analysis of Web Page and Weh Site Constancyand Permanence, journal of the American Society for Information Science 50(February 1999): 162-80.

THE MKROSCOPF- At'in TTJ/; MOVINC, TARcn

18. Budd, Thorp, and Donohew, Coulciit Anali/sis, 33-36.19. Louisa Ha and E. Lincoln James, "InteracHvity Re-examined: A

Baseline Analysis of Early Business Web Sites," Journal of Broaiicastiii^ &Electronic Media 42 (fall 1998): 457-74.

20. Web Techniques, "Web Statistics," Cyberatlas 2, no. 5 {availableonline at: http://www.cyberatlas.com.

21. Lauren A. Clyde, "The Library as Information Provider: The HomePage," The Electronic Library 14 (December 1996): 549-58.

22. Sally J. McMillan, "Who Pays for Content? Funding in InteractiveMedia," lournal of Computer-Mediated Communication 4 (September 1998):available online at: h t tp : / /www.ascusc .org/ jcmc/vol4/ issuel /mcmillan.html.

23. Birgit Wassmuth and David R. Thompson, "Banner Ads in OnlineNewspapers: Are They Specific?" (paper presented at the American Acad-emy of Advertising annual conference, Albuquerque, NM, 1999).

24 . Brian L. Massey and Mark R. Levy, "Interactivity, Online Journal-ism, and English-Language Web Newspapers in Asia," ]ournalisnt & MrtssCommunication Quarterly 7b (spring 1999): 143.

25. Bonita Dostal Neff, "Harmonizing Global Relations: A Speech ActTheory Analysis of PRForum," Public Relations Review 24 (3, 1998): 351-76;Steven R. Thomsen, "@ Work in Cyberspace: Exploring Practitioner Use ofthe PRForum," Public Relntion^ Revinv 22 (2, 1996): 115-31.

26. Louisa Ha, "Active Participation and Quiet Observation of AdforumSubscribers, journal of Adzvrtising Education 2 (fall 1997): 4-20.

27. Steven M. Schneider, "Creatinga Democratic Public Sphere throughPolitical Discussion: A Case Study of Abortion Conversation on the Internet,"Social Science Computer Review 14 (winter 1996): 373-93.

28. Andrew Winzelberg, "The Analysis of an Electronic Support Groupfor Individuals with Eating Disorders," Computers in Human Behavior 13(summer 1997): 393-407.

29. Michael D. Mehta and Dwaine Plaza, "Content Analysis of Porno-graphic Images Available on the Internet," The Information Sfjdcfi/13 (spring1997): 153-61; Marty Rimm, "Marketing Pornography on the InformationSuperhighway: A Survey of 917,410 Images, Descriptions, Short Stories, andAnimations Downloaded 8.5 Million Times by Consumers in over 2000 Citiesin Forty Countries, Provinces, and Territories," The Georgetown Law Journal 83(May 1995): 1849-1934.

30. John E. Newhagen, John W. Cordes, and Mark. R. Levy,"Nightly@nbc.com: Audience Scope and the Perception of Interactivity inViewer Mail on the Internet," lournal of Communicafion 45 (summer 1995):164-75.

31. Christine Ogan, "Listserver Communication during the Gulf War:What Kind of Medium is the Electronic Bulletin Board?" Journal of Broadcast-ing & Eh-ctronic Media 37 (spring 1993): 177-97.

32. Louis j . Kruger, Steve Cohen, David Marca, and Lucia Matthews,"Using the Internet to Extend Training in Team Problem Solving/' BehaviorResearch Methods, Instruments, & Computers 28 (spring 1996): 248-52.

33. R. Alan Hedley, "The Information Age: Apartheid, Cultural Impe-rialism, or Global Village?" Social Science Computer Review 17 (spring 1999):78-87.

34. McMillan, "Who Pays for Content?"35. Wassmuth and Thompson, "Banner Ads in Online Newspapers."36. Charles Frazer and Sally J. McMillan, "Sophistication on the World

96 lauKNAiJSM & MASS CfiMMUNiCAnof^ QuARTFUiy

Wide Web: Evaluating Structure, Function, and Commercial Goals of WebSites," in Adz'ertising and the World VJide Web. ed. Esther Thorson and DavidSchumann (Hillsdale, NJ: Lawrence Erlbaum) 119-34; Chang Liu, Kirk P.Arnett, Louis M. Capelia, and Robert C. Beatty, "Web Sites of the Fortune 500companies; Facing Customers through Home Pages," Information & Manage-ment 31 (1997): 335-45.

37. Clyde, "The Library as Information Provider"; Massey and Levy,"Interactivity, Online Journalism," 138-51.

38. Xigen Li, "Web Page Design and Graphic Use of Three Ll.S. News-papers," Journalism & Mass Communication Quarterly 75 (summer 1998): 353-65.

39. McMillan, "Who Pays for Content?"40. Wassmuth and Thompson, "Banner Ads in Online Newspapers."41. See Frazer and McMillan, "Sophistication on the World Wide Web";

McMillan, "Who Pays for Content?"; Sally J. McMillan and Kathryn B.Campbell, "Online Cities; Are They Building a Virtual Public Sphere orExpanding Consumption Communities?" (paper presented at the AEJMCAnnual Conference, Anaheim, CA, 1996).

42. Chilton R, Bush, "The Analysis of Political Campaign News," jour-nalism Quarter}]/ 28 (spring 1951): 250-52.

43. Judit Bar-Ilan, "The Mathematician, Paul Erdos (1913-1996) in theEyes of the Internet," Scicntomctrics 43 (March 1998); 257-67.

44. McMillan and Campbell, "Online Cities."45. Carrie Heeter, "Implications of New Interactive Technologies for

ConceptualizingCommunication,"inM('diflUsem//ipiM/t>rmflfiOMA;?('; Emerg-ing Patterns of Adoption and Consumer Use, ed, Jerry L. Saivaggio and JenningsBryant (Hillsdale, NJ: Lawrence Erlbaum, 1989), 217-35.

46. Massey and Levy, "Interactivity, Online Journalism," and McMillan,"Who Pays for Content?"

47. Rachel K. Gibson and Stephen J. Ward, "U.K. Political Parties and theInternet; 'Politics as usual' in the new media?" Press/Politics 3 (fall 1998); 14-38.

48. Nancy Tannery and Charles B. Wessel, "Academic Medical CenterLibraries on the Web," Bulletin of the Medical Library Association 86 (October1998): 541-44.

49. Wassmuth and Thompson, "Banner Ads in Online Newspapers."50. Budd, Thorp, and Donohew, Content Analysis, 66.51. Krippendorff, Content Analysis, 7452. Mark Elliott, "Classifying Family Life Education on the World Wide

Web," Family Relations 48 (January 1999): 7-13.53. Li, "Web Page Design."54. Ha and James, "Interactivity Re examined."55. Bar-Man, "The Mathematician, Paul Erdos."56. Bates and Lu, "An Exploratory Profile of Personal Home Pages."57. Clyde, "The Library as Information Provider."58. McMillan, "Who Pays for Content?"59. Gibson and Ward, "U.K. Political Parties and tbe Internet,"60. Elliott, "Classifying Family Life Education,"61. Holsti, Content Analysis for the Social Sciences ami Humanities.62. Koehler, "An Analysis of Web Page and Web Site Constancy."63. Clyde, "The Library as Information Provider."64. Wassmuth and Thompson, "Banner Ade. in Online Newspapers."65. A separate issue, not directly addressed in this .study, is the use of

THE MtcRoscopE AND THE MOVING TARGET 97

computer-based content analysis for examining the digital content of Websites. None of the studies examined in this article utilized this technique.With computeri;2ed analysis of content, reliability is assured—the computerwill obtain the same results every time the data is analyzed. With this formof content analysis, emphasis must be placed on validity rather fhan onreliability,

66. Wassmuth and Thompson, "Banner Ads in Online Newspapers."

98 jniKNAi KM & MA^^ OmMUNiCAnoN QuM