Lazy Preservation, Warrick, and the Web Infrastructure

Frank McCown

Old Dominion UniversityComputer Science Department

Norfolk, Virginia, USA

Internet Archive TutorialJCDL 2007Vancouver, BCJune 19, 2007

14

• McCown, et al., Brass: A Queueing Manager for Warrick, IWAW 2007.

• McCown, et al., Factors Affecting Website Reconstruction from the Web Infrastructure, ACM IEEE JCDL 2007.

• McCown and Nelson, Evaluation of Crawling Policies for a Web-Repository Crawler, HYPERTEXT 2006.

• McCown, et al., Lazy Preservation: Reconstructing Websites by Crawling the Crawlers, ACM WIDM 2006.

Available at http://warrick.cs.odu.edu/

15

What Types of Websites Are Lost?

Marshall, McCown, and Nelson, Evaluating Personal Archiving Strategies for Internet-based Information, IS&T Archiving 2007.

31

Success of website recovery each week

*On average, we recovered 61% of a website on any given week.

42

Overlap with Internet Archive

46

Database

Perlscript

config

Static files (html files, PDFs,

images, style sheets, Javascript, etc.)

Web Infrastructure

Web Server

Dynamicpage

Recoverable

Not Recoverable

47

Injecting Server Components into Crawlable Pages

Erasure codesHTML pages Recover at least

m blocks

49

50