Digital Libraries and the Open Archives Initiative Louisiana State University June 30, 2000 Edward...

Digital Librariesand the

Open Archives Initiative

Louisiana State UniversityJune 30, 2000

Edward A. Fox

fox@vt.edu http://fox.cs.vt.edu

CC CS DLRL Internet TIC

Virginia Tech, Blacksburg, VA, USA

Acknowledgements (Selected) Sponsors: ACM, Adobe, ARL, Belgian Science Found., CLIR,

DARPA, IBM, LANL, Microsoft, NSF, OCLC, SPARC, US Dept. of Ed. (FIPSE), …

VT Faculty/Staff: Tony Atkins, Thomas Dunbar, John Eaton, Gwen Ewing, Peter Haggerty, Gary Hooper, Gail McMillan, Len Peters, James Powell, …

VT Students: Emilio Arce, Fernando Das Neves, Brian DeVane, Robert France, Marcos Goncalves, Scott Guyer, Robert Hall, Neill Kipp, Paul Mather, Tim McGonigle, Todd Miller, Constantinos Phanouriou, William Schweiker, Ohm Sornil, Hussein Suleman, Patrick Van Metre, Laura Weiss, …

Virginia Tech Background

Largest university in Virginia, land-grant, football, town population 35K plus 25K students

Blacksburg Electronic Village, since 1992, with > 80% of community on Internet

Net.Work.Virginia, largest ATM network, with over 750 sites, for education, research, government

LMDS, Local Multipoint Distribution Service, gigabit wireless networking - 1/3 of Virginia

Math Emporium, 500 workstations Faculty Development Initiative, round 2 Hosting First Joint Conference on Digital Libraries,

www.jcdl.org, Summer 2001 @ Hotel Roanoke, VA

Remember!

Digital libraries introduction

Digital libraries to enhance learning

OAi (help establish enormous international cooperative of data and service providers)

Digital Libraries

SGML (1985)

PDF(1992)

NSF DLI (1994)

LibraryCancellations

(1988)

UniversityScholarlyElectronic

Pub. (1988)

Info.Literacy(1995)

ImprovingEducation Internet

(1984)

WWW(1994)

Multimedia(1986)

Digital LibrariesShorten the Chain from

Editor

Publisher

A&I

Consolidator

Library

Reviewer

DLs Shorten the Chain to

Author

Reader

Digital

LibraryEditor

Reviewer

Teacher

Learner

Librarian

How do universities anddigital libraries relate?

Each U. will have its own digital libraries. Hence there will be large numbers (i.e., critical mass).

All students will learn how to use and how to “feed” digital libraries (and bring those habits to future work as needs and skills).

All digital library problems (esp. federation, flexibility, personalization) appear at U’s (so they are a good type of testbed, with willing collaborators in-place for developing solutions).

Start with NDLTD, extend to NUDL

Digital Libraries --- Virginia Tech

MARIAN (NLM) CS DL Prototype - ENVISION (NSF, ACM) TULIP (Elsevier, OCLC) BEV History Base (NSF, Blacksburg) DL for CS Education - EI (NSF, ACM) WATERS, NCSTRL (NSF) NDLTD (SURA, US Dept. of Education) CSTC (NSF, ACM), CRIM (NSF, SIGMM) WCA (Log) Repository (W3C) VT-PetaPlex-1 (Knowledge Systems)

NCSTRL

http://www.ncstrl.org Networked Computer Science Technical

Reference Library CS Technical Reports 1994 merger of CSTC + WATERS 1998 integration with LANL server (CoRR) Federated search, mirrors, Dienst protocol

Digital Libraries --- Objectives

World Lit.: 24hr / 7day / from desktop Integrated “super” information systems: 5S:

streams, structures, spaces, scenarios, societies Ubiquitous, Higher Quality, Lower Cost Education, Knowledge Sharing, Discovery Disintermediation -> Collaboration Universities Reclaim Property Interactive Courseware, Student Works Scalable, Sustainable, Usable, Useful

Benefits

Ease of use Effectiveness

“The benefits of digital libraries will not be appreciated unless they are easy to use effectively.” - IITA Workshop report

DLs: Why of Global Interest? National projects can preserve antiquities and

heritage: cultural, historical, linguistic, scholarly Knowledge and information are essential to

economic and technological growth, education DL - a domain for international collaboration

– wherein all can contribute and benefit– which leverages investment in networking– which provides useful content on Internet & WWW– which will tie nations and peoples together more

strongly and through deeper understanding

DL Challenges

Preservation - so people with trust DLs

Supporting infrastructure - networks, ...

Scalability, sustainability, interoperability

DL industry - critical mass by covering libraries, archives, museums, corporate info, govt info, personal info - “quality WWW” integrating IR, HT, MM, ...

– Need tools & methods to make them easier to build

Computing (flops)Digital content

Com

mun

icat

ions

(ban

dwid

th, c

onne

ctiv

ity)

Locating Digital Libraries in Computing andCommunications Technology Space

Digital Libraries technologytrajectory: intellectualaccess to globally distributed information

less more

D ig ita l L ib ra r y C o n te n t

A rtic le s ,R e p o rts,

B o o ks

T e xtD o cum e n ts

S p ee ch ,M u s ic

V id eoA u d io

(A e ria l)P h o tos

G e og rap h icIn fo rm ation

M o d e lsS im u la tio ns

S o ftw a re ,P ro g ra m s

G e no m eH u m a n,a n im a l,

p la n t

B ioIn fo rm ation

2 D , 3 D ,V R ,C A T

Im ag es a ndG ra p h ics

C o n te n tT yp e s

Definitions

Library ++ (library+archive+museum+…) Distributed information system + organization

+ effective interface User community + collection + services Digital objects, repositories, IPR management,

handles, indexes, federated search, hyperbase, annotation

Definition: Digital Libraries are complex systems that

help satisfy info needs of users (societies) provide info services (scenarios) organize info in usable ways (structures) present info in usable ways (spaces) communicate info with users (streams)

5S Layers

Societies

Scenarios

Spaces

Structures

Streams

Document Models, Representations, and Accesses

Doc = stream + structure + use-scenario; hybrid (paper/electronic), digital only

Multilingual: content, summary, metadata Multimedia: structure, quality (oS), search Structured: MARC, SGML, by user: MVD Distributed collection: Kleisli, CIMI, Z39.50 Federated search: collecting, picking site(s),

parallel search / fall-back, fusing results Access: IPR, payment, security, scenarios

Architectural Issues

Internet middleware Independent system / part of federation Decompositions vary

– search engine, browser, DBMS, MM support– repository, handle server, client– information resources + mediators, bus or agent

collection + client with workspace/environment Metrics: e.g., for federated search

Standards

Protocols/federation– Z39.50, CIMI– Dienst, NCSTRL– OAi protocol

Metadata– TEI: inline, detailed (structure in stream)– MARC: two-level, fine-grained– Dublin Core: high-level, 15 elements– RDF: describing resources/collections, annotation– OAMS and others used in OAi

Remember!

Digital libraries introduction

Digital libraries to enhance learning

OAi (help establish enormous international cooperative of data and service providers)

Enhancing Learning with DLs

DigitalLibraries

In te rac tiveExperiences

E n hanc ingL earning

N D L T DN e tw o rke d D L

o f Th e se s &D isse rta tio ns

S tu de n t P o rfo liosS e lf-A rch iv ing

G ra y L ite ra tu re(D e p t. o f E d u c .)

W 3 C W C AR e p o s ito ry

L o g s, T oo ls,P u b lica tio ns

C S T CC S

T e a ch ingC e n te r

C R IMC u rricu lu mR e so u rcesIn te r. M M

C o m p u te rS c ie n ce

(w ith N S Fa n d A C M )

DigitalLibraries

In te rac tiveExperiences

E n hanc ingL earning

Enhancing Learning with DLs

DigitalLibraries

A u tho ring(te x t, m ark u p ,h yp erm ed ia ,

ca ta lo g in g -D C )

S u b m itt ingW o rk (E T D )(M eta da ta ,P D F , X M L)

P re s erv ing(u s in g s td s,m ig ra tin g ,

ve rs ion in g )

A dd ing toD ig ita lL ib ra ry

(s tu de n t)

D isc o v e rin g ,B ro w s in g ,S e a rch in g ,R e trie v ing

A nn o ta tin g ,D o w n lo ad in g ,

In s ta llin g ,F e e db a ck

5 S F ra m e w o rk:S o c ie tie s ,S c e na rio s,

S tre am s,S p ac e s,S tru ctu res

U s in gD ig ita l L ib ra ry

(d ire c t)(in fo lite ra cy)

In d ire c tly U s ingD ig ita l L ib ra ry(e m b ed de d ,b y ag en t, . . .)

U s in g D LC o n ten ts (to o ls,d a ta se ts , en v 's,co u rse w a re , . . .)

C o lla bo ra tion(in /a ro u nd D L

a n d its a rt ifa c ts -d is tan c e e du c .)

O th erIn te rac tiveL e arn ingA c tiv it ies

In te rac tiveExperiences

E n hanc ingL earning

NSF Education Innovation (EI)

NSF “Interactive Learning with a Digital Library in Computer Science” (1993-98)

45 online courses (esp. Internet, IR, MM, Professionalism, overall EI project pages): 100+K accesses/wk

Tools: SWAN (visualization), QUIZIT Evaluation

– traditional– network logging and analysis– tools for visualization

Digital Library Courseware

http://ei.cs.vt.edu/~dlib/ WWW pages or large PDF copy files Online quizzes based on book by Michael Lesk

(Morgan Kaufmann Publishers) Contents based on book, with several other

popular topics added (e.g., agents) Separate pages to supplement: Definitions,

Resources (People, Projects), and References

CS -> CSTC -> CRIM NSF and ACM Education Committee are funding

a 2 year project “A Computer Science Teaching Center” - CSTC - http://www.cstc.org/

College of NJ, U. Ill. Springfield, Virginia Tech Focus initially on labs, visualization, multimedia Multimedia part is also supported by a 2nd grant

to Virginia Tech and The George Washington University: http://www.cstc.org/~crim/ (with curricular guidelines also under development)

CS Teaching Center (CSTC) Instead of building large, expensive multimedia packages, that become

obsolete and are difficult to re-use, concentrate on small knowledge units.

Learners benefit from having well-crafted modules that have been reviewed and tested.

Use digital libraries to build a powerful base of support for learners, upon which a variety of courses, self-study tutorials & reference resources can be built. [See NSF NSDL - National Science (math, engineering, technology education) Digital Library (formerly SMETE-lib) at http://www.dlib.org/smete/public/smete-public.html]

ACM Education Board and SIG support, new NSF grant with COLLEGIS Research Institute and others …

Browsing (1)

Browsing (2)

CRIM Rationale

MM field needs properly trained personnel Support this with resources + curricula Together these help us move toward a DL

for Interactive MM -> CS -> NSDL Benefits will go to teachers (who have more

to build upon) and students (who will have a richer environment for learning

CRIM Project Activities

Workshops, other ways to involve community WWW site including DL in CSTC re MM

– Devised cataloging schema, designed interface

– Referring to all MM syllabi and curriculum

– Inviting learning resources for the CRIM DL, with reviews, reuse certifications

Publish report on MM curriculum through ACM and IEEE, after careful review

Introducing into CC2001: information retrieval, hypertext/hypermedia, multimedia, digital libraries

Curriculum Resources in Interactive Multimedia (CRIM)

MM field needs properly trained personnel Support this with resources + curricula Benefits will go to teachers (who have more to

build upon) and students (who will have a richer environment for learning

CSTC, CRIM have led to ACM Journal of

Educational Resources in Computing, JERIC Together these help us move forward: DL for

Interactive MM -> CS -> NSDL

SMETE Library -> NSDL(from www.dlib.org to NSF DLI-2)

Context: Global movement toward Digital Libraries (see April 1998 CACM)

NSF effort: Science, Mathematics, Engineering, and Technology Education Digital Library (focussed on undergraduates)– 3 workshops, yearly increasing funds / new calls

NSDL will operate as a distributed federation, with separate parts for each key discipline, and should lead to a global effort.

Selected NSDL Projects/Topics

COLLEGIS Res. Inst. IMS, CS, Math, Viz., …

Columbia University Earth sciences

Stanford University Medicine (images)

U. California Berkeley Engineering

University of Maryland K-12 education

U. Texas at Austin Physical anthropology

Remember!

Digital libraries introduction

Digital libraries to enhance learning

OAi (help establish enormous international cooperative of data and service providers)

Open Archives initiative

OAiwww.openarchives.org

openarchives@openarchives.org

OAi Philosophy

Self-archiving = submission mechanismLong-term storage system = archiveOpen interface = harvesting mechanismData provider + service providerStart with “gray literature”

– e-prints/pre-prints, reports, dissertations, …

Tiered Model of Interoperability

Mediator services

Metadata harvesting

Document models

Repository of Digital Objects

RepositoryAccessProtocol

handle

Digital object

terms and conditions

Open Archives initiative History

xxx at LANL = Los Alamos National Laboratory (Ginsparg) for high-energy physics - 1991

CSTR + WATERS = NCSTRL (Lagoze) - 1994 xxx + NCSTRL = CoRR collaboration - 1998 UPS (Universal Preprint Service) – 1999 mtg

– Herbert Van de Sompel (U. Ghent, SFX) …– Dublin Core (DC), XML– Dienst protocol and software (Lagoze)

Renamed late 1999 as OAi

Open Archives (protoproto)

ArXiv & Los Alamos National LabCogPrints & U. SouthamptonNACA & NASA (reports)NCSTRL & Cornell U.NDLTD & Virginia TechRePEc & U. SurreyTotal of around 200K records

Original Open Archives Members Caroline Arms, Library of Congress Leslie Carr, University of Southampton Mark Doyle, American Physical Society Dale Flecker, Harvard University Edward A. Fox, Virginia Tech Michael Friedman, HighWire Press, Stanford U. Paul M. Gherman, Vanderbilt U. & SPARC Paul Ginsparg, Los Alamos National Lab. & xxx Stevan Harnad, University of Southampton Thomas Krichel, University of Surrey & RePEc Carl Lagoze, Cornell University …

Original Open Archives Memberscont’d

Rick Luce, Los Alamos National Laboratory Clifford Lynch, Coalition for Networked Info. Kurt Maly, Old Dominion University Michael Nelson, NASA Langley Research Center John Ober, California Digital Library Bob Parks, Washington University & EconWPA Herbert Van de Sompel, University of Ghent Eric F. Van de Velde, Caltech Don Waters, The Andrew W. Mellon Foundation Ken Weiss, California Digital Library

Open Archives Future EconWPA (U. Washington) e-biomed -> PubMed Central (NIH) PubScience (DOE) Clinical Medicine Netprints (+ other HighWire Press holdings ) University ePub (California Digital Library) All public e-prints (MIT) Scholar’s Forum (Caltech) Int’l: CERN, Germany, India, Mexico, … Goal: millions of books/articles/reports / yr

Approaches to Open Archives

Build ByDiscipline

Build By Institution

Approaches to Open Archives

Build ByDiscipline

Build By Institution

AuthorCategoryInterdisciplinaryYearLanguageQuery …

Open Archives initiative (OAi) xxx@LANL, high-energy physics (Ginsparg, 1991) CSTR + WATERS = NCSTRL (Lagoze,1994) xxx + NCSTRL = CoRR collaboration (1998) Universal Preprint Service protoproto, Oct. 21-22, 1999, Santa Fe

– led by LANL, CNI, DLF, Mellon --> OAi Santa Fe Convention (see Feb. D-Lib Magazine article) Follow-on mtgs: 6/3@San Antonio, 9/21@Lisbon (ECDL) Archives -> Open Archives

– Support unique archive identifiers– Implement Open Archives Metadata Set (DC-based, using XML)– Implement Dienst harvesting interface (based on Dienst protocol)– Register the archive

Build tools, layer other services: linking, searching, …

Figure 1. Layers Related to Open Archives Initiative

Services

…

Search/Browse

Authoring Citation Checking Submission

Metadata Creation

Editorial: Reviewing, Certification

Registry

Archives: Name, ID, Description, Terms and Conditions, …

Metadata Formats: Name, XML DTD, …

…

Archive Formats: Name, Standard, Preservation Process, …

Protocols Tools

Services

Copy-Edit / Add Value Citation DB Updating

Authority Control

Preservation Conversion

Text/MM Editing

Gazetteer Cataloging

Collaboration

Annotation

Summarization

Citation / Linking

SFX

CiteSeer

Repository NCSTRL Repository

…

EconWPA Repository

RePEc Repository

Repository for NDLTD Open Archives Harvesting Protocol

Metadata Formats: OA Metadata Set, NDLTD Standard (DC-based) Set

Transaction Log

Training Resources

VT Partition

Record (Metadata)

Record (Full Content)

… …

UVA Partition

Metadata Content

Caltech Partition

Metadata Content

Mechanisms Sharing

– Join federation, run software– Make metadata and archive available

Aggregating– By discipline– By institution– By genre

Automating– Workflow– Harvesting and providing services– Federated searching– Dynamic linking (e.g., with SFX)

Report on Open Archives work in

progress atVirginia Tech

With students:Hussein Suleman (hussein@vt.edu)Dave Watkins (dwatkins@cs.vt.edu)

Robert France (france@vt.edu)Marcos Andre Goncalves (mgoncalv@cs.vt.edu)

VT View of the Open Archives initiative (OAi)

Enable sharing of publication metadata and full-text by digital libraries

Standardize low-level mechanisms to share contents of libraries

Build higher-level user-centric and administrative services in meta-libraries

Install organizational mechanisms to support the technical processes

Virginia Tech Projects

MARC XML-DTD Computer Science Teaching Centre (CSTC) W3C Web Characterization Repository OAi Repository Explorer Networked Digital Library of Theses and

Dissertations (NDLTD)

MARC XML-DTD

XML Transport format for US-MARC records

Standardized metadata exchange format for traditional library services joining OAi

CS Teaching Center (CSTC)

Collection of reviewed online resources used to aid in teaching of Computer Science

Supports author submission and peer-review process for new ACM Journal of Educational Resources In Computing (JERIC)

Connected with NSDL (NSF 00-44)

http://www.cstc.org

W3C Web Characterization Repository

Online database of metadata related to publications, tools and data sets dealing with Web characterization

Project of the Web Characterization Activity working group of the World-Wide-Web Consortium (www.w3c.org/WCA)

http://purl.org/net/repository

OAi Repository Explorer

Serves as a compliancy testAllows browsing of open archives using only OAi

protocolSends requests on behalf of user, parses and

checks responses and displays browsable interfaceWill detect most discrepancies in protocol

http://purl.org/net/explorer

NDLTD

Work has begun on interoperability between Virginia Tech and partners in Germany

Wrappers have been created to harvest data from remote sites which use other protocols

Harvested data to be stored in a central OAi-compliant database (work in progress)

Extending Services - 1 of 2

Working with publishers

– Motivate students: awards, …

– Publicize support of NDLTD ACM, ACS, IEEE-CS, Elsevier, …

– Allow students to increase level of access Arranging preservation

– Mirroring worldwide

– Involving long-term trusted parties

Extending Services - 2 of 2

Adding services currently prototyped– annotation and SDI (routing) capabilities

– Dublic Core metadata, crosswalk to MARC

– support for XML, *ML, preservation

– harvesting, federated search

Adding other services planned– building/using citation DB (CiteSeer, SFX, …)

– implementing plagiarism check (like “SCAM”)

Remember!

Digital libraries introduction

Digital libraries to enhance learning

OAi (help establish enormous international cooperative of data and service providers)