Overview of Digital Libraries: From Requirements to Theory to System to Projects JCDL -- Portland, Oregon, USA Tutorial 4 -- July 14, 2002 Edward A. Fox

Overview of Digital Libraries:From Requirements to Theory to

System to Projects

JCDL -- Portland, Oregon, USATutorial 4 -- July 14, 2002

Edward A. Fox

[email protected] http://fox.cs.vt.edu

CS DLRL Internet TIC

Virginia Tech, Blacksburg, VA, USA

Outline• Virginia Tech context• Why DLs? What are DLs? (5S theory)• Case Study: WCA• Case Study: Education: CSTC -> NSDL• Case Study: NDLTD• Accessibility and Visualization• DL Software: MARIAN• DL Hardware: PetaPlex• Interoperability: OAI, ODL

Acknowledgements (Selected)

• Sponsors: ACM, Adobe, IBM, Microsoft, NSF, OCLC, SOLINET, SURA, US Dept. of Ed. (FIPSE), …

• VT Faculty/Staff: Marc Abrams, Tony Atkins, Thomas Dunbar, Debra Dudley, John Eaton, Gwen Ewing, Peter Haggerty, H. Rex Hartson, Deborah Hix, Gary Hooper, Sunny Kim, JAN Lee, Mann-Ho Lee, Gail McMillan, Len Peters, James Powell, Shalini Urs, …

• VT Students: Emilio Arce, Fernando Das Neves, Brian DeVane, Robert France, Marcos Goncalves, Scott Guyer, Robert Hall, Neill Kipp, Paul Mather, Tim McGonigle, Todd Miller, Constantinos Phanouriou, William Schweiker, Ohm Sornil, Hussein Suleman, Patrick Van Metre, Laura Weiss, Wensi Xi, …

Virginia Tech Background

• Largest university in Virginia, land-grant, football, town population 35K plus 26K students

• Blacksburg Electronic Village, since 1992, with > 80% of community on Internet

• Net.Work.Virginia, with sites for education, research, government• LMDS, Local Multipoint Distribution Service, gigabit wireless

networking - 1/3 of Virginia• Math Emporium, 500 workstations• Faculty Development Initiative, round 3• Torgersen Hall, $30M Advanced Communications and Information

Technology Center, with DLRL

Internet TechnologyInnovation Center

Supported by Virginia’s Center for Innovative Technology

Statewide University Partners - Governing Board:

• Christopher Newport University• William Winter, William Muir, Virginia Electronic Commerce Technology

Center / Southeastern Virginia Network (VECTEC/SEVAnet)• George Mason University

• Steven Ruth, International Center for Applied Studies in IT (ICASIT)• Old Dominion University – Kurt Maly (CS Head), …• University of Virginia

• Alf Weaver, Internet Commerce Group (InterCom)• Jim French, Internet Digital Library

• Virginia Tech• Edward Fox, Digital Library Research Laboratory (DLRL), CC, CS• Scott Midkiff, Center for Wireless Telecomm. (CWT), VTISC, ECpE

ITIC @ VTResearch Areas

• Collaboration (e.g., group decision support)• Community networking (e.g., BEV)• Internet access (e.g., statewide network)• Information services (e.g., digital libraries)• Modeling and simulation (e.g., Web traffic)• Usability (e.g., human factors engineering)• Virtual environments (e.g., CAVE, visualization)

Digital Libraries --- Virginia Tech• MARIAN (NLM, NSF)• CS DL Prototype - ENVISION (NSF, ACM)• TULIP (Elsevier, OCLC)• BEV History Base (NSF, Blacksburg)• DL for CS Education - EI (NSF, ACM)• WATERS, NCSTRL (NSF)• NDLTD (SURA, US Dept. of Education, NSF)• CSTC (NSF, ACM), CRIM (NSF, SIGMM)• WCA (Log) Repository (W3C)• VT-PetaPlex-1 (Knowledge Systems)• NSDL (NSF): CITIDEL, DL-in-a-Box, GetSmart• AmericanSouth.Org (Mellon)


Digital Libraries

SGML (1985)

PDF(1992)

NSF DLI (1994)

LibraryCancellations

(1988)

UniversityScholarlyElectronic

Pub. (1988)

Info.Literacy(1995)

ImprovingEducation Internet

(1984)

WWW(1994)

Multimedia(1986)

SynchronousScholarly Communication

Same time, Same or different place

Asynchronous, Digital Library Mediated Scholarly Communication

Different time and/or place

Borgman et al.:Workshop Report onSocial Aspects ofDigital Libraries: http://www-lis.gseis.ucla.edu/DL/

InformationLifeCycle

Information Life Cycle

AuthoringModifying

OrganizingIndexing

StoringRetrieving

DistributingNetworking

Retention/ Mining

AccessingFiltering

UsingCreating

Computing (flops)Digital content

Com

mun

icat

ions

(ban

dwid

th, c

onne

ctiv

ity)

Locating Digital Libraries in Computing andCommunications Technology Space

Digital Libraries technologytrajectory: intellectualaccess to globally distributed information

less more

D ig ita l L ib ra r y C o n te n t

A rtic le s ,R e p o rts,

B o o ks

T e xtD o cum e n ts

S p ee ch ,M u s ic

V id eoA u d io

(A e ria l)P h o tos

G e og rap h icIn fo rm ation

M o d e lsS im u la tio ns

S o ftw a re ,P ro g ra m s

G e no m eH u m a n,a n im a l,

p la n t

B ioIn fo rm ation

2 D , 3 D ,V R ,C A T

Im ag es a ndG ra p h ics

C o nte n tT yp e s

Integrated Integrated CCLINC CCLINC Translingual Information SystemTranslingual Information System

Integrated Integrated CCLINC CCLINC Translingual Information SystemTranslingual Information System

DARPA

Extraction

What is th

e north korean

movement in th

e front li

ne?

CCLINC SERVER

Info Detection

Summarization

It seems that North Korea launch a missile againAfter North Korea launched a Daipodong missilelast month, NK is perceived to proceed to an additionaltest launch. Korea, US and Japan enter into an alertstate, and prepare for a joint response policy. Korea estimates that the additional launch will be on 09/05. Japan estimates that NK’s missile range is short. USinformation says that there is no sign of launch yet.

Translation

What is th

e status of nk

missile la

unch against japan?

BugHanI IlBonE Ddo MiSaIlEul

BalSaHan Deus HaDa

2-w

a yS

pe e

c h T

ran

s ati

on

Structured Video Browser(making video into hypermedia)

www.learn.umd.edu

• IBrowse

• Expository multimedia• Narrative Structures

ICUInformation and CommunicationUniversity

Users Web Search Engines

WWW

Servlet Engine

Web Server

OSDB

Search Server

Servlet Servlet ServletMPEG-7

DescriptionModule

1

2

3

4

5

3’

4’

5’

MPEG-7 Image Library Systems Tech. M

PE

G-7 Im

age Library System

s

MP

EG

-7 Video Library S

ystems T

ech.

ICUInformation and CommunicationUniversity

MPEG-7 Video Library Systems Tech.

Video Data

Description GeneratorDescription Schemes

Design Tool

DescriptionScheme

MetaDatabase

VideoDatabase

Retrieval ServerModule

PlayerP

resentation

Module

Architecture

How is numeric information created, distributed, consumed?

Basic Example

About• enumerate founders

know numbers, computers, and the Web.

• They saw the Web’s potential to solve a big problem:

Working with numericinformation istoo hard!

How does enumerate help?

RDL• Value - What is the number? • Format - $100 [in thousands]• Semantics - How it translates or corresponds to

other formats• Provenance – created by whom and when? • Measure - scale about the number:

• Units: feet, meters, $, pounds, RBI• Magnitude: thousands, millions, billions• Modifiers: number been manipulated?

• Structure - relationship of numbers to each other

Capture intelligence about data Save data + intelligence in a file Transmit or store the file Build software to decode the file and

automate data tasks RXL does the same thing for analytics

How does enumerate help?

The enumerate formula

enumerate’s interactive data can be analyzed instantly

You Do ItYou Do It Numerator! Publish™

Applications RDL Editor – including Applet / HTML

Tag Wizard DataPublisher™ Enterprise

Views & Applets Standard Chart View Standard Table View Combination View Full Feature View Native XML/RDL Views

AmericanSouth.Org – Roles, ContentSOLINET Libraries (Data

Providers)Scholars

Intellectual Organization Controlled vocabulary Metadata extension

development

Collection Decisions Selection Criteria

Selection Criteria Controlled

vocabulary

Central Server Maintenance Local Server Maintenance Provision of Context

Metadata Repository Metadata Creation/Maintenance

Organizational Structure and

Annotation Tools

Central Interface Design/Maintenance

Local Interface Design/Maintenance

Selection of Other Annotation

Tools

Central Indices Creation/Maintenance

Local Indices Selection of Thesauri

Coordination of Metadata Gateway

Development

Gateway Implementation Concept Mapping

Digital Objects

Content Area Description Audio

Digital

Finding Aid

MSS Other

Photo

Video

MF

Print

Total

African-American cultural life 6 4 6 9 4 12 3 10 18 72

Agricultural crisis of late 19th century

1 1 3 1 1 4 8 19

Codification of segregation laws 1 3 2 1 1 8 16

Configuration of white supremacy 1 3 3 3 1 9 20

Cultural values and activities 3 1 5 17 4 15 1 5 20 71

Disenfranchising movements 1 2 2 1 2 1 6 15

Educational movements 6 1 1 18 6 21 3 5 27 98

Emergence of Holiness & Pentecostal Groups

1 1 1 7 10

Emergence of new musical forms 3 1 1 1 2 8

Emergence of organized groups expressing farmers concerns

2 2 1 8 13

Expansion of Southern evangelical Protestant Churches

3 1 9 3 9 11 23 59


Digital

Finding Aid

MSS Other

Photo

Video

MF Print

Total

Expansion of industrial activity 6 12 5 10 5 14 52

Forms of inter-racialism 1 1 1 2 1 4 10

Great Migration & its relationship to worsened race relations in the South

3 3

Growth of business 1 5 12 1 13 5 15 52

Growth of cities & towns 1 1 5 12 4 13 1 2 18 57

Interplay of economic interest among regions

1 1 4 1 2 1 6 16

Local literature 3 1 2 17 4 7 3 31 68

Lost Cause monument movement 3 2 3 8

Political relationships between Populist & other groups

1 2 2 4 9


Digital

Finding Aid

MSS

Other

Photo

Video

MF Print

Total

Popular magazines & newspapers 2 2 1 13 17 35

Reactions of African-American leaders to Segregation

2 1 2 4 1 2 1 1 10 24

Relationship among Southern Populists & those in the West

1 1

Relationship between new racial system of 1890s and other

2 4 1 8 15

Role of immigration 1 1 2 6 4 2 9 25

Survival of African-American communities & Culture

2 2 1 5 7 1 2 13 33

Women’s Groups 2 1 10 1 5 1 4 9 33

Total Each Format 41 14 51 161 38 133 13 79 301 831

Digital LibrariesShorten the Chain from

Editor

Publisher

A&I

Consolidator

Library

Reviewer

DLs Shorten the Chain to

Author

Reader

Digital

LibraryEditor

Reviewer

Teacher

Learner

Librarian

Digital Libraries --- Objectives

• World Lit.: 24hr / 7day / from desktop• Integrated “super” information systems: 5S:

streams, structures, spaces, scenarios, societies • Ubiquitous, Higher Quality, Lower Cost • Education, Knowledge Sharing, Discovery• Disintermediation -> Collaboration • Universities Reclaim Property• Interactive Courseware, Student Works• Scalable, Sustainable, Usable, Useful

Benefits

• Ease of use

• Effectiveness

• “The benefits of digital libraries will not be appreciated unless they are easy to use effectively.” - IITA Workshop report

DLs: Why of Global Interest?

• National projects can preserve antiquities and heritage: cultural, historical, linguistic, scholarly

• Knowledge and information are essential to economic and technological growth, education

• DL - a domain for international collaboration• wherein all can contribute and benefit• which leverages investment in networking• which provides useful content on Internet & WWW• which will tie nations and peoples together more

strongly and through deeper understanding

Application

Domain

Related Institutions

Examples Technical Challenges Benefit / Impact

PublishingPublishers, Eprint

archivesOAI Quality control, openness Aggregation, organization

Education

Schools, colleges, universities

NSDL, NCSTRL Knowledge management,

reuseabilityAccess to data

Art, Culture

Museum AMICO, PRDLA Digitization, describing,

catalogingGlobal understanding

ScienceGovernment,

Academia, Commerce

NVO, PDG, SwissProt, UK

eScience,European Union Commission

Data modelsreproducibility, faster reuse, faster

advance

(e) Governme

nt

Government Agencies (all levels)

Census Intellectual property rights,

privacy, multi-nationalAccountability, homeland security

(e) Commerce

, (e) Industry

Legal institutionsCourt cases,

patents Developing standards

Standardization, economic development

History, Heritage

Foundations American Memory Content, context,

interpretation

Long term view, perspective, documentation, recording, facilitating, interpretation,

understanding

Cross-cutting

Library, Archive

Web, personal collections

Multi-language, preservation, scalability, interoperability, dynamic

behavior, workflow, sustainability, ontologies,

distributed data, infrastructure

Reduced cost, increased access, pereservation, democratization, leveling, peace, competitiveness

Reagan Moore

Ed Fox

June

2002

for

NSF

Libraries of the FutureJCR Licklider, 1965, MIT Press

World

Nation

State

City

Community

DL Challenges

• Preservation - so people with trust DLs

• Supporting infrastructure - networks, ...

• Scalability, sustainability, interoperability

• DL industry - critical mass by covering libraries, archives, museums, corporate info, govt info, personal info - “quality WWW” integrating IR, HT, MM, ...

• Need tools & methods to make them easier to build

DL Examples

• IBM Digital Library

• Virtua (www.vtlc.com)

• Greenstone (www.greenstone.org)

• Eprints (www.eprints.org)

• Many systems in NSF DLI projects

• VT systems: MARIAN, CSTC, NDLTD• Work on ODL, DL-in-a-box, CITIDEL, NCSTRL

Definitions

• Library ++ (library+archive+museum+…)

• Distributed information system + organization + effective interface

• User community + collection + services

• Digital objects, repositories, IPR management, handles, indexes, federated search, hyperbase, annotation

Definition: Digital Libraries are complex systems that

• help satisfy info needs of users (societies)

• provide info services (scenarios)

• organize info in usable ways (structures)

• present info in usable ways (spaces)

• communicate info with users (streams)

5S Layers

Societies

Scenarios

Spaces

Structures

Streams

Document Models, Representations, and Accesses

• Doc = stream + structure + use-scenario; hybrid (paper/electronic), digital only

• Multilingual: content, summary, metadata• Multimedia: structure, quality (oS), search• Structured: MARC, SGML, by user: MVD• Distributed collection: Kleisli, CIMI, Z39.50• Federated search: collecting, picking site(s),

parallel search / fall-back, fusing results• Access: IPR, payment, security, scenarios

Architectural Issues

• Internet middleware• Independent system / part of federation• Decompositions vary

• search engine, browser, DBMS, MM support• repository, handle server, client• information resources + mediators, bus or agent

collection + client with workspace/environment• Metrics: e.g., for federated search

Standards

• Protocols/federation• Z39.50, CIMI• Dienst, NCSTRL• OAI protocol

• Metadata• TEI: inline, detailed (structure in stream)• MARC: two-level, fine-grained• Dublin Core: high-level, 15 elements• RDF: describing resources/collections, annotation• OAMS -> DC and others used in OAI

Digital Library Courseware

• http://ei.cs.vt.edu/~dlib/

• WWW pages or large PDF copy files

• Online quizzes based on book by Michael Lesk (Morgan Kaufmann Publishers)

• Contents based on book, with several other popular topics added (e.g., agents)

• Separate pages to supplement: Definitions, Resources (People, Projects), and References

Topical Outline - Foundations

• Early visions

• Definitions

• Resources

• References

• Projects

Topical Outline – IR Areas

• Search, Retrieval, Resource Discovery• Information storage and retrieval• Boolean vs. natural language• Search engines• Indexing, phrases, thesauri, concepts• Federated search and harvesting, OAI• Integrating links and ratings• Crawlers, spiders, metasearch, fusion

• Details following – Li Wang indep. study

What is a Crawler?

• A Program• An Important Module For Web Search Engine• Crawls On The Web According To Its Algorithm• Retrieves Web Pages• Gets Useful Information• Stores The Web Pages For Future Refining

Jobs For Threads

Get A New URLFrom Buffer

Contact The ServerFor File Type

Download TheFile

Parse TheWeb Page

Put New URLsInto Buffer

Advanced Functions

• Backward Linkage Information Collector

A Web Page

Topical Outline - Multimedia

• Multiple media types, representations

• Text, audio, image, video, graphics, animation

• Capture, digitization, standards, interchange

• Compression, content-based retrieval

• Playback (Real), SMIL, QoS

• JPEG, MPEG (and versions)

Topical Outline - Architectures

• Distributed, centralized

• Modular, componentized

• Bus (InfoBus), hierarchical, star

• Mediators, wrappers (TSIMMIS)

• Light weight protocols

• Architecture of OAI and XOAI

Topical Outline – Interfaces

• Taxonomy of interface components

• Workflow

• Visualization

• Environments

• Design

• Usability testing

Topical Outline – Metadata

• MARC

• Dublin Core

• RDF

• IMS

• OAI (Open Archives Initiative)

• Crosswalks, mappings

• Ontologies

• Topics maps, concept maps

Topical Outline – Epub, SGML, XML

• Authoring

• Rendering, presenting

• Structure

• Tagging, Markup, DOM

• Semi-structured information

• Dual-publishing, eBooks

• Styles (XSL, XSLT)

• Structure queries

Topical Outline – Databases

• Extending database technology

• Structured and unstructured info

• Multimedia databases

• Link databases

• Performance

• Replicated storage, I2-DSI (details following)

OAI and I2-DSI (Ryan Richardson)

•OAI – metadata harvesting •OAI-PMH•Data providers/Service providers

•I2-DSI – mirroring and replication•Can we put them together to get benefits of both?•Is this the first of many higher level mirroring schemes for Internet2, that provide independence from lower level representation issues?

I2-DSI Architectural Diagram

Mirror Mirror Mirror Mirror

User

Internet

OAI Server

Distributed Director

I2-DSI Technical Issues

• Resumption tokens• Mirrors are stateless, so any mirror can answer

(any part of) a request

• Every client request to the mirror is logged, to enable log comparison among the mirror archives

• Mirroring time is dependent on # of records sent in each chunk

I2-DSI interface

Topical Outline – Agents

• Protocols

• Knowledge interchange

• Negotiation, registries

• Distributed issues

• Ontologies (standard upper)

• Webbots (automatic indexing)

Topical Outline – Economics

• E-commerce

• Sustainability

• Preservation and archiving• DLF, Besser, Lorie, Gladney

• Self-archiving

• Open collections

• Economic models, business plans

Topical Outline – IPR

• Intellectual property rights (IPR)

• Legal issues

• Terms and conditions

• Copyright

• Patents, trademarks

• Distributed rights management

• Security

Topical Outline – Social Issues

• Cooperation, collaboration• Annotation, ratings• Digital divide• Educational applications• Cultural heritage• Museums (AMICO)• Organizational acceptance• Personalization• Internationalization


W3C Web Characterization Repository

• Online database of metadata related to publications, tools and data sets dealing with Web characterization

• Project of the Web Characterization Activity working group of the World-Wide-Web Consortium (www.w3c.org/WCA)

• http://purl.org/net/repository


CS -> CSTC -> CRIM• NSF and ACM Education Committee are funding

a 2 year project “A Computer Science Teaching Center” - CSTC - http://www.cstc.org/

• College of NJ, U. Ill. Springfield, Virginia Tech

• Focus initially on labs, visualization, multimedia

• Multimedia part is also supported by a 2nd grant to Virginia Tech and The George Washington University: http://www.cstc.org/~crim/ (with curricular guidelines also under development)

CS Teaching Center (CSTC)

• Instead of building large, expensive multimedia packages, that become obsolete and are difficult to re-use, concentrate on small knowledge units.

• Learners benefit from having well-crafted modules that have been reviewed and tested.

• Use digital libraries to build a powerful base of support for learners, upon which a variety of courses, self-study tutorials & reference resources can be built.

• ACM support led to Journal of Educational Resources in Computing (JERIC), accessible from www.cstc.org

Browsing (1)

Browsing (2)

SMETE Library -> NSDL(from www.dlib.org to NSF DLI-2)

• Context: Global movement toward Digital Libraries (see April 1998 CACM)

• NSF 00-44 effort: Science, Mathematics, Engineering, and Technology Education Digital Library (focussed on undergraduates)• 3 workshops, yearly increasing funds / new calls

• NSDL will operate as a distributed federation, with separate parts for each key discipline, and should lead to a global effort.

Selected NSDL Early Projects/Topics

COLLEGIS Res. Inst. IMS, CS, Math, Viz., …

Columbia University Earth sciences

Stanford University Medicine (images)

U. California Berkeley Engineering

University of Maryland K-12 education

U. Texas at Austin Physical anthropology

www.CITIDEL.org• Computing and Information Technology Interactive

Digital Education Library• Builds on CSTC, JERIC, ACM DL, IEEE DL, ResearchIndex. NCSTRL, …

• Led by Virginia Tech, with co-PIs:• Fox (director, DL systems)• Lee (history)• Perez (user interface, Spanish support)

• Partners• College of New Jersey (Knox)• Hofstra (Impagliazzo)• Villanova (Cassel)• Penn State (Giles)

DIGITAL LIBRARY SERVICES

REPOSITORIES

USER PORTALS

Overview of CITIDEL architecture

Union Metadata Repository

OAI Data

Provider

Laboratories Repository

Applets Repository

Papers Repository

Syllabi Repository

. . .

Digital Library Services

OAI Data

Harvester

Distributed repository structure

Annotations

OAI Data

Harvester

EDUCATORS

ADMINISTRATORS LEARNERS

Multilingual Searching

Revising Annotating Filtering Browsing Administering

Filtering Profiles User Profiles

Union Metadata

OAI Data

Provider

Remote and Peer Digital Libraries (eg. NSDL -CIS)

PORTALS

SERVICES

REPOSITORIES

Digital library architecture for localand interoperable CITIDEL services

CITIDEL -> NSDL

• Computing and Information Technology Interactive Digital Education Library

• A collection project in the National STEM (science, technolgy, engineering, and mathematics) education Digital Library – NSDL (www.nsdl.nsf.gov) -> LEARNS

NSDL Information ArchitectureEssentially as developed by the Technical Infrastructure Workgroup

referenceditems &

collections

referenceditems &

collections

Special Databases

NSDLServicesNSDL

ServicesOther NSDLServices

CI Services

annotation

CI Services

discussion

CI Services

personalization

CI Services

authentication

CI Services

browsing

Core Services:information retrieval

Core Collection-Building Services

harvesting

Core Collection-Building Services

protocols

Core Services:metadata gathering

Portals &ClientsPortals &

ClientsPortals &Clients

Usage Enhancement

Collection Building

User Interfaces

NSDLCollections

NSDLCollections

NSDLCollections

CoreNSDL“Bus”

“The network is the library.”

A Learning Environments and ResourcesNetwork for SMET Education (LEARNS)

LEARNS Connects:

Users: students, educators, life-long learners

Content: structured learning materials; large real-time or archived datasets; audio, images, animations; primary sources; digital learning objects (e.g. applets); interactive (virtual, remote) laboratories; ...

Tools: search; refer; validate; integrate; create; customize; publish; share; notify; collaborate; ...

LEARNS Supports:

Users

Content

Tools

(profiles)

(metadata)

(protocols)

Learning communities

Customizable collections

Application services

LEARNS Enables:Environments for

• Communication

• Collaboration

• Creation

• Validation

• Evaluation

• Recognition

• ...

• Discovery

• Stability

• Reliability

• Reusability

• Interoperability

• Customizability

• ...

of Resources

AND

GoalCore Integration Track

(FY00 pilots, FY01 full)Collections TrackServices TrackTargeted Research Track

LEARNS

operational

by 2002

Expectations of NSDL ProgramTracks

• Core Integration: coordinate a distributed alliance of resource collection and service providers; and ensure reliable and extensible access to and usability of the resulting network of learning environments and resources

• Collections: aggregate and actively manage a subset of the digital library’s content within a coherent theme / specialty

• Services: increase the impact, reach, efficiency, and value of the digital library in its fully operational form

• Targeted (Applied) Research: have immediate impact on one or more of the other three tracks

Collections

• Discovery of content

• Classification and cataloguing• Acquisition and/or linking; referencing• Disciplinary-based themes define a natural body of content,

but other possibilities are also encouraged

• Access to massive real-time or archived datasets

• Software tool suites for analysis, modeling, simulation, or visualization

• Reviewed commentary on learning materials and pedagogy

Services• Help services, frequently asked questions, etc.

• Synchronous/asynchronous collaborative learning environments using shared resources

• Mechanisms for building personal annotated digital information spaces

• Reliability testing for applets or other digital learning objects

• Audio, image, and video search capability

• Metadata system translation

• Community feedback mechanisms


A Digital Library Case Study

• Domain: graduate education, research

• Genre:ETDs=electronic theses & dissertations

• Submission: http://etd.vt.edu

• Collection: http://www.theses.org

Project: Networked Digital

Library of Theses & Dissertations

(NDLTD) http://www.ndltd.org

The Networked Digital Library of Theses and Dissertations

www.NDLTD.org

Leader of the Worldwide ETD(Electronic Thesis and Dissertation) Initiative

Training AuthorsExpanding Access

Preserving KnowledgeImproving Graduate Education

Enhancing Scholarly CommunicationEmpowering Students & Universities

GradProgram

IT Ed.(Tech)Library

NDLTD

Media

ETD Web Sitehttp://www.ndltd.org/

ETDs Got Your Interest?

Graduate Students

Singapore AMChronicle of Higher Ed.National Public RadioNY Times ...

U. Laval

Key Ideas: Networked infrastructure

Scalability

Education is the rationale

University collaboration

Workflow, automation

Authors must submitMaximalAccess

PDF, SGML, MM,MARC, DC, URNs,Federated search

Standards

8th graders vs. grads

What led to today’s meeting?• 1987 mtg in Ann Arbor: UMI, VT, …• 1992 mtg in Washington: CNI, CGS, UMI, VT and 10

universities with 3 reps each• 1993 mtg in Atlanta to start Monticello Electronic Library

(regional, US Southeast): SURA, SOLINET• 1994 mtg at VT: std: PDF + SGML + multimedia objects• 1996 funding by SURA, US Dept. of Education (FIPSE)• 1997 meetings in UK, Germany, ...• 1998 – 1st symposium – Memphis (20)• 1999 – 2nd symposium – Blacksburg (70)• 2000 – 3rd symposium – St. Petersburg (225)• 2001 – 4th symposium – Caltech (200)• 2002 – 5th symposium–BYU; 2003–Berlin; 2004–Kentucky

What are the long term goals?

• 400K US students / year getting grad degrees are exposed / involved

• 200K/yr rich hypermedia ETDs that may turn into electronic portfolios (images, video, audio, …)

• Dramatic increase in knowledge sharing: literature reviews, bibliographies, …

• Services providing lifelong access for students: browse, search, prior searches, citation links

• Hundreds/thousands of downloads / year / work

ETDs: Library Goals

• Improve library services• Better turn-around time • Always available

• Reduce work • catalog from e-text • eliminate handling: mailing to UMI, bindery

prep, check-out, check-in, reshelving, etc.• Save space

• Aiding universities to enhance graduate education, publishing and IPR efforts

• Helping improve the availability and content of theses and dissertations

• Educating ALL future scholars so they can publish electronically and effectively use digital libraries (i.e., are Information Literate and can be more expressive)

What are we doing?

NDLTD

Computer Resources

Research

Literature

Student Prepares Thesis/Dissertation

Student Defends & Finalizes ETD

My Thesis

ETD

Student Gets CommitteeSignatures and Submits ETD

Signed

Grad School

Graduate School Approves ETD, Student is Graduated

Ph.D.

Library Catalogs ETD, Access isOpened to the New Research

WWW

NDLTD

Status of the Local Project

• Approved by university governance Spring 1996; required starting 1/1/97

• Submission & access software in place

• Submission workshops for students (and faculty) occur often: beginner/adv.

• Faculty training as part of Faculty Development Initiative

• Over 4000 ETDs in collection – some have audio, video, large images, software, …

Archiving ETDs

• Every 15 minutes back-ups made of not-yet-approved submissions

• Hourly back-ups of newly approved ETDs

• Weekly back-ups of entire ETD collection

• Copies stored on-site and off-site

VT ETD Cataloging

• same as current cataloging policies, except:• author-assigned keywords (not LCSH)• generic (not LC) call no.• fields/subfields as required for computer files• full abstracts

• time savings• cataloger familiar with computer files• equipment, software for word processing• 5 minutes avg. (10-15 minutes for paper TDs)

Library Costs

• $12/vol. for paper thesis processing• catalog, bind, security strip, label, shelve• @950 vols./yr. = $11,466

• $3.20/vol. ETD processing• cataloging @950 vols./yr. = $3040

• $.07/vol. shelving• $.04/vol. circulation

Costs/Savings at VT

• Graduate School stopped shipping to the library 3000 copies of paper TDs/year

• Library stopped binding, shelving, and circulating 3000 copies of TDs/year

• 166 ft of shelf space saved/year by the library

• VT used existing equipment in Library (vs. start-up costs for staff, hardware and software from from a zero-base estimate: $65,000 – see http://scholar.lib.vt.edu/theses/)

Popular Works 1996458 Seevers, Gary L. Identification of Criteria for Delivery of Theological Education Through Distance Education: An International Delphi Study (Ph.D., Educational Research and Evaluation, April 1993; 1353Kb)

432 Hohauser, Robyn Lisa. The Social Construction of Technology: The Case of LSD (MS in Science and Technology Studies, Feb. 1995; 244Kb)

390 Childress, Vincent William. The Effects of Technology Education, Science, and Mathematics Integration Upon Eighth Grader's Technological Problem-Solving Ability (Ph.D. in Vocational and Technical Education, July 1994; 285Kb)

310 Kuhn, William B. Design of Integrated, Low Power, Radio Receivers in BiCMOS Technologies (Ph.D. in Electrical Engineering, Dec. 1995; 2Mb)

287 Sprague, Milo D. A High Performance DSP Based System Architecture for Motor Drive Control ( MS in Electrical Engineering, May 1993; 878Kb)

165 Wallace, Richard A. Regional Differences in the Treatment of Karl Marx by the Founders of American Academic Sociology (MS in Sociology, Nov. 1993; 479Kb)

150 McKeel, Scott Andrew. Numerical Simulation of the Transition Region in Hypersonic Flow (Ph.D. in Aerospace Engineering, Feb. 1996; 3Mb)

Popular Works 19979920 Liu, Xiangdong. Analysis and Reduction of Moire Patterns in Scanned Halftone Pictures (Ph.D. in Computer Science, May 1996; 6.6Mb)

7656 Petrus, Paul. Novel Adaptive Array Algorithms and Their Impact on Cellular System Capacity (Ph.D. in Electrical Engineering, March 1997; 5Mb)

2781 Agnes, Gregory Stephen. Performance of Nonlinear Mechanical, Resonant-Shunted Piezoelectric, and Electronic Vibration Absorbers for Multi-Degree-of-Freedom Structures (Ph.D. in Engineering Mechanics, Sept. 1997; ? + 7926Kb)

2492 Gonzalez, Reinaldo J. Raman, Infrared, X-ray, and EELS Studies of Nanophase Titania (Ph.D. in Physics, July 1996; 4607Kb)

1877 Shih, Po-Jen. On-Line Consolidation of Thermoplastic Composites (Ph.D. in Engineering Mechanics, Feb. 1997; 3.3Mb)

1791 Saldanha, Kevin J. Performance Evaluation of DECT in Different Radio Environments (MS in Electrical Engineering, Aug. 1996; 3.2Mb)

1431 DeVaux, David. A Tutorial on Authorware (MS in CS, April 1996; 2.3Mb)

1394 Kuhn, William B. Design of Integrated, Low Power, Radio Receivers in BiCMOS Technologies (Ph.D. in Electrical Engineering, Dec. 1995; 2518Kb)

National / Regional Projects• Australia

• U. New South Wales (lead)• U. of Melbourne• U. of Queensland• U. of Sydney• Australian National U.• Curtin U. of Technology• Griffith U.

• Germany• Humboldt University (lead)

• 3 other universities

• 5 learned societies: Math, Physics, Chemistry, Sociology, Education

• 1 computing center

• 2 major libraries

• OhioLINK: 79 colleges/univs• Consorci de Biblioteques

Universitàries de Catalunya, as group, www.cbuc.es: 9 sites

• India• Brazil• UK (British Library, JISC,

Edinburgh)• UNESCO (especially Latin

America, Eastern Europe, Africa)

OhioLINK

• Statewide Consortium

• Represents 79 colleges, universities, libraries

• Public Universities

• Private Universities and Colleges

• 2-Year Colleges

• Only a few (e.g., Miami U. of Ohio) are also NDLTD members on their own

US University Members (64)• Air University (Alabama)• Baylor University• Boston College• Brigham Young University (part, whole)• Caltech• Clemson University• College of William & Mary• Concordia University (Illinois)• Drexel University• East Carolina University• East Tenn. State U. – required fall 2000• Florida Institute of Technology• Florida International University• Florida State University• George Washington University• Georgetown University• Louisiana State University• Marshall University (W. Va.)• Miami University of Ohio• Michigan Tech• Mississippi State University• MIT• Montana State University• Naval Postgraduate School (CA)• New Jersey Inst. of Technology• New Mexico Tech• North Carolina State University – required 9/2002• Northwestern University• Penn. State University• Regis University• Rochester Institute of Tech.• Texas A&M

• U. of Central Florida• U. of Colorado Health Science Center• U. of Florida• U. of Georgia• U. of Hawaii, Manoa • U. of Illinois, Urbana-Champaign• U. of Iowa• U. of Kentucky – required in CS only• U. of Maine – required in CS, Spatial Info Sci/Eng• U. of Missouri-Columbia• U. of North Texas – required since 8/99• U. of Nevada, Las Vegas• U. of New Orleans• U. of North Texas• U. of Oklahoma• U. of Pittsburgh• U. of Rochester• U. of South Florida• U. of Tennessee, Knoxville• U. of Tennessee, Memphis• U. of Texas at Austin – required in 2001• U. of Virginia• U. of West Florida• U. of Wisconsin - Madison • Vanderbilt U.• Virginia Commonwealth U.• Virginia Tech - required since 1/97• Wake Forest U.• West Virginia U. - required fall 1998• Western Michigan U.• Worcester Polytechnic Inst.

Other Countries - 60 Members

• Australia• Belgium• Brazil• Canada• China, Hong Kong• Columbia• France (Université Lyon2)• Germany• India (Hyderabad)• Italy• Korea• Mexico

• Netherland• Norway• Russia• Singapore• S. Africa (Rhodes U.)• S. Korea• Spain• Sudan• Sweden• Taiwan• UK

Institutional Members (17)• British Library• Cinemedia• Coalition for Networked Information (CNI)• Committee on Institutional Cooperation (CIC)• Consorci de Biblioteques Universitàries de Catalunya• Diplomica.com• Dissertation.com• Dissertationen Online (Germany)• ETDweb, a Division of Answer4.com• Ibero-American Science & Technology Education Consortium (ISTEC)• National Documentation Centre (NDC), Greece• National Library of Portugal (for all universities)• OCLC Online Computer Library Center• OhioLINK• Organization of American States (SEDI/OAS)• Southeastern Library Network (SOLINET)• UNESCO (www.unesco.org/webworld/etd)

Type 1 Members University Requires ETDs

• Adobe Acrobat and/or XML/SGML tools• Automated submission & processing• Archive/access through UMI, (OCLC,)

Virginia Tech, ...• (Local) WWW site, publicity• (Local) Assistance provided as requested:

email, phone, listserv(s)

Type 2 Members University Agrees to Require ETDs

• Like Type 1 but set date not reached• Usually has an option or pilot• May: wait for new AY; start with all who enter

after; …• Build grass roots support

• Advisory committee: representative? expert?• Champions to spread by word of mouth• Approval: Senates, Commissions, Deans, Students• Publicity to reach community

NDLTD Members, Types 3-7

• 3. Part of university requires ETDs• 4. University allows ETDs• 5. University investigating, has pilot• 6. University consortium joins:

• CIC (Big 10 coordinating body)

• 7. Non-university organization joins• CNI (Coalition for Networked Info.)

Counts of ETDs at Selected U’s (2001)University/Institution ETD Collection size

ADT: Australian Digital Thesis Program (Australia) 238

University of Bergen (Norway) 45

California Institute of Technology 2

Consorci de Biblioteques Universitaries de Catalunya (Spain) 151

East Tennessee State University 106

Humboldt-University (Germany) 430

Louisiana State University 3

Mississippi State University 33

MIT 62

North Carolina State University 301

Pennsylvania State University 83

Pontifical Catholic University (PUC) (Brazil) 90

Gerhard Mercator Universitat Duisburg (Germany) 126

Universitat Politecnica de Valencia (Spain) 189

University of Florida 174

(continued)

Counts of ETDs at Selected U’s (cont’d)

University/Institution ETD Collection size

University of Georgia 121

University of Iowa 6

University of Kentucky 19

University of Maine 27

University of North Texas 337

University of South Florida 25

University of Tennessee 12

University of Tennessee, Knoxville 28

Uppsala University (Sweden) 178

Virginia Tech 3393

West Virginia University 1006

Worcester Polytechnic Institute 83

TOTAL 7268

Counts of UniversityScanned ETD Collections

University/Institution ETD Collection Size

MIT 5,581

National Documentation Center, Greece 12,000

New Jersey Institute of Technology 26

University of South Florida 150

TOTAL 17,763

VT ETD Access Logs

1997/98 1998/99

Increase1997/98-1998/99 1999/00

Increase1998/99-1999/00

Requests for PDF files(mostly full ETDs) 221,679 481,038 117.0% 578,152 20.2%

Requests for HTML files(mostly tables of contents and abstracts) 165,710 215,539 30.1% 260,699 21.0%

Requests for multimedia 1,714 4,468 160.7% 12,633 182.7%

Distinct files requested 6,419 21,451 234.2% 16,409 -23.5%

Distinct hosts served 29,816 57,901 94.2% 87,804 51.6%

Average data transferred daily 156 MB 219 MB 40.4% 382 MB 74.4%

Data transferred 55 GB 78 GB 40.4% 137 GB 75.6%

VT ETD Access by Int’l Sites

International Domain

1997/981997/98

rank1998/99

1998/99rank

Increase1997/98-1998/99

1999/001999/00

rank

Increase1998/99-1999/00

UnitedKingdom 6,735 1 11,347 1 68.5% 25,583 1 125.5%

Malaysia 876 16 4,190 6 378.3% 16,147 2 285.4%

France 2,138 7 4,797 5 124.4% 14,960 3 211.9%

Germany 6,727 2 3,374 9 -49.8% 14,384 4 326.3%

Canada 3,413 4 9,632 3 182.2% 13,543 5 40.6%

Spain 590 18 3,647 8 518.1% 9,918 6 171.9%

Italy 1,430 12 3,095 10 116.4% 9,300 7 200.5%

Multimedia Use in ETD Collection

File type Examples Count

Still image BMP, DXF, GIF, JPG, TIFF 328

Video AVI, MOV, MPG, QT 58

Audio AIFF, WAV 18

Text PDF, HTML, TXT, DOC, XLS 7601

Other Macromedia, SGML, XML 51

Access Choices at VT (7/2000)

Worldwide54%

VT-Only24%

Patent3%

Mixed19%

Who are sponsors / cooperators?• Funding, Donations of hardware/software

• SURA• US Dept. of Education (FIPSE)• Adobe Systems• IBM• Microsoft• OCLC

• Others Serving on Steering Committee• National/Regional Projects: Australia, French

speaking group, Germany, IberoAmerica (ISTEC), UK (UTOG)

• CGS, National Lib. Canada, NSF, OAS, SOLINET, UMI, UNESCO, ...

For professional societies

• Like “writing across the curriculum”, e.g., Chemical Markup Language, MathML, …

• Besides writing: computing/communications, information literacy, personal digital library management, tool use, research methods, collaboration, archiving/preservation

• Data sets, communities of users of them

• Classification systems / browsing / searching

• NRC’s “Issues for Science and Engineering Researchers in the Digital Age”, 57 pages

Relationship with publishers

• Concern of faculty and students that still wish to publish books or journal articles, voiced: campus, Chronicle, NPR, Times

• Solution: Approval Form gives students, faculty choices on access, when to change access condition; use IPR controls in DL

• Solution: by case, work with publishers and publisher associations to increase access• AAP, AAUP• AAAS, ACM, ACS, Elsevier, ...

Some responses from publishers

• ACM: need to acknowledge copyright• Elsevier: need to acknowledge copyright• IEEE-CS: endorse initiative• ACS: After first publication, can release• Textbook publishers: different market,

manuscript significantly reworked• General: restricting access to local campus

will not cause any problems

How does this relate to UMI?

• Generally, they are independent decisions.• 1987 UMI workshop was first to explore ETDs.• UMI wrote support letter for US Dept. of Ed. proposal. • UMI is on Steering Committee.• ProQuest Direct pilot of scanning works started 1/1/97,

with free 2 yr access to front part.• We are collaborating on:

• accepting electronic author submissions

• standards (e.g., representation)

ETD Initiative (and UMI)

StudentsLearn aboutDL, EPub

TDsbecome more

expressive

N. Amer. (T)Ds areaccessible, archived

Global TDsbecome more

accessible,archived

UMI

Universities

User Search Support(multilingual, XML)

NDLTD W orld FederatedSearch

Virg in ia Tech ...(univ)

D issertationsO nline

(G erm any)

O hioLink(lib / un iv group)

Portugese N L ...(national lib)

Austra lia(regional)

O AS,ISTEC(Latin

Am erica)

UserInterface

Note: All groups shown are connected with NDLTD.

www.theses.org

• James Powell student project, D-Lib Magazine description in Sept. 1998

• XML description of each site• type of search engine / service

• language

• coverage (for resource discovery)

• Adding Z39.50 gateway capability and integrating with MARIAN, along with Harvest and Open Archives protocols

Access Approaches

• Goal: Maximize access and services, e.g., by encouraging:

• UMI centralized services• VTLS: free union collection of ETD vmetadata• OCLC: free union collection of TD metadata• Distributed service: Dienst, Z39.50• Regional services (e.g., OhioLinkh)• Local servers with browse, search

• From local catalogs to local archives• WWW robot indexing and search services

Access Possibilities

Websearchengines

librarycatalogclients

www.theses.org

www.openarchives.org

3rd

PartyServices(e.g.,UMI)

VirginiaTech

NationalLibrary ofPortugal

CBUC(Spain)

OhioLink

MIT NationalProjects:AU, GE, …

Why might a university want to be involved?

• To improve graduate education / better prepare your students / increase their knowledge (epub, DLs, IPR) and visibility

• To enhance university infrastructure (DL)• To unlock university information• To save money for students and for the

university / improve workflow• To build an important digital library (of ETDs)

NDLTD Members and ETD-MS

• NDLTD members will• Share metadata for their ETDs

• Providing that in either ETD-MS

• Or if they use a version of MARC locally, work to have that eventually shared in either MARC21 or UNIMARC

• Run OAI, either locally or in consortia, so their metadata can be harvested, according to necessary terms and conditions

Complex to Simple

MARC ($50) Dublin Core (DC)

+thesis

ETD-MS

• ETD Metadata Standard• XML-encoded metadata standard

(content and encoding) for Electronic Theses and Dissertations (ETDs)

• in part conforming to Dublin Core (DC)

• using RDF

• using UNICODE

• Will specify relationship with MARC

ETD-MS Schema Includes

• Elements not in dces (Dublin Core Element Set)• e.g., thesis.degree

• Elements with wildly divergent semantics• e.g., thesis.advisor rather than dc.contributor

• Relationships to other elements

• Controlled vocabularies• e.g., {Bachelors, Masters, Doctorate, Other} for

thesis.degree.level

• Labels in multiple languages

ETD Encoding Decisions

• Text• UNICODE (with language identifiers)

• Structure• MARC (MARC-21 or UNIMARC)

PLUS• XML / RDF / DC + ETD

• Multimedia• Following international standards• Other schemes may not be amenable to preservation

RDF for ETDs

• WWW Consortium (W3C)’s RDF: Resource Description Framework

• NUDL ETD metadata realized as an RDF application profile• Specifying elements from DC element set

• Plus new elements from a registered ETD schema

• Constraints & policies attached to both• (e.g., “Full title,” “Name as it appears on title page,”

“Repeatable”)

• Links to authority records encoded as URIs• XML syntax as per RDF standard

OCLC and ETD-MS

• Identify TDs in WorldCat (4.3M)

• Through OAI make available metadata for WorldCat TDs in both DC and ETD-MS

• Provide an authority service for personal names for NDLTD

• Coordinate with other authority services such as LC

VTLS and ETD-MS

• Support NDLTD through a union catalog service implemented with Virtua

• Accept metadata in MARC21 or UNIMARC, and help identify other converters for other types

• Accept metadata in one other format, namely ETD-MS, collected using OAI (harvesting)

• Accept data in various character sets, with UNICODE preferred, but in some cases the submitter may be required to convert

Multiple objectives

• Sharing research results

• Decrease costs, increase services

• Increase knowledge of users

• Adding to author knowledge/skills

• Epub, DL, IPR

• Enhancing organization’s infrastructure

• CS department, library

• University, Laboratory

How can a university get involved?

• Select planning/implementation team• Graduate School

• Library

• Computing / Information Technology

• Institutional Research / Educ. Tech.

• Send us letter, give us contact names• www.ndltd.org/join

• Adapt Virginia Tech solution• Build interest and consensus

• Start trial / allow optional submission

Contact Our Project Team

[email protected]

Phone CallVisit

Video Tape

Convene Local Planning Group

ETD

Build Local ETD Site

Digital Library

Policies

Inspection/Approval

Workshop/Training

ETD

ETD

Support Offered

• Software, documentation, tech support• Email, listservs ([email protected])• UNESCO training, Guide (www.etdguide.org)• NDLTD Committees

• Steering• Standards• Conference• Strategic Planning

NUDL (www.nudl.org)Int’l Research Support

• Networked University Digital Library• Partners: Germany, Mexico (Puebla and

Monterrey), Brazil• Problems: Multilingual search, high

performance DLs, requirements/usability, …

• Start with ETDs, then expand to other student works, portfolios, data sets, (CS) courseware, ...

Future Work - 1 of 2

• Working with publishers to increase level of access as much as possible

• Interoperability tests among universities and with UMI to provide integrated services

• Study with testbed that emerges, to improve information retrieval, browsing, interface, and other types of user support

• Evaluation, improving learning experience, spread to worldwide initiative, sustainable support and coordination

Future Work - 2 of 2

• Adding services currently prototyped• annotation and SDI (routing) capabilities• Dublin Core metadata, crosswalk to MARC

• Adding other services planned• building and using citation database (w. SFX)• implementing plagiarism check (like “SCAM”)

• Developing NUDL as a sustainable self governing global institution (w. committees)

Why ETD?Short Answer

• For Students:• Gain knowledge and skills for the Information Age

• Richer communication (digital information, multimedia, …)

• For Universities: • Easy way to enter the digital library field and benefit thereby

• For the World: • Global digital library – large, useful, many services

• General:• Save time and money

• Increased visibility for all associated with research results

The Process?Short Answer

• For Students:

• Plan on ETD from day 1

• Secure knowledge from: workshops, online info, colleagues

• Work with faculty to plan approach

• PDF? XML? TEI? Multi/hypermedia? Data sets? Viz?

• Get signed approval form: access, ©, proxy assignment

• After defense and approval, submit ETD to university

• For Universities:

• Form team

• Adapt solution from work at other universities, attend ETD conference

• Pilot -> Option -> Requirement


Accessibility Activities / Plans

• Interface design (simple, 3D, VR)• Usability studies• Generic multi-lingual support• Support for those with disabilities• Hybrid collection (paper, MARC,

abstracts, full-text, multimedia)• Disciplinary classifications, tools• Visualization of results, collection

CAVE Experiments

• Use a familiar metaphor• building / floor / room / shelf / book

• Rearrange orderings / shelving• use categories, clustering, ranking • use visualization: colors and gaps• study space mappings: physical, logical

• Simplify movement for key tasks

CAVE-ETD

• CAVE-ETD is a simulation of a library that runs in a CAVE (VR environment).

• Populated with a subset of ETD records.

Main Foyerroom

room

room

room

Book Browsing

Reading Book Abstract

ENVISION

• NSF “A User-Centered Database from the Computer Science Literature” (1991-93)

• Collected bib/typesetter data, converted to SGML• Scanned thousands of page images• MARIAN search engine - can be made available (also

applied to the Virginia Tech library catalog) used as part of a prototype object-based DL, with tailored visualization interface (L. Nowell dissertation)

Envision Results Window

Envision – New Version

Envision – New Versions - Clusters

SPIRE Visualization

VIDI:A Lightweight Protocol Between

Visualization Systems and Digital Libraries

Jun Wang

Virginia Tech CS MS Thesis

Spring 2002

Problem Concerned

• Scenario

DL1

DL2

DL3

VIS1

VIS2

VIDI Protocol Design Features

• Enabling interoperability• Lightweight• Extended OAI Protocol• Flexible implementations enabled• General• XML, HTTP• Standard time formats• Dual usage of commands• Simple and Easy!

VIDI Protocol Request Verbs

• Identify (DL, VIS)

• ListMetadataFormats (DL)

• ListVisdataFormats (VIS)

• ListTransformers (VIS)

• RequestResultSet (DL)

Extend OAI Protocol

OAI OAI & VIDI VIDI•GetRecord

•ListIdentifiers

•ListRecords

•ListSets

•Identify

•ListMetadataFormats

•ListVisdataFormats

•ListTransformers

•RequestResultSet

Implementation Roles and Times

• Implementing protocol

• Devising general approaches to protocol use for DL-VIS environments

• Applying protocol in representative cases• ENVISION-ODL• ENVISION-MARIAN

Implementation Process

1. Analyze metadata format in DL

2. Analyze visdata format in VIS

3. Write transformer (if not in registry)

4. Decide on command flow

5. Implement protocol commands

Command Flow Used In Prototype

<back

ENVISION-ODL (II)

Connect ENVISION with:

• ODL

A DL implementing OAI protocol, which means we can issue OAI requests and receive responses to retrieve the data

ENVISION-MARIAN

Connect ENVISION with:

• MARIAN• A DL having multiple collections (NDLTD,

DIRLINE, CITIDEL, VT catalog,…)• User Authentication

Future Work

• SOM decoupling

DL VIS

Transformer


MARIAN

• Multiple Access Retrieval of Information with Annotations

• (Marian the Librarian …)• Evolved from CODER system to a distributed

Online Public Access Catalog (OPAC), then DL backend, now becoming a full DL system

• From C/C++ to Java• Future: NDLTD, NUDL, PetaPlex• Use for campus collection management• Use for www.theses.org as centralized system with

gateway services: OAi, Harvest, Z39.50, …

MARIAN

• Digital Library Search & Retrieval System• Principles

• Network representation• Class-based retrieval• Weight-valued functions and weighted sets

• Interoperability• System: wrappers and harvesting• Syntax: OAI standards (XML, Unicode, …)• Structure: information networks• Semantics: class-based retrieval

: collection views

System & Syntactic Interoperability

GermanPhysDis

Collection

5SL Source

Description

wrapper wrapper

Harvestprotocol

VT OAI

Collection

MARIAN Mediation Middleware

MIT ETDCollection...

Open Archives

protocol

wrapper...Dienst

protocol

SOIF

DublinCore RFC1807

NDLTD/NUDL/Digital Library User

Queries + Results

GreekHellenic Dissertations

Collection

wrapper

MARCZ39.50

protocol

WrapperGenerator

Local Data Store

Search ServicesRecommendation Services, etc

AnalysisIndexingLinking

MARIAN – Part of Class Hierarchy

Structural Interoperability through Information Networks

Composite Subject

Subject Entry

Corporation

Conference

PhysDis-ETD

Person

VT-ETD

Title

Abstract

VT-ETD

Title

Notes

MIT-ETD

Title

Subject

Person

Keyword Set

(SOIF Format)

(MARC-21 Format) (ETD-MS Format)

(RFC1807 Format)

Individual

Subject

DC.Title

CrawlerTitle

DC.Description

CrawlerDescrip tion

Body

Headings

DC.Subject

DC.Creator

Keyword Set

Individual

Headings

DC.Subject

DC.Creator PhysDis-ETD

DC.T i tle

Crawler Ti tle

ThesisDissertation

Tit le

Descrip tion

Subject

Individual

DC.Descr ipt ion

CrawlerDescr ip tion

Body

Individual

HasDcCreator

HasCrawlerAuthor

HasDcSubject

HasHeadings

HasKeywords

HasAuthor

HasSubject

Subc lasses

Subc lasses

Keyword Set

Subc lasses

1.0

0.8

0.9

1.0

0.8

1.0

0.8

1.0 0.8

0.9

PhysDis Collection View

MARIAN Layers

Database Layer

Search Engine Layer

User Information Layer

User Interface Layer

User User User User

MARIAN Parallelism

Java part response time vs. query rate comparation

(type 1 requests)

01000200030004000

0 100 200 300 400 500

query rate (#/min)

resp

onse

tim

e (m

s)

all modules in one machine one "webgate"

two "webgate"s four "webgate"s


PetaPlex

• Digital Library Machine (“super” object store): Parallel computer / storage utility

• Research: inverted files, video server, …• Knowledge Systems Incorporated is supplying

VT-PetaPlex-1 with 2.5 terabytes through 100 nodes:

• Net connection + 25GB disk + 233 MHz Pentium + Linux

PetaPlex Complex

FRONT END MACHINERS/6000, 1G RAM, 4 Proc.

Nanoserver

Nanoserver

Nanoserver

Nanoserver

Nanoserver

Nanoserver

Nanoserver

Nanoserver

Nanoserver

Nanoserver Nanoserver

Nanoserver

Nanoserver Nanoserver

Service

Machine 1

Service

Machine 2

Service

Machine 3

Service

Machine 4

PetaPlex Top View

4 ft.

side

PetaPlex Side View

4 ft. wide

8 ft.

high

Roles:* Support* Cooling* Power

15

shelves

PetaPlex Service Machine Possibilities

• Front-end provides handle/repository abstraction through hashing

• Small object server• Large object server

• video on demand• streaming audio

• Information retrieval server• Proxy / cache server (e.g., 1 terabyte server

of 1000 worldwide for Comsat/Intelsat)

Sornil & Mather Dissertations

• Mather: efficiently handling very large numbers of objects of varying sizes

• Sornil: efficiently handling IR for very large dynamic collections, large numbers of users, high transaction rates, large inverted files• modeling and simulation• data organization • parallelization of algorithms, alone and in

combination for retrieval (related) tasks

Comparison Network of

Workstations (NOW)

Beowulf PetaPlex

Archi- tecture

Cluster of general purpose workstation class machines using off-the-shelf network interconnect

General purpose PCs, interconnected with a custo- mized network

Special purpose architecture tuned for superstorage. Uses a mix of off-the-shelf PC components and specialized network interconnects.

Cost per node

Workstation prices. Between $2000-$2500/node

Mid to low-end PC prices. Between $1200-$1800 per node

Mass produced components will reduce price to around $100/node

Target area

Computation Computation Storage, computation is a secondary function

Filesystem support

UNIX flavors UNIX flavors Replaces location dependant files with location independent fine-grained URN named objects


Open Archives Initiative

OAIwww.openarchives.org

[email protected]

OAi Philosophy

• Self-archiving = submission mechanism• Long-term storage system = archive• Open interface = harvesting mechanism• Data provider + service provider• Start with “gray literature”

• e-prints/pre-prints, reports, dissertations, …

Open Archives Initiative (OAI)• xxx@LANL, high-energy physics (Ginsparg, 1991)• CSTR + WATERS = NCSTRL (Lagoze,1994)• xxx + NCSTRL = CoRR collaboration (1998)• Universal Preprint Service protoproto, Oct. 21-22, 1999, Santa Fe

– led by LANL, CNI, DLF, Mellon --> OAi• Santa Fe Convention (see Feb. D-Lib Magazine article)• Follow-on mtgs: 6/3@San Antonio, 9/21@Lisbon (ECDL)• Archives -> Open Archives

• Support unique archive identifiers• Implement Open Archives metadata set (DC, using XML)• Implement OA harvesting protocol (derived from Dienst protocol)• Register the archive

• Build tools, layer other services: linking, searching, …

Open Archives (protoproto)

• ArXiv & Los Alamos National Lab• CogPrints & U. Southampton• NACA & NASA (reports)• NCSTRL & Cornell U.• NDLTD & Virginia Tech• RePEc & U. Surrey• Total of around 200K records

Original Open Archives Members

• American Physical Society

• California Digital Library

• Caltech

• Coalition for Networked Info.

• Cornell University

• Harvard University

• Library of Congress

• Los Alamos Nat’l Lab

• Mellon Foundation

• NASA Langley Research Cntr

• Old Dominion University

• Stanford University

• U. of Ghent

• U. of Surrey

• U. of Southampton

• Vanderbilt University

• Virginia Tech

• Washington University

Open Archives Future

• EconWPA (U. Washington)• e-biomed -> PubMed Central (NIH)• PubScience (DOE)• Clinical Medicine Netprints (+ other HighWire Press holdings )• University ePub (California Digital Library)• All public e-prints (MIT)• Scholar’s Forum (Caltech)• Int’l: CERN, Germany, India, Mexico, …• Goal: millions of books/articles/reports / yr

Harvesting vs. Federation

• Competing approaches to interoperability• Federation is when services are run remotely on remote

data (e.g. Federated searching)

• Harvesting is when data/metadata is transferred from the remote source to the destination where the services are located (e.g. Union catalogues)

• Federation requires more effort at each remote source but is easier for the local system and vice versa for harvesting

• OAI currently focuses on harvesting

Metadata vs. Data

• Data refers to digital objects or digital representations of objects

• Metadata is information about the objects (e.g. title, author, etc.)

• OAI focuses on metadata, with the implicit understanding that metadata usually contains useful links to the source digital objects

Technical Umbrella for Practical Interoperability…

ReferenceLibraries

PublishersE-Print

Archives

…that can be exploited by different communities

Museums

OAI – Repository Perspective

Required: Protocol

DODO DO DO

MDO

MDO MDOMDOMDO

MDOMDOMDO

OAI – Black Box Perspective

OA 1

OA 2

OA 4

OA 3

OA 5OA 6

OA 7

Tiered Model of Interoperability

Mediator services

Metadata harvesting

Document models

Repository of Digital Objects

RepositoryAccessProtocol

handle

Digital object

terms and conditions

Approaches to Open Archives

Build ByDiscipline

Build By Institution

Approaches to Open Archives

Build ByDiscipline

Build By Institution

AuthorCategoryInterdisciplinaryYearLanguageQuery …

Author′s toolswww.physik.uni-oldenburg.de/EPS/mmm

DiscoveryCurrent

AwarenessPreservation

Service Providers

Data Providers

Meta

data

harv

estin

g

The World According to OAI

Figure 1. Layers Related to Open Archives Initiative

Services

…

Search/Browse

Authoring Citation Checking Submission

Metadata Creation

Editorial: Reviewing, Certification

Registry

Archives: Name, ID, Description, Terms and Conditions, …

Metadata Formats: Name, XML DTD, …

…

Archive Formats: Name, Standard, Preservation Process, …

Protocols Tools

Services

Copy-Edit / Add Value Citation DB Updating

Authority Control

Preservation Conversion

Text/MM Editing

Gazetteer Cataloging

Collaboration

Annotation

Summarization

Citation / Linking

SFX

CiteSeer

Repository NCSTRL Repository

…

EconWPA Repository

RePEc Repository

Repository for NDLTD Open Archives Harvesting Protocol

Metadata Formats: OA Metadata Set, NDLTD Standard (DC-based) Set

Transaction Log

Training Resources

VT Partition

Record (Metadata)

Record (Full Content)

… …

UVA Partition

Metadata Content

Caltech Partition

Metadata Content

Mechanisms

• Sharing• Join federation, run software• Make metadata and archive available

• Aggregating• By discipline• By institution• By genre

• Automating• Workflow• Harvesting and providing services• Federated searching• Dynamic linking (e.g., with SFX (OpenURLs))

VT View of the Open Archives Initiative (OAI)

• Enable sharing of publication metadata and full-text by digital libraries

• Standardize low-level mechanisms to share contents of libraries

• Build higher-level user-centric and administrative services in meta-libraries

• Install organizational mechanisms to support the technical processes

• Insights from 5S (streams, structures, scenarios)

Virginia Tech Projects

• MARC XML-DTD

• Computer Science Teaching Centre (CSTC)

• W3C Web Characterization Repository

• OAI Repository Explorer

• NDLTD

• Open Digital Libraries, XOAI-PMH

MARC XML-DTD

• XML Transport format for US-MARC records

• Standardized metadata exchange format for traditional library services joining OAI

Key Features of the OAI Metadata Harvesting Protocol

• definitions & concepts

• repository

• record

• identifier

• datestamp

• set

• protocol features

• HTTP encoding

• metadata prefix & schema

• flow control

• protocol requests

• supporting requests

• harvesting requests

repository

repos i tory

OAI protocol

harves ter

supportdata

harvestingdata

items

identifiers

oai-identifier = oai:archive-identifier:record-identifier

Registered URI

Scheme

Archive Identifier:Registered within

OAI

Unique ID within archive:

(syntax is archive-specific)

example = oai:ncstrl:ncstrl.cornellcs/TR94-1418

locally unique key for extracting a record from a repository

selective harvesting - datestamps

repos i tory

harvest withindate range

record

record

selective harvesting - sets

repos i tory

harvest within setS1

recordrecord

record

S2

OAI Tools

• Related resources, e.g., XML, Unicode

• Servers and utilities, e.g., ARC, Kepler, EPrints

• XML Schema Validator

• Repository Explorer• Interactive Browsing• Testing of parameters• Multiple views of data• Multilingual support• Automatic test suite

ARC (arc.cs.odu.edu)

Kepler Architecture

arc Service Provider

HTTP (Only Supports OAI request)

OAI Layer

File based Repository Personal Editor

Archivelet

archivelet archivelet

DP handle IP:Port

DP handle IP:Port

Registration Server

Service Provider

mappingtable

Active

Active

User

4. Query

2. Data provider locate

3. Metadata Harvest

5. Full Text fetch

1. Register/Notify

OAI-based NCSTRL architecture

XSV Schema Validator

OAI Repository Explorer

• Serves as a compliancy test

• Allows browsing of open archives using only OAI protocol

• Sends requests on behalf of user, parses and checks responses and displays browsable interface

• Will detect most discrepancies in protocol

• http://purl.org/net/explorer

RE1.3

OAI Repository Explorer

• Serves as a compliancy test

• Allows browsing of open archives using only OAI protocol

• Sends requests on behalf of user, parses and checks responses and displays browsable interface

• Will detect most discrepancies in protocol

• http://purl.org/net/explorer

Request, Response – OAI, VT ETDshttp://scholar.lib.vt.edu/theses/OAI/cgi-bin/index.pl?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai:VTETD:etd-520112859651791

Request

Response

Case Study: NCSTRL Costs/BenefitsStakeholders Sample Potential Cost Sample Potential Benefit

Providers Faculty Lower value for P&T Faster publishing

Students Less recognition Broader set of outlets

Practitioners Limited relevance Ease of publishing, > quantity

Users Faculty Lower quality of work Broader access to resources

Students Higher access costs (vs. department available material)

Lower access costs (vs. journal available material)

Departments New maintenance costs Broader visibility

University libraries Additional access costs Access to new resources

Practitioners More difficult access Access to new resources

Case Study: NDLTD

• Multiple independent university/library/ center-based collections of electronic documents

Virginia Tech

Rhodes U.

U.Waterloo

International

ETD

Library

OAI

Metadata

Harvesting

Protocol

ETD Union Collection (OAI)

VIRTUA

Merged Metadata Collection

MARIAN

Virginia Tech ETD Archive

Duisburg ETD

Archive

HumboldtETD

Archive

Future: recommender, …

… OAI Data Provider

OAI Service Provider

OAI Harvesting

LEGEND

Union Catalog Creation

NDLTD Site / Member

Local DB

OAI Server

Local Search / Brow se

Student Entry

NDLTD Central

OAI Harvester

Name Authority Service

(e.g. OCLC)

MARIAN Union

Catalog

VTLS Union Catalog

MARC DB

Virtua

Conversion

Alternate MARC Transport (f tp?) tapes?)

Librarian Verif ication / Validation / Enrichment / Maintenance

DL Components

User Interfaces

Workflow Mgr

DBMS

Search Engines, Classifiers, …

Data, MM Info

Gateways

Repository

Rights Mgr

MM/ HT Renderer

Open Digital Library (ODL) Hypothesis (Hussein Suleman)

• Can we leverage the successful model of the OAI Protocol for Metadata Harvesting to alleviate our architectural problems ?

Maybe … if

Digital Libraries can be modeled as• networks of extended Open Archives, where• each extended Open Archive is a• source of data and/or a provider of services.

Open Digital LibrariesXOAI-PMH

• Dissertation work of Hussein Suleman (member of OAI technical committee)

• Extending the OAI protocol• Supporting rapid development of DLs using

networks of components• Demonstrated with NDLTD, CSTC• Described in Dec. 2001 D-Lib Magazine

article, and article submitted for publication

Component System Approach• (Open) DL = Network of Extended OAs

Local Archive

Data Input

Remote Archive

Browse

Metadata Repository

Search Recommend

Resource Discovery

User Interface

OAI/ODL archive

OAI/ODL protocol

leg

end

Example Architecture (NDLTD)

Humboldt

Duisburg

MIT Filter

MIT

Browse

Union Catalog

Search Recent

User Interface

User Interface

OAI/ODL archive

OAI/ODL protocol

leg

end

Virginia Tech

PhysNet

CalTech

Dresden

ODL Demonstration - FrontPage

ODL Demonstration - Search

ODL Demonstration - Browse

Summary• Virginia Tech context• Why DLs? What are DLs? (5S theory)• Case Study: WCA• Case Study: Education: CSTC -> NSDL• Case Study: NDLTD• Accessibility and Visualization• DL Software: MARIAN• DL Hardware: PetaPlex• Interoperability: OAI, ODL

Selected Links• CITIDEL

• www.citidel.org• NCSTRL

• www.ncstrl.org• NDLTD

• www.ndltd.org• NSDL

• www.nsdl.org• Virginia Tech Digital Library Courseware

• http://ei.cs.vt.edu/~dlib • Virginia Tech Digital Library Research Laboratory (DLRL)

• http://www.dlib.vt.edu• (5S, 5SL, AmericanSouth.Org, CSTC, ENVISION, MARIAN,

NDLTD, NSDL, OAI, ODL)• Virginia Tech DLRL OAI Projects

• http://www.dlib.vt.edu/projects/OAI/• Repository Explorer

• http://purl.org/net/oai_explorer

More Links• ARC Cross-Archive Search Service

• http://arc.cs.odu.edu/

• Dublin Core Metadata Initiative• www.dublincore.org

• E-Prints DL-in-a-box• www.eprints.org

• Open Archives Initiative• http://www.openarchives.org

• OAI Metadata Harvesting Protocol• http://www.openarchives.org/OAI/openarchivesprotocol.htm

• XML Schema Validator• http://www.w3.org/2001/03/webdata/xsv

• XML Tools at W3C• http://www.w3.org/XML/#software

Documents

Overview of Digital Libraries: From Requirements to Theory to System to Projects JCDL -- Portland, Oregon, USA Tutorial 4 -- July 14, 2002 Edward A. Fox