Chaitan Baru Richard Marciano - MNHS

San Diego Supercomputer CenterNational Partnership for Advanced Computational Infrastructure

Projects at SDSC

Chaitan BaruRichard Marciano

{baru,marciano}@sdsc.edu

Data Intensive Computing GroupSan Diego Supercomputer Center


Projects at SDSC

• National Archives and Records Administration, NARA• Persistent Archives and Electronic Records

• NHPRC/NARA• XML and GIS

• aXioMap• I2T: An Information Integration

Testbed for Digital Government


Projects at SDSC (… cont)

• AMICO• In conjunction with the California Digital Library

(CDL)• Part of the NSF DLI-2 project

• ESRI• Community of Science, Inc.• Networked Earthquake Engineering

Simulation (NEES)• NSF program


NARA & NSF

N ational Partnership for A dvanced Co mputatio nal Inf rastructure

NPACI is a Highly Leveraged National“Partnership of Partnerships”

47 institutions, up from 37

20 states, up from 18

4 countries, up f rom 1

5 national labs

Many projects (new and old)

Vendors and industry

Government agencies


Information Based Computing

Archival StorageApplications

Digital Library

Data Storage

Information Management

CollectionBuilding

CDLUCB - ElibUCSB - ADLStanford - SDLIPU Michigan - UMDL

Digital SkyNeuroscience

Protein Data BankMolecular Structures

Earth Systems Science

Applications Digital Libraries


Information Management Hierarchy• Persistent Archives

• Storage of information model, data model, along with data• Data Grid

• Access to data in a different administration domain• Digital Library - Presentation / Information Discovery

• Interlib - ADEPT, UC Berkeley Digital Library• Data Collection

• Extensible Meta-data catalog - EMCAT• Data handling

• SDSC Storage Resource Broker - SRB• Archival Storage

• High performance storage system - HPSS


Common Information Model

• eXtensible Markup Language (XML) • Use tags to define semantic context for components of the

data set• Document Type Definition (DTD)

• Provides semi-structured representation for organizing tags that can be applied to groups of digital objects

• Development of standards for tags• Digital sky, Protein Data Bank, Neuroscience brain images• California Digital Library - Art Museum Image Consortium


Hierarchy of Information Contexts• Digital object context

• Meta-data to define the structure of the object• When publishing a digital object, must also publish the

context of the object• Use collections to organize objects

• Meta-data to define the structure of the collection • When publishing a collection, must also publish the

information needed to organize the collection.• Use presentation context to control access

• Meta-data to define structure of presentation


Persistent Object Preservationfor Archives (POP-A)

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��MIX

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

SRB/EMCAT

Electronic Records Archives: Conceptual View

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

Metadata

ArchivalRepository

��

��

��

��

OrderFulfillment

System

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

ReferenceWorkbench

Query

Rebuild

Present

��

��

��

��

��

��

Tapes

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

AccessioningWorkbench

Accession

Verify

Wrap & Containerize

Describe

CollectionDisks

Internet

Collection

Collection

��

��

��

��Archival Research Catalog

EMCAT: Extensible Meta-data Catalog GSI: Grid Security Infrastructure MIX: Mediation of Informationusing XML

SRB: Storage Resource Broker

��

��

��

��

RecordsSchedules

GSI


Collections Studied

E-Mail Postings SDSC

Tiger/Line’92 Census

104th Congress Bills House

105th Congress Bills House

Electronic Archive Project NARA

Combat Area Casualties File NARA

Patent Data USPTO

Image Collection (AMICO) CDL Calif.

Joint Interoperability Test Defense


E-mail Postings Demo____NARA_article_begin____:___________________Path: news.sdsc.edu!newshub.csu.net!newshub.sdsu.edu! newsfeed.berkeley.edu! news.cis.ohio-state.edu!news.rootsweb.com!rootsweb-gwFrom: [email protected]: soc.genealogy.hispanicSubject: Passenger Lists for Ships from Spain To CubaDate: 22 Mar 1999 16:20:37 -0800Organization: RootsWeb Genealogical DataCooperativeLines: 7Message-ID: <[email protected]>NNTP-Posting-Host: localhostMime-Version: 1.0Content-Type: text/plain; charset=US-ASCIIContent-Transfer-Encoding: 7bitX-Trace: bl-1.rootsweb.com 9221484373147 127.0.0.1 (23 Mar 1999 00:20:37 GMT)X-Complaints-To: [email protected]: 23 Mar 1999 00:20:37 GMTXref: news.sdsc.edu soc.genealogy.hispanic:3156

Does anyone know where I can get passengers lists forships that transported Spaniards to Cuba circa 1860's?Any help would be appreciated.

Thanks,

Cheryl Sanchez-Sivers

____NARA_article_end____:_____________________


E-mail Postings Demo

• http://srb.npaci.edu

• 40 million E-mail records:• ingested, archived, dynamically rebuilt within a month


NHPRCNational Historical Publications & Records Commission

“Methodologies for the “Methodologies for the LongLong--Term Preservation of and Access toTerm Preservation of and Access toSoftwareSoftware--Dependent Electronic Records”Dependent Electronic Records”

funded January 1, 2000

a 3-year project


Introduction

A project funded by the National Historical Publications and Records Commission (NHPRC):

• to conduct research on the long-term preservation of and access to software-dependent data objects (SD-DO), and

• to develop prototypes that will lead to the creation of useful tools for archivists to preserve and provide access to electronic records over the long-term.


Goals• The organizing principles we adhere to are:

!Electronic records need to be input as infrastructure independent digital objects (II-DO)

!Context information about relationships (collection information)needs to be maintained

!The structure of a collection can be derived through an adaptiveprocess

!New electronic records can then be validated against the collection structure

!Error handling mechanisms can be developed!Mechanisms to handle the natural evolution of a collection's

structure must be developed


Issues Investigated

• II-DO creation based on the analysis of text, compound, spatial E-records

• II-DO creation & management automation• Archivists’ Workbench (AW) framework:

• prototype of infrastructure independent management• Issues:

• robustness• structural change• scalability


Approach

• Use of markup standards (SGML, XML, …)• XML advantages:

• allows us to model SD-DO as semistructured information • more flexible• tolerates structural variations

• allows for a common information interchange format across the WWW


Plan of Work• Research key functions of an AW:

• using different classes of SD-DO:• textual records: ASCII, US Congress, HTML• compound records: E-mail, pdf, Word, Excel• spatial records: GIS data

• Key functionality includes:• Ingestion• Structural inference (DTD creation)• II-DO validation• Error handling• DTD evolution


Major Tasks

a. text |b. compound |----- recordsc. spatial |

• I. Input I.a. I.b. I.c.• II. DTD Creation II.a. II.b. II.c.• III. Document Validation III.a. III.b. III.c.• IV. Error Handling IV.a. IV.b. IV.c.• V. DTD Evolution V.a. V.b. V.c.


Three-year Timeline


Deliverables• Technology Demos:

• Year 1: AW for: • text without evolution• html without evolution

• Year 2: AW for:• text with DTD evolution• compound docs without DTD evolution• preliminary error handling• ingestion of spatial objects

• Year 3: AW for: • all types of information


Advisory Board & SDSC• Objectives:

• exchange information• develop a working relationship

• Feedback of interest:• attributes for long-term preservation of SD-DO• relationships to be maintained across DO & collections• sources of data to drive project (evolution, …)• usability of tools• dissemination of results in the archival and records

management community


aXioMap

Application of XML for Interactive Online Mapping


Spatial XML Markup Languages

• Metadata Standards• FGDC / XML DTD• ANZMETA / XML DTD

• Geography Markup Language (GML) 1.0• OGC Working Draft 17-Jan-2000

• Web Mapping Testbed (WMT): NIMA, USACoE, FGDC, NASA, USDA, USGS ...

• Digital Earth (www.digitalearth.gov)

• AXL (ArcXML) pre-release• part of ESRI ArcIMS 31-Jan-2000


GML• GML: XML specification to encode geo. info.

For both Data Storage & Data Transport• Initial release deals with OGC Simple Features:

• vector geodata: e.g. digital map info (streets, population, land use zones, property lines, watersheds, etc.)

• GML is not concerned with the visualization of geographic features (drawing of maps)

GMLin XML

Direct rendering Graphicformat

Transformation into a vector graphics rendering format

• SVG• VML• VRML

Direct routing w.o. viz. Numerical model


Web Vector Graphics: VML & SVG• April ‘98--Adobe, IBM, Netscape, Sun:

• PGML (Precision Graphics Markup Language)• May ‘98--Hewlett-Packard, Macromedia, Microsoft, Visio

• VML (Vector Markup Language)==> October ‘98--SVG working group formed

SVG contains six predefined objects:Rectangle, circle, ellipse, polyline, polygon, & line

==> Adobe: Illustrator, Photoshop, GoLive


Selected VML Demos

• http://www.qolsandiego.net/maproom/northcity/main.htm• http://www.elzaresearch.com/gis/primaries/


I2T: An Information Integration Testbed for Digital Government

• To be funded by the NSF Digital Government program at $720K for 3 years (unofficial)

• Personnel• Chaitan Baru, SDSC, PI• Yannis Papakonstantinou, CSE/UCSD, co-PI• Amarnath Gupta, SDSC, co-PI• Bob Hollebeek, U.Penn, co-PI• Richard Rockwell, U.Michigan, co-PI• Bertram Ludaescher, Richard Marciano, Ilya Zaslavsky,

Senior Personnel


Government Partners

• U.S. Census Bureau• NARA• USGS• Department of Community and Economic

Development, State of Penna• Department of Labor and Industry, State of

Penna• SANDAG (San Diego Association of

Governments), San Diego County


Research Topics

• Extending the MIX system to support integration and mediation of geospatial information sources• Modeling of GIS information in XML• Spatial extensions to XML query languages• Dealing with heterogeneity in

• accuracy• resolution• feature space• schema

• XML-based representations of GIS to support long-term archiving


Research Topics• Tools for DTD-guided wrapping of unstructured

text• Specifically, investigate conversion of Census codebooks

to XML based on the Data Documentation Initiative (DDI) DTD

• Novel applications of the I2T infrastructure• Census Integrated Statistical Information system

(FERRETT)• Sociology workbench--access to remote survey

information. Ability to read DDI-encoded survey codebooks and other XML data sets

• Distributed decision support and data mining applications


Users: government, research, education, general public

Various surveys

. . . . Federal . . . . State LocalWrapped sources from multiple government levels

Census USGS NARA PA State SANDAG

Census IntegratedInfo. Systems BBQ GUI Data Mining Sociology

Workbench

Mediator

Spatial Mediator

Conflation Mediator,Data Analysis andother plug-ins

Archive Access

The I2T Testbed

Research partners:• SDSC• UCSD CSE• U.Penn, NSCP• U.Michigan, ICPSR

Agency informationsources


The AMICO Digital Library Project

http://www.amico.orghttp://www.npaci.edu/DICE/AMICO

Art Museum Image Consortium55,146 objects (750 MB)

53,763 thumbnail images (319 MB)

57,609 full tiff images (180 GB)


AM

ICO

Con

sort

ium

of

26

mus

eum

sAGO_ Art Gallery of OntarioAIC_ Art Institute of ChicagoAKAG Albright-Knox Art Gallery, Buffalo, NYASIA Asia SocietyBMFA Boston Museum of Fine ArtsCCP_ Center for Creative Photography, U. ArizonaCMA_ The Cleveland Museum of ArtDMCC Davis Museum and Cultural Center, Wellesley College, MAFASF Fine Arts Museums of San FranciscoGEH_ George Eastman House, Rochester, NYJPGM J. Paul Getty Museum, Los Angeles, CALACM Los Angeles County Museum of ArtLOC_ Library of CongressMACM Musée d'art contemporain de Montréal MBAM Musée des beaux-arts de MontréalMCAS Museum of Contemporary Art, San DiegoMIA_ The Minneapolis Institute of ArtsMMA_ The Metropolitan Museum of ArtNGC_ National Gallery of Canada, Ottawa/OntarioNMAA National Museum of American Art, Smithsonian InstitutionPMA_ Philadelphia Museum of ArtSFMO San Francisco Museum of Modern ArtSJMA San Jose Museum of ArtTFC_ The Frick Collection, NYWAC_ Walker Art Center, Minneapolis, MNWMAA Whitney Museum of American Art, NY


Raw Metadata Structure- catdata: 8 files

16,604 year1.d99042914,430 year1.d99051222,938 year1.d99052054,303 year1.d990627

15 year1.d99070854,298 year1.d990731

93 year1.d990806657 year1.d990813

- tiffmetadata: 23 files2963 AGO_.tiffmetadata.txt1016 AIC_.tiffmetadata.txt894 AKAG.tiffmetadata.txt187 ASIA.tiffmetadata.txt

7591 BMFA.tiffmetadata.txt401 CCP_.tiffmetadata.txt

1455 CMA_.tiffmetadata.txt56 DCMC.tiffmetadata.txt

470 DMCC.tiffmetadata.txt10141 FASF.tiffmetadata.txt2137 GEH_.tiffmetadata.txt1459 JPGM.tiffmetadata.txt1013 LACM.tiffmetadata.txt

20654 LOC_.tiffmetadata.txt86 MACM.tiffmetadata.txt50 MBAM.tiffmetadata.txt31 MCAS.tiffmetadata.txt

1440 MIA_.tiffmetadata.txt550 MMA_.tiffmetadata.txt

1507 NGC_.tiffmetadata.txt1416 NMAA.tiffmetadata.txt154 PMA_.tiffmetadata.txt158 SFMO.tiffmetadata.txt86 SJMA.tiffmetadata.txt68 Such.tiffmetadata.txt

396 WAC_.tiffmetadata.txt37069 replacements.txt57499 replacements2.txt

- thumbmeta: 52,689 filesAGO_.1016.25_thum.met*AGO_.1016.32_thum.met*AGO_.1016.39_thum.met*…...WAC_.994C_thum.metWAC_.996C_thum.metWAC_.998C_thum.metWAC_.99C_thum.met*WMAA.1557_56_thum.metWMAA.31_426_thum.met


Tape Data StructureSix DLT7000 tapes. Each tape contains a tar file of TIFF images of approximately 30GB.

Tape 1: AGO_, BMFA, LACM

Tape 2: CMA_, LOC_, AIC_, AKAG, ASIA

Tape 3: CCP_, GEH_, MACM, MCAS, MMA_, NMAA, SFMO, TFC, WMAA, DMCC, JPGM, MBAM, MIA_, NGC_, PMA_, SJMA, WAC_

Tape 4: FASF

Tape 5: FASF

Tape 6: FASF





AMICO Metadata Conversion Steps

Merge“Raw” Metadata files:- catdata (8 files),- tiffmetada (23 files),- thumbmeta (52,689 files)

Convert toXML

Split-by-museums 1

XML fileper museum

Split-by-file size

MultipleXML files

per museum

eXcelonDump&Load

Utility

eXcelonData Server

Split-by-machines

1 XML fileper museum

Multiplemuseum XML

files per machine

3 XML files:- 1 catdata- 1 tiffmetadata- 1 thumbmeta

eXcelonData Server

eXcelonData Server

ConsolidatedMetadata files:- 1 catdata- 1 tiffmetadata- 1 thumbmeta

Tape Read


XMAS query

California Digital Library (CDL) PrototypeThe Art Museum Image Consortium (AMICO)

MIXmView based onAMICO DTD

Wrapper

MARCDatabase

XML doc

AMICO XMLDatabase

AMICO XMLDatabase

SRB/MCAT

HPSS

Request forimage (X.509)

tif file

BBQ Interface(slide carousel

interface)

(XMAS: XML Matching and Structuring query language)


Architecture for “interactive” archives

Metadata server

HPSS

• eXcelon• Oracle 8i• DB2

Networkattached

DiskTB

capacity

Storage Area Network(SAN)


Current catalog metadata count (per museum)


Catalog & Image count (per museum)


Catalog & Image count (per museum)


Average tiff size in MB (per museum)


Community of Science, Inc.

• www.cos.com• Specifying XML DTD standards for Current

Research Information Systems (CRIS)• Investigate tools for

• wrapping unstructured text • mapping from source formats to CRIS DTD

• Enable creation of warehouse of research information and enable e-commerce


ESRI

• ESRI develop ArcInfo, ArcView, ArcIMS GIS products

• Project involves evaluating the ArcXML (AXL) standard

• Keeping AXL abreast of developments in the Geography Markup Language (GML) standard proposed by OpenGIS Consortium

• Identifying issues in mapping AXL to other XML standards, e.g. WAP (Wireless Application Protocol)


NEES

• Proposal to NSF’s Networked Earthquake Engineering Simulation (NEES) program

• Develop NeesML, an XML-based standard for representing earthquake engineering simulation metadata and data

• NeesML will facilitate the creation of a NEES Curated Database, a warehouse of earthquake engineering simulation information

Documents

Chaitan Baru Richard Marciano - MNHS