8
NLW Large Ingest Experiences Glen Robson [email protected]

Introduction Wills Project Between: 1543 – 1858 182, 404 Wills 816, 325 Images 998, 729 Fedora Objects Recorded amount ingested per hour

Embed Size (px)

DESCRIPTION

Object LayoutObject Layout METS DC RELS-EXT Zoomify M Reference M Archive E Will Page 1 METS DC RELS-EXT Zoomify M Reference M Page 2 METS DC RELS-EXT Zoomify M Reference M Archive E Page 3 METS DC RELS-EXT Archive E

Citation preview

Page 1: Introduction  Wills Project  Between: 1543 – 1858  182, 404 Wills  816, 325 Images  998, 729 Fedora Objects  Recorded amount ingested per hour

NLW Large Ingest Experiences

Glen Robson [email protected]

Page 2: Introduction  Wills Project  Between: 1543 – 1858  182, 404 Wills  816, 325 Images  998, 729 Fedora Objects  Recorded amount ingested per hour

Introduction

Wills Project Between: 1543 – 1858

182, 404 Wills 816, 325 Images 998, 729 Fedora Objects

Recorded amount ingested per hour

Page 3: Introduction  Wills Project  Between: 1543 – 1858  182, 404 Wills  816, 325 Images  998, 729 Fedora Objects  Recorded amount ingested per hour

Object Layout

METS

DC

RELS-EXT

Zoomify MReference

MArchive E

Will

Page 1

METS

DC

RELS-EXT

Zoomify MReference

M

Page 2METS

DC

RELS-EXT

Zoomify MReference

MArchive E

Page 3

METS

DC

RELS-EXT

Archive E

Page 4: Introduction  Wills Project  Between: 1543 – 1858  182, 404 Wills  816, 325 Images  998, 729 Fedora Objects  Recorded amount ingested per hour

Hardware

Tape : Scalar i2000, 3 Copies: Offsite Copy (University of Aberystwyth) 1 Copper Room In the machine 1

Optical: Plasmon G638 (Write Once)

Fedora 2.2.4

MPT Store Oracle

Primary ServerFedora 2.2.4 MPT Store

Oracle

Secondary Server

Raid ArrayReferenced

Datastreams

Mounted External Datastreams (Http Referenced)

RHEL 5

Page 5: Introduction  Wills Project  Between: 1543 – 1858  182, 404 Wills  816, 325 Images  998, 729 Fedora Objects  Recorded amount ingested per hour

Ingest

From processing server REHL 5 8 * Duel Core Processors

Single Thread Create object hierarchy Upload Images (Zoomify and PNG)

Page 6: Introduction  Wills Project  Between: 1543 – 1858  182, 404 Wills  816, 325 Images  998, 729 Fedora Objects  Recorded amount ingested per hour

Findings

Page 7: Introduction  Wills Project  Between: 1543 – 1858  182, 404 Wills  816, 325 Images  998, 729 Fedora Objects  Recorded amount ingested per hour

Lessons Learned

Resource Index Optimise Database Simultaneous Ingest? Mysql?

Page 8: Introduction  Wills Project  Between: 1543 – 1858  182, 404 Wills  816, 325 Images  998, 729 Fedora Objects  Recorded amount ingested per hour

Questions