Plugin Lifecycle

Andrew FabianMetaArchive Annual Membership

MeetingAtlanta, Georgia

Friday October 24, 2008

Plugin Lifecycle

START

Local Development and Testing

Commit toRepository

New CollectionsIn Conspectus

IsolationTesting

Plug-inModifications

ConspectusModifications

Production! END

Local Development

Note: There is a presentation on this topic by Daniel Culpepper.

• Write the Plugin with LOCKSS Plugin Tool.

• Test-crawl with that tool.– Crawling will happen from your desktop computer,

not your LOCKSS box. Make sure firewalls are OK with this.

• This implies setting up a Manifest page on the site to be crawled.

• When finished you have an .XML Plug-in file.

Subversion

• Subversion synchronizes files between MetaArchive developers.

• You’ll need a Subversion client.– TortoiseSVN for Windows– SCPlugin for Macs– svn (command-line) for everything

• After a plugin is written, add it to the MetaArchive repository.

• Releases (testing and production) are made from the official repository only.

The Conspectus Database• This is the MetaArchive’s online master list of

what gets harvested.

• It contains:– Title– Descriptive Metadata– Base URL– Plugin Identifier– Extra Parameters

• Add a new record for the collection you’re getting ready to harvest.– These are fairly detailed records. Make sure to have the

appropriate descriptive metadata on hand.

The Conspectus Database

The Conspectus Database

Isolation Testing

• This tests two things:– The final combination of the Plugin, Collection,

and production software– Recrawls

• MetaArchive Sandbox:– VMWare Virtual Machine– Gets Plugins from Subversion, Collections from

the Conspectus– Runs the MetaArchive LOCKSS daemon, but

does not communicate with the rest of the network

Isolation Testing

Demo

Going Live

• Almost the same process as Isolation Testing

• Ask MetaArchive admins to release the plugin and the collection on a weekly phone call.

• They will install the plugin for the entire network, and allocate space on a set of MetaArchive storage nodes to cache the new collection.

Plugin ModificationsIn Production

• Go through normal development cycle

• Make sure to increment the Plugin Version Number!

• Ask admins to update the Plugin from Subversion– All MetaArchive caches will load the new plugin within a day.– For faster updates, trigger a manual crawl of the “Plugins”

AU from the DebugPanel.

Conspectus ModificationsIn Production• Edit Descriptive Metadata at any time

• Add additional AUs at any time– Add a new line to the “Extra Parameters”

section– Announce the new AU via email and weekly

phone call

• Changing existing AU parameters can break things. Instead, add a new AU and then delete the old.

Deletion

• If a collection no longer needs to be harvested, and the backup copy is no longer useful, it can be deleted.

• Deletion is tricky, because there are several steps:– Delete Collection from Conspectus– Delete AUs from all participating MetaArchive Caches– Delete files from disk of all participating MetaArchive

caches

• Detailed instructions are in the MetaWiki.

Questions?

• Answers– Yes– fixed in next version – No– 42– legacy support– it’s not a bug, it’s a feature