© DSpace User Group Meeting April 21, 2006 — Bergen, Norway William Reilly, Larry Stone — MIT Libraries 2006 v. 20060421_0945 Technical Introduction To

©

DSpace User Group Meeting

April 21, 2006 — Bergen, Norway

William Reilly, Larry Stone — MIT Libraries

2006v. 20060421_0945

Technical Introduction To and

Initial Use Of the

Lightweight Network Interface (LNI) (—to DSpace!)

DSUG Bergen 2006 Lightweight Network Interface (LNI) 2

Today's Talk

1. Context: CWSpace project

2. Web Services

• "Lightweight Network Interface"

3. WebDAV & DSpace

• "Technical Introduction To…"

4. MIT OpenCourseWare Submit (example)

• "Initial Use Of…"

5. Appeal for Use Cases, Future Dev!


Context: CWSpace project


Project Overview

Vision: Increase value in Institute publication: OCW– Also: Increase

value in Open Source platform: DSpace

* CLE = Collaborative Learning Environment

Goal: Archive OCW to MIT’s DSpace– Also: Make content

available to CLEs *

CWSpace: MIT iCampus project, nearing completion

Method: Interoperability of systems– Use Content Packaging standard– Use Web Services standard


Perceived Need: Interoperability

• Obvious observation: Higher Ed. Technologies increasingly seeking integration with Library & Repository holdings

• Not so obvious(?): Libraries seeing their charter to include collection, preservation, dissemination of Teaching & Learning materials (?)


Interop: What’s Available

• Normalized data model (out of chaos)– "Sections" fit well to publication organization

• Normalized content files (.PDF)– Reduced ability to disaggregate, re-purpose

1 of 4


Interop: What’s Achievable 2 of 4


Interop: Challenges 3 of 4


Interop: Possible Resolution? 4 of 4


Web Services: "Lightweight Network Interface"


DSpace dev: LNI & PKGers


DSpace dev: LNI & PKGers


What was the Problem…?

Content Providers wanted…Networked Access: Programmatically access the DSpace

object model, but not running on the DSpace server

Basic (Lightweight Network Interface (LNI))

[ Access, Deposit, Limited Admin. Modifications ]• Submit

• Retrieve Item w. Metadata

• Build own U/I

• Collection Mgt, Map Items

Advanced (possible future LNI (or other WS!) dev.)

[ Replace, Withdraw, Delete, 3rd Party, Workflow ]• Specialist metadata

• Automated metadata

• Quality control checks

• Format migration …

DSpace

Content Provider


What Is the LNI?The "Lightweight Network Interface" is:

• A remote API to control DSpace

• It's a separate module at the "application layer" — peer to

the Web U/I, can run in same Tomcat

• It's essentially WebDAV: Extensions to HTTP– A SOAP (with WSDL) equivalent is provided

• It's a mapping: — – a "complete and comprehensive" view of the DSpace

public object API…(the "business logic" layer)

– …as mapped to WebDAV semantics


How Do I Use It?

• LNI is not for direct, interactive access *

– e.g. HTTP browser

• Instead, the LNI is accessed via an application you develop

• Independence: Language, platform, machine– Code: Perl, C, Java, other WebDAV libraries– OSes: "Web Services" all about connecting different platforms

• Further "independence": choice of WebDAV or SOAP equivalent

• Server side? Minimal — install LNI & let it run. – Use existing DSpace authentication, e-People


WebDAV & DSpace:"Technical Introduction To"

a.k.a.

Micro-Intro-Tutorial-Review…


WebDAV & LNI

• WebDAV = "Distributed Authoring and Versioning" – using a Web server, rendered "DAV-able"

• WebDAV — 1999 spec. from the IETF (Internet Engineering Task Force) RFC 2518– http://webdav.org/specs/rfc2518.html

• LNI implementation of WebDAV: – Compliance Level 1 (no Versioning or Locking)– To date: No DELETE nor MKCOL; other limitations… – …room for future development! ;^)

• LNI: DSpace model mapped to WebDAV model …


DSpace: Model

• DSpace digital archive

• Differs from file system– Submit URI not

known– Workspace– Workflow (optional)– Metadata– Bundles– Handle to URL

resolution (& vice versa)


WebDAV: Model, Basics• Each

Resource named by a URL

• Properties are "name value" pairs

• Properties are extensible

• Property values can be found, and modified

DAV = "Distributed Authoring and Versioning"

URL


WebDAV: Model, Fuller• Resources

can be Collections or leaf nodes

• Collections can nest

• Everything can be discovered descending from the DAV server's root '/' resource


DSpace Mapped to WebDAV


DSpace Translated to WebDAV

• When you have a Handle (e.g. from other services like SRW/U, OAI-PMH), then use LNI "lookup" functionality to convert it to LNI WebDAV URL.

• To get a Handle (given a LNI URL), do PROPFIND on that URL and read the "Handle" property value.

LNI URL

Handle


DSpace Implemented as WebDAV

1. Properties– WebDAV

– Custom for DSpace

2. DSpace LNI URLs

3. WebDAV Methods


Properties: WebDAV

• DAV:creationdate• DAV:displayname• DAV:getcontentlength• DAV:getcontenttype• DAV:getlastmodified• DAV:resourcetype• …

http://www.webdav.org/specs/rfc2518.html

Akin to file system properties,with add'l. metadata


Properties: Custom DSpace (1 of 3)

• ALL DSpace Objects:– dspace:type

• "BITSTREAM", "ITEM", "COMMUNITY" etc.

– DAV:resourcetype • "COLLECTION" etc.

– dspace:displayname– dspace:current-user-privilege-set



• Community DSpace Objects– dspace:logo

– dspace:short_description

– dspace:side_bar_text

– dspace:handle

– …



• Bitstream DSpace Objects– DAV:getcontentlength = getSize()– dspace:format = getFormat().getID()– dspace:sequence_id = getSequenceID()– …

• Many more DSpace Objects (some 10)– With many more Properties (some 87)

http://wiki.dspace.org/LightweightNetworkInterface


URLs: DSpace LNI (1 of 3)

• Site– http://uni.edu/dspace/dav

• Community– http://uni.edu/dspace/dav/dso_1721.1$46

• Collection– http://uni.edu/dspace/dav/dso_1721.1$3549



• Item– http://uni.edu/dspace/dav/

dso_1721.1$3549/dso_1721.1$5543– http://uni.edu/dspace/dav/dso_1721.1$5543

• Bitstream– http://uni.edu/dspace/dav/

dso_1721.1$5543/bitstream_13– http://uni.edu/dspace/dav/

dso_1721.1$5543/bitstream_13.pdf



Nota Bene

The LNI URLs for DSpace objects are:

• only for use with the LNI.

– They are therefore not to be used:• as persistent URLs (e.g. bookmark, store for later use)

• as interactive URLs (e.g. browser)

• opaque — you must use the LNI to resolve to Handles

• Therefore please to not:• Roll your own LNI URLs to send

• Infer Handles & Etc. from LNI URLs received


WebDAV Methods in LNI• PROPFIND - very powerful request with multiple

functions:– list names of properties– return property values– recurse through 1 or more levels of a "collection" resource.

• PROPPATCH - changes value of or deletes properties on one resource.

• GET - retrieves contents of a resource• PUT - replace or add a new resource to a collection• COPY - map a resource ("Item") to a different Collection.

• As noted, equivalent SOAP methods are provided• Not Implemented: Locking, Versioning.


Initial Use: OpenCourseWare "Submit"


Current DSpace (pre-LNI)Content Provider, currently

• Batch import (& export)

• End user Web U/I

• dspace-admin Web U/I

• SRW, OAI-PMH

DSpace

Batch Importer

OCW Comm.

"Biology (7)"

"Physics (8)"

DSpace Object API

Web U/I Web, butNon Auto-

mated

Non Std.Non Web

Workflow Adm

Item


DSpace with LNIContent Provider using LNI:• LNI SOAP end-point URL

• LNI WebDAV URLs

• dspace-admin (partial)

• SRW, OAI-PMH– (Not replicated in LNI)

• Still available: – Batch import (& export)– End user Web U/I

DSpace

DSpaceLNI.class

OCW Comm.

"Biology (7)"

DSpace Object API

DSpace LNI WebDAV

- Web.- Stds.

- Batch-able.

"Physics (8)"

Item


Submission: OverviewOpenCourseWare Client

Needs 3 Values:

1. LNI SOAP end-point URL http://userid:[email protected]/dspace-lni/lni/DSpaceLNI

2. Handle for OCW Community = hdl:1721.1/125

3. Dept. Name (MIT "Course" Number) for Course to be submitted = "Physics (8)"

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

"Biology (7)"

DSpace Object API

DSpace LNI WebDAV

Lang. -Indep.

Platform -Indep.

"Physics (8)"

Item


Submission: 1. LookupRequest: OCW Comm. HANDLE

LNI lookup() = http://dspace.mit.edu/dspace-lni/dav/lookup/handle/1721.1/125

Response: OCW Comm. LNI URL http://dspace.mit.edu/dav/dso_1721.1$125

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

dso_1721.1$125

"Biology (7)"dso_1721.1$197

"Physics (8)"dso_1721.1$253

DSpace Object API

DSpace LNI WebDAV

Lookup


Submission: 2. CollectionsRequest: PROPFIND on LNI URL for children

DSpace Collections' "displaynames":<propfind>

<prop>

<DAV:displayname/>

</prop>

</propfind>

Response: MULTISTATUS<multiStatus>

<resource>

<href URI="dso_1721.1$197"/>

<displayname>Biology (7)</displayname>

</resource>

<resource>

<href URI="dso_1721.1$253"/>

<displayname>Physics (8)</displayname> </resource>...

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

dso_1721.1$125

"Biology (7)"dso_1721.1$197

"Physics (8)"dso_1721.1$253

DSpace Object API

DSpace LNI WebDAV

Iterate…


Submission: 3. Submit (PUT)Request: PUT to DSpace Collection LNI

URL, including specification of Packager Type (IMS-CP):

PUT /dspace/dav/dso_1721.1$253?package=OCW-IMSCP

....package contents in body...

Response: LNI URL for DSpace Item in "Location" Header:

HTTP/1.1 201 OK

Location: http://dspace.mit.edu/dspace/dav/dso_1721.1$836

....other headers....

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

dso_1721.1$125

"Physics (8)"dso_1721.1$253

DSpace Object API

DSpace LNI WebDAV

IngestItemdso_1721.1$836


Submission: 4. Item HandleRequest: PROPFIND on DSpace Item LNI

URL for the DSpace Handle:

<propfind>

<prop>

<dspace:handle/> </prop>

</propfind>

Response: Properties Requested

=== PROPERTIES Successfully returned:

dspace:handle = "hdl:1721.1/836"

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

dso_1721.1$125

"Physics (8)"dso_1721.1$253

DSpace Object API

DSpace LNI WebDAV

Reverse Lookup

Itemdso_1721.1$836

1721.1/836

OCW Content Mgt. System

Stores DSpace Handle

SQL Server Table: hdl:1721.1/836


Submission: SummaryIn this 4-step OpenCourseWare

submission example, we've seen:

• SOAP endpoint URL• DSpace Handles• DSpace LNI URLs• LNI Lookup() function• WebDAV Properties• WebDAV Methods• PROPFIND (LNI request)• MULTISTATUS (LNI response)• LNI "Reverse Lookup"• WebDAV Headers (Location;

Success/Fail)

DSpace

DSpaceLNI.class

OCW Comm.1721.1/125

"Physics (8)"1721.1/253

DSpace Object API

DSpace LNI WebDAV

Item1721.1/836


Subsequent Use: [ your use case here ]


LNI with Other Protocols

• Search– SRU/W { + LNI }

• Dublin Core metadata– OAI-PMH { + LNI }– METS Packager (DC only) { using LNI }


LNI Security Considerations

• Security– SSL (see wiki page)– Stateless– E-Person userid & pw with each invocation

• Authorization– E-Person– <DAV:current_user_privilege_set>– Area for further development!


Code Status• http://wiki.dspace.org/LightweightNetworkInterface_2fClient

Downloads

/!\ This code should not be considered stable. It is subject to change at any time, and may not work with the current development or released version of DSpace.

Use it for exploration and evaluation only. There are not yet any final plans for releasing it; much depends on the course of the AddOnMechanism project.

• Patch to DSpace 1.4alpha source (28-Mar-06)• New files to add to DSpace 1.4 alpha (28-Mar-06)• Pre-built client library JAR and WSDL -- not needed when building

from source.

http://wiki.dspace.org/LightweightNetworkInterface_2fClient


Future Development

• Other DSpace objects– Bitstream Format Registry– Metadata Registry …

• Other WebDAV methods– PUT (for Replacement)– DELETE– MKCOL …

• Other Use Cases, Needs–Basic, Advanced …

©

DSpace User Group Meeting

April 21, 2006 — Bergen, Norway

William Reilly, Larry Stone — MIT Libraries

2006

Lightweight Network Interface (LNI)

Questions, & Etc.

http://wiki.dspace.org/LightweightNetworkInterfacehttp://cwspace.mit.edu/docs/ProjectMgt/Reports/DSpaceUserGroupBergen2006/MIT-LNI-DSUG-2006.ppt.htm


Abstracthttp://dsug2006.uib.no/index.php?p=pres#william_reilly

The "Lightweight Network Interface" (LNI) is a work in progress to provide (yet) another way to gain networked access to the DSpace application-level API. Initially developed to support the requirement that "Web Services" be used on the CWSpace project (archiving MIT's OpenCourseWare (OCW) to DSpace), the ensuing vibrant (!) discussion regarding possible technical approaches (SOAP, RESTful, WebDAV, XML over HTTP) led us to define some high level goals for how this new interface ought to be shaped (e.g. platform-neutral; based on mature standards; readily extensible; work with (not replicate) existing remote APIs (SRW, OAI-PMH); comprehensive view of DSpace model; etc.).

Dubbed the "Lightweight" network interface, the intent has been to largely adopt the robust and proven protocols (and "verbs") of WebDAV and HTTP, and to establish a proposed mapping of WebDAV's Resource-centric view onto the DSpace object model. (Note that a set of SOAP methods has also been developed on top of the WebDAV functionality, such that in fact either approach is supported.) The details of the resulting proposed API were posted on a lengthy page to the DSpace Wiki; a "smoketest" Java client to the LNI was posted to the CWSpace Wiki (along with detailed instructions on how to work with SSL and the LNI).

This presentation introduces the technology, the API, the rationale for the approach, and a discussion of the mapping to the DSpace data model as well as example uses of the LNI to DSpace (e.g. submit; disseminate; copy to another collection; list collections; etc.).

Closely related DSpace development activity of the Plugin Manager, Packager Plugins, and Crosswalk Plugins will be addressed briefly as well, as these are used in conjunction with the LNI on the CWSpace project for submission (and dissemination) of IMS Content Packages of OCW content (courseware websites).

http://dsug2006.uib.no/index.php?p=pres#william_reilly

Documents

© DSpace User Group Meeting April 21, 2006 — Bergen, Norway William Reilly, Larry Stone — MIT Libraries 2006 v. 20060421_0945 Technical Introduction To