Embed Size (px)
Citation preview
A New Kind of Catalog
Charley PennellPrincipal Cataloger for Metadata
North Carolina State University
North Carolina Library Association 2007
Where is this talk headed?
Local motivation National trends What is Endeca? Features Does Endeca work? Where are we going from here? Where is everybody else going?
Why a new catalog?What was wrong with the old one?
A little TRLN catalog primer
TRLN libraries (Duke, NCCU, NCSU, UNC-CH) jointly develop and maintain BIS, 1985-1992
DRA implemented for catalog (UNC & Duke continue Acq/Serials modules), 1991-1993
No integrated keyword/browse capability, 1993-1999
Web2 catalog implemented, 1999- Sirsi & DRA “merge” in 2002; Taos DOA
A little TRLN catalog primer 2
NCSU & NCCU to Unicorn; Duke to Aleph; UNC-CH to Millenium, 2003-2004
Sirsi/Dynix merger, 2004: vendor focus shifts (even more) toward school/public market
While agreeing to continue to support Web2, S/D increasingly looking to merge all product catalogs into single interface
What was the catalog lacking?
Simplicity: a simple, hopefully uncluttered interface Interactivity: ways to interact with results to get
better results Forgiveness: just fix my typos and case errors,
don’t make me feel stupid! Response time: always Real-time sorting: the limit is how many?!! Relevance ranking: as if! Web services: use the Web to repurpose data,
enable mash-ups, add-ons & improvements
Which interface is ready for immediate use?
0
10
20
30
40
50
60
70
80
90
1stQtr
2ndQtr
3rdQtr
4thQtr
East
West
North
So, why DOES everyone think that the catalog sucks stinks?
"Most integrated library systems, as they are currently configured and used, should be removed from public view."
- Roy Tennant, OCLC
The old model
The integrated library system
Historically, the ILS developed as an inventory control system for use by library staff only
First library automation systems (Plessey, CLSI, Geac, Innovative) were designed around circulation or acquisitions functions
Interaction time was calibrated to the slow pace of backroom work where the audience was basically captive
Staff focus on known-item searching, not resource discovery
The catalog as part of the ILS
The first integrated OPACs were veneers on top of existing inventory management systems—patrons & staff competed for system resources! They still do!
First OPACs allowed for browse only; early keyword searching restricted to certain fields (A/T/S) only
Libraries with no IT support were stuck with what their vendor provided and the enhancement process for improvements
Libraries with IT support created their own systems: BIS, NOTIS, Clarement Colleges, Georgetown, PALS, DOBIS/LIBIS
The state of the ILS in 2007 Customer demands for increasing
functionality in a marketplace with little $$ to spend has reduced the ILS vendor pool through mergers and buyouts
New functionality (multi-search, ERMS, E-Ref, ILL, etc.) increasingly being met by stand-alone and third party applications
Increasing competition from open source (Koha, Evergreen, Scriblio, LibraryThing) and e-commerce
Q: Is our dogged adherence to MARC the only thing keeping the remaining ILS vendors afloat?
The state of the catalog 2007 Library users’ search expectations have been
conditioned by interactions with commercial Websites and Google, with which Libraries can barely afford to compete, but must
Libraries are becoming increasingly virtual as users interact with us online (e-resources, Second Life)
User expectations for online experiences are more interactive, instantaneous, and inviting
Perhaps most importantly…
The information resources represented in the catalog represent a shrinking percentage of what end users need or want
Calhoun’s Aristotelian vs. Copernican views of the catalog
What do users want from the OPAC?
Make subject searching in online catalogs easier using post-Boolean probabilistic searching with automatic spelling correction, term weighting, intelligent stemming, relevance feedback, and output ranking
Streamline users' book selection decisions at the catalog by adding tables of contents and back-of-the-book indexes to cataloging (i.e., metadata) records
Reduce the many failed subject searches by expanding the online catalog with full texts—journal and newspaper articles, encyclopedias, dissertations, government documents, etc. Increase finding strategies in online catalogs through the library classification-- Markey, Karen (2007). “The online library catalog: Paradise lost and paradise regained”, D-Lib Magazine, 13(1/2).
“Many researchers express surprise at the brevity (from one to three words) of the queries people submit to online systems. Belkin tells why so few words make up their queries, "Precisely because of the inquirer's lack of knowledge about a problem area, it is impossible to specify what would resolve it." For Belkin, the saving grace is the inquirer's ability to recognize what he or she wants or does not want during the course of the search. Therein lies an important solution to the problem—information systems that report results for easy eyeballing and instantaneous recognition of relevant possibilities.” – Karen Markey
What is an Endeca?
A software company based in Cambridge, MA
A search and information access technology provider for a number of major e-commerce websites
Developers of the Endeca Information Access Platform
Endeca features
Commercial-strength search/sort speeds
Site customizable relevance ranking
Faceted browse True browsing (LC
classification) Spell-checking ”Did you mean?” Automatic word
stemming
Endeca at NCSU Libraries Went live in January 2006 Works with a text version
of a daily snapshot of Libraries’ MARC & other metadata
Used to improve the discovery portion of the library catalog
Interoperates with ILS for holdings, current availability status
Web2 interface still present for known item & authority searching
Implementation timeline
License / negotiation: Spring 2005 Acquire: Summer 2005 Implementation:
August 2005 : vendor training September 2005 : finalize requirements October 2005 – January 2006 : design and
development January 12, 2006 : go-live date
Widen to TRLN partners: Winter 2008
Implementation Team
Implementation Team brought together from IT, DLI, Cataloging, Collections, Reference, Circulation
Worked on indexing, UI, usability testing, etc. Areas of contention
Number of initial search boxes (1 or 2) Order, grouping of facets Placement of classification hierarchies, breadcrumbs Use of “search” and “browse” on tabs
Visualization aided by Tito’s wireframes
8th (and Final) Revision: Aggregate holdings information by library.
Reduces complexity of continuing and online resources.
Brief view vs. Full view gives user choice about displaying holdings.
NCSU Endeca features
Facets
Results
Call # browse
Breadcrumbs
Features we started with
Faceted browse Availability facet Breadcrumbs Spell check / Did you mean Hierarchical subject browse based on LCC Fuzzy link to live Web2 data New book browse for titles added in last
week only
Features that we’ve added
New book browse based on relative date (last week, last month, last three months)
RSS feeds based on user results “Search within” results Send search to TRLN partners Static unique link to live Web2 data
Relevance ranking
Based on locally customizable algorithm: Most relevant: query exactly as entered For multi-term searches: phrase match Field match
title match more relevant than notes match Other factors:
number of fields matched weighted frequency static ordering (publication date, circulation stats)
Faceting at the NCSU Libraries
Follows on what we have learned from the commercial Web search model
Mines metadata already available via MARC record, local class number, ILS item categories, circ status, and date stamping
Required massive clean-up of 6xx subdivisions Allows both pre- and post-coordinate limits Uses table mapping to enable drilling down through
call number results
Facet refinements
Availability Author Library Format Language
New(ness)
LC Classification Subject: Topic Subject: Genre Subject: Region Subject: Era
A single facet need not represent data from a single field
Single Unicorn item types (Book, Kit, Manuscript, Map, Data set)
Multiple Unicorn item types (Audio, Microform, Thesis/Dissertation, Software & Multimedia, Videos)
Leader byte 07 (Bib lvl): Journal, Magazine
Library (Online)
Ranking facet results by number of postings makes sense in a short list, but not in a long list
The author facet is less useful in some types of searches …
… than others!
Technical overview
Raw MARC data
NCSU exports and reformats
Flat text files
Data Foundr
yParse text
files Indices
MDEX Engine
NCSU Web Application
HTTP
HTTP
Information Access Platform
MARC ingest
MARC flat text file(s) for ingest by Endeca. Transformation accomplished with MARC4J. Opportunity to manipulate data on the back-end.
Transformed data
The end result…
Video
Other Endeca library catalogs
Phoenix Public Library: http://www.phoenixpubliclibrary.org/
McMaster University: http://libcat.mcmaster.ca
Florida Center for Library Automation http://catalog.fcla.edu/
Individual Florida universities http://fs.catalog.fcla.edu/, etc.
Does Endeca work?
Problems: authority control
Endeca is a keyword search engine; “browse” can only be effected using sort options
There is no authority control within Endeca itself, rather it relies on AC within ILS
To make use of available metadata, subjects were split along subdivisions. Authors were not
Talks were held with the vendor to explain the potential for drawing on authority x-refs to collocate searches
Problems: subject context
Problems with wrong delimiter values (esp. $v) Problems maintaining context in atomized LCSH
One-way relationships English language$vDictionaries$xSpanish
Chronological headings devoid of geographic context Cuba$xHistory$yRevolution, 1959
Phrase headings expressed in multiple subdivisions Prisoners$xAbuse of
Problems: subject hierarchies
Chronological hierarchy not built into $y “19th century” does not subsume 1800-1809, 1801-1861, 1809-1817, 1815-
1861, 1817-1825, Civil War, 1861-1865, etc. Geological periods exist as text only (Ordovician, Pleistocene, etc.)
Some chronological headings are expressed as text in 650$a Middle Ages Nineteen sixties
Geographic hierarchy not consistent between 651 and 650 $zNorth Carolina$zRaleigh $aRaleigh (N.C.)
BT/NT/RT relationships from authority file lacking
Some potential solutions
Search behavior education FAST (Faceted Application of Subject
Terminology) Web2 x-refs to redirect searches to Endeca Combining $z hierarchies Hierarchy lists
What do our users think?
“The new Endeca system is incredible. It would be difficult to exaggerate how much better it is than our old online card catalog (and therefore that of most other universities). I've found myself searching the catalog just for fun, whereas before it was a chore to find what I needed.”
- NCSU Undergrad, Statistics
“The new library catalog search features are a big improvement over the old system. Not only is the search extremely fast, but seemingly it's much more intelligent as well.”
- NCSU faculty, Psychology
Usability testing
Task Difficulty: Old Catalog
Easy43%
Medium12%
Hard22%
Failed23%
Task Difficulty: New Catalog
Easy59%
Medium12%
Hard7%
Failed22%
Usability testingAverage Task Duration:
Old vs New Catalog00:00.0 00:43.2 01:26.4 02:09.6 02:52.8 03:36.0
Task 1
Task 2
Task 3
Task 4
Task 5
Task 6
Task 7
Task 8
Task 9
Task 10
Old Catalog
New Catalog
Usage statistics
Searches by Field Type: J uly 06 - J an 07
0
60,000
120,000
180,000
240,000
300,000
360,000
420,000
Keyword(default)
I SBN Title Author Subject Multi-Field
Search and Navigation
Search 67%Navigation 8%
Search -> Navigation 25%
Newness wearing off?
March ‘06 - May ‘06
July ‘06-January ‘07
Requests by Search Type
Search -> Navigation
29%
Navigation 20%
Search 51%
Navigation by Dimensions
Subject: Topic26%
Availability2%
LC Classification21%
Format10%
New10%
Library10%
Subject: Genre6%
Subject: Era2% Language
3%
Subject: Region4%
Author6%
July 06 – Jan 07
Navigation by Dimension (most used)
0 20,000 40,000 60,000 80,000 100,000 120,000 140,000
Availability
Subject: Era
Language
Subject: Region
Author
Subject: Genre
Library
New
Format
LC Classification
Subject: Topic
Requests
July 06 – Jan 07
Navigation by Dimension (order of UI presentation)
32,650
16,009
12,257
22,818
54,476
57,667
34,096
145,589
120,644
9,286
0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 160,000
Author
Language
Subject: Era
Subject: Region
Library
Format
Subject: Genre
Subject: Topic
LC Classification
Availability
Requests
July 06 – Jan 07
Where are we going from here?
Future directions
Additional hierarchies (geographic names, dates) Make use of NAF, SAF, particularly cross-reference
structure Massage underlying metadata
Addition of Date Cataloged – Done! Addition of LC Class numbers to e-resources – Done! FRBR work numbers/records? – Tested! FAST headings?
Accommodation of true browse for all indexes
Future opportunities Expanding the scope of the implementation to the
10M records in TRLN (Duke, NCCU, NCSU, UNC-Chapel Hill)
Enrich catalog through external web services: book jackets, reviews, TOC, etc. – Amazon, OCLC.
LibraryThing, Bowker Syndetics Build use-case based cross-application shopping
cart functionality Integrate catalog w/other tools through web services
—“Free the Data”
Web services…
Mobile device searching
Where is everybody else going?
Catalogs detaching themselves from ILS Detached data lends itself to experimentation Don’t have to throw out baby with bathwater when
better interfaces come out Data itself safe and secure in ILS
MARC becoming superfluous; MARC’s granularity NOT!
Social interaction: reviews, folksonomic tags, ratings
Phoenix Public Library on Endeca
III’s new faceted catalog, Encore
ExLibris Primo at Vanderbilt
Athens County, OH—Koha Zoom open source
Georgia PINES—Evergreen open source
Casey Bisson’s Scriblio
Danbury Public powered by LibraryThing
OCLC WorldCat Local at UW
Thanks for listening!
Charley Pennell
Principal Cataloger for Metadata
NCSU Libraries
North Carolina State University
Raleigh, NC 27695-7111
More info at: http://www.lib.ncsu.edu/endeca/
