Converting Local Portal Data to MySpace Anne L. Highsmith, Consortia Systems Coordinator & Prabha Acharya, Programmer Analyst II Texas A&M University

Converting Local Portal Data to MySpace

Anne L. Highsmith, Consortia Systems Coordinator &

Prabha Acharya, Programmer Analyst IITexas A&M University

Converting Local Portal Data to MySpace ELUNA 2007

Points to Cover

Background Data Conversion Issues Testing on a Cloned Database Conversion Process Results and Future Plans


Background (1)

TAMU had a local Electronic Resources Locater (ERL) function to list ejournals and databases.

Users searched ERL and clicked on links to access eresources

Users could store the links in their MyPortal record on the library’s website


Background (2)

Metalib went into production at TAMU in January 2006 as Search Now.

TAMU had discussions with Ex Libris staff as to how MyPortal links could be converted to Metalib First proposal, for Ex Libris staff to do a major

conversion, not feasible Decided to wait for Metalib 4.0 and X-Portal

APIs to store data to MySpace


Rude awakening!

Fall of 2006, Ex Libris staff informed TAMU there would be no API to store to MySpace

Alternatives Margaritas? Convert them ourselves?


Conversion options

Assumed conversion would have to be: User-initiated Title by title

Discovered a way do the conversion directly, without involving the end user


Data Conversion Issues

Patrons Journals Databases


Data Conversion Issues - Patrons

How to match the MyPortal patron with the Metalib user? MyPortal patrons identified by uin (University

ID Number) Our Metalib implementation uses uin as user

login Needed to match the uin with the

z312_source_id


Data Conversion Issues - Journals

How to match MyPortal journals with Metalib? Basic journal data stored in z11 table

Wrote Perl script (journals.pl) to extract journal identifier and issn from z11

Used MS Access to match MyPortal data, keyed by uin & issn, with Metalib data

User journal data stored in z312_jnl_001 Wrote Perl script (journals_by_patron.pl) to

extract z312_source_id (uin) and journal identifier from Metalib user record in z312 table


Data Conversion Issues - Databases

How to match MyPortal databases with Metalib? During creation of Metalib IRDS, stored the

ERL key in the IRD cataloger note Loaded IRD extract into MS Access

Matched Metalib user extract with IRD extract Basic database data stored in z122 table

Wrote Perl script (databases_by_patron.pl) to extract z312_source_id (uin) and Metalib identifier (TEX #)


Data Conversion Issues - Final

As a result of matching ERL data and Metalib data, created a file that contained 1 or 2 entries for each user whose data had to be converted: Tab-separated record beginning with uin and

list of database id numbers Tab-separated record beginning with uin and

list of journal id numbers


Testing on a Cloned Database


Testing – Cloning the database (1)

TAMU has a test Metalib DB on separate server

Cloned production DB to test DB so could test in a realistic environment mpsync - documented in Appendix D in the

System Administrators Guide & Knowledge Base entry #1240.

exp_current_library & imp_current_library, documented in 7.8 in the SysAdmin Guide

Further doc – “Working with Test/Failover Installations of MetaLib”


Cloning – Export from Production

Created files on prod for export to test Run mpsync to copy IRDs

TAMU has approx. 4500 IRDS Elapsed time – 58 sec.

Run export current library for vir00 TAMU has 32,000+ user records Elapsed time – 1 hr.

Run export current library for jnl01 TAMU has approx. 44,000 journal records Elapsed time – 2:25 min.


Cloning – Import into Test

Import IRDS created by mpsync Elapsed time – 3 hrs. 688 new records added Matched existing keys. Appeared to create

some duplicates Import User Data

Elapsed time – 40 min. Drops & re-adds tables

Import Journal Data Elapsed time – 3 min. Drops & re-adds tables


Conversion Process

Quality control File preparation Loading new patrons Convert databases and ejournals


Conversion – Quality Control

Wrote a Perl script (pre_migration.pl) to collect stats and set up QC measures. For each user it reports: Users who don’t have Metalib record # of databases and journals already in Metalib # of databases and journals to convert # of users that have more than maximum

number of databases or ejournals


Conversion – File Preparation

Testing demonstrated that It takes about 3 seconds per resource to be migrated.

With about 13000 resources that would be about 10 hours

So, wrote Perl script (breakup_myportal_file.pl) to break the input file into several files of about 250 records each.


Conversion – Loading new patrons

Since TAMU uses external authentication rather than patron load, some older MyPortal patrons not in Metalib database

Using file created during QC stage, create_xml_patron_load.pl formats user records for Metalib patron load utility Set expiry date to 1 year in future Based on a review of records created on that

date, approx. 65 have since logged in.


Conversion - Convert databases and ejournals (1) Run databases_by_patron.pl to get

current lists from user records for comparison


Conversion - Convert databases and ejournals (2) Using save_to_bookbag.cgi, for each

user Log in to Metalib Send request to Metalib pds to get pds handle Send request to Metalib, using just obtained

pds handle, to get session id Construct url of the form

http://metalibhost.edu/V/sessionid?pds_handle=pdshandle&func=find-db-add-res&resource=resourceid


Conversion - Convert databases and ejournals (3) Set session id in cookie For each database

Check to see if it duplicates db already stored If not, store db to user’s clipboard

For each journal Store to user’s record


Conversion - Convert databases and ejournals (4) Do QC to make sure that all of the

appropriate databases and ejournals are stored

Exit the session Error logging revealed that for some users, 1

database or ejournal was not stored – these were re-run as an added input file


Final results

Processing all files took most of the week, off and on

Each file, limited to 250 records, took about 1 hour to process

Only post-production problem had nothing to do with conversion – ezproxy error.

Once users have logged into library portal, can now go directly to My Databases or My Ejournals list within Metalib


Future plans

Conversion of Voyager My Bookbag to e-shelf


Contact & download info

Anne L. Highsmith

Consortia Systems Coordinator

Texas A&M University

[email protected]

URL to download perl code, cgi, etc.http://libraryasp.tamu.edu/users/ahighsmi/

presentations/save_to_bookbag.zip

Documents

Converting Local Portal Data to MySpace Anne L. Highsmith, Consortia Systems Coordinator & Prabha Acharya, Programmer Analyst II Texas A&M University