Upload
rosamond-cameron
View
217
Download
0
Tags:
Embed Size (px)
Citation preview
Connecting Applications with the NSDL
Technical Network Services
Annual Meeting 2008
NDR API
NCS
Presentation overview
• Introduction (Jim Blake)• Collaborative Tools
– NSDL Wiki (Katy Ginger)– ExpertVoices (Sharon Clark)– OnRamp (Lynette Rayle) – NSDL Collection System (Jonathan Ostwald)– Strand Map Service (John Weatherly)
• Backend Services– NDR API (Aaron Birkland) – Search Service (Jim Blake)
• Conclusion (Jim Blake)
Annual Meeting 2008
NDR API
NCS
TNS operations and services
• Operate infrastructure and library– Provide NCore platform for use by NSDL
grantees– Operate NSDL.org to support
dissemination of grantee collections
* Swiped out of context from yesterday's presentation by Lagoze, Krafft, Sumner and Wright
Annual Meeting 2008
NDR API
NCS
TNS operations and services – (paraphrased)
• Mix and Match software stack– "multi-level commoditization"
• Discoverability
Annual Meeting 2008
NDR API
NCS
The stack
Fedora
NDR API
NDR Toolkit
WordPressplug-ins
WordPress
Fedora
NDR API
NDR Toolkit
WordPressplug-ins
yourWordPress
Fedora
NDR API
NDR Toolkit
yourapplication your
application
Fedora
NDR API
Annual Meeting 2008
NDR API
NCS
Discoverability
• More than a repository, it's a brand name.
Annual Meeting 2008
NDR API
NCS
Discoverability
• More than a brand name, it's a web site with high page rank:
http://wiki.nsdl.org/index.php/How_To_Improve_Rankings_With_Blog_Posts
Annual Meeting 2008
NDR API
NCS
Discoverability
• Less obvious– Improved page rank
• All resources are reachable from the site maps at nsdl.org
– More hits• Site maps are crawled by Google and Yahoo.
Annual Meeting 2008
NDR API
NCS
Case study 1: NSDL.org
• Most of the NCore software was initially written to support NSDL.org
• Changing viewpoint:– from writing NCore for NSDL.org– to using NCore in NSDL.org
Annual Meeting 2008
NDR API
NCS
The NCore platform
Search Service
Strand Map
Service
NSDLCollection
System
NSDL Data Repository
NDR-API
Fedora: Native Interface
ExpertVoices
NSDLWiki
OAI/RSSIngest
OAIServer
OnRamp
Annual Meeting 2008
NDR API
NCS
Case study 2: IPY
• E-Zine
• Expert Voices blog
• Funded development for – OnRamp functionality– Expert Voices theme– etc.
Annual Meeting 2008
NDR API
NCS
e-zine via OnRamp
Annual Meeting 2008
NDR API
NCS
blog via Expert Voices
Annual Meeting 2008
NDR API
NCS
Case study 3:Curriculum Customization Service
• Digital library services w/o the NSDL Data Repository• Collection management with the NCS• Strand maps• DDS search (Digital Discovery System)
Annual Meeting 2008
NDR API
NCS
Concept structure via NCS
Annual Meeting 2008
NDR API
NCS
Education Standards via SMS
Annual Meeting 2008
NDR API
NCS
related resources via DDS
Annual Meeting 2008
NDR API
NCS
the "stack"
NCS SMS DDS
Annual Meeting 2008
NDR API
NCS
Presentation overview
• Introduction (Jim Blake)• Collaborative Tools
– NSDL Wiki (Katy Ginger)– ExpertVoices (Sharon Clark)– OnRamp (Lynette Rayle) – NSDL Collection System (Jonathan Ostwald)– Strand Map Service (John Weatherly)
• Backend Services– NDR API (Aaron Birkland) – Search Service (Jim Blake)
• Conclusion (Jim Blake)
Annual Meeting 2008
NDR API
NCS
NSDL Wiki Features
• Collaborative online environment to organize, create, & annotate resources
• Create links directly to NSDL resources on your Wiki pages• Group specific spaces (restricting to a group is possible)• Individual user accounts• Search extension enables linking to NSDL
– find resources to talk about– insert links to resources
• NDR Extension enables the addition of:– new resources to the library– referenced resources to the library– metadata about resources
Annual Meeting 2008
NDR API
NCS
NSDL Wiki Uses• NSDL’s Classic Articles in Context (CAC)
– Landmark works of scientific inquiry via dynamic learning spaces for teachers and learners
– Links to related NSDL teaching resources and bibliographies of contemporary articles that cite classic papers
– A venue for collaborative discussion and contribution directed by its users
• NCORE documentation• MatDL – soft matter wiki is a local implementation by
an NSDL Pathway
Annual Meeting 2008
NDR API
NCS
Create DiscoverDescribe
Identify AnnotateRelateContributeRepresent
Aggregate
Annual Meeting 2008
NDR API
NCS
Create DiscoverDescribe
Identify AnnotateRelateContributeRepresent
Aggregate
Annual Meeting 2008
NDR API
NCS
Create DiscoverDescribe
Identify AnnotateRelateContributeRepresent
Aggregate
Annual Meeting 2008
NDR API
NCS
Create DiscoverDescribe
Identify AnnotateRelateContributeRepresent
Aggregate
Annual Meeting 2008
NDR API
NCS
NDR API
Create DiscoverDescribe
Identify AnnotateRelateContributeRepresent
Aggregate
Annual Meeting 2008
NDR API
NCS
NSDL Wiki Technical• Based on MediaWiki 1.11 and 1.12
• Extensions are available to download from SourceForgehttps://sourceforge.net/project/showfiles.php?
group_id=204090&package_id=268147
• NSDL Wiki Documentation:https://wiki.nsdl.org/index.php/Community:NSDLWiki
Annual Meeting 2008
NDR API
NCS
Presentation overview
• Introduction (Jim Blake)• Collaborative Tools
– NSDL Wiki (Katy Ginger)– ExpertVoices (Sharon Clark)– OnRamp (Lynette Rayle) – NSDL Collection System (Jonathan Ostwald)– Strand Map Service (John Weatherly)
• Backend Services– NDR API (Aaron Birkland) – Search Service (Jim Blake)
• Conclusion (Jim Blake)
Annual Meeting 2008
NDR API
NCS
Expert Voices• A blogging tool with integrated NSDL search,
resource linking, and publication to NDR
• Moderated conversations are designed to tie NSDL resources to science news and add context for resources that enhance discovery, selection, and use
EV has two interrelated goals: • provide a community discussion/presentation forum• create context for science resources in the library
Expert Voices Gateway: http://expertvoices.nsdl.org
Annual Meeting 2008
NDR API
NCS
Features
• Wordpress Multi-User• Published entries can become NSDL resources• Blog entries can have linked references to NSDL
resources• Levels of authority (Roles)• Integrated with NSDL Shibboleth-based community sign-
on (Wordpress plug-in)• Blog(s) available as RSS feed(s)
Annual Meeting 2008
NDR API
NCS
Example Blog Uses
• Connecting current news to related library resources http://expertvoices.nsdl.org/connectingnews/
– Encourage teachers to use current events as teaching opportunities– Contextual links to NSDL resources– Includes related NSES standards – RSS from the blog is used on the MSP website
• Topic-based discussionhttp://expertvoices.nsdl.org/polar/
– Provide ideas for connecting science and literacy topics– Generates the News Section for Beyond Penguins and Polar Bears eZine
• NSDL community members as contributors http://expertvoices.nsdl.org/pats-picks/
– Link and review web resources for STEM educators
• Teacher/student discussion forum http://expertvoices.nsdl.org/cornell-info204/
– supplemental blog for a Cornell University course
Annual Meeting 2008
NDR API
NCS
NSDL Resource LinksProvides more information about a resource in the library
Annual Meeting 2008
NDR API
NCS
NSDL NDR ConnectionAdd a blog post and its associated resources to the NSDL
Annual Meeting 2008
NDR API
NCS
NSDL NDR ConnectionFind blog post and its referenced resources in NSDL search
Blog Post
Referenced Resource
Annual Meeting 2008
NDR API
NCS
Referenced Resources
<dct:references> in metadatahttp://ndr.nsdl.org/api/get/2200/20070314153845593T/format_nsdl_dc
Relationships in objectshttp://ndr.nsdl.org/api/get/2200/20070314153845593T
... <dct:references xsi:type="dct:URI">
http://www.polarbearsinternational.org/in-the-news/threatened-species/</dct:references> <dct:references xsi:type="dct:URI">
http://nsf.gov/news/special_reports/arctic/index.jsp</dct:references> ...
... <relationships> <nsdl:annotates>2200/20070314154210749T</nsdl:annotates> <nsdl:metadataProvidedBy>2200/20070314153843382T</nsdl:metadataProvidedBy> <nsdl:metadataFor>2200/20070314153844301T</nsdl:metadataFor></relationships>...
Annual Meeting 2008
NDR API
NCS
NCore WPMU Plugins
Plugins available to integrate other WPMU installations with NCore data repository
• NSDL Search Popup (nsdl-searchPopup): Enhances the URLs in your post with NSDL resource links.
• NSDL Site Admin (nsdl-admin): Provides admin tools to add blogs as resources to the NSDL Data Repository.
• NSDL NDR Popup (nsdl-ndrPopup): add a resource and its metadata to the NSDL Repository.
http://wiki.nsdl.org/index.php/Community:ExpertVoices
Annual Meeting 2008
NDR API
NCS
Presentation overview
• Introduction (Jim Blake)• Collaborative Tools
– NSDL Wiki (Katy Ginger)– ExpertVoices (Sharon Clark)– OnRamp (Lynette Rayle) – NSDL Collection System (Jonathan Ostwald)– Strand Map Service (John Weatherly)
• Backend Services– NDR API (Aaron Birkland) – Search Service (Jim Blake)
• Conclusion (Jim Blake)
Annual Meeting 2008
NDR API
NCS
Presentation overview
• Introduction (Jim Blake)• Collaborative Tools
– NSDL Wiki (Katy Ginger)– ExpertVoices (Sharon Clark)– OnRamp (Lynette Rayle) – NSDL Collection System (Jonathan Ostwald)– Strand Map Service (John Weatherly)
• Backend Services– NDR API (Aaron Birkland) – Search Service (Jim Blake)
• Conclusion (Jim Blake)
Annual Meeting 2008
NDR API
NCS
The NSDL Collection System (NCS)
• Web application for creating, editing, managing and developing collections.
• Communicates directly with NDSL data repository (NDR)
• Features include
– Metadata Editor
– Educational Standards– Collection Management– Web services
NCS Metadata Editor
Chunked records
Incrementalvalidation
Controlledvocabularies
RequiredFields
Bestpractice
s
Projectlogo
User Interface programmatically constructed from XSD schema
Annual Meeting 2008
NDR API
NCS
Collection Management
• Web-based access supports distributed cataloging and collection management
• Per-collection workflow support enables coordinated management processes
• Roles and permissions allows managed access to information and system functionality
Annual Meeting 2008
NDR API
NCS
Web Services
MetadataEditor
MetadataEditor
Search / Browse, Manage
Search / Browse, Manage
LocalRepository
LocalRepository
IndexIndex
PutRecord
PutRecord
SearchSearch
OAIOAI
StandardsSuggestorStandardsSuggestor
NDRNDR
NCS
Annual Meeting 2008
NDR API
NCS
Use Case: NSDL
• Collection Management– NCS manage collections metadata for each NSDL
collection– 20 out of 150 plus NSDL collections are directly
managed in the NDR via the NCS
• NSDL.org – Recommend a Resource inserts recommendations
into the NCS using the put record web service– Collections and Pathways views pull data from NCS
web service
Annual Meeting 2008
NDR API
NCS
Use Case: Curriculum Customization Service (CCS)
Learning application in support of Science Curriculum for the Denver Public School District
Curriculum objects are cataloged in the CCS NCS
Complex and dynamic displays pull data from CCS NCS via web service
Annual Meeting 2008
NDR API
NCS
NCS Technical• Java-based Application runs on Linux, PC, Mac
• NSDL Wiki Documentation• https://wiki.nsdl.org/index.php/Community:NCS
• Local Install• Download link is found on wiki page
• To create a collection in the NCS hosted by NSDL at Cornell University, contact NSDL at http://nsdl.org/about/contactus/
Annual Meeting 2008
NDR API
NCS
Presentation overview
• Introduction (Jim Blake)• Collaborative Tools
– NSDL Wiki (Katy Ginger)– ExpertVoices (Sharon Clark)– OnRamp (Lynette Rayle) – NSDL Collection System (Jonathan Ostwald)– Strand Map Service (John Weatherly)
• Backend Services– NDR API (Aaron Birkland) – Search Service (Jim Blake)
• Conclusion (Jim Blake)
Annual Meeting 2008
NDR API
NCS
The Strand Map Service• A service for creating interactive applications that enable
educators and learners to– Explore the relationships and interconnections between
important STEM concepts and learning goals– Locate resources and other curriculum components
supporting specific concepts and learning goals– Enhance science content knowledge, pedagogical content
knowledge (educators), and preparation for learning (learners)
– Design coherent instructional materials supporting recognized learning goals and informed by research on student learning (educators)
• Service APIs let developers embed the maps in Web sites and display educational resources and other information in the maps.
Annual Meeting 2008
NDR API
NCS
Content from Project 2061/AAAS
• Benchmarks: Describe what learners should know, or be able to do, at key stages in their education across the STEM disciplines
• Strand maps: Learning progression diagrams illustrating how student understanding changes over time
• Curriculum components: Materials and information to improve teachers’ science content knowledge, pedagogical content knowledge, and instructional practices
Annual Meeting 2008
NDR API
NCS
SMS Service
• Integrate the AAAS Atlas on-line
Annual Meeting 2008
NDR API
NCS
Technical Components of the Service
• Computational model of interconnected concepts– Depict how ideas change over time -> Learning Progressions and
Knowledge Maps
– Integrate curriculum components -> prior research, learning resources, standards, misconceptions, assessments
• Visualization algorithms incorporating AAAS graphical conventions
• JavaScript and REST service APIs for use by developers
Annual Meeting 2008
NDR API
NCS
SMS Architecture
Annual Meeting 2008
NDR API
NCS
JavaScript and REST APIs
Developers can:• Place the interactive Strand Maps in any Web
page• Display educational resources and other
information in the maps– Associate resources with learning goals
• Access and display map visualizations– Interactive JavaScript, SVG, PDF, JPG, PNG,
TIFF• Access and traverse the data graph: chapters,
maps, grades, strands, benchmarks– XML, JSON
APIs used to place interactive maps in web pages, embed content and
control map features
Benchmark details
Related benchmarks
Map Navigator
NSES standardsEmbedded
content
Embedded content
API is used to insert content in the bubble and tabs
Student misconceptions
Search and browse for maps
Use Case: Curriculum Customization Service• Enhances key concepts in the Denver Public Schools
curriculum with the benchmarks
Use Case: Educational Benchmarks Collection• NSDL collection containing each benchmark with related
educational resources
Use Case: NSDL Science literacy Maps• Connects NSDL resources with the benchmarks
Annual Meeting 2008
NDR API
NCS
Using the Strand Map Service
• Available for developers: – Web service APIs– Documentation– Working code examples
Go to:http://strandmaps.nsdl.org/cms1-2/docs/
Annual Meeting 2008
NDR API
NCS
Presentation overview
• Introduction (Jim Blake)• Collaborative Tools
– NSDL Wiki (Katy Ginger)– ExpertVoices (Sharon Clark)– OnRamp (Lynette Rayle) – NSDL Collection System (Jonathan Ostwald)– Strand Map Service (John Weatherly)
• Backend Services– NDR API (Aaron Birkland) – Search Service (Jim Blake)
• Conclusion (Jim Blake)
Annual Meeting 2008
NDR API
NCS
The Repository
• Central storage, organization, and access platform for data
• Serves as the “patch bay” through which NCore components share data and interoperate.
• Has an object model specializing in organizing data from multiple sources around shared resources.
• Access is through a model-centric web service API
Focus on this section is on how data flows through the repository: From an OAI harvest, blog post, or wiki page to Search discovery.
Annual Meeting 2008
NDR API
NCS
Into the RepositoryOAI Harvest
Service
Pathway or Project
OAI
Repository API
Repository
NCS
Annual Meeting 2008
NDR API
NCS
Out of the repository
OAI
Search
API
Strand maps
Nsdl.org, Wiki, EV
NCS
Annual Meeting 2008
NDR API
NCS
A Closer LookMetadata
Propertiesoai:id = abc123
Datastreams
Relationships
format_nsdl_dc
nsdl:metadataFor
nsdl:providedBy
Resource
MDProviderBlob of nsdl_dc xml, same as OAI metadata payload in the case of Harvest service
Required fields are populated by harvest service via API
Annual Meeting 2008
NDR API
NCS
A Closer LookMetadata
Propertiesoai:id = abc123ncs:status = swell
Datastreams
Relationships
format_nsdl_dcformat_ncs_itemformat_dcs_data
nsdl:metadataFor
nsdl:providedBy
Resource
MDProviderNCS, for example, stores its “full” metadata and audit trails in additional datastreams
Applications may introduce new elements for their own use in the NDR object
Annual Meeting 2008
NDR API
NCS
A Closer LookMetadata
Propertiesoai:id = abc123ncs:status = swell
Datastreams
Relationships
format_nsdl_dcformat_ncs_itemformat_dcs_data
nsdl:metadataFor
nsdl:providedBy
Resource
MDProvider
Content
The API allows read/write access to all parts of every object (subject to permissions and access rights).
...of course, elements of the object that aren’t part of the core model are just along for the ride, useful to applications that understand them, benign to others.
API
Annual Meeting 2008
NDR API
NCS
SearchingMetadata
Propertiesoai:id = abc123ncs:status = swell
Datastreams
Relationships
format_nsdl_dcformat_ncs_itemformat_dcs_data
nsdl:metadataFor
nsdl:providedBy
Resource
MDProvider
Content
The API allows searching over properties and datastreams of all objects
API search results are repository objects..
API
Annual Meeting 2008
NDR API
NCS
SearchingMetadata
Propertiesoai:id = abc123ncs:status = swell
Datastreams
Relationships
format_nsdl_dcformat_ncs_itemformat_dcs_data
nsdl:metadataFor
nsdl:providedBy
Resource
MDProvider
Content
The search service allows search over nsdl_dc metadata, and resource content (and a few other facts, such as identity of the resource’s collection).
Search results are, for the most part, all nsdl_dc metadata describing a single resource.
Search
Annual Meeting 2008
NDR API
NCS
Searching and Indexing
Metadata
Propertiesoai:id = abc123ncs:status = swell
Datastreams
Relationships
format_nsdl_dcformat_ncs_itemformat_dcs_data
nsdl:metadataFor
nsdl:providedBy
Search
API
Other datastream content may be searched by externally-deployed applications, such as DDS, or custom indexes.
Annual Meeting 2008
NDR API
NCS
Presentation overview
• Introduction (Jim Blake)• Collaborative Tools
– NSDL Wiki (Katy Ginger)– ExpertVoices (Sharon Clark)– OnRamp (Lynette Rayle) – NSDL Collection System (Jonathan Ostwald)– Strand Map Service (John Weatherly)
• Backend Services– NDR API (Aaron Birkland) – Search Service (Jim Blake)
• Conclusion (Jim Blake)
Annual Meeting 2008
NDR API
NCS
The NSDL Search Service
• A Lucene search index, containing over 2 million records, exposed as a web service.– One record for each resource in the repository
(active resources with metadata)• A Nutch database of crawled web pages, exposed as
a web service.– parsed text from over 800,000 pages
(restricted by robots.txt and non-text formats)• A GUI front-end webapp to assist with queries and to
display results.
Annual Meeting 2008
NDR API
NCS
Fully exposed Lucene query syntax
• http://ndrsearch.nsdl.org/search?q=frogs
• Default seach hits all text fields:q=frogs
• Search specific fieldsq=dc\:title:frogs
• Boolean combinationsq=frogs AND spaghetti NOT ellipsoids
• Proximity searchesq="deformed frogs"~5
Annual Meeting 2008
NDR API
NCS
The stack
Fedora
NDR API OAIserver
SearchService
Nutchdatabase
Searchfront end
Annual Meeting 2008
NDR API
NCS
Reaching behind the GUI
Annual Meeting 2008
NDR API
NCS
Reaching behind the GUI
Annual Meeting 2008
NDR API
NCS
Search Technical
• Lucene index:– 2,000,000 records in 20 gigabytes– updated nightly from the repository and the Nutch
database• Nutch database with parsed text from over 800,000
pages– crawl for several hours daily, complete every 30
days, approx.• NSDL Wiki documentation:
https://wiki.nsdl.org/index.php/Community:Search
Annual Meeting 2008
NDR API
NCS
Presentation overview
• Introduction (Jim Blake)• Collaborative Tools
– NSDL Wiki (Katy Ginger)– ExpertVoices (Sharon Clark)– OnRamp (Lynette Rayle) – NSDL Collection System (Jonathan Ostwald)– Strand Map Service (John Weatherly)
• Backend Services– NDR API (Aaron Birkland) – Search Service (Jim Blake)
• Conclusion (Jim Blake)
Annual Meeting 2008
NDR API
NCS
The challenge moving forward
1. How to best use what we have2. Where to go from here
Annual Meeting 2008
NDR API
NCS
How to use what we have?
• A change in mindset:– from "NCore is software developed in order to
implement the NSDL"– to "NSDL is a digital library implemented using
NCore software
Annual Meeting 2008
NDR API
NCS
How to use what we have?
• Generifying, modularizing, adding configurability– How can we make these tools more widely useful?
• Focus on all levels of documentation– Adopt a user-centric approach
• Eat our own dog food– Install from our documents– Use released NCore software on our production
servers.
Annual Meeting 2008
NDR API
NCS
Where to go from here?
• Building the roadmap• Are you our next use case?