(ATS4-PLAT05) Accelrys Catalog: A Search Index for AEP

Ton van Daelen

Sr. Director, Platform Product Management

ton.vandaelen@accelrys.com

The information on the roadmap and future software development efforts are intended to outline general product direction and should not be relied on in making a purchasing decision.

Outline

• Search use cases

• Deployment architecture

• Solr search index

• Search syntax

• Administration

• Demo – Pro client UI

– Web UI

– Admin UI

Accelrys Catalog Vision

Pro Client – Pers Productivity

Search from Pro Client Examples that use the ‘Http Connector’ component PilotScript referencing ‘rsplit()’ Protocols using MAO data

Web User

Search from Web Port Recent protocols Popular protocols Protocols searching ‘Corporate Database”

Admin

Administer Generate index

Update frequency

Next steps: • Mail Users • Post report

Search Generate index

Update frequency

Canned reports Security issues

Bad design Bad documentation

Catalog

Xml log

The Size of the Challenge

• 10-100 Pro client users

• 50-1000 Web users

• 1-10 servers

• -> 5000+ protocols to be managed

Admin Use Cases …

• Bad design practices. Find protocols that: – have shortcuts as copies

– have saved checkpoints

– store passwords

– have components that are owner access only

– don’t have top level parameters (Web Port)

– have component with absolute file paths

• Bad documentation practices. Find protocols that: – don’t have help text (or default help)

– have components with missing captions

More Admin Use Cases

• General queries. Find protocols:

– with components that are deprecated (ad hoc / report)

– not run in n days

– not changed in n days

– by client type (pro client, web port, web service, Notebook, Isentris, …)

– with components with GUID x

– with SQL components with specific DSN

Introduction to Text Searching

• Unstructured or minimally-structured searches

– Think “Google”

– Keyword-based, non-relational; wide range of user input

– Based on lookups using pre-built word (token) indexes

Introduction to Text Searching (cont’d)

• Strategies to make searches more effective

– Stop word removal: and, the, by, for, of, …

– Stemming: startedstart, clusterscluster, etc.

– Synonym aliasing: oncology=cancer, MB=megabyte, etc. (supported but only minimally implemented; extensible)

– Language-specific document and query processing (support for Asian languages)

Apache Solr

• Open source text search server

• Part of Apache Software Foundation

• Uses and extends Lucene Java search library

• Hosted by a web application server

• http://lucene.apache.org/solr/

Solr: Under the Hood…

• Schema – XML specification of document fields and their types – Specifies how fields are tokenized and processed for indexing

• Solr config file – XML specification of query and result set processing rules – E.g. field weights

• Optional auxiliary files – Stop words, synonyms, protected words (unstemmed)

• Host application container – For AEP this is Tomcat

Tokenization and Filtering

• Tokenization options in Solr – Break on whitespace – Break on all non-letter characters – Break on case changes (for CamelCaseTokenization) – Break on character set changes (alphanum/ideographic/katakana)

• Additional filters – Lowercase filter: converts all characters to lowercase – CJK bigram filter: outputs adjacent character pairs for Asian languages – Stem filter: applies stemming rules (many language-specific variants)

• Field indexing and query processing use same tokenization – Better search results may be obtained by using slightly different analysis for indexing

versus querying

• See http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters

Customizing Solr

Mapping XMLDB to Solr Documents

• XML Database = Component/Protocol Database • For each item in XMLDB, an indexing protocol

– reads the item from the database – creates data record properties corresponding to Solr fields – joins in statistics from usage log – converts the data record to a JSON “document” – POSTs the document to Apache/Tomcat/Solr via HTTP

• Weighting – Protocol name and description have higher weight – Proximity has higher weight

• name: protocol or component name • path: location in XMLDB • type: “component” or “protocol” • parameters: names of parameters • author: user who created protocol/component • modifieddate: data protocol/component last changed • runcount: number of times protocol has been run • lastrun: date protocol was last run • uses: list of components used by protocol • alltext: composite field for keyword search

Some Catalog Fields (defined in schema)

Administration

• Configure servers

• Specify update interval

• Manual rebuild

Configuring Accelrys Catalog

• Configuration (admin portal)

– AEP servers to index

– Indexing schedule

• Note

– Indexer runs as scheduled service

– Indexing takes ~1 to 3 minutes per 1000 XMLDB items

– Two index copies; users can continue search while index is rebuilt

– Tomcat and Solr automatically installed and launched with Apache

Limitations

• Usage info can be incorrect because log file doesn’t store full protocol path (“Protocol 1” !)

• No indexing at runtime – it can take a day before index is updated

Demo

Example Queries

• MAO type:"Component“ – Any components referencing ‘MAO’

• uses:"Xml Reader" -author:Accelrys – Components/protocols that have an xml reader and are not

authored by Accelrys

• lastrun:[*TO NOW-6MONTH] – Last run at least six months prior

• runcount:0 – Never been run

• Accelrys Catalog is powerful search technology built into AEP

• Become a beta tester (beta-2)

• Plan for 9.0 upgrade now

• (ATS4-PLAT10) Planning your deployment for a 64 bit world

Summary