Semantic Indexing and Search for Content Management Systems with Apache Stanbol

Semantic Indexing and Search for Content

Management Systems

Suat Gönül, SRDC suat@apache.org

A. Anil Sinaci, SRDC sinaci@apache.org

prepared by

presented by

1 / 36

About me

• PhD student at

• Senior Software Engineer at

• FP7 Projects:

2 / 36

Agenda

Stanbol Overview

Main scenario

Two-layered architecture

Storage

Revision management

Indexing

Application of the architecture

3 / 36

Stanbol Overview

Semantic Services

Traditional CMS

Different means of interaction

4 / 36

Stanbol Design

RESTful API

OSGi Services

5 / 36

Semantic Indexing & Search Scenario

Enhanced content

Plain content

6 / 36

CMS Structure

Plain content

7 / 36

Configuring Index

8 / 36

Domain Specific Enhancements

… <rdf:Description rdf:about="urn:enhancement-810e14ac-af3d-3310-b9bb- 74233df94ef5"> <j.6:extracted-from rdf:resource="urn:content-item-sha1-d20174d4d24878f60b02bd8b03acc677404a7998"/> <j.6:confidence rdf:datatype="http://www.w3.org/2001/XMLSchema#double">0.8</j.6:confidence> <rdf:type rdf:resource="http://fise.iks-project.eu/ontology/EntityAnnotation"/> <rdf:type rdf:resource="http://fise.iks-project.eu/ontology/Enhancement"/> <j.6:entity-label>Heart disease</j.6:entity-label> <j.2:created rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2012-04-18T05:34:27.329Z</j.2:created> <j.2:creator rdf:datatype="http://www.w3.org/2001/XMLSchema#string">org.apache.stanbol. enhancer.engines.keywordextraction.engine.KeywordLinkingEngine</j.2:creator> <j.6:entity-reference rdf:resource="http://purl.bioontology.org/ontology/SNOMEDCT/Ins_161639008"/> <j.2:relation rdf:resource="urn:enhancement-a4f0b4e7-4992-bef2-3aa3-7b8010d0a56a"/> </rdf:Description> …

9 / 36

Semantic Indexing & Search

Enhanced content

10 / 36

Two-layered Architecture

11 / 36

Storage Layer

Plain content

Content check

CMS Enhanced content

12 / 36

Storage Layer

Separation of Storage layer provides

Storage of items having different granularities

ContentItem, Entity

Opportunity of integration with standard based CMSes

JCR, CMIS

13 / 36

Revision Management

IndexingSource keeps track of revisions of stored items

When an item is created, create also a corresponding revision

When an item is updated or deleted increase the corresponding, existing revision

Used while synchronizing the Storage and Indexing layers

14 / 36

Revision Management - ChangeSet

IndexingSource provides the list of changed items based on a given revision through the ChangeSet structure

getChanges(revision:long, batchSize:int):ChangeSet

ChangeSet keeps IDs of items not items themselves

15 / 36

Revision Management - ChangeSet

An example: After step 1:

1 : urn:contentItem.1 //added 1 : urn:contentItem.2 //added 1 : urn:contentItem.3 //added After step 2: 1 : urn:contentItem.1 2 : urn:contentItem.2 //updated 2 : urn:contentItem.3 //removed After step 3: 1 : urn:contentItem.1 2 : urn:contentItem.2 3 : urn:contentItem.3 //added 3 : urn:contentItem.4 //added

getChanges(0, …) =>

{urn:contentitem.1,

urn:contentitem.2, urn:contentitem.3, urn:contentitem.4}

getChanges(2, …) =>

{urn:contentitem.3, urn:contentitem.4}

16 / 36

Revision Management - Epoch

IndexingSource keeps an epoch indicating a common revision for all stored items

A change of the epoch means that whole data source of the IndexingSource has changed

For instance, data source of the IndexingSource is replaced with the new DBPedia dump

Revision registry is repopulated for the new items

17 / 36

Indexing Layer

18 / 36

Indexing Layer – Semantic Index

Poll changes Notify changes

Incremental indexing

Persists the revision of latest indexed item and corresponding epoch

Requests changes as of the latest persisted revision

In case of epoch change, changes are requested from scratch

19 / 36

Indexing Layer – Semantic Index

State of the index

UNINIT

INDEXING

ACTIVE

REINDEXING

20 / 36

OSGi Based Design

21 / 36

Application of Two-layered Approach

The first step is interaction with already deployed CMS

JCRIndexingSource connects to the CMS using the standard JCR API

Checks the content in the CMS

22 / 36

The JCRIndexingSource wraps a CMS document as a ContentItem to be processed by Stanbol.

… Heart valves keep blood flowing in a one-way direction by opening to let the proper amount of blood flow through and then closing to prevent backflow. From the right ventricle, blood is pumped through another valve and then into the lungs, where it receives oxygen. …

23 / 36

Corresponding enhanced ContentItem

… Heart valves keep blood flowing in a one-way direction by opening to let the proper amount of blood flow through and then closing to prevent backflow. From the right ventricle, blood is pumped through another valve and then into the lungs, where it receives oxygen …

{jcr:lastModifiedBy=admin, jcr:lastModified=2012-06-08T11:16:38,jcr:mimeType=text/plain,…}

{"@subject": "urn:enhancement-33ccf589-9982-3cba-4c88-8281b73e1096", "@type": ["Enhancement", "TextAnnotation"], "confidence": 0.6, "created": "2012-06-08T09:09:10.474Z", "creator": "org.apache.stanbol.enhancer.engines.keywordextraction.engine.KeywordLinkingEngine", "end": 1529, "extracted-from": "urn:content-item-b38798a5-9856-485b-880b-6f8a6a0b9eb7", "selected-text": { "@literal": "of blood flow", "@language": "en" }, "selection-context": { "@literal": "Heart valves keep blood flowing in a one-way... "@language": "en" },},...

24 / 36

In the second step, LDPath is used to configure an index configuration.

25 / 36

In the second step, LDPath is used to configure an index configuration.

26 / 36

LDPath

Using the previous LDPath, a Solr based SemanticIndex will be configured.

Each line in the LDPath instance provides configuration of an index field

snomed_has_finding_site = umls-skos:has_finding_site :: xsd:string;

Index field name

RDF path to be processed for the current RDF resource

Field configurations.

27 / 36

Collection additional information for recognized named entities

… <rdf:Description rdf:about="urn:enhancement-810e14ac-af3d-3310-b9bb- 74233df94ef5"> <j.6:extracted-from rdf:resource="urn:content-item-sha1-d20174d4d24878f60b02bd8b03acc677404a7998"/> <j.6:confidence rdf:datatype="http://www.w3.org/2001/XMLSchema#double">0.8</j.6:confidence> <rdf:type rdf:resource="http://fise.iks-project.eu/ontology/EntityAnnotation"/> <rdf:type rdf:resource="http://fise.iks-project.eu/ontology/Enhancement"/> <j.6:entity-label>Heart disease</j.6:entity-label> <j.2:created rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2012-04-18T05:34:27.329Z</j.2:created> <j.2:creator rdf:datatype="http://www.w3.org/2001/XMLSchema#string">org.apache.stanbol. enhancer.engines.keywordextraction.engine.KeywordLinkingEngine</j.2:creator> <j.6:entity-reference rdf:resource="http://purl.bioontology.org/ontology/SNOMEDCT/Ins_161639008"/> <j.2:relation rdf:resource="urn:enhancement-a4f0b4e7-4992-bef2-3aa3-7b8010d0a56a"/> </rdf:Description> …

28 / 36

The same LDPath, which was used to configure the index, is used to get additional information for recognized named entities

<rdf:Description rdf:about="http://purl.bioontology.org/ontology/SNOMEDCT/Ins_161639008"> <j.3:ctv3id>14O7.</j.3:ctv3id> <j.3:UMLS_CUI>C0580320</j.3:UMLS_CUI> <j.0:notation>161639008</j.0:notation> <j.3:TUI>T033</j.3:TUI> <j.3:isa>At risk of disease</j.3:isa> <j.1:label>Heart disease</j.1:label> <j.1:semanticType>Finding</j.1:semanticType> <j.3:isPrimaryTopicOf rdf:resource="http://purl.bioontology.org/ontology/SNOMEDCT/Ins_161639008.meta"/> </rdf:Description>

snomed_isa = umls-skos:isa :: xsd:string;

snomed_semantic_type = snomed:semanticType :: xsd:string;

29 / 36

All semantically related knowledge is indexed along with the actual content in the customized index

The underlying SOLR index can be queried directly through its RESTful services

30 / 36

Faceted search interface over the index

31 / 36

Application of Two-layered Approach All facets

32 / 36

33 / 36

34 / 36

35 / 36

Thank you for listening!

Questions?

36 / 36

Semantic Indexing and Search for Content Management Systems with Apache Stanbol

Investor Relations

INDEXING* INDEXING*

Apache Stanbol - the community vision

Comparing Ontotext KIM and Apache Stanbol

Apache Solr: Indexing and Searching (Draft last modified 10/26

Marwan Al-Namari Hassan Al-Mathami. Indexing What is Indexing? Indexing is a mechanisms. Why we need to use Indexing? We used indexing to speed up access

Near Real Time Indexing Kafka Message to Apache Blur using

Indexing Techniques Indexing Techniques in Warehousing …H.Haddouti/UB_Tree.pdf · Indexing Techniques Indexing Techniques in ... Processing Relational OLAP Queries with UB-

FamilySearch Indexing: Indexing

Apache Solr: Indexing and Searching 1. 2. 3

Indexing 3-dimensional trajectories: Apache Spark and Cassandra integration

Automated Linking Data with - Apache Stanbol · 2012-09-20 · Entity-Linking + Disambiguation (e.g. by using Solr MLT) Disambiguation of already linked Entities More Domain speciﬁc

BROWSING THE PDS IMAGE ARCHIVE WITH THE IMAGING ATLAS … · Apache Solr (“Solr”) [2], an open-source indexing plat-form built upon another Apache product, Lucene [3]. The PDS

Greenplum GPText 1.2.0.0 User’s Guide · 2020-07-20 · Data Loading and Indexing Workflow.....5 Querying Data Workflow ... parallel-processing database server with Apache Solr

Use Custom Vocabularies with the Stanbol …...Use Custom Vocabularies with the Stanbol Enhancer Rupert Westenthaler, Salzburg Research, Austria Mittwoch, 07. November 12 About Me

English Wikipedia On Hadoop Cluster - Virginia Tech...Loading this dataset onto a system, such as an Apache Hadoop cluster, and indexing it into Apache Solr, would allow researchers

Storing, Indexing and Querying Large Provenance Data Sets as RDF Graphs in Apache HBase Artem Chebotko Joint work with John Abraham and Pearl Brazier University

Structuring Medical Records with Apache Stanbol · PDF fileStructuring Medical Records with Apache Stanbol ... Stanbol, Apache ManifoldCF ... Cassandra Reference Cases

SMART CONNECTOR TECHNOLOGY FOR FEDERATED SEARCH · ABOUT APACHE SOLR AND MONGODB Apache Solr • Open source enterprise search platform built on Apache Lucene™ - java-based indexing

Salton2-1 Automatic Indexing Hsin-Hsi Chen. Salton2-2 Indexing indexing: assign identifiers to text items. assign: manual vs. automatic indexing identifiers:

data-intensive applications Apache Beam: portable and ...€¦ · Cache: Redis, Memcached (in progress) Databases: Apache HBase, Cassandra, Hive (HCatalog), Mongo, JDBC Indexing: