(Meeting Overview)

(Meeting Overview)

Arjen P. de Vries, Georgina Ramirez, Johan List

Djoerd Hiemstra, Vojkan Mihajlovic, Mila Boldareva,

Maurice van Keulen

Overview• Cirquid Goals

• Multi-model DBMS Architecture

• Region algebras• For XML path traversal?• For ranking in IR?

• GALAX Architecture (+example)

Goals• Develop efficient and flexible system

that integrates information retrieval and data retrieval• ‘structure + content’

• Two parts:• Database architecture (Arjen & Djoerd)• Optimization (Henk Ernst)

Example QueryFOR $article IN

document("collection.xml")//article

WHERE $article/text() about

‘Willem-Alexander dating Maxima’

AND $article[@language = ‘English’]

AND $article[@pub-date between ‘31-1-2003’ and ‘1-3-2003’]

RETURN <result>$article</result>

Basic Assumption• Coupled IR+DB system architecture

is not desirable and efficient

• Possible Alternatives:• Express entire combined algorithms in

DBMS query language• Exploit DBMS extension mechanism for IR• Flexible and transparent integration of

IR in query engine

Multi-model DBMS Architecture

Conceptual Layer

Logical Layer

Physical LayerSuffix Array Staircase-Join

X-Path

…

LM IR …

Cirquid Focus• X-Path extension and IR Language

Modeling extension• Suitable for collection-based processing• Maintain data independence• Based on region algebra

(1, 1:23, 0) (1, 8:22, 1) (1, 14:21, 2) … …

(1, 2:7, 1) (1, 9:13, 2) (1, 15:20, 3) … …

<section>

<title>

(1, 3, 2) … …

(1, 4, 2) … … “retrieval”

“information”

Node index

Word index

<section> <title> Information Retrieval Using RDBMS </title> <section> <title> Beyond Simple Translation </title> <section> <title> Extension of IR Features </title> </section> </section></section>

1

2 3 4 5 6 7

89 10 11 12 13

14

15 16 17 18 19 20

21

22

23

Containment, direct containment, tight containment, proximity

XML IndexingA

B

T1

C

D

T2 T3

E

T4

A

B C

D ET1

T2 T3 T4

B

T1

C

D

T2 T3

E

T4

A

Node index

Word index

OID S E P

A 0 0 13 -

B 1 1 3 1

C 2 4 12 1

D 3 5 8 2

E 4 10 11 2

… … … … …

T1 2

T2 6

T3 7

T4 9

LM IR on Regions• Extend region representation with a

probability value• Extend DB with rules how the

probabilities are computed• E.g.:

P(A ranked_combining B) =count(#B in A) / count( * in A )

• Background model [W3C flawed?]• Prob(A ranked_containing B in collection C)

Issues• Tokenization etc. part of schema?!• ‘Content independence’ through

declarative specification of RM?• Define term-prob ::=

FOR $n in //*LET $rtf = count($n/text() contains

Q),$rlen = count($n/text()) RETURN <p>$rtf/$rlen</p>

More Issues• Adjacency? Proximity? Tag name???

• Region representation?• Pre-post? Stretched pre-post?• Byte offset?

• Reduce cost of materialization of results by scanning original collection file?

• Allows direct use of suffix array… but is it efficient? For what queries?

System Development Plan• Focus on query plan generation

• All the way from conceptual to physical!• Inspiration sources: Moa and RAM• Generate for both MonetDB and ‘normal’ RDBMS;

also X-100?

• Initial goal• Tijah – be pragmatic, must handle INEX 2003!

• Integrate with existing Xquery processor:• Galax Open Source implementation• Investigate also Konstanz system

• Galax project, started in 2000 in Bell Labs.http://db.bell-labs.com/galax/

• Implements (most of):• XQuery 1.0 and XPath 2.0 Data Model• XQuery 1.0 and XPath 2.0 Functions and Operators• XQuery 1.0 : An XML Query Language• XML Query Use Cases• XML Schema Part1: Structures & Part2: Datatypes

• A Typed Implementation: Static & Dynamic• A functional implementation (O’Caml).

Galax Architecture (+example)EXAMPLE

• Use case: Relational• Xquery: Return the item number and the description of all the bicycles.

<result> { for $i in $items//item_tuple where contains($i/description, "Bicycle") return <item_tuple> {$i/itemno} {$i/description} </item_tuple> } </result>

Galax Architecture (+example)

XQuery Parser

Parsing Layer

XML Parser

XQuery Expression

XML Schema Description

XMLDocument

XML Schema AST

XQuery AST

Galax Architecture (+example)

XQueryParser

Parsing Layer

XML Parser

XQuery Expression

XML Schema Description

XMLDocument

XML Schema AST

XQuery AST

Mapping Layer

XQuery Mapping to

the Core

Type System Mapping

XQuery Core Internal Structure

XQuery Type System Internal Structure

Galax Architecture (+example)

XQueryParser

Parsing Layer

XML Parser

XQuery Expression

XML Schema Description

XMLDocument

XML Schema AST

XQuery AST

Mapping Layer

XQuery Mapping to

the Core

Type System Mapping

XQuery Core Internal Structure

XQuery Type System Internal Structure

Static Type

Checker

Static Error for non well-typed queries

Type of Query Result

(Static) Evaluation Layer

element result { element item_tuple { element itemno {xsd:int}, element description {xsd:string} }*}

Galax Architecture (+example)

element result { for $i in ( glx:distinct-docorder(

(let $glx:sequence := (glx:distinct-docorder(($items))) return let $glx:last := (fn:count(($glx:sequence))) return for $glx:dot at $glx:position in ($glx:sequence) return glx:distinct-docorder(

(let $glx:sequence := (glx:distinct-docorder( (descendant-or-self::node()))) return let $glx:last := (fn:count(($glx:sequence))) return for $glx:dot at $glx:position in ( $glx:sequence) return child::item_tuple))))) return if (fn:boolean((let $glx:v1 := (fn:data((glx:distinct-docorder((let $glx:sequence := ( glx:distinct-docorder(($i))

…

Normalized Expression (XQuery Core)

Algebra

• At a logical level, not at the physical.

• Use of regular-expression types.

• Iteration construct based on the notion of monad.

• Notation similar to path navigation in XPath.

Algebra: some operators

• Projection: book0 / author

• Iteration: for b in bib0/book do book [b/author,b/title]

• Selection: where e1 then e2

• Aggregation: avg, count, max, min, sum.

• Joins: nested for loops

• Structural Recursion: match p

case b: …

case c: …

else …

Some Optimization Rules• Goal:

• To eliminate unnecessary FOR or MATCH expressions • Enable other optimizations by reordering or distributing computations.

• Some rules:• FOR simplification

• For v in () do e ()• For v in e do v e• For v in (e1,e2) do e3 (for v in e1 do e3) , ( for v in e2 do e3)

• IF simplification cexpr1 := true cexpr2

If cexpr1 then cexpr2 else cexpr3

cexpr1 := false cexpr3

• LET simplification used_count $v Expr2 => 0 Expr2

Let $v := Expr1 return Expr2 used_count $v Expr2 => 1 Expr2 [ Expr1 / $v ]

Galax Architecture (+example)Optimized Normalized Expression (XQuery Core)

element result { for $i in (glx:distinct-docorder((let $glx:dot := ($items) return for $glx:dot in (descendant-or-self::node()) return child::item_tuple))) return if ( fn:contains((fn:data((glx:distinct-docorder((let $glx:dot := ($i) return child::description))))),("Bicycle")) ) then ( element item_tuple { glx:distinct-docorder((let $glx:dot := ($i) return child::itemno)), text { "" }, glx:distinct-docorder((let $glx:dot := ($i) return child::description))} ) else (())}

Galax Architecture (+example)

XQueryParser

Parsing Layer

XML Parser

XQuery Expression

XML Schema Description

XMLDocument

XML Schema AST

XQuery AST

Mapping Layer

XQuery Mapping to

the Core

Type System Mapping

XQuery Core Internal Structure

XQuery Type System Internal Structure

Static Type

Checker

Static Error for non well-typed queries

Type of Query Result

(Dynamic) Evaluation Layer

Query Processor

Data Model Query Result

XML Parser

XML AST

XML Data Model Loader XML Data Model

Internal Structure

Validation

OUR MAPPING OUR QP

Road Ahead• But…

• Goal, again, is to be mainly pragmatic first• Deeper research starts after initial QP

generator has been bootstrapped from existing system

• Risk:• Too much engineering• Algebra in Galax not suited for optimization

Research issues• Should the ‘semi-structured semi-

monad’ algebra be adapted to enable more set-oriented processing?

• Gives IR application rise to new physical operators???