Native XML Databasefor Information Systems

Chris WallaceISD3

March 2006

Chris Wallace, ISD3 2

Exploring the design space

• Native XML database (NXD)– Storing, querying and updating XML documents without

mapping into relations– Schema-free– Trees are to NXD what tables are to RDBMS– Tables are trees

• Information Systems– Focus on semi-structured data (mixture of simple data

items, text and complex nested structures)– Searching, derived data, visualisation– Process support– Large problem space variously supported by

spreadsheets, word documents, ad-hoc databases, increasingly web-integrated data.

• “design as a conversation with the materials in the situation” (Schon)

Chris Wallace, ISD3 3

eXist Native XML Database• Open source Java • European team of developers led by Wolfgang

Meier• Documents (files) are organised in collections

(folders) in a file store– XML Documents stored in an efficient, B+ tree structure

with indexes– Non-XML resources (XQuery, CSS, JPEG ..), etc can be

stored as binary• Deployable in different ways

– Embedded in a Java application– Part of a Cocoon pipeline– As web application in Apache/Tomcat– With embedded Jetty HTTPserver (as on stocks)

• Multiple Interfaces– REST – to Java servlet – SOAP– XML:RPC

Chris Wallace, ISD3 4

NXD case studies

• FOLD– modules, programmes, scheme operations,

staff, organisational structures, events• Family photos and history

– Integration of meta-data on family photos with family history (births, deaths and marriages)

• ISD3 Assignment – a web-based calculator– e.g. a currency converter

• State-machine simulator– Coke machine

Chris Wallace, ISD3 5

ISD3 Coursework 1

• Develop a simple web-based calculator• Not just a programming exercise

– User interface design • Users language, units, not raw data• User interaction design

– Data design• choice of representation of domain facts

– Veracity• Relationship between data in database and domain

being modelled• How is veracity monitored and maintained

– Process• Examine some of the XP processes

– Test-driven development

Chris Wallace, ISD3 6

Application Design• Data model is one simple table:

– Currency code, name and symbol– Latest conversion rates from GBP to currency X

• Currency Coding– Use ISO4217 e.g. XE.COM list

• Core algorithm:– Conversion from N X to ? Y is

• N * rate(X to GBP) * rate(GBP to Y) i.e.• N * (1/rate (GBP to X) * rate(GBP to Y)

• Currency rates to be updated by an administrator– (not via a web-service)

• Interface is to be a sticky form:– Input form and output result on one page– Input form default values from last input

• Veracity management– Rates must be dated and sourced –

Chris Wallace, ISD3 7

Technical Decisions

• Choice of platform:– PL/SQL and Oracle– ASP.NET and SQL Server– JSP (Java Servlet Page) and JDBC to

Postgres(say)– PHP and MySQL– XML and Native XML Database (eXist)

• Calculation location:– client-side in ECMAScript (aka JavaScript)– server-side

• with/without Ajax

Chris Wallace, ISD3 8

Two approaches

• PHP-MySQL– Define and create MySQL table– Write PHP script to provide interface and

access the database using SQL– Write editor for the Currency table

• XML– Create MS Excel spreadsheet of

currencies – Convert to XML in Excel and save– Write XQuery script to provide interface

Chris Wallace, ISD3 9

Development Processfor currency converter

• XP Practices:– ‘Spike’ Simple end to end implementation– Incremental development

• Setup eXist database – Using the Admin interface:

• Create a directory for application• Create a subdirectory for currency data

• Create XML dataset(s) in Excel • Upload to eXist• Write the XQuery script cur.xql• Upload to eXist• Execute in browser

Chris Wallace, ISD3 10

Currency Data in XML

• Start MS Excel 2003• Create the spreadsheet with column headings• Convert to List (needs XML add-in)• Save as XML data

Code Name Rate

GBP Sterling 1

USD US Dollars 1.7423

EUR Euro 1.46331

NZD NZ Dollars 2.60665

JPY Yen 205.852

Chris Wallace, ISD3 11

XQuery

• W3C candidate recommendation– http://www.w3.org/TR/xquery/

• Designed by, amongst others, Don Chamberlin– http://www.research.ibm.com/journal/sj/414/

chamberlin.pdf• A functional programming Language

– Based on XPath (tree-access language)– Integrate, select, update, compute and

construct XML documents– cf PL/SQL

• http://www.w3.org/XML/Query/

Chris Wallace, ISD3 12

Write the XQuery script

• Use the admin interface to test simple queries

• Use a syntax aware editor if possible– JEdit– Dreamweaver– Java Client interface to eXist– PFE32

Chris Wallace, ISD3 13

Executing an XQuery

eXist DB

cur.xql

XQuery Engine

parameters

html

Client Browser eXist: Server

Get cur.xql +parameters

servlet

fetch cur.xql

render

User clickslink

cur.xql?fromAmount=100&fromCode=USD&toCode=EUR

XSLT

Chris Wallace, ISD3 14

XQuery Script (1)declare namespace request="http://exist-db.org/xquery/request";

let $fromAmount := request:request-parameter("fromAmount",“100"), $fromCode := request:request-parameter("fromCode","GBP"), $toCode := request:request-parameter("toCode","EUR"), $currencies := collection('/db/calculator/currencies')//Currency, $fromCurrency := $currencies[Code=$fromCode], $toCurrency := $currencies[Code=$toCode], $toAmount := round(xs:decimal($fromAmount) * xs:decimal($toCurrency/Rate) div xs:decimal($fromCurrency/Rate) )return

Chris Wallace, ISD3 15

XQuery Script (1)declare namespace request="http://exist-db.org/xquery/request";

let $fromAmount := request:request-parameter("fromAmount",“100"), $fromCode := request:request-parameter("fromCode","GBP"), $toCode := request:request-parameter("toCode","EUR"), $currencies := collection('/db/calculator/currencies')//Currency, $fromCurrency := $currencies[Code=$fromCode], $toCurrency := $currencies[Code=$toCode], $toAmount := round(xs:decimal($fromAmount) * xs:decimal($toCurrency/Rate) div xs:decimal($fromCurrency/Rate) )return …

Default

Return node sequence of all Currency

elements in this doc Filter Condition

Cast

Chris Wallace, ISD3 16

XQuery Script (2)

return<html><head><title>Currency Calculator</title></head><body><h1>Currency Calculator</h1><form method ="get"><table border="1"><tr><td>Amount to Convert</td> <td><input type="text" name="fromAmount“ value="{$fromAmount}"/> </td></tr>

Embedded XQuery

XML

Chris Wallace, ISD3 17

XQuery (3)

<tr><td>From Currency</td><td><select name="fromCode">{for $currency in $currencies let $code := data($currency/Code), $name := data($currency/Name) return if ($code = $fromCode) then <option value="{$code}" selected="yes">{$name}</option> else <option value="{$code}" >{$name}</option>}</select></td></tr>

FLWOR expression

conditional

Chris Wallace, ISD3 18

Round two - enhancements

• Add another currency– ZAR Rand 10.4767

• Add new columns– Source and date/time

• Update spreadsheet – Add columns and data

• Update XQuery script – cur2.xql– Add source – Sources and oldest date

Chris Wallace, ISD3 19

Round 3 – Currency Table

• Same document used for different purpose:– currency.xsl– curtable.xql– Run it curtable.xql

Chris Wallace, ISD3 20

FOLD - Modules and Programmes

+ Module

- moduleCode : String

+ Module Specification

- version : Year

- faculty : Faculty

- field : Field

- title : String

- credits : CreditsType

- level : LevelType

- syllabus : RestrictedHTML

- readingStrategy : RestrictedHTML

+ 1..1+ 1..*

+ definition

+ ProgrammeStructure

- version : Year

+ Programme

- programmeCode : String

- ucasCode : String [0..1]

+ 1..1

+ 1..*+ s tructure

+ Stage

+ 1..1

+ 1..* {ordered}

+ OptionGroup

- id : String

- comment : String [0..1]

- minCredits : int

- maxCredits : int

+ 1..1

+ 1..* {ordered}

+ Core

+ 1..1

+ 1..* {ordered}

+ 1..*

+ 1..*

+ core

+ Option

+ 1..1

+ 1..* {ordered}

+ 1..*

+ 1..*

+ optional

+ Module Combination

- comment : String

+ 1..1

+ 0..1+ pre-requis ite

+ 1..1

+ 0..1

+ co-requisite

+ 1..*

+ 1..*

+ e

xpre

ssio

n

This is a boolean expression such as ( m1 and m2 and (m4 or (m5 and m6))

+ Learning Outcome

- assessed in Comp A : boolean

- assessed in Comp B : boolean

- specification : RestrictedHTML

- outcomeType : Learning Outcome

+ 1..1

+ 1..* {ordered}

+ Reading item

+ Book

- authors : String

- title : String

- year : String

- source : String

+ WebSite

- url : URL

- text : String

+ 1..1

+ 1..1

+ 1..1

+ 1..*+ Excluded

The FOLD

Chris Wallace, ISD3 21

FOLD Design

• Some table data for maintenance in spreadsheet form. Forces some compromises in the data structure– Multi-valued fields (not nested)– Specific roles (not generic)

• Module specifications, Programme Structures, Organisational Groups – deeply structured documents – complex relationships

Chris Wallace, ISD3 22

Research Areas

• Design practice for NDX– ‘Pattern language’ to help map from conceptual

model to multiple XML schemes– Identifier design– Structuring documents by responsibility and

versions

• NDX in organisational use– Social effects of distributed responsibility– Visualisation of complex relationships – Handling integrity problems – accept

inconsistency as a way of life– Management of veracity

Chris Wallace, ISD3 23

Process support

• Short term – Process support– Form generation– Linkage to process documentation

• Medium term – Process monitoring– Online capture of significant dates

• Coursework hand-in date• Date exam sent to moderator• Date coursework returned to students

– Derived information• Workload prediction based on coursework schedule and student

numbers• Display of latest coursework returned and SMS message to

students– State-based process modelling

• Exam moderation• Long term- Process management

– Workflow – Process enactment software

Chris Wallace, ISD3 24

Next 6 weeks

• Web Services – Application integration

• Process modelling– Use cases and scenarios– State machines

• Business Process Modelling