Presenting Statistical Data Using XML Office for National Statistics, United Kingdom Rob Hawkins,...

Presenting Statistical Data Using XML

Office for National Statistics, United KingdomRob Hawkins, Application Development

Presenting Statistical Data Using XML

Supporting paper for MSIS 2005

Reference: CES/AC.71/2005/23• Mission• Requirements and Examples• Component Design• The Future

• Questions at the end please !

Mission

Develop a Java component, which• queries a large relational database (300 million

statistical values and supporting metadata)• produces a variety of statistical outputs• enables a range of user interactions with the

outputs• can be integrated into any Java-based web

application• performs and scales well on busy public-facing

web-sites• is flexible and extensible

Mission

Key customer: UK Neighbourhood Statistics Programme (NeSS)

• Web-site to provide better information for neighbourhood renewal

Also, build the component so that it can be re-used for other web projects

Requirements and Examples

HTML tables

Requirements and Examples

Charts

Requirements and Examples

Thematic maps

Requirements and Examples

Print-friendly examples

Requirements and Examples

Metadata Linking• Requirement to access contextual metadata from items in

tables, charts, and maps

Requirements and ExamplesNeighbourhood Profiles (a report per area, with multiple outputs on same page)

Component Design

• Simple API hides internal complexity• Internal re-design can be carried out without affecting the

client-code• Written as a “framework” for re-use• Stateless and not dependent on web-site that uses it• Design is suited to being encapsulated by a Web Service

Use of XML and XSLT

• Instead of building the output directly from relational database queries, we first create the results as XML

• We then use XSLT transformations to convert the XML into the required form (HTML, CSV, SVG, etc)

• Re-usability of the solution is improved by use of a common, multi-dimensional XML Schema

• As long as XML conforming to the schema can be produced, any data source can be used, e.g. legacy sources

• The entire process is dynamic, carried out when the web-site user clicks “go” – querying the database, building the XML, running the XSLT transformation

• This is all run on the server – there is no transformation taking place on the client

Simplified design diagram

HTML, XML

XML Transformer

XML & XSLT

XRender Service Layer

Data Request(What I want)

Presentation Request

(How I want it)

Chart (URL)

HTML table HTML,SVG, CSV

Client code

CSV, etc.

SVG Map

GIS Service

PopChart

ArcXMLXML

HTML

Process Layer

Data and Metadata

Presentation Rules (XSLT)

Common XML Schema

Data and XML Cache

XML

XSLT

Re-usable framework

Business Data

Sources & Stored

Procedures

Business Data

Access Objects turn

database queries into

XML

Generic Objects

Model and Build the Common

XML Schema

Generic Process objects Coordinate the Transformations

Business Process Objects build the client response

Business Configuration XML used to

tailor the output

Generic XSLT chains

Business XSLT chains

Business API layer

Environmental configuration

XML

Business Cascading

Style Sheets

Business JavaScript

Generic Utility Objects e.g. for

Caching, automating the transformations, building charts

“Plug-and-Play” Design“Business” = Business-domain specific code and configuration – e.g. for NeSS requirements

Performance and Scalability

• High-volume public web-site – peaks at hundreds of concurrent user requests

• Requirement to render most output within a couple of seconds as a worst-case average

• Techniques used:– Oracle index tuning and query hints– Memory caches to minimise database access. – XSLTC (Pre-compiles XSLT scripts) – Balance and optimise the work done by Java and by XSLT

The future

• Further ongoing enhancements for NeSS• Re-use component on UK “National Statistics Online” web-

sites• Share ideas (and code?) with other National Statistics

Institutes ?

End of presentation

Any questions ?