Upload
alexandra-peters
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
INT-2: XQuery Levels the Data Integration Playing Field
Carlo (Minollo) InnocentiDataDirect XML Technologies,
Program Manager
© 2007 Progress Software Corporation2 INT-2: XQuery Levels the Data Integration Playing Field
Agenda
The Problem Why XQuery XQuery for Java API (XQJ) XQuery for Data Integration XQuery Demos, Code Walk-throughs Summary
© 2007 Progress Software Corporation3 INT-2: XQuery Levels the Data Integration Playing Field
The user submits a request for a report
about their stock holdings
The web server needs to fetch user’s personal data, stock holdings and live stock data to compile a report to send
back to the userA public service offers
live (delayed) stock prices
A Typical Data Integration Problem
DB1 DB2 DB3
Different repositories are used for different parts of the information necessary to create a stock holdings report
© 2007 Progress Software Corporation4 INT-2: XQuery Levels the Data Integration Playing Field
TomCat server
Stock price web service
dBASE IV OpenEdge®
DatabaseSQL Server
Some implementation constraints…
HTML SOAPthrough AXIS
dBASE IVAPIs
JDBC
Java™/JSP codeaccessing the various
Java APIs and generating the HTML report
Java OpenClient or
JDBC
© 2007 Progress Software Corporation5 INT-2: XQuery Levels the Data Integration Playing Field
Data Source
Data Source
Data Source
Data Source
Data Source
Data Consumer
Data Consumer
Data Consumer
Data Consumer
Data Consumer
Data Source
Data Source
Data Source
Data Source
Data SourceEDI Message Web Service
RDBMSOpenEdgeDatabase
XML Document
<XML>
Data Access Layer
AJAXClient
DynamicHTMLClient
WebServiceClient
PublishingApps
RESTClient
A dangerous approach
© 2007 Progress Software Corporation6 INT-2: XQuery Levels the Data Integration Playing Field
Data Source
Data Source
Data Source
Data Source
Data Source
Data Consumer
Data Consumer
Data Consumer
Data Consumer
Data Consumer
Data Source
Data Source
Data Source
Data Source
Data SourceEDI Message Web Service
RDBMSOpenEdgeDatabase
XML Document
<XML>
AJAXClient
DynamicHTMLClient
WebServiceClient
PublishingApps
RESTClient
XML
XQuery
The XQuery Vision
© 2007 Progress Software Corporation7 INT-2: XQuery Levels the Data Integration Playing Field
Agenda
The Problem Why XQuery XQuery for Java API (XQJ) XQuery for Data Integration XQuery Demos, Code Walk-throughs Summary
© 2007 Progress Software Corporation8 INT-2: XQuery Levels the Data Integration Playing Field
What is XQuery?
W3C Query Language for XML
• Native XML Programming Language• “The SQL for XML”• Designed to query, process, and create XML
High level functionality
• Find anything in an XML structure • Querying and combining data • Creating XML structures• Functions• User-defined function libraries
© 2007 Progress Software Corporation9 INT-2: XQuery Levels the Data Integration Playing Field
XQuery Basics
Path Expressions: Finding Data
doc("holdings.xml")/holdings/entry
FLWOR Expressions: Querying and Combining Data
for $h in doc("holdings.xml")/holdings/holding, $c in doc("companies.xml")/companies/companywhere $h/userid = "Minollo" and $c/ticker = $h/stocktickerreturn $c/name
© 2007 Progress Software Corporation10 INT-2: XQuery Levels the Data Integration Playing Field
XQuery Basics
Path Expressions, FLWOR Expressions, and XML Constructors
for $h in doc("holdings.xml")/holdings/entry, $c in doc("companies.xml")/companies/companywhere $h/userid = "Minollo" and $c/ticker = $h/stocktickerreturn <company ticker="{ $c/ticker }"> { $c/companyname } { $c/annualrevenues } </company>
© 2007 Progress Software Corporation11 INT-2: XQuery Levels the Data Integration Playing Field
Functions and Modules
A Function in a Library Module
module namespace stock="http://tagsalad.com/stocks";
declare function stock:companies($user as xs:string) { for $h in doc("holdings")/holdings/entry, $c in doc("companies")/companies/company where $h/userid = $user and $c/ticker = $h/stockticker return <company ticker="{ $c/ticker }">
… </company>};
© 2007 Progress Software Corporation12 INT-2: XQuery Levels the Data Integration Playing Field
Functions and Modules (2)
Importing and Using a Library Module
import module namespace stock="http://tagsalad.com/stocks";
stock:companies("Minollo")
© 2007 Progress Software Corporation13 INT-2: XQuery Levels the Data Integration Playing Field
Why XQuery?
Native Support for XML• Conventional programming and query languages are
not designed for XML• No more parse, navigate, cast, repeat – XML is the
native datatype
Designed for Data Integration• Native XML and non-XML data can be used the same
way• Vastly simplifies development when input includes
XML, relational, EDI…– Requires support from implementation for the data
sources you need
© 2007 Progress Software Corporation14 INT-2: XQuery Levels the Data Integration Playing Field
Why XQuery?
XML Output is Directly Useful• XML is becoming the industry standard for data
exchange• Dynamic Web Sites• Publishing Applications• Web Messages• We normally don’t exchange SQL tables or present
them to users!
Programmer Productivity• Readable, declarative code – transparent, easier to
maintain• 7 to 20 times less code than Java + SQL + JDBC +
XML APIs
© 2007 Progress Software Corporation15 INT-2: XQuery Levels the Data Integration Playing Field
Why XQuery?
Performance• Declarative code – can be optimized by the
XQuery Engine
• Relational database vendors and experts very involved in the design
• Actually performance depends on the implementation …
© 2007 Progress Software Corporation16 INT-2: XQuery Levels the Data Integration Playing Field
Benefits of XQuery
Data Integration is harder without XQuery!• Every data source is different
• Many applications use several languages and APIs to address data sources (e.g. Java+JDBC+DOM, SQL, Perl, XSLT…)
• Mediating among data sources accounts for a lot of code
• XQuery treats all data sources as XML
© 2007 Progress Software Corporation17 INT-2: XQuery Levels the Data Integration Playing Field
Benefits of XQuery
Processing XML is harder without XQuery!• Most programming languages don’t know XML
structures• Parse, navigate, cast, repeat• XML is the native data structure for XQuery
XML Reporting is harder without XQuery!• XML input and output may have very complex structure• Many different desired XML outputs• Data Integration, Native XML Processing are needed• XQuery gives full query processing for any XML input
and output
© 2007 Progress Software Corporation18 INT-2: XQuery Levels the Data Integration Playing Field
Agenda
The Problem Why XQuery XQuery for Java API (XQJ) XQuery for Data Integration XQuery Demos, Code Walk-throughs Summary
© 2007 Progress Software Corporation19 INT-2: XQuery Levels the Data Integration Playing Field
What is XQJ?
XQuery API for Java (XQJ) – JSR 225 “The JDBC for XQuery”
© 2007 Progress Software Corporation20 INT-2: XQuery Levels the Data Integration Playing Field
Benefits of XQJ
Industry Standard, similar to JDBC• No need to learn a new proprietary API for each
product and each version
• Can build on existing JDBC knowledge
Lets XQuery fit into any Java architecture Queries can be created or parameterized at run-time
• Example: A portfolio for a given user at a given date
Interfaces are designed for use in J2EE applications• Example: Results can be retrieved as DOM, SAX,
StAX, or text
© 2007 Progress Software Corporation21 INT-2: XQuery Levels the Data Integration Playing Field
Agenda
The Problem Why XQuery XQuery for Java API (XQJ) XQuery for Data Integration XQuery Demos, Code Walk-throughs Summary
© 2007 Progress Software Corporation22 INT-2: XQuery Levels the Data Integration Playing Field
An XQuery architecture
© 2007 Progress Software Corporation23 INT-2: XQuery Levels the Data Integration Playing Field
DataDirect XQuery
**
High performance Scalable Embeddable Plugs into any Java
architecture Accesses almost any data
source No dependency on servers Standards-based
© 2007 Progress Software Corporation24 INT-2: XQuery Levels the Data Integration Playing Field
HOLDINGS
USERID TICKER SHARES
Jonathan PRGS 23
Minollo PRGS 4000000
Jonathan AMZN 3000
Minollo AMZN 3000
<portfolio > <company ticker="AMZN"> <companyname>Amazon.com, Inc.</companyname> <annualrevenues>7780</annualrevenues> </company> <company ticker="EBAY"> <companyname>eBay Inc.</companyname> <annualrevenues>22600</annualrevenues> </company> <company ticker="IBM"> <companyname>Int'l Business Machines C</companyname> <annualrevenues>128200</annualrevenues> </company> <company ticker="PRGS"> <companyname>Progress Software</companyname> <annualrevenues>493.4</annualrevenues> </company></portfolio>
COMPANIES
TICKER NAME REVENUES
AMZN Amazon.com, Inc. 7780
EBAY eBay Inc. 22600
PRGSProgress Software 493.4
YHOO Yahoo! Inc. 10700
Highly optimized for relational sources Minimizes retrieval of data
• No more rows than needed• No more columns than needed
Uses database functionality • Joins• Sorting• Etc..
Optimizes for each SQL dialect Efficient JDBC retrieval
• Embeds DataDirect JDBC technology• Optimizations added to support XQuery
Supports incremental retrieval Optimizes for XML hierarchies
• Sort-merge algorithm• Minimal cost of XML construction
Leverages SQL library Supports hints
XQuery can be fast for relational data!
© 2007 Progress Software Corporation25 INT-2: XQuery Levels the Data Integration Playing Field
XQuery can be fast for XML files!
<?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <soap:Body> <GetQuotesResponse xmlns="http://swanandmokashi.com"> <GetQuotesResult> <Quote> <CompanyName>APPLE COMPUTER</CompanyName> <StockTicker>AAPL</StockTicker> <StockQuote>74.17</StockQuote> <LastUpdated>9/14/2006 4:01pm</LastUpdated> <Change>1.17</Change> <PercentChange>1.82%</PercentChange> <OpenPrice>N/A</OpenPrice> <DayHighPrice>N/A</DayHighPrice> <DayLowPrice>N/A</DayLowPrice> <Volume>0</Volume> <MarketCap>63.266B</MarketCap> <YearRange>47.87 - 86.40</YearRange> <ExDividendDate>21-Nov-95</ExDividendDate> <DividendYield>N/A</DividendYield> <DividendPerShare>0.00</DividendPerShare> </Quote> </GetQuotesResult> </GetQuotesResponse> </soap:Body></soap:Envelope>
General XQuery rewrites• Constant-folding, elimination of common
sub-expressions, loop rewrites, ordering rewrites, etc…
Document projection• XML construction accounts for much of
the cost• Don’t build parts of the document that
the query doesn’t need! Document streaming
• Discard parts of the document when no longer needed
• Makes memory usage near constant with size of file
Multiple Gigabytes can be queried
© 2007 Progress Software Corporation26 INT-2: XQuery Levels the Data Integration Playing Field
XQuery can use XML Converters
• EDI File
ISA+00+DATADIRECT+00+STYLUS2006+01+DATA DIRECT +01+STYLUS STUDIO +060504+1212+~+00503+200654321+0+I+:'GS+BF+DATADIRECT+STYLUS2006+20060504+121212+256+X+005030'ST+105+3389'BGN+28+1024+20060504+121212+GM'NM1+2L+4+Progress Software Corporation'N3+14 Oak Park Drive'N4+Bedford+MA+01730+US+AA'REF+1Z+PRGS'NM1+2L+4+Apple Computer, Inc.'N3+1 Infinite Loop'N4+Cupertino+CA+95014+US+AA'REF+1Z+AAPL'SE+11+3389'GE+1+256'IEA+1+200654321'
doc("adapter://EDI?ticker-request.edi")
<X12> <ISA> <ISA01><!--I01: Authorization Information Qualifier-->00<!--No Authorization Information Present (No Meaningful Information in I02)--></ISA01> <ISA02><!--I02: Authorization Information-->DATADIRECT</ISA02> <ISA03><!--I03: Security Information Qualifier-->00<!--No Security Information Present (No Meaningful Information in I04)--></ISA03> <ISA04><!--I04: Security Information-->STYLUS2006</ISA04> <ISA05><!--I05: Interchange ID Qualifier-->01<!--Duns (Dun & Bradstreet)--></ISA05> <ISA06><!--I06: Interchange Sender ID-->DATA DIRECT </ISA06> <ISA07><!--I05: Interchange ID Qualifier-->01<!--Duns (Dun & Bradstreet)--></ISA07> <ISA08><!--I07: Interchange Receiver ID-->STYLUS STUDIO </ISA08> <ISA09><!--I08: Interchange Date-->060504<!--2006-05-04--></ISA09> <ISA10><!--I09: Interchange Time-->1212</ISA10> <ISA11><!--I65: Repetition Separator-->~</ISA11> <ISA12><!--I11: Interchange Control Version Number-->00503<!--Standards Approved for Publication by ASC X12 Procedures Review Board through October 2005--></ISA12> <ISA13><!--I12: Interchange Control Number-->200654321</ISA13> <ISA14><!--I13: Acknowledgment Requested-->0<!--No Interchange Acknowledgment Requested--></ISA14> <ISA15><!--I14: Interchange Usage Indicator-->I<!--Information--></ISA15> <ISA16><!--I15: Component Element Separator-->:</ISA16> </ISA> <GS> <GS01><!--479: Functional Identifier Code-->BF<!--Business Entity Filings (105)--></GS01> <GS02><!--142: Application Sender's Code-->DATADIRECT</GS02> <GS03><!--124: Application Receiver's Code-->STYLUS2006</GS03> <GS04><!--373: Date-->20060504<!--2006-05-04--></GS04> <GS05><!--337: Time-->121212</GS05> <GS06><!--28: Group Control Number-->256</GS06> <GS07><!--455: Responsible Agency Code-->X<!--Accredited Standards Committee X12--></GS07> <GS08><!--480: Version / Release / Industry Identifier Code-->005030<!--Standards Approved for Publication by ASC X12 Procedures Review Board through October 2005--></GS08> </GS>
Convert non-XML format to XML on-the-fly!
• EDI message types• Comma-delimited or tab-delimited files• dBase• RTF• mbox• Batch conversions are supported• Custom conversions
© 2007 Progress Software Corporation27 INT-2: XQuery Levels the Data Integration Playing Field
declare function local:amazon-listing($isbn){ <tns:Request> <tns:Condition>All</tns:Condition> <tns:DeliveryMethod>Ship</tns:DeliveryMethod> <tns:FutureLaunchDate/> <tns:IdType>ASIN</tns:IdType> <tns:ItemId>{ $isbn }</tns:ItemId> <tns:ResponseGroup>Medium</tns:ResponseGroup> </tns:Request>};
let $loc := <location address="http://soap.amazon.com/onca/soap?Service=AWSECommerceService" soapaction="http://soap.amazon.com" />let $payload := local:amazon-listing("0395518482")return ws:call($loc, $payload)
Leverage existing SOA architecture in queries!
• Integrate queries with web services• Easily generate complex web service requests• Vastly increases the reach of your queries
XQuery can access Web Services
© 2007 Progress Software Corporation28 INT-2: XQuery Levels the Data Integration Playing Field
Questions
so far?
© 2007 Progress Software Corporation29 INT-2: XQuery Levels the Data Integration Playing Field
The Problem Why XQuery XQuery for Java API (XQJ) XQuery for Data Integration XQuery Demos, Code Walk-throughs Summary
Agenda
© 2007 Progress Software Corporation30 INT-2: XQuery Levels the Data Integration Playing Field
TomCat server
Stock price web service
dBASE IV OpenEdgeDatabase
SQL Server
A Data Integration problem
HTML SOAPthrough AXIS
dBASE IVAPIs
JDBC
Java/JSP codeaccessing the various
Java APIs and generating the HTML report
Java OpenClient or
JDBC
© 2007 Progress Software Corporation31 INT-2: XQuery Levels the Data Integration Playing Field
Data Source
Data Source
Data Source
Data Source
Data Source
Data Consumer
Data Consumer
Data Consumer
Data Consumer
Data Consumer
Data Source
Data Source
Data Source
Data Source
Data SourceEDI Message Web Service
RDBMSOpenEdgeDatabase
XML Document
<XML>
Data Access Layer
AJAXClient
DynamicHTMLClient
WebServiceClient
PublishingApps
RESTClient
A dangerous approach
© 2007 Progress Software Corporation32 INT-2: XQuery Levels the Data Integration Playing Field
Data Source
Data Source
Data Source
Data Source
Data Source
Data Consumer
Data Consumer
Data Consumer
Data Consumer
Data Consumer
Data Source
Data Source
Data Source
Data Source
Data SourceEDI Message Web Service
RDBMSOpenEdgeDatabase
XML Document
<XML>
AJAXClient
DynamicHTMLClient
WebServiceClient
PublishingApps
RESTClient
XML
XQuery
The XQuery Vision
© 2007 Progress Software Corporation33 INT-2: XQuery Levels the Data Integration Playing Field
The DataDirect XQuery Solution
TomCat server
Stock price web service
dBASE IV ProgressDatabase
SQL Server
HTML
© 2007 Progress Software Corporation34 INT-2: XQuery Levels the Data Integration Playing Field
Step by step
XQuery to aggregate data from the multiple data sources
XQuery to publish an HTML or XSL-FO (PDF) report directly
Pipelining multiple XQueries with validation steps
Exposing an XQuery Web Service and consuming it from OpenEdge®
© 2007 Progress Software Corporation35 INT-2: XQuery Levels the Data Integration Playing Field
Step by step
XQuery to aggregate data from the multiple data sources
XQuery to publish an HTML or XSL-FO (PDF) report directly
Pipelining multiple XQueries with validation steps
Exposing an XQuery Web Service and consuming it from OpenEdge
© 2007 Progress Software Corporation36 INT-2: XQuery Levels the Data Integration Playing Field
Step by step
XQuery to aggregate data from the multiple data sources
XQuery to publish an HTML or XSL-FO (PDF) report directly
Pipelining multiple XQueries with validation steps
Exposing an XQuery Web Service and consuming it from OpenEdge
© 2007 Progress Software Corporation37 INT-2: XQuery Levels the Data Integration Playing Field
Step by step
XQuery to aggregate data from the multiple data sources
XQuery to publish an HTML or XSL-FO (PDF) report directly
Pipelining multiple XQueries with validation steps
Exposing an XQuery Web Service and consuming it from OpenEdge
© 2007 Progress Software Corporation38 INT-2: XQuery Levels the Data Integration Playing Field
Agenda
The Problem Why XQuery XQuery for Java API (XQJ) XQuery for Data Integration XQuery Demos, Code Walk-throughs Summary
© 2007 Progress Software Corporation39 INT-2: XQuery Levels the Data Integration Playing Field
Benefits of XQuery
Data Integration is harder without XQuery!• Every data source is different
• Many applications use several languages and APIs to address data sources (e.g. Java+JDBC+DOM, SQL, Perl, XSLT…)
• Mediating among data sources accounts for a lot of code
• XQuery treats all data sources as XML
© 2007 Progress Software Corporation40 INT-2: XQuery Levels the Data Integration Playing Field
Benefits of XQuery
Processing XML is harder without XQuery!• Most programming languages don’t know XML
structures• Parse, navigate, cast, repeat• XML is the native data structure for XQuery
XML Reporting is harder without XQuery!• XML input and output may have very complex structure• Many different desired XML outputs• Data Integration, Native XML Processing are needed• XQuery gives full query processing for any XML input
and output
© 2007 Progress Software Corporation41 INT-2: XQuery Levels the Data Integration Playing Field
DataDirect XQuery
© 2007 Progress Software Corporation42 INT-2: XQuery Levels the Data Integration Playing Field
Getting Started …
Examples & Tutorialshttp://www.xquery.com• XQuery Tutorial
• XQJ Tutorial
• DataDirect XQuery Tutorial
© 2007 Progress Software Corporation43 INT-2: XQuery Levels the Data Integration Playing Field
Questions?
© 2007 Progress Software Corporation44 INT-2: XQuery Levels the Data Integration Playing Field
Thank you foryour time
© 2007 Progress Software Corporation45 INT-2: XQuery Levels the Data Integration Playing Field