Data Integration in Service Oriented Data Integration in Service Oriented ArchitecturesArchitectures
Rahul PatelRahul PatelSr. Director R & D, BEA Systems
Liquid Data – XML-based data access and integration for enterprise data
Project Alchemy – Mobile, disconnected applications for enterprise data
June 8, 2004
Service Oriented ArchitectureService Oriented Architecture
Employees Customers Partners
Specialty InfrastructureDirectory, Search, Content, etc
Enterprise ApplicationsSFA, CRM, Custom, Legacy, Mainframe
Database & Tuxedo
Data & Transaction Services Infrastructure Services
Message Bus (Transport, Routing, Transformation)
Shared Business Services
1
Composite Applications, Portals and B2B Services
Man
agem
ent
Sec
uri
ty
Enterprise Data Services Enterprise Data Services
Customer
(DB)
Order Mgt
(DB)
Customer
Portal
GetCust_by_Custid
GetCust _by_Orderid
Update_Address
Create_Order
Customer:Jack
Customer
(Package
App)
Order Mgt
(Web
Service)
Division 1
Division 2
Requires accessing and integrating data from multiple resources
Service Clients
Data Services on Logical Model Physical Model
AddressesCustomer
Profile
Pending Payments
Credited Payments
Orders
History
Data Services PlatformData Services Platform
Repository
Enterprise Data
JDBC
Model DrivenIntegration(XQuery)
Customer
Order
Payment
ProgrammingModel(SDO)
Web Srvcs
Adapters
Custom
J2E
E,
We
b S
erv
ice
, X
Qu
ery
, J
MS
•Logical models capture the data access & integration complexity ONCE•SAME data model, programming model, & API for whole enterprise
Enterprise Data ServicesEnterprise Data ServicesModel Based ApproachModel Based Approach
Model-based unification of disparate data sources– Define virtual model(s)– Map physical data source models to virtual model– Data Mediator converts operations against virtual model
intoQueries and transactions against individual sources,
plusMiddleware coordination and computations to finish the
job Technology for Enterprise Data Services
– Data model mapping and integration– SOA-appropriate programming model– Translation of data models, translation of queries– Performance and Caching
Choosing Appropriate TechnologiesChoosing Appropriate TechnologiesXML is the AnswerXML is the Answer
Required Capabilities Standards
XMLA standard for data format and data interchange
A standard for querying both relational & non-relational data
XQuery
A standard for interfacing into applications
Web Services, JCA
XML SchemaA standard for describingand modeling data
A standard for publishing available services
Web Services
Data Services InfrastructureData Services InfrastructureMust tie all the pieces togetherMust tie all the pieces together
Various parts of data architecture– XMLSchema (shape)– Logical Views (nodes in the information model)– Relationships (arcs in the information model)– Data sources (enterprise data source)– XQuery functions/source (aggregate/transform logic)– Programming model for these (SDO)– Business logic (Java, Workflows, etc.)– Publishing the models for application use (Web Services, SDO,
Controls..)– Policies (meta-data, security, etc.)
How to connect all these in a meta-data environment– Dependencies– Integrated development / management experience– A logical of reusable unit of data representing a particular business
entity – e.g. Single view of Customer
Model Driven “Read” Data ServicesModel Driven “Read” Data ServicesOptimized Distributed Queries on Logical ModelOptimized Distributed Queries on Logical Model
SQL sub-query pushdown to relational sources– Selects, projects, functions,
inner/outer joins, sorts, aggregates
Distributed join optimization– Batched parameter passing
Join, Index Join, Merge Join
Slow data source handling– Intra-query function result
caching– Parallel invocation of slow
functional sources
Query result cache for stored queries (optional)– Consistency based on TTL plus
invalidation API
Check security & cache
Cache results
Create optimized Execution plan
Merge & Transform Data
1
2 4
5
Data ServiceRequest
QueryResult
Get data from underlying sources
Sub-query Sub-query Function Call
3
Model Driven “Write” ServicesUpdate Logic specified on the same Model
Physical Source 1
Physical Source 2
Update_Address
Update_CreditCard
MappingMeta-data
MappingMeta-data
Decomposition
Change LogTime StampSource Info.Update Plan
User Defined Logic,Validation, Rules, etc.
User Defined Logic,Validation, Rules, etc.
User Exit on View