DATA INTEGRATION USING SOA : A REVIEW

Rachmad SanuriSTMIK ELRAHMA

AbstractJust like a business need for strategic planning to implement Service-Oriented Architecture

(SOA). Many early SOA initiatives have focused on high-level application integration to more effectivelybroker processes, messages, services and enable cost effective reusability.

At the granular level of data even in the early SOA implementations are still a lot of developmentremains an unsolved problem related to heterogeneous data that varies by format, semantics, and hierarchiesamong multiple applications. The result can be cost and time overruns to reverse-engineer data integrationmechanisms, limitations in the functionality of the SOA, and inconsistent data across the enterprise.

This paper examines how an enterprise data integration platform enriches a service-orientedarchitecture, and how an SOA provides an ideal framework for implementing data integration technologyacross the enterprise.Keywords: SOA, data integration, service-oriented

INTRODUCTIONOne of the most intriguing new information systems methodological approaches in

IT to come along is SOA. The emergence of the SOC paradigm and Web servicetechnology, in particular, has aroused enormous interest in SOA [1]. As organizations growand become more complex, consumers make greater demands of their business functionsand models. Organizations have continually enhanced and expanded their ITinfrastructures more and more as information has become available in electronic form.SOA activities focus on hypothesizing new ways of working by developing practicalimplementation scenarios that build on and exploit existing information systems andnetworking technology. Organizations have recognized that collecting and disseminatingdata represents the gathering of very valuable “services” and the potentially tremendousopportunities for the organization to collect and reuse this data.

SOA DefinitionService-oriented architecture (SOA) is a set of principles and methodologies for

designing and developing software in the form of interoperable services. These services arewell-defined business functionalities that are built as software components (discrete piecesof code and/or data structures) that can be reused for different purposes. SOA designprinciples are used during the phases of systems development and integration, [4].

“A service-oriented architecture is essentially a collection of services. These servicescommunicate with each other. The communication can involve either simple data passingor it could involve two or more services coordinating some activity. Some means ofconnecting services to each other is needed” Mark. D. Hensen(2008)[2].

Of the various definitions on a conclusion that there is, SOA is a paradigm forenhanced flexibility continues (improved flexibility). SOA is not a concrete architecture: itis something that brings us to the concrete architecture. We can call this paradigm, style,philosophy, concepts, representations, or strategies. SOA is an approach (approach),system values, ways of thinking that will lead us towards a real decision when designing thesoftware architecture.

Analogy of SOAThe analogy of a SOA like every citizen of a city, in the context of SOA, can be

called as a service. A cleaning service providing service (service) hygiene, a sales provideservice sales, cashier receives cash payments from customers, courier delivery of servicegroceries, accountants prepare financial statements organization, managers manageorganizations, and many such services that are owned / provided by citizens. Every citizen,can also act as a consumer (consumer) for the service provided. In general, each provider(provider) service can also act as a consumer. For example, a cleaning service to act as aconsumer at the time he purchased from a service provider fried food, fried food vending.And one thing that is very important in the concept of SOA is that the consumer does nothave to know how to implement service provider. A passenger bus does not have to knowhow to run a bus.

Business Drivers Behind SOASOA provide great efficiency and economies of scale. In practice, however, SOAs

have swallowed millions of dollars of investment in IT infrastructure while failing to satisfybusiness demands for reduced costs, greater agility, and competitive advantage inincreasingly fast-paced global marketplaces.

SOA themselves are not the problem. The fundamental problem is the complexityof IT infrastructure and the absence of an integrated architectural foundation that enablesinteroperability among silos of heterogeneous applications[5] built up over years and evendecades. In many organizations, well over 50 percent of the IT budget is devoted tobuilding and maintaining points of integration among legacy systems specific to finance,supply chain, customer management, and other mission-critical functions. CIOs and ITmanagers are confronted with:

1. High costs for IT development, deployment, and maintenance2. Poor IT infrastructure flexibility in meeting new business demands3. Inconsistent, inaccurate business data across the enterprise

Integrating such fragmented architectures has emerged as one of the greatest challengesfacing IT and business, with bottom-line business implications. Business prosperityincreasingly depends on a global view of customers, suppliers, products, and partners—anideal not achievable without integration.

Figure 1. Data Fragmentation

Facing competition and brisk merger and acquisition activity, businesses in manyindustries struggle with data fragmentation. In retail and manufacturing, multiple orderentry or order fulfillment systems and poorly integrated supply chains are common.Financial services and insurance companies struggle to deliver services consistently overmultiple channels, including the Web, call centers and retail offices. Virtually every businessis challenged to capitalize on cross-sell and up-sell opportunities, and meet the dataintegrity requirements and other regulations.

RESULTS AND DISCUSSIONService-oriented architecture offers an elegant solution. Though not a new concept,

SOA has gained widespread acceptance with the advent of open, industry-standardprotocols such as Extensible Markup Language (XML), Simple Object Access Protocol(SOAP), and Web Services Description Language (WSDL).

Figure 2. SOA Architecture

SOA is a way of reorganizing a portfolio of previously siloed software applications[5]and support infrastructure into an interconnected set of services, each accessible throughstandard interfaces and messaging protocols. It is an approach to capture, define, andexpose a collection of services and service components available throughout anorganization. SOA is an architectural discipline that centers on the notion that IT assets aredescribed and exposed as services. In simplest terms, a service is a unit of work done by aservice provider to achieve desired results for a service consumer as displayed in Figure 1.Standards such as Simple Object Access Protocol (SOAP), Web Service DescriptionLanguage (WSDL), Universal Description, Discovery and Integration (UDDI), ExtensibleMarkup Language (XML) and Web Service Security (WS-Security) allow organizations toexchange data in a vendor neutral manner using well defined and widely adopted standards.Service components refer to individual processes, data, applications, and technology that anorganization can bundle to deliver a particular service.

These standards offer a highly flexible layer of abstraction that can reduce developmenttime and cost by liberating developers from writing low-level middleware “plumbing” andpromoting reuse of components. They provide greater ease of use and have seen greateruptake than earlier architectural approaches, such as object-oriented DCOM and CORBA.At its most basic, a Web services-based SOA will cover:

1. Loosely coupled services. A layer of abstraction between the technical serviceimplementation and client (e.g., a Web portal) that eliminates the need forcustomized, “tightly coupled” interoperability

2. Leverage of IT assets. Component-based services may be wrapped and reused fordeployment across multiple projects and applications to reduce development timeand cost

3. Use of open standards. Web services standards such as XML, SOAP, and WSDLprovide the interoperability that enables an SOA to work across heterogeneousplatforms

Figure 3. Fragmented Data Solution: Enterprise Data Integration Architecture.

Keys SOA Strategic PrinciplesOrganizations should develop enterprise architectures to support organizational

analysis and identification of performance gaps, discover opportunities for collaboration,and avoid duplicative investments within and across the organization. Organizations shouldalso classify their business and IT operations in an integrated and consistent way.

As shown in Table 1, these SOA principles serve as a bridge between the businessand IT “layers” of the enterprise architecture. Organizational services are the foundation tosupport the reuse of applications, application capabilities, components, and businessservices. More importantly, well defined services help define SOA concepts clearly, whichhelp reconcile differing management and IT perspectives considerably, and is thus a goodstarting point for introducing SOA within an organization.

Here are the four principles that can provide assistance and guidance in helping toreconcile differing SOA viewpoints:

Table 1. SOA PrinciplesPrinciple Description

Services can include both businessprocesses and IT services, and canexistat all levels of the architecture.

Services should include both businessprocesses and IT services, and can existat all levels of the architecture. This is adeliberate expansion of the conventionalunderstanding of SOA in the IT world and reflectsthe broader SOA purpose.

Service Components arecharacterizedby how well they meet the definitionof aservice component, and by their levelofcomplexity.

Service components can be, and oftenare, hierarchical or networked,aggregating lower level components intocomplex, high-level services. They arealso characterized by how well they meetyour organizations definition of a servicecomponent, and by their level ofcomplexity.

Services Components must betraceablethroughout the organization’sEnterpriseArchitecture.

All services must map to businessprocesses and applications within theEnterprise Architecture.

Services, service components, andservice level agreements (SLAs)shouldbe defined, cataloged, and published.

Services, service components, and SLAsshould be defined, cataloged, andpublished to make their capabilitiesknown to potential customers. It followsthat higher-level services and servicecomponents should be configured,managed, and governed to provide astable and reliable service experience forcustomers.

The first principle on which to build a SOA is that services can include businessprocesses, IT elements, or both. It suggests that services can exist at all levels of thearchitecture. A service component is defined as a self-contained business process or servicewith predetermined functionality that may be exposed through a business or technologyinterface. This definition makes clear that services can be provided by a combination ofbusiness and technology elements, not necessarily by IT alone. An example of a businessprocess that works as a service would be a cashier at a fast food restaurant. The cashieroffers well-defined services to all customers and is “reusable” in the sense that it isreplicated throughout all the fast food chain’s locations. IT elements could include atimesheet application within a financial management system as an example of a serviceprovided through IT.

Dilemmas in Service-Oriented ArchitecturesSOA implementations that lack a data integration foundation have often resulted in

massive hand-coding efforts to redress granular data complexities underlying the businesslogic, resulting in further proliferation of brittle point-to-point connectivity. Others may behandicapped in data functionality, or exacerbate problems of inconsistent data across theenterprise.

To avoid costly setbacks and implement an SOA that provides integration at boththe application level and the data level, IT decision makers should define businessrequirements for data integration in an SOA, scrutinize the capabilities of SOA solutions todeal with complex data issues, and recognize functional distinctions between EAI, dataintegration, and other SOA technologies.

In particular, IT architects should consider five elements: Data Semantics: The business context behind data definitions for concepts such as

customer address, product category, or employee type Data Quality: Improving the accuracy and consistency of the “dirty data” common

in disparate applications and legacy systems Data Governance: Data and metadata lineage, management, documentation,

reporting, and auditing tools that help satisfy Sarbanes-Oxley and other regulatoryrequirements

Data Access: Broad reach into structured, semi-structured, and unstructured data inhierarchical and relational databases, mainframe systems, fi les and documents, andapplications

Bulk Data Processing: Support for processing large data volumes, includingchanged data capture (updating only changed data for faster performance)

A service-oriented architecture offers a framework for EAI and enterprise dataintegration technologies to complement each other, with EAI orchestrating processes andtransactions, and a data integration platform executing complex data integration functions.Without data integration technology as a foundation, an SOA suffers limitations in itsability to fully access and leverage data, and deliver on its potential as a transformativearchitecture for IT and business.

Toward SOA :Service-Oriented Data IntegrationFrom its roots as a single-purpose ETL (extraction, transformation, and loading)

tool for data warehousing, enterprise data integration technology has evolved over the past10 years to become a comprehensive platform to access, transform, and integrate data froma wide variety of heterogeneous systems, as well as integrate data in real time throughinteroperability with Java Message Service (JMS) and other messaging systems. Thetechnology has a proven track record of dealing with extremely complex data integrationissues and huge volumes of data in order to provide a consistent, accurate view ofinformation to the business.

Today, data integration technology has evolved to supply the rich data-levelfunctionality required for an optimal SOA and adheres to the three key principles of SOA:•Loose coupling: Data integration services in an SOA allow a client application to accessbusiness-relevant data wherever it resides, in whatever form, in a consistent and accuratemanner across the enterprise• Leverage of IT assets: Component-based data integration services may be wrapped andreusedacross multiple systems without extensive rework• Use of open standards: Support for XML, SOAP, and WSDL enables data integrationtechnologyto interoperate seamlessly throughout the SOA stack as both a provider and consumer ofdataintegration services

In doing so, the data integration engine handles under the covers the complex data-level “heavy lifting” that maximizes the business relevance and utility of the information:• Reconciles disparate semantic definitions• Executes complex transformations• Ensures data quality and consistency across systems.

An Ideal Data Integration Ecosystem FrameworkA service-oriented architecture offers an ideal framework for implementing a

common data integration approach across the enterprise. In an SOA, data integrationtechnology takes advantage of the layer of abstraction that enables its components andservices to be wrapped and reused without extensive hand-coding.

Migrating to an SOA environment also presents the opportunity to readily connectdisparate instances of data integration technology with Web services protocols,JDBC/ODBC, and JMS. For instance, it’s not uncommon for organizations to have dataintegration software deployed on multiple servers for separate purposes, such as datawarehousing, synchronization among applications, or batch transfer of mainframe flat files.

With separation of business logic and underlying data, the open architecture of theSOA encourages leveraging and reusing these data assets by delivering them ascomponentized data services.

Anatomy of Service-Oriented Data IntegrationA sound data foundation for an SOA may be found in an enterprise data

integration platform that offers mature and specialized technologies engineered expresslyto address data issues. This platform is comprised of three functional components: auniversal data access layer, a metadata repository and services, and a Data IntegrationServices Engine. This trio operates in concert to coordinate and deliver a range of dataservices [figure 4].

Figure 4. With a Service-Oriented Enterprise Data Integration Architecture, All Back-endIT Assets Are Accessed through Data and Application Services over Standard Web

Services Protocols

Universal Data AccessThe ideal data integration platform would provide pre-built connectivity to a wide

variety of packaged applications, mainframe systems, relational databases, and semi-structured and unstructured data. It should provide nearly unlimited data access viatraditional physical as well as virtual data integration approaches, while minimizing the costand complexity of accessing data regardless of where it resides.

The maturity of today’s leading data integration platforms helps to greatlyaccelerate the design, deployment, testing, and maintenance of data access to sourcesincluding:

Packaged applications (SAP, Oracle, J.D. Edwards, Siebel, PeopleSoft) Relational databases (Oracle, Microsoft SQL Server, IBM DB2) Unstructured data (Excel, PDF, Lotus Notes, PowerPoint) Semi-structured and industry-specific data (XML, EDI, ACORD, HL7, FIX, Swift)

Universal data access should cover not just sources of data, but also flexiblemethods of access that suit business objectives:• Scheduled (traditional access for bulk movement of data)• Real-time (near-instantaneous access on an event-driven or request/response basis)• Changed data (only data updated since last access)• On-demand (federated access to data when it is requested by a client or user)

Metadata Repository and ServicesThe data integration platform should extend beyond data to its metadata—the

“glue” that describes data values and their semantic meaning. It should provide a drag-and-drop user interface that enables developers to rapidly build business logic, processes, andtransformations for data and make them available for reuse. It should have at its core ascalable metadata repository that stores and manages data models, transformations,workflows, and other artifacts to provide:• A mechanism to reconcile data semantics among disparate systems• Reduced development time and cost• Data lineage for reporting, auditing, and data governance

The metadata repository serves as a universal data interaction framework thatbrokers the translations between high-level service definitions and more granular datadefinitions and mappings. A key value is abstracting the logical model from the actualimplementation of the data integration logic, which allows for better management of thebusiness rules and transformations that form the data integration solution.

A data integration platform is unique in providing rich metadata management andanalysis vital to understanding the genesis and lineage of enterprise information,determining the ripple effects of changed data, and pinpointing weak links in the datainfrastructure. An SOA offers organizations that have neglected metadata an opportunityto reconcile the business meaning of data, to better understand how data and processes arederived and the interdependencies between them, and to exploit metadata for greateraccuracy and efficiencies.

Ensuring Enterprise-Class DeploymentAt the heart of a data integration platform is a high-performance engine for

delivering data integration services, which offers a variety of flexible data delivery

mechanisms and scalability for large-volume data transformations and movement overmultiple concurrent sessions. Leading data integration platforms will provide:

Conventional batch mode movement Changed data capture (moves only updated data for improved performance) Real-time data capture and movement Built-in data partitioning for multi-CPU parallel processing Support for distributed, multi-node grid systems High availability, fault tolerance, and failover

The data integration engine also fortifies security at the data layer through robustauthentication and authorization, support for identity management protocols such asLDAP (Lightweight Directory Access Protocol) and Active Directory, and RSA dataencryption.

The Dynamic Delivery of Data Integration ServicesWith universal data access, metadata repository, and data integration services, the

data integration platform furnishes the specialized data services and functionality essentialto service oriented data integration:

Data Profiling: Thorough, accurate information about the content, quality, andstructure of data in virtually any system

Data Cleansing: Resolving missing data fields, correcting conflicting data, andmanaging data relationship and hierarchy resolution to ensure the quality of datasupplied to the business

Data Transformation: Creating a consistent view of data irrespective of its source,structure, and business semantics

Data Federation: Federated access and integration of data when data cannot becopied, or to avoid data latency or cost issues of building a physical data store

Data Movement: Flexible, “right-time” delivery of data wherever it is needed Data Lineage, Data Reporting, and Metadata Analysis: Reporting, analysis, and

auditing of data and metadata for improved visibility and control

SOA Practical Effect SamplesData-enabled SOA can deliver tangible benefits to both business and IT are

customer data integration or CDI, and order entry and fulfillment1. Customer Data IntegrationIn mergers and acquisitions, companies are pressed to quickly leverage a new customerbase to establish credibility, build loyalty, and capitalize on cross-sell and up-sellopportunities. Let’s say two newly merged divisions of the same company, each with itsown customer base, are looking for cross-sell opportunities between the divisions. Thecompany would be faced with the daunting task of consolidating vast stores of customernames, address, business history, and other information stored in incompatible andgeographically dispersed systems. A Web services-based SOA offers an expedient means ofintegrating the systems between these two divisions. However, unless a data integrationplatform is at the foundation, the merged entity faces the risk of inconsistent and “dirty”data—conflicting customer names, addresses, and business history. A data integrationplatform supplies the data access, cleansing, transformation, and movement servicesrequired for an optimal transition and functions as the nucleus of its CDI initiative

Figure 5. Data Quality, Data Movement, and Data Transformation Services Are Executedfor SOA-Based Customer Data Integration Solutions

2. Order Entry and FulfillmentAccording to a recent industry survey, the average company today has multiple

order entry systems and as many order fulfillment applications, the majority of them poorlyintegrated or hand-coded with limited reusability. For example, a manufacturer envisionsan SOA environment to better orchestrate ordering and fulfillment, improve developerproductivity, and offer customers a more efficient ordering experience.

In this scenario, the manufacturer recognizes that its SOA should extend beyondprocess orchestration to better integration of the data underlying the various operationalsystems in the order entry, supply chain, and order fulfillment processes. It implements adata integration platform to better synchronize the order and customer data across thesevarious systems in an optimized manner to minimize ordering bottlenecks, satisfy toprevenue-generating customers, and streamline the overall manufacturing, distribution, andaccounts receivable functions.

Figure 6: Demand Data Movement and Set-Based Transformation Services Are Executedfor Order Management Solutions in an SOA

The Migration Towards SOAA service-oriented architecture is not built in a day. It is, after all, an architectural

transformation and as such compels a strategic assessment of IT infrastructurecomponents, an alignment of IT resources and business objectives, and education amongIT professionals. Organizations are also likely to confront organizational, cultural, andownership issues that transcend departmental boundaries in both IT and business. To helpensure a smooth transition to a SOA, IT may benefit from focusing on incrementalprojects with high ROI and by establishing an integration competency center (ICC).

Establish Integration Competency CentersThere has been a major shift towards coordinating data integration projects across

the entire enterprise. This shift has resulted in the emergence of integration competencycenters (ICCs). ICCs are an organizational approach designed to increase agility and lowercosts by creating a central pool of skilled resources, promoting reuse, sharing best practices,and establishing common processes and standards for integration [3]. They are particularlyrelevant in the evolution towards SOA and shared services.

Figure 7: An ICC Is the Proven Approach to Building and Managing SOAs in anEnterprise

To support ICCs, a data integration platform must support universal data access toaddress:

The full breadth of data integration initiatives across an enterprise Mission-critical performance, scalability, and availability Global IT team collaboration, capturing data workflows and logic as metadata for

reuse

An Ideal SOA supports and encourages organizations to establish ICCs. An ICCserves as a center of excellence that helps IT and business managers develop reusablepractices, accelerate integration projects, share resources, refine best practices, andsubstantially reduce costs. While the specific structure of an ICC will vary depending on anorganization’s needs and existing structure, ICCs have four basic models:

1. Best Practices: This ICC model defines processes for data integration initiatives andrecommends appropriate technology, but does not own technology or engage inactual implementation activities.

2. Technology Standards: This ICC model standardizes development processes oncommon technology standards, enabling greater reuse of work from project toproject. Although neither technology nor people are shared, standardization createssynergies among disparate project teams.

3. Shared Services: In this ICC model, processes are defined, architecture isstandardized, and a centralized team maintains shared work and environment, butmost development work occurs in the distributed lines of business.

4. Central Services: In this ICC model, standards and processes are defined,technology is shared, and a centralized team is responsible for all developmentwork on integration initiatives.

CONCLUSIONIt is uniquely designed for enterprise-wide deployment and distributed architectures

and provides an ideal foundation for service-oriented data integration by delivering:(1)Universal data access to virtually any enterprise information. (2)Comprehensive dataintegration services including data profiling, transformation, movement and federation.(3)Improved data quality, lineage, auditing, and security (4)Robust metadata managementcapabilities. (5)Reusability for rapid deployment, greater IT productivity, and high ROI.(6)Open, platform-neutral architecture and standards-based interoperability

Data integration is a critical component of an enterprise IT organization’s SOA. Toachieve SOA’s goals of loose coupling and reusability, it is critical for client applications tohave the ability to access business-relevant data wherever it resides, and in whatever form isrequired, and in a consistent and accurate manner. Data integration services, powered by adata integration platform that provides universal data access and a metadata-drivenarchitecture, provide that capability.

Implementation of this SOA is likely to be, and probably should be, incremental.More progress needs to be made at the development level. Organizations need to developthe implementation, governance and configuration management aspects of an SOAmethodology. Their efforts must address governance, security considerations, configurationmanagement, and submission of data and data components. This SOA implementationapproach must also identify the actions and elements needed to implement the associatedinfrastructure to manage these efforts and make SOA a practical reality.

REFERENCES[1] Krafzig, D., Banke, K., and Slama. D. Enterprise SOA: Service-Oriented Architecture Best

Practices. Prentice Hall, 2005

[2] Mark D. Hensen, SOA Using Java Web Service, Prentice-Hall, 2009

[3] Nayak, N., Linehan, M., Nigam, A., Marston, D., Jeng, J., Wu F., Boullery, D., White,L., Nandi, P., Sanz. J. Core Business Architecture for a Service- Oriented Enterprise.IBM Systems Journal, 46(4):723-742, 2007

[4] http://en.wikipedia.org/wiki/Service-oriented_architecture

[5] http://en.wikipedia.org/wiki/Information_silo)