W H I T E P A P E R

How to Obtain Flexible, Cost-effective Scalability andPerformance through Pushdown ProcessingUnder the Hood of the Pushdown Optimization Option

Now Available Through Informatica PowerCenter 8

This document contains Confidential, Proprietary and Trade Secret Information (“Confidential Information”) ofInformatica Corporation and may not be copied, distributed, duplicated, or otherwise reproduced in any mannerwithout the prior written consent of Informatica.

While every attempt has been made to ensure that the information in this document is accurate and complete,some typographical errors or technical inaccuracies may exist. Informatica does not accept responsibility for anykind of loss resulting from the use of information contained in this document. The information contained in thisdocument is subject to change without notice.

The incorporation of the product attributes discussed in these materials into any release or upgrade of anyInformatica software product—as well as the timing of any such release or upgrade—is at the sole discretion ofInformatica.

Protected by one or more of the following U.S. Patents: 6,032,158; 5,794,246; 6,014,670; 6,339,775;6,044,374; 6,208,990; 6,208,990; 6,850,947; 6,895,471; or by the following pending U.S. Patents:09/644,280; 10/966,046; 10/727,700.

This edition published April 2006

Table of Contents

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2

Historical Approaches to Data Integration . . . . . . . . . . . . . . . . . . . . . . . . .4

The Combined Engine- and RDBMS-based Approach to Data Integration . .5

How Pushdown Optimization Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

Overview of Pushdown Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

Two-Pass Pushdown Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Partial Pushdown Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

Full Pushdown Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

Platform-specific Pushdown Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

Limitations on the Types of Transformations that Can Be Pushed to the Database . . . . . .9

Benefits of Pushdown Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Increased Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Increased IT Team Productivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

Reduced Risk and Enhanced Flexibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12

Conclusion and Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12

1Pushdown Optimization

White Paper

Executive SummaryOver the next five to 10 years and beyond, the two dominant variables in the enterprise dataintegration equation are painfully clear—more data and less time. Given these, what’s the rightdata integration strategy to effectively manage terabytes or even hundreds of terabytes of datawith enough flexibility and adaptability to cope with future growth?

Historically, data integration was performed by developing hand-coded programs that extractdata from source systems, apply business/transformation logic and then populate the appropriatedownstream system, be it a staging area, data warehouse or other application interface.

Hand-coding has been replaced, in many instances, by data integration software that performsthe access, discovery, integration, and delivery of data using an “engine” or “data integrationserver” and visual tools to map and execute the desired process. Driven by acceleratedproductivity gains and ever-increasing performance, “state of the art” data integration platforms,such as Informatica® PowerCenter®, handle the vast majority of today’s scenarios quite effectively.

PowerCenter has enjoyed wide acceptance and use by high-volume customers representingcompanies and government organizations of all sizes. Based on this use, Informatica hasidentified performance scenarios where processing data in a source or target database—insteadof within the data integration server—can lead to significant performance gains. These scenariosare primarily where data is “co-located” within a common database instance, such as whenstaging and production reside in a single Oracle relational database management system(RDBMS) or where a large investment has been made in database hardware and software thatcan provide additional processing power.

With these scenarios in mind, Informatica Corporation set out to deliver a solution that delivers the best of both worlds without incurring undo configuration and management burden; a solution that best leverages the performance capabilities of its data integration server and/or the processing power of a relational database interchangeably to optimize the use of available resources.

2

“Helping to overcome the challenges of implementing data integration as anenterprise-wide function, PowerCenter 8 offers key new features that can enable near-universal data access, deliver greaterperformance and scalability, and signifi-cantly increase developer productivity. The push-down logic will allow us to take further advantage of our database processing power.”

Mark CothronData Integration Architect,Ace Hardware

Informatica has developed a solution that offers IT architects flexibility and ease of performanceoptimization through “push down” processing into a relational database using the samemetadata-driven mapping and execution architecture: the PowerCenter Pushdown OptimizationOption now available through Informatica PowerCenter 8. PowerCenter 8 is the latest release ofInformatica’s single, unified enterprise data integration platform for accessing and integratingdata from virtually any business system, in any format, and delivering that data throughout theenterprise at any speed.

This white paper describes the flexibility, performance optimization, and leverage provided by thePowerCenter 8 Pushdown Optimization Option. It examines the historical approaches to dataintegration and describes how a combined engine- and RDBMS-based approach to dataintegration can help the enterprise:

• Cost-effectively scale by using a flexible, adaptable data integration architecture

• Increase developer and team productivity

• Save costs through greater leverage of RDBMS and hardware investments

• Eliminate the need to write custom-coded solutions

• Easily adapt to changes in underlying RDBMS architecture

• Maintain visibility and control of data integration processes

After reading this paper, you will understand how pushdown processing works, the option’stechnical capabilities, and how these capabilities will benefit your environment.

3Pushdown Optimization

White Paper

Historical Approaches to Data IntegrationHistorically, there have been four approaches to data integration:

1. Hand-coding. Since the early days of data processing, IT has attempted to solve integrationproblems through development of hand-coded programs. These efforts still proliferate in manymainframe environments, data migration projects, and other scenarios where manual labor isapplied to extract, transform, and move data for the purposes of integration. The high risks,escalating costs, and lack of compliance associated with hand-coded efforts are welldocumented, especially in today’s environment of heightened regulatory oversight and theneed for data transparency. Early on, solutions for automation emerged to replace hand-coding as an alternative cost effective solution.

2. Code generators. The first early attempts at increasing IT efficiency led to the development ofcode generation frameworks that leveraged visual tools to map out processes and data flowbut then generated and compiled code as the resultant run-time solution. Code generatorswere a step-up from hand-coding for developers, but this approach did not gain widespreadadoption as solution requirements and IT architecture complexity arose and the issues aroundcode maintenance, lack of visibility through metadata, and inaccuracies in the generationprocess led to higher rather than lower costs.

3. RDBMS-centric SQL Code generators. An offspring of early generation code generatorsemerged from the database vendors themselves. Using the database as an “engine” and SQLas a language, RDBMS vendors delivered offerings that centered on their “flavor” of databaseprogramming. Unfortunately, these products exposed the lack of capability of the SQLlanguage and the database-specific extensions (e.g., PL/SQL, stored procedures) to handlecross-platform data issues; XML data; the full range of functions such as data quality,profiling, and conditional aggregation; and the rest of the complete range of business logic needed for enterprise data integration. What these products did prove was that forcertain scenarios, the horsepower of the relational database can be effectively used for data integration.

4. Metadata-driven engines. Informatica pioneered a data integration approach that leveraged a data server, or “engine,” powered by open, interpreted metadata as the workhorse fortransformation processing. This approach addressed complexity and met the needs forperformance. It also provided the added benefit of re-use and openness due to its metadata-centricity. Others have since copied this approach through other types of engines and languages, but it wasn’t until this metadata-driven, engine-based approach was widelyadopted by the market as the preferred method for saving costs and rapidly delivering on data integration requirements that extraction, transformation, and loading (ETL) was establishedas a proven technology. Figure 1 shows this engine-based data integration approach.

4

THE POWERCENTER PUSHDOWNOPTIMIZATION OPTION

Automatically generates and “pushes down”mapping logic

• Generates database-specific logic that

represents overall data flow

• Pushes the execution of the logic into the

database to perform data transformation

processing

Provides a single design environment with an easy-to-use GUI

• Decouples data transformation logic

from the physical execution plan

• Controls where processing takes place

• Dynamically creates and executes

database specific transformation language

• Allows you to preview the processing

you can push to the database

Leverages a single, unified data integration platform

• Applies pushdown optimization to all data

integration processing available on the

PowerCenter platform, including data

cleansing, data profiling, and unstructured

and semi-structured data processing

DataSources

MetadataRepository

DataIntegrationServer

DataTarget

Figure 1: Informatica Pioneered the Metadata-driven Engine Approach to Data Integration

The Combined Engine- and RDBMS-based Approach to Data IntegrationUsing an engine-based approach, Informatica PowerCenter has become the industry performanceleader for enterprise data integration. This leadership has been demonstrated in industrybenchmarks, with continued success in complex, high-volume customer environments and inhead-to-head evaluations with other competitive offerings. Performance capabilities, such assource-specific partitioning, 64-bit support, threaded architecture, and continued testingand refinement of the data server, led to organizations to choose PowerCenter to meet theirmost strenuous requirements.

In reviewing requirements for the latest version of the product, PowerCenter 8, Informaticaevaluated certain scenarios in which it made sense, while processing transformations in arelational database, to limit the movement of data out and subsequent back in the databaseduring “data co-resident” periods. It is with these scenarios in mind that Informatica developedthe pushdown optimization capabilities to round out the optimal performance architecture of its enterprise data integration platform.

Pushdown optimization is enabled through PowerCenter’s metadata-driven architecture, whichdecouples the data transformation logic from the physical execution plan. This uniquearchitecture allows processing to be “pushed down” inside an RDBMS when possible.

PowerCenter 8 is the only software on the market that offers engine-based and RDBMS-basedintegration technology in a single, unified platform. This approach ensures a broad spectrum ofdata integration initiatives and enables IT to save costs through intelligent use of existingcomputing resources. Both approaches are required for organizations looking to develop anIntegration Competency Center where all integration efforts are developed and/or managed byan expert team faced with varying solution requirements.

Although processing is spread between the data integration engine and the database engine,with Power Center 8 developers use a single design environment and the same standard set ofPowerCenter tools. For example, a developer can design the data flow using the PowerCenterDesigner, and can design job workflow using the PowerCenter Workflow Manager. Metadatacontinues to be generated and managed within PowerCenter. By simply selecting pushdownoptimization in the PowerCenter graphical user interface (GUI), developers can control whereprocessing takes place, and database-specific transformation language will be dynamicallycreated and executed as appropriate. Pushdown optimization ensures that existing IT assets are fully utilized, helping organizations maximize their investment in RDBMS horsepower.

How Pushdown Optimization Works The Pushdown Optimization Option increases systems performance by providing the flexibility to push data transformation processing to the most appropriate processing resource, whetherwithin a source or target database or through the PowerCenter server. This section explains howpushdown processing works, including two-pass, partial, and full pushdown processing. Itdescribes platform-specific pushdown processing and outlines the limitations on the types oftransformations that can be pushed to the database.

5Pushdown Optimization

White Paper

A SQL code generator-only approach to dataintegration hampers IT’s ability to deliver onthe various needs of the enterprise due to the limitations of SQL as a comprehensivelanguage for data integration efforts.

6

Overview of Pushdown ProcessingSeparating logical business logic from physical run-time execution, the Pushdown OptimizationOption is coupled with the creation and management of workflows. Workflows tie the executionof a metadata-based mapping to an actual physical environment. This environment spans notonly the PowerCenter Data Integration Services that may reside on multiple hardware systems,but also the relational databases where pushdown processing will occur. As shown in Figure 2,data integration solution architects can configure the pushdown strategy through a simple drop-down menu in the PowerCenter 8 Workflow Manager.

Pushdown optimization can be used to push data transformation logic to the source or targetdatabase. The amount of work data integration solution architects can push to the databasedepends on the pushdown optimization configuration, the data transformation logic, and themapping configuration.

When pushdown optimization is used, PowerCenter writes one or more SQL statements to thesource or target database based on the data transformation logic. PowerCenter analyzes thedata transformation logic and mapping configuration to determine the data transformation logicit can push to the database. At run time, PowerCenter executes any SQL statement generatedagainst the source or target tables, and it processes any data transformation logic withinPowerCenter that it cannot push to the database.

Using pushdown processing can improve performance and optimize available resources. Forexample, PowerCenter can push the data transformation logic for the mapping seen in Figure 2to the source database.

Figure 2: Data Integration Solution Architects Can Configure the Pushdown Strategy through a Simple Drop-DownMenu in the Powercenter 8 Workflow Manager

Figure 3 shows a mapping that can be pushed to the source database.

The mapping contains a filter transformation that filters out all items except for those with an IDgreater than 1005. PowerCenter can push the data transformation logic to the database, and itgenerates the following SQL statement to process the data transformation logic:

INSERT INTO ITEMS(ITEM_ID, ITEM_NAME, ITEM_DESC, n_PRICE) SELECT ITEMS.ITEM_ID,ITEMS.ITEM_NAME, ITEMS.ITEM_DESC, CAST(ITEMS.PRICE AS INTEGER) FROM ITEMS WHERE(ITEMS.ITEM_ID >1005)

PowerCenter generates an INSERT SELECT statement to obtain and insert the ID, NAME, andDESCRIPTION columns from the source table, and it filters the data using a WHERE clause.PowerCenter does not extract any data from the database during this process. BecausePowerCenter does not need to extract and load data, performance improves and resources are maximized.

Two-Pass Pushdown ProcessingPushdown processing is based on a two-pass scan of the mapping metadata. In the first pass,PowerCenter starts scanning the mapping objects starting with source definition object, movingtowards the target definition object. When the scan encounters an object containing datatransformation logic that cannot be represented in SQL, the scanning process stops, and alltransformation upstream of this transformation are grouped together with equivalent SQL forexecution inside the source system.

In the second pass, PowerCenter scans in the opposite direction (i.e., from the target definitionstowards the source definitions). When the scan encounters an object containing datatransformation logic that cannot be represented in SQL, the scanning process stops, and alltransformation objects downstream of this transformation are grouped together with equivalentSQL for execution inside the target system. PowerCenter executes any remaining datatransformation logic.

When you configure PowerCenter to use pushdown optimization, it can process the datatransformation logic using full or partial pushdown optimization.

7Pushdown Optimization

White Paper

Figure 3: Sample Mapping Pushed to Source Database

Partial Pushdown ProcessingPartial pushdown processing occurs when either the source and target systems are in differentdatabase instances, or only some of the data transformation logic can be represented in SQL. In such cases, some processing may be pushed into source database, some processing occursinside PowerCenter, and some processing may be pushed to the target database.

In Figure 4, all transformations up to and including the aggregate transformation are pushed intothe source database. The Update Strategy transformation is executed within PowerCenter, andthe Expression transformation is executed inside the target database.

Figure 4 shows an example of partial pushdown processing.

Figure 4: Mapping for Partial Pushdown Processing

Full Pushdown ProcessingFull pushdown processing occurs when the source and target relational database managementsystems are the same instance, and data transformation logic can be completely represented in SQL. In this case, PowerCenter pushes down the entire mapping processing inside thedatabase system.

Figure 5 shows example mapping that is fully processed inside the database system.

Figure 5: Mapping for Full Pushdown Processing

In Figure 5, the sources and targets are the same instance, and the data transformation logiccan be pushed to the database. The work of the filtering, joining, and sorting the data isperformed by the database, freeing PowerCenter resources to perform other tasks. However, thetransformation logic is represented in PowerCenter, so it is platform independent and easy tomodify. The visual representation makes it simple to review the flow of logic, and the PushdownOptimizer Viewer allows you to preview the SQL statements PowerCenter will execute at run time.

8

Platform-specific Pushdown OptimizationWhen pushdown optimization is applied to specific database type, the PowerCenter DataIntegration Services generate SQL statements using native database SQL. Standards-basedgeneration for ODBC is also supported, and PowerCenter generates SQL statements using ANSISQL. PowerCenter can generate a greater variety of transformation functions when a specificdatabase type is used and ensures optimal generation of the fastest execution plan.

Limitations on the Types of Transformations that Can Be Pushed to the DatabasePowerCenter includes a number of transformations that perform functions that cannot beperformed within a database. Figure 6 summarizes the transformation types that can be pushedto the database.

Figure 6: Transformation Types that Can Be Pushed to The Database Using PowerCenter

With the PowerCenter Pushdown Optimization Option, data integration solution architectscanleverage both the database and PowerCenter’s capabilities by pushing some transformationlogic to the database and processing other data transformation logic using PowerCenter.

9Pushdown Optimization

White Paper

Transformation Source-side Target-sideAggregator x

Application Source Qualifier

Custom

Expression x x

External Procedure

Filter x

Java

Joiner x

Lookup x x

Normalizer

Rank

Router

Sequence Generator

Sorter x

Source Qualifier x

Stored Procedure

Target x

Transaction Control

Union x

Update Strategy

XML Generator

XML Parser

XML Source Qualifier

For example, users might have a mapping that filters and sorts data and then outputs the datato an XML target. To utilize database and PowerCenter capabilities to their fullest potential, dataintegration solution architects might push the transformation logic for the Source Qualifier, Filter,and Sorter transformations to the source database, and then the extract the data to output it tothe XML target.

Figure 7 shows a mapping that uses database capabilities and PowerCenter’s XML capabilities.

Figure 7: Mapping Pushes Transformation Logic to the Source and Writes to an XML Target

Benefits of Pushdown Optimization The PowerCenter Pushdown Optimization Option offers many benefits, including:

• Increased performance by using optimal resources

• Increased ease-of-use with a metadata-driven architecture that provides metadata lineage

• Increased IT team productivity with simplified debugging and performance tuning

• Reduced risk and enhanced flexibility through database neutrality

Increased Performance The PowerCenter Pushdown Optimization Option increases systems performance by providing the flexibility to push data transformation processing to the most appropriate processingresource, whether within a source or target database or through the PowerCenter server. With this option, PowerCenter is the only enterprise data integration software on the market thatallows data integration solution architects to choose when pushing down processing offers a performance advantage.

With the PowerCenter Pushdown Optimization Option, data integration solution architects canchoose to push all or part of the data transformation logic to the source or target database.Data integration solution architects can select the database they want to push transformationlogic to, and they can choose to push some sessions to the database, while allowingPowerCenter to process other sessions.

For example, let’s say an IT organization has an Oracle source database with very low useractivity. This organization may choose to push transformation logic for all sessions that run onthis database. In contrast, let’s say an IT organization has a Teradata® source database withheavy user activity. This organization may choose to allow PowerCenter to process thetransformation logic for sessions that run on this database. In this way, the sessions can be tuned to work with the load on each database, optimizing performance.

With the PowerCenter Pushdown Optimization Option, data integration solution architects canalso use variables to choose to push different volumes of transformation logic to the source ortarget database at different times during the day. For example, partial pushdown optimizationmay be used during the peak hours of the day, but full pushdown optimization is used frommidnight until 2 a.m. when activity is low.

10

Increased IT Team Productivity With its unique metadata-driven architecture, the PowerCenter Pushdown Optimization Optionincreases IT team productivity in several ways:

• Ease-of-use on different platforms. PowerCenter’s metadata-driven architecture allowstransformation logic to be easily ported to different platforms. The same transformation logiccan easily be performed on different databases. The same session can be assigned differentdatabase connections, and the same data transformation can be performed without rewritingcode or using different SQL syntax.

• Ease of maintenance. PowerCenter’s metadata-driven architecture makes it easy to track datafor the purposes of error-logging, data cleansing, or data profiling. In addition, metadata forrepository objects is also maintained in the PowerCenter repository. Modifications to repositoryobjects, import and export metadata can be tracked and a history of changes to repositoryobjects can be maintained.

• Simplified debugging and performance tuning. When data transformation logic is configuredin PowerCenter, the data transformation logic is represented in a mapping, which provides avisual representation of the data flow, making it simple to debug and edit the transformationlogic. Because PowerCenter is a single, unified platform, different functions can be applied to the same metadata without exiting the PowerCenter GUI. For example, a developer mightcreate a mapping to represent data transformation, and then launch the Data Profiling tool to assess the status of the data. Later, the developer can open the Workflow Manager toperform the transformation and launch the Workflow Monitor to track the data as it movesfrom the source to the target.

A tool called the Pushdown Optimization Viewer lets data integration solution architects previewthe flow of data to the source or target database. This tool allows data integration solutionarchitects to preview the data flow, the amount of transformation logic that can be pushed to the source or target database, and the SQL statements that will be generated at run time as wellas any messages related to pushdown optimization. Figure 8 shows the mapping from Figure 6displayed in the Pushdown Optimization Viewer.

Figure 8: Pushdown Optimization Viewer

11Pushdown Optimization

White Paper

Reduced Risk and Enhanced FlexibilityIT organizations typically support several different relational databases. Even when they are able to standardize on a single RDBMS, changing business conditions—resulting from mergersand acquisitions, cost cutting, etc.—dictate that they need to be prepared to support multiplerelational databases architectures. IT organizations need to be able to fully leverage thecapabilities of each type of database, and yet stay agile enough to rapidly integrate other types of databases as the need arises. New regulatory and governance requirements also dictate increased visibility and control into the business rules applied to data as it movesthroughout the enterprise.

PowerCenter reduce the risk of changing database architectures and enhances flexibility by beingdatabase-neutral. PowerCenter’s metadata-driven architecture extends to mappings that leveragethe Pushdown Optimization Option. The appropriate database-specific logic can be easilyregenerated post-database change, providing flexibility of choice and ease of change. Leveragingmetadata analysis and reporting, rather than having business logic tied to vendor-specific hand-coded logic, enables effective data governance and transparency.

Conclusion and Next StepsToday’s challenges to save costs and also drive revenue are pushing organizations to examinetheir current data integration infrastructure needs and choose solutions that provide flexibilityand maximum leverage of current assets.

Informatica PowerCenter provides IT organizations with the flexibility to optimize performance in response to changing runtime demands, peak processing needs, or other dynamic aspects of the production environment, helping IT organizations achieve cost-effective scalability andperformance. By delivering a combined engine-centric and RDBMS-centric approach to dataintegration in a single, unified platform, PowerCenter 8, with its Pushdown Optimization Option,ensures optimal performance for the broad spectrum of data integration projects and helps ITsave costs through the intelligent use of existing computing resources.

With the Pushdown Optimization Option, PowerCenter 8 can help the enterprise:

• Cost-effectively scale through a flexible, adaptable data integration architecture

• Increase developer and team productivity

• Save costs through greater leverage of RDBMS and hardware investments

• Eliminate the need to write custom coded solutions

• Easily adapt to changes in underlying RDBMS architecture

• Maintain visibility and control of data integration processes

To find out how using a combined engine- and RDBMS-centric approach can benefit your data integration initiatives, or to find out more about PowerCenter 8, please visit us atwww.informatica.com or call us at (800) 653-3871.

12

13Pushdown Optimization

White Paper

Worldwide Headquarters, 100 Cardinal Way, Redwood City, CA 94063, USAphone: 650.385.5000 fax: 650.385.5500 toll-free in the US: 1.800.653.3871 www.informatica.com

Informatica Offices Around The Globe: Australia • Belgium • Canada • China • France • Germany • Japan • Korea • the Netherlands • Singapore • Switzerland • United Kingdom • USA

© 2006 Informatica Corporation. All rights reserved. Printed in the U.S.A. Informatica, the Informatica logo, and, PowerCenter are trademarks or registered trademarks of Informatica Corporation in the United States and in jurisdictionsthroughout the world. All other company and product names may be tradenames or trademarks of their respective owners.

J50701 6650 (04/25/06)