Lean Integration: Translating an Innovative Agile … Integration at MISRC[1...1 Lean Integration: Translating an Innovative Agile Approach into Business Value MIS Research Center

1

Lean Integration:Translating an Innovative Agile Approach into Business Value

MIS Research Center SeminarMarch 25 , 2011

John SchmidtVP, Global Integration Services

1

InformaticaThe #1 independent leader in Data Integration

• 2009 Revenue: $500 million

• 5-year AverageG h R 18%

$500

$550

Growth Rate: 18% per year

• Customers: 3,931

• 84 of Fortune 100

• 87%+ of Dow Jones

• Government organizations in 20 countries

• Partners: 400+$250

$300

$350

$400

$450

2

• Major SI, ISV, OEM and On-Demand Leaders

• Employees: 1,755

• Direct in 28 Countries $150

$200

$250

2005 2006 2007 2008 2009

2

Our Singular MissionEnabling The Information Economy

We enable organizations to reduce IT costs and gain a competitive advantage in today’s global information economy by empowering them to access, integrate and trust all their

information assets.

3

The Role of IT in Business

“CEO’s CFO’s and CIO’s all recognize there isCEO s, CFO s and CIO s all recognize there is zero separation between business strategy and IT execution today. There is pretty much nothing you can do, from cutting costs or growth without the enablement of IT in one way, shape or form.”

S S h k b k

4

Steve Schuckenbrock, President, Dell Large Enterprise Business

Reference: Financial Times Special Report, Wednesday, October 27, 2010

3

The Data Driven Enterprise

Partner Trading Network(B2B)

Cloud Computing TraditionalEnterprise

HIPAA

SEPA

NACHA

5

Problem: Data chaos results from unnecessary complexity

Root Cause - Integrations manufactured as custom point solutions

6

4

Craftsman Approach – Custom Development

7

Manufacturing WorldSkilled manual workersManufacture by hand

Custom products (works of art)

Integration WorldSkilled IT Specialists

Hand codingCustom integration points

Assembly Line Approach – This is ICCs now

BenefitsConsistent Quality

Lower Cost

8

Integration WorldDedicated skilled labor

Standard ToolsPeriodic Large Releases

Manufacturing WorldDedicated skilled labor

Standard ToolsMass production

5

Modern Factory Approach – Automated Flow of Materials and Information

BenefitsLow cost AND high qualityExactly the way you want it

Flow of Materials

Flow of Information

9

Manufacturing WorldAutomation – new roles

JIT WorkflowMass-Customization

Integration WorldTemplate-driven self-service solutions

Agile/iterative DevelopmentConfigurable re-useable objects

It is Critical That You Have an Expert and Sustainable Integration Capability

Efficiency Spectrum

Hand codingIntegration

Tools (EAI,...)

Projects& Ad hoc

integration

ProgramManagement

Operational Efficiency

IntegrationTechnology

ManagementPractices

Low High

Integration Platform

Competency Center

Integration Factory Platform

LeanIntegration

10

Fast(quick & dirty)

ImprovedQuality

CostEfficiency

Faster, Better AND Cheaper

BusinessBenefits

Point-in-time Methods Sustaining Methods

1990’s 2000’s 2010’s

6

Cultural Change is easy – isn’t it?

11

Transform complexity and chaos to an Agile, Data-driven Enterprise

Competitive Weapon

Sharedf Dat

a

No Compromise

Complexity Agility

Shared Resource

Controlled cost

Unmanaged silos

Perc

eptio

n of

Business Empowerment

ANDIT Control

StandardizationAND

Innovation

12

Methodology

Transformations

Technology Hand coding IntegrationTools

Integration Platform

Integration Factory Platform

Independent toInterdependent

Tactical toStrategic

Managed toCultural

Project Program CompetencyCenter

LeanIntegration

7

Analyst Perspective

“Taking a production line approach to integrations makes aTaking a production line approach to integrations makes a lot of sense…they should be more widely considered and implemented.”

Philip Howard, Research Director - Data ManagementBloor Research, January 7, 2009

13

Lean Integration Principles

Eliminate Waste Continuous Improvement

Automate Processes Empower the Team

14

Build Quality In

Optimize the Whole

Plan for Change

8

1. Eliminate WasteExample: Value Added Ratio – Before Lean

Scenario: Simple Change Request to the Data Warehouse

5 min 5 min 2 hours 1 hour 1 hour 50 minValue-Add

15

1 week 2 weeks 2 weeks 1 week 2 weeks

NonValue-Add

1. Eliminate WasteExample: Most non-value added activities eliminated

5 min 5 min 2 hours 1 hour 1 hour 50 min

4 hours 8 hours 0 weeks 4 hours 16 hours

Value-Add

NonValue-Add

16

Lead Time Reduction = = = 97% Original Lead Time – Current Lead Time

Original Lead Time

1344 - 32

1344

Lead Time Reduced from 2 months to <1.5 days

9

Archiving reduces datafile size (GB)

1,000

1,200

1,400

Without archiving With archiving …and truncating temp tables

1. Eliminate WasteKeeping data in production that no-one uses is waste

0

200

400

600

800

Current Year 1 Year 2 Year 3

Data that is not used still costs a lot:• Disk capacity (production, test, DW)

B k t (t l b t )

Projected savings by eliminating data waste

$12,907,437

17

• Backup costs (tapes, labor, storage)• Business slow-down by end-of-period jobs• Application upgrade costs• Impact on real-time transactions• Compliance costs• Data quality analysis and resolution time

$1,603,277

$4,925,769$6,378,391

Year 1 Year 2* Year 3 3-year Total

Take Away: Eliminating Waste is not about cutting out fat

“D ’t thi k f l i i di t t l i ht thi k“Don’t think of lean as requiring a diet to lose weight; think of it as focusing the people, process, technology, and architecture that you possess on the projects that have the most impact for the business.”

Mike Gualtieri, John Rymer, Jeffrey Hammond, Top Five Changes For Application Development

18

Forrester, January 4, 2010

10

2. Automate ProcessCreate assembly lines for rapid customization

• Generating custom wizards for templates to maximize productivity for end users– developers and/or analysts

• Ability to build your own wizards based on patterns without coding

IT Designs Templates and Patterns – Users create custom solutions fast

y y p g

• Common data integration and event processing patterns

2. Design Template1. Understand and Define Common Patterns

3. End Users Quickly Configure and Customize in Assembly Line-Fashion

Examples:• Slowly changing dimension in

warehousing• Data file interchange with supply

19

Mapping Template Example: Slowly Changing Dimension Wizard

Huge Productivity Gains for End Users

• Data file interchange with supply chain partners

• Time-based event correlation

Take Away: Automation is a key to simplification

“Factory IT couples lean management techniques y p g qand process improvements with advances in cloud computing and software development to simplify the operating environment and improve productivity and cost performance.”

Reshaping IT management for turbulent times,Roger Roberts,Hugo Sarrazin, Johnson Sikes, McKinsey

20

Quarterly, November, 2010

11

3. Empower the TeamProblem: IT is the bottleneck for data requestsSolution: Give the business self-service capabilities

Business Self-Service

Enables business self-service, improving

business agility

21

business agility

IT maintains governance and oversight

IT personnelSupport

Take Away: Self-service improves business agility AND gives IT better governance

“Lean is about ensuring IT is more closely aligned to the Business.”

Stephen Pritchard, Financial Times

22

Source: Financial Times Special Report, Wednesday, October 27, 2010

12

4. Continuously ImproveProblem: How can groups improve themselves?Solution: Use visual management of the entire integration lifecycle.

Object reuse Project X Development Project Y Development

Data Quality MetricsData Volumes by ProjectDevelopment Activity

23

Take Away: Don’t use Lean to cut staff

“Best-practice executives view lean as a performance improvement strategy rather than merely a cost-cutting exercise.”

Alexander Peters Forrester Research

24

Alexander Peters, Forrester Research

Source: Financial Times Special Report, Wednesday, October 27, 2010

13

5. Build Quality InProblem: Quality issues accumulate without business involvementSolution: Involve business in fixing highest priority issues

Line of business manager

Data Quality

Data Steward

25

ITDeveloper

Take Away: Metadata is an enabler for IT automation

“Metadata allow systems to adapt more quickly toMetadata allow systems to adapt more quickly to changing requirements by pushing power into the data, and out onto users.”

Brian Foote and Joseph Yoder, Big Ball of Mud, Addison-Wesley Software Patterns Series, 2000

26

Reference: Financial Times Special Report, Wednesday, October 27, 2010

14

6. Plan for ChangeInsulate consuming applications from change

Applications PortalsAnalytics

Service ServiceService

Service

ServiceServiceServiceService

Service

ServiceService Service

ServiceService ServiceService

ServiceService Service

ServiceService

CUSTOMER ORDER PRODUCT INVOICE

27

Build Quality

In

ContinuouslyImprove

Optimize The

Whole

PlanFor

Change

Focus on Customer Value and Eliminate

Waste

Lean IntegrationProven Value Proposition: Better, Cheaper & Faster

AutomateProcesses

Empower the Team

Cloud Services The InformaticaPlatform

Mapping ArchitectFor Visio

PCAE - Metadata Manager

Data QualityV9 Data ServicesDatabase Archiving

28

Implemented data quality metrics and monitoring system;

Saved est.$3M in development costs

and avoided $20M in AML regulatory fines

PlatformFor Visio Manager

Saved $1.4M in first 6 months because of improved customer master data quality; Reduced SKU’s by

40% by reducing old or obsolete parts

Centrally managed virtual views

speed up time to add products to

DW portfolio from 1700 hours to 40

hours

Business analysts use Informatica

Cloud Services to self-serve data for their own analysis, cutting lead-times to information by

90%

4 month payback from storage reduction,

and rapid retrieval of archived claims for improved customer

service

After re-working their ETL processes to always use design patterns, achieved 8x efficiencies on

all new projects and re-work on old

projects

Reduced impact analysis time

between 85-95%, reduced time for business analyst inquiries by 50%, reduced end-user support time by

over 65%

Informatica Confidential and Proprietary

15

Manufacturing Production Lines

Bread factoryBottling Line

Car Assembly Line

29

Leading Practice: Integration Systems are Managed Like First-Class Business Systems

Business SystemsIntegration Systems

IntegrationHub

TransformationHub

Application Integration

Enterprise Resource Planning Systems

Sales & Marketing Systems

CustomerSystems

Product and Order Management

Systems

Integration Systems

Human Resource Systems

Finance & Accounting

Systems

Distribution and Inventory Systems

External Systems (Software as a

Service)

Engineering & Manufacturing

Systems

Integration Repository

Portal Process Management

DataWarehouse

Business Analytics

Master Data Management

Extract Transform Load

Data Services

Managed File Transfer

Enterprise Service Bus

Information Security

Enterprise Messaging

InformationPortal

&

Metadata Manager

Process State Engine

CloudServices

Business to BusinessData Quality

30

IntegrationHub

TransformationHub

Integration RepositoryPortal

Application Integration

Process Management

Directory &Single Sign-on

Content Management

Business Glossary

Service Registry

ActivityMonitoring

Workflow

Search Complex Event Processing

Data Stewardship

16

Informatica Platform Mapping

Integration SystemsIntegration

HubTransformation

HubApplication Integration

DataWarehouse

Extract Transform Load

CloudServices

UltraMessaging

CEP

Integration Repository

Portal Process Management

Business Analytics

Master Data Management

Data Services

Managed File Transfer

Enterprise Service Bus

Information Security

Enterprise Messaging

InformationPortal

Metadata Manager

Process State Engine

Business to BusinessData Quality

PowerCenter

B2B DataExchange

ILM

CloudServices

B2B DataExchangeB2B DataExchangeB2B DataExchange

PowerCenterPowerCenterPowerCenterPowerCenter

31

Directory &Single Sign-on

Content Management

Business Glossary

Service Registry

ActivityMonitoring

Workflow

Search Complex Event ProcessingData Stewardship

PowerCenter

Data Quality

MDM

PowerCenterPowerCenterPowerCenterPowerCenter

Data Quality

Informatica Factory Platform Products in Eight Growth Categories

Data Quality Informatica Cloud PlatformData Data Data Data Quality

Global Address Assessment

Informatica Cloud PlatformLoader SynchronizationReplication

Global Address Verification

32

Data Explorer

Data QualityData Transformation

Data Exchange

IdentityResolution

Data Services

PowerCenter

Data Archive

Data Subset

Data Privacy

RulePoint

RuleCast

Real-TimeAlert Manager

PowerExchange MDMAdvanced XMLLBM Streaming

UME Persistence

UME Queueing

17

Production Line for Application Modernization

Data Quality Data Data Data Data Quality

Global Address Informatica Cloud PlatformAssessmentLoader SynchronizationReplication


Informatica Cloud Platform

33

Data Explorer


Data Exchange

IdentityResolution

Data Services

PowerCenter

Data Archive

Data Subset

Data Privacy

RulePoint

RuleCast



UME Persistence

UME Queueing

Production Line for Supply Chain Integration





34

Data Explorer


Data Exchange

IdentityResolution

Data Services

PowerCenter

Data Archive

Data Subset

Data Privacy

RulePoint

RuleCast



UME Persistence

UME Queueing

18

Production Line for Channel Integration (ESB)





35

Data Explorer


Data Exchange

IdentityResolution

Data Services

PowerCenter

Data Archive

Data Subset

Data Privacy

RulePoint

RuleCast



UME Persistence

UME Queueing

Production Line for Business Intelligence COE

Data Data Data Informatica Cloud PlatformData Quality Data Quality

Global Address Loader SynchronizationReplication

Informatica Cloud PlatformAssessment


36

Data Explorer


Data Exchange

IdentityResolution

Data Services

PowerCenter

Data Archive

Data Subset

Data Privacy

RulePoint

RuleCast



UME Persistence

UME Queueing

19

Good technology alone does not guarantee success

“Deploying technology, by itself, is a poor strategy for improving the implementation of interfaces. An effective strategy for integration is established by using these critical success factors: • Establish an ICC. • Develop an integration "city plan" consisting of:

• Policies that implement the necessary governance • A set of best practices that is continually refined by measuring

the result of applying the best practices in integration projects • A technology architecture that establishes standards for the

technologies to be used during application integration

37

• Have integration competency staff drive the selection of the integration technologies that are to be deployed. “

The Seven Critical Success Factors for Application Integration,Jess Thompson, Gartner, November 10, 2010

Lean methods in a nutshell

38

20

Lean methods – a more complete list

39

As-Is Value Stream Map

CR Review Committee

GMNA Applications Team(Irina)

Data Warehouse Team

Change Request

Confirmation Request

Telephone Tag

Status Request

Status Update

Automated Workflow/Tracking (Cust satisfaction)

Data Dictionary to clarify rqmnts N tif C t

Status Request

Status Update

DOIT Corporation: Value Stream Map (AS-IS) for Change Request ProcessMonday, December 20, 2010

Semi-Weekly Review

Requirements Review

Production Deployment

Test ExecutionTesting HandoffDesign & Development

Integration Team Manager

Assign Resource

Test Team Manager

Test SchedulingData Warehouse Team

Production CR Submission

Production Execution

Add CRTo List

ApprovedChanges

ForwardCR Request

To Developer

Clarify Requirements

Requirements Clarification

Approved CR& Design Docs

Design Docs& Schedule

Change Management Board

CR Approval

Daily ETL Batch Run

Test Case Development

Status Request

Status Update

Test Results Distribution

CR Approval& Schedule

Charge Request

Bypass Committee for

simple changes (8 days)

to clarify rqmnts (13 days)

Design Document

Approved Designs

Bypass Council for simple CR’s

(26 days)

Notify Customer(Cust Satisfaction)

(5 days)

Automated Regression

Testing(21 days - requires

investment)

Architecture Review Council

Design Approval

40

Development Team

1

Review

Infrastructure Team

DeploymentTest TeamDevelopment Team

Development Team

Development

CR’sP1x12P2x35P3x124

Automatic Daily ETL Batch Run

Execution

30 Minutes

8.8 Days

15 Minutes

8.5 Days

90 Minutes

12.8 Days

180 Minutes

1 Day

15 Minutes

26 Days

180 Minutes

13.3 Days

Work Time = 510 Minutes (8.5 hrs or 0.35 Days)

Lead Time = 75.6 Days0.3 Days

Test Team

Development

Value Ratio: Work Time / Lead Time = 0.5%Notes:(1) Lead Time includes 5 delay in customer notification(2) Lead Time could be reduced to 24 days with just process changes and using existing tools(3) Lead Time could be reduced to 3 days with a capital investment for automated testing

21

Integration Factory Organizational Structure

41

Helpful resources: Best practice referencesAvailable at amazon.com or integrationfactory.com

• Integration Competency Center (2005): A permanent cross-functional team operating as a shared service function supportingas a shared service function supporting multiple organizational units and sustaining integration in a coordinated manner.

• Lean Integration (2010): A teachable, sustainable management system that

42

g yemphasizes creating value for customers, continuous improvement, and eliminating waste as a sustainable data management and integration practice.

22

43

CIO Magazine, October 2010Staff Pick for IT Management & Leadership

44

23

Sharing Knowledge Among Users

The go-to destination to buy and sell proven data integration, data quality, and data data integration, data quality, and data management solutions (Blocks).

Get promoted! Reduce your time to market with pre-built solutions

Get ahead of the competition! Increase your sales reach with Informatica Marketplace

45

Maximize your Informatica investment with Informatica Marketplace - www.informaticamarketplace.com

Integration Maturity Assessment Survey

Integration Maturity Assessment

50.0

60.0

e LeanLean

Integration Maturity Assessment

50.0

60.0

e

0.0

10.0

20.0

30.0

40.0

Financia

l Man

agem

ent

Integ

ration M

ethod

ology

MetaData

Manage

ment

Inform

ation

Archite

cture

Busines

s Proce

ss

Modelin

g Mana

gement

Integ

ration Sys

tems

A f S b 1 2010

MIN

- M

AX -

Aver

age Lean

Sustaining

Program

Project

Lean

Sustaining

Program

Project0.0

10.0

20.0

30.0

40.0

Financia

l Man

agem

ent

Integ

ration M

ethod

ology

MetaData

Manage

ment

Inform

ation

Archite

cture

Busines

s Proce

ss

Modelin

g Mana

gement

Integ

ration Sys

tems

MIN

- M

AX -

Aver

age

46

http://vip.informatica.com/?elqPURLPage=7960

As of September 1, 2010

24

Impact: IT Costs are Increasing

47

Takeaways

• Lean is Easy – don’t try to be “perfect”

• Learn a Lean technique and start using it• Learn a Lean technique and start using it

• Invest in Factory tools

48

25

49

Reference Models are an essential element of an effective Business Framework

Solutions defined in terms of

How does a framework help?

• Faster projects

• Higher quality solutions

IFW FOUNDATION MODELS



• Decreased cost through reuse of technology assets

• Tighter integration of business and IT – more

flexible and agile organization

• Common Business Language

• Up-to-date Risk and

50

IBM’s IFW

FSDM FSFM FSWM

IFW PROCESS & INTEGRATION MODELS

IFW PROCESS MODELS

FS-BOM

FS-IDM

IFW DATA MODELS

BDWM

BSTs

IFW ODS

Banking


Up to date Risk and Compliance requirements

integrated throughout

26

Case Study: Wachovia’s Business Architecture

51

Information Architecture Framework

Layer 4 – Enterprise ViewOverarching consistent context for executives & information stewards 4

ness

Vie

ws

Reference ModelsBusiness Context DiagramFunction/Information Matrix

M t D t Li t

Metadata(federated

repository of relevant models)Physical Data Models

File/Table PropertiesStorage Information

Logical Data ModelsEntity Relationship Diagram

Information ModelsBusiness GlossaryCanonical Model

Transformation ModelsInterface Specifications

Transformation rulesProtocol Definitions

Interaction ModelsLogical Data Map

Sequence diagrams

Process ModelsBusiness Event Model

Operational Workflow Model

Inte

grat

ion

View

s

Layer 3 - Business ViewDomain models for business owners and project sponsors

Layer 2 – Solution ViewArchitecture models for specific

systems and solutions

1R 1M

2R 2M

3R 3M

Layer 1 –Technology ViewTechnical models for developers,

engineers and operations staff

Bus

inMaster Data List4

52

Data at Rest Data in Motion

A layered approach is needed to manage data complexity, provide multiple stakeholder views, and maintain traceability to business requirements.

27

Data Governance Methodology

2

One-Time Effort Repeat for every program

10Implement Projects


1Organize

Governance Committee

Define Governance Framework

4Assign

Organizational

5Scope

Program

3Develop

Enterprise Models

6Assess

Baseline & Data Quality

8Plan Migration

Roadmap


7Develop Target

Architecture

9Develop Program Models

53

gRoles

Entry Points1. Enterprise Initiative (Large-scale program with defined ROI – e.g. Merger or major business transformation)2. Top-level Directive (C-level or higher – e.g. one view of customer/business, control IP, data security, etc.)3. Scale-up DQ and MDM Projects (Leverage successful solutions that demonstrate the value for a specific

business area and scale them up to cross-functional enterprise-wide initiatives)

Documents

Lean Integration: Translating an Innovative Agile … Integration at MISRC[1...1 Lean Integration: Translating an Innovative Agile Approach into Business Value MIS Research Center