Fully Automated SOA ETL Metadata Capture Soln

Copyright © 2007 CA. All rights reserved. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies.

A Fully Automated Metadata Capture Solution

Session # DG105SNFocus Area DADIM


Abstract

The Harris Information Warehouse Environment (HIWE) Reporting environment in cooperation with the Allfusion Repository for Distributed Systems has delivered a fully automated metadata capturing solution, a first of its kind at BMO. The technical foundation of this solution is based on Service Oriented Architecture (SOA) implemented using Simple Object Access Protocol (SOAP) as the communication framework. This session will provide an architecture / technical overview of the automation of the end-to-end processes involved; from the initial creation of metadata in reports hosted on Microsoft Reporting Service platform; to the final Enterprise Metadata Repository website.


Company

BMO Financial Group (NYSE, TSX: BMO), Established in 1817 as Bank of Montreal, BMO Financial Group is a highly diversified North American financial services organization. With total assets of $320 billion at October 31, 2006 and almost 35,000 employees, BMO provides a broad range of retail banking, wealth management and investment banking products and solutions. Canadian clients are served through BMO Bank of Montreal, BMO Nesbitt Burns, BMO InvestorLine and BMO Harris Private Banking. In the United States, clients are served through Chicago-based Harris Bank, an integrated financial services organization that provides banking and investment services. BMO’s North American investment and corporate banking division, BMO Capital Markets, provides a full suite of financial products and services to North American and international clients.


Biography

Mark PahuljeBank of Montreal

7 yrs. in Metadata Management Group

Winners of Wilshire Award for Best Practices in Metadata Management 2006

In previous life, 5yrs experience a BI Consultant, during which time developed MIND and won award as profiled in "DB2's Bank of Montreal to save $270M with Web-enabled business intelligence" using Microstrategy’s Business Intelligence Platform.

Winner of User Interface Engineering’s Advanced Design & Usability Techniques team challenge at BMO, involving 250 participants.

Aspire to ride “Le Tour de France”[email protected]


Agenda

Background Info (5mins) Presentation (45mins)

Metadata Management overview HIWE project overview Metadata Architecture overview Walkthrough of a Capture Process Parsing Problem Walkthrough of Metadata Website

QA (10mins)


BMO’s Metadata Team

The Metadata Management group is part of the Information Management department within the Enterprise Systems organization of BMO FG’s Technology and Solutions.

embedded within the Bank’s systems development life cycle as part of the CMMI program

part of standards process for ensuring the integrity of customer information

engaged in deliver specific types of metadata through external projects and business initiatives.

Our involvement in projects ranges from managing metadata within traditional IT systems development, data marts or vendor solutions to implementing data quality initiatives, or delivering information strategies and architectures.

In spite of these consulting services, Metadata Management is not considered to be a standalone program.


We embed governance and quality as part of an overall information strategy aligned to business goals

Value CreationBusiness Intelligence

Centre of Competency

Architecture Renewal

Data MartConsolidation

AccessTechnology Strategy

(Metadata)

Governance & Quality

Policy & StandardsGovernance Process

Info StewardshipIntegrated Change

Management Best in Class Delivery

Global ResourcingCMMI

Key enterprise initiatives

InformationInto Action:

Competitive advantageReal Outcomes

Trusted Information

Best in Class People

High Performance Culture

Resource Management


BMO Financial Group honored with 2007 WILSHIRE AWARD for Metadata Best Practices

LOS ANGELES—January 18, 2007—Wilshire Conferences, Inc. and The Data Administration Newsletter (TDAN.com) have announced the winner of the 2007 Wilshire Award for Metadata Best Practices. From among a pool of exceptionally accomplished nominees, the panel of eight judges has chosen the BMO Financial Group (BMO) as this year’s recipient for its implementation of an outstanding, multi-faceted metadata initiative.

Candidates are evaluated on numerous best-practice criteria including (though not limited to) the following:

Overall business impact/benefits Return on objective Measurable return on investment Scale of the solution Number and breadth of applications

served by the solution Length of time the solution has been in place Innovative problem solving How well the submission addresses

the questions asked above


Harris Information Warehouse Environment (HIWE) Project Mission

Implement a new Data Mart and supporting infrastructure - replace Alltel Information Warehouse not renewed in 3ys

Enhance Marketing & Information Decision Support (MIDS)

Reduce Operational Costs and time to produce information

Support aggressive growth strategy Support Enterprise-wide information needs As much as 30% of MIDS resources has to

be dedicated to supporting the environment to compensate for current limitations

Time to meet information delivery and performance of the system is inadequate to meet client expectations and demands.


HIWE Project Timeline

Metadata Completed

HIWE Project is initiated

HIWE planning phase is

completed and Stage 1 Started

MIDS writes high level andDetailed Business Requirements

Harris Standard Staging development

Harris Standard Staging History

Load and MIDS/Basel UAT

Starts

HSS load completed except MLS and Pershing

Harris Marketing Datamart development

MLS and Pershing load complete

Stage 2 – Report Environment, Report Replacement, Data Mining work space, Additional Interfaces, Change Records

UNICA Implementation

Stage 2 Funding

Datamart Loading Starts

SRA #1 signed for Stage 1

Datamart Loading

Complete

O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J| | | | | | | | | | | | | | | | | | | | | | | | | | | c | | | | | | | | | | | | | | | | | |

2002 2003 2004 2005 2006

Harris Information Warehouse Environment – High Level Application Architecture

CWAN

Event Triggering

S/W

CWC

Jan 27, 2003

Standard StagingTransformation Env.

Standard Staging Area

Meta Data Collection Processes

ETL

(Abinitio)

Reconciliation

Subject Files including :BMO & Harris Common Dimensions, LOB data

Exchange - Valex

VXSI & ControlTables

Campaign Management

Customer ProfitabilityProcessing

MIDS Marketing Data Mart Transformation Env.

ETL(Abinitio)

Reconciliation

CustomerProduct

Market Analysis

Model Building Data Mining Env.

SAS Base, Stat. Access

Connect,Graph

SAS Enterprise

Distributing Channel

Performance

Event Triggering

BMO IntranetExecutiveDash Board

Other Web App.Dream Screen

Microsoft Reporting Services

CAMeta Data

Repository

Firewall

MIDS, LOB Users

IBM SP690 - AIX, UDB Data Mart Env.

IBM SP690 - AIX ETL Env.

AIX - UDB, MFS

IBM SP690 - AIX, ETL Env.

ESP

DATA

Rention

Credit RiskBasel Data Mart

Norman

Micro StrategyNarrow Caster

Cell PhonePDA etc..

Not in current scope

ExternalApp. ???

Not in current scope

User Desk Top ApplicationsBrowser - Netcape / I.E.Campaign Management - Exchange Valex OLAP & Adhoc Reporting – Microsoft Reporting SevicesSecurity - SSH clientClient Office tools - Lotus Notes, MS Office ProDB SQL Access - UDB Client, Command CenterData Mining- SAS Enterprise Miner (server), Agnoss - Knowledge studioHIWE Administration tools - Trillium Control Center, CWC client interface, ESP, Data Retention Facility, 3270 Emulator, FTP

Alltel - IW (Oracle) to be decommission

IW

IBM SP690 - AIX, UDB Data Mart Env.

Trillium

ProMod

Basel

SAS datasets

History Data

Model tables

Marketing Data Base

MIDS Marketing Data Base

Reports: Risk Management ; Cool

Micro StrategyMetadata

Event Data

Customer Transactions

Privacy Preference Tracking

Annual Aggregates

Account / HHProfitability

Marketing Analysis

Product Pricing

Customer Profiles

LOB requestedExtracts

Monthly

1. IM2. ST3. AM4. CL5. RE6. ECM7. ACOM8. TR29. FSR10. SBA11. VRU12. Pershing13. BC14. IB15. Debit Card16. Hogan-CLS17. Hogan-DDA18. Hogan-FCA19. Hogan-FPS

Current Source System Feeds

Furture Datafeeds

Daily

1. NRM2. TLR3. ATM

Bi-Monthly

1. GL2. PRI

Quarterly

1. IXI

Annually

1. Dunn& Brad Street2. OXX3. Business Owner Link4. PROS BUS

Semi-Annually

1. ACX

On-requet

1. PROS2. Personnel Files

Monthly

1. KO2. NCCS3. MRP


Enterprise Repository Architecture

Population Application

Oracle DBMS

Data Data SourcesSources

AdminAdminUserUser

OCIF, BIW, Basel…

Repository Database contains 500 tables to model date types. During a load any number of tables maybe appended too.

When completed, results are immediately available to web users and repository clients.

Windows ClientsData Tier

Metadata Management - The Roadmap to your Information Assets

ActiveX/COM*Assembler

*C/C++*Cobol

COOL:DBA*Excel

Spreadsheets *DB2-MVS

*Word Documents

*ErwinETI

IDMS*GlossaryInformix

*JCLJava

MS SQL*ODBC

*OracleParadigm Plus

PeopleSoftPowerBuilder

SybaseVisual BasicVisual C/C++

*LDAP* currently

owned

Data Source Types

Web Web ClienClien

tt

Repo Repo ClientClient

Corporate Data Artefacts

PresentationTier

BMO Corporate Repository - Custom Web Application

Advantage Repository Engine processes request and loads the data into the database.

Basel ETL….

Basel Erwin Model…

Web ServerEnd

Users

Advantage Repository

Engine

Repo Repo ClientClient


Standard Scanning Process Diagram

AllFusion Repository

ERwin ERX File

PCAF Reuse File

*.pcr

PCAF Control File

*.pcc

PCAF Data File

Output.pcd(temporary)


HIWE Meta Model Extensions


Web Service Oriented Architecture 101

Web Services Description Language

Universal Description Discovery and Integration

Simple Object Access Protocol

HIWE Reporting Web Service Provider

HIWE Reporting Web Service Consumer

Wide Screen Shot With a Header

Automated Metadata Scan Architecture

HIWE Reporting Web Service Provider

Metadata Repository Database

1. HIWE Web Consumer Apprequests and consumes soap message from Microsoft Reporting Web Service provider & parses & matches meta-model relationship chain, if found in repository db. }}} Output: A file in .csv format text file.

PCC & PCD

2. General Custom Scan App reads a CSV file and produces proprietary CA Allfusion Repository for Distributed Systems load files. }} } Output: Load files (.pcc,.pcd)

Advantage Repository Engine (ARE)

WindowsScheduler

3. PCAFLDC is a generic out-of-the-box vendor cmd line loader utility that reads repository files (.pcc, .pcd, .pcr) into the Metadata Repository database processed by ARE service. }}} Output: Log file (.log)

CSVEmail Alert

log

4. HIWE Sweeper Appalerts Metadata team via email to any errors or completions of previous apps.

PCR

XSD

XML

New

XML

http://images.google.ca/imgres?imgurl=http://jamesburnsdesign.com/images/3DExamples/clock.jpg&imgrefurl=http://jamesburnsdesign.com/3dgraphics.html&h=500&w=500&sz=41&tbnid=dpA_SwyPlHbSIM:&tbnh=127&tbnw=127&hl=en&start=5&prev=/images%3Fq%3Dclock%2B3d%26imgsz%3Dsmall%257Cmedium%257Clarge%257Cxlarge%26svnum%3D10%26hl%3Den%26lr%3D%26rls%3DGGLG,GGLG:2005-20,GGLG:en%26sa%3DG

http://images.google.ca/imgres?imgurl=http://jamesburnsdesign.com/images/3DExamples/clock.jpg&imgrefurl=http://jamesburnsdesign.com/3dgraphics.html&h=500&w=500&sz=41&tbnid=dpA_SwyPlHbSIM:&tbnh=127&tbnw=127&hl=en&start=5&prev=/images%3Fq%3Dclock%2B3d%26imgsz%3Dsmall%257Cmedium%257Clarge%257Cxlarge%26svnum%3D10%26hl%3Den%26lr%3D%26rls%3DGGLG,GGLG:2005-20,GGLG:en%26sa%3DG


BA User’s Report 2590


HIWE Report XSD


Web Consumer Consumes Sample Reports XML file

* Footnote; Verdana 9pt, Bold


HIWE Web Consumer Email


HIWE Web Consumer CSV


HIWE GCS Mapping File XML



2. General Custom Scanner Email


Sample PCD Generated File

PLATINUM Compressed Aggregate Format Data File v.2.0.0//E,ELEMENT,12,3,C11,Region Name,C10,PRODUCTION,S0T,0,12,1,DESCRIPTION,11,Region NameE,REL_DT,13,3,C8,VARCHAR2,C10,PRODUCTION,S0R,RPT_FLD,14,12,2,2,C55,Secured Loans Report (For Metadata testing

2).DERIVED_3,C9,DERIVED_3T,0,14,1,DERIVATION,37,=Trim(Fields!RGN_NM.Value) ' Total'T,0,14,1,WHERE_CLAUSE,37,=Trim(Fields!RGN_NM.Value) ' Total'R,RP_HS_FL,15,4,14,0E,TABLE,16,3,C5,RGN_C,C10,PRODUCTION,S1R,REL_COL,17,12,13,4,C6,RGN_NM,S80,S0,S0R,TBL_COL,18,16,17,0R,FLD_SRC,19,14,18,0R,RPT_FLD,20,1,2,2,C55,Secured Loans Report (For Metadata testing

2).DERIVED_4,C18,Cur Bal (Subtotal)T,0,20,1,DERIVATION,30,=Sum(Fields!CUR_BAL_AMT.Value)T,0,20,1,WHERE_CLAUSE,30,=Sum(Fields!CUR_BAL_AMT.Value)R,RP_HS_FL,21,4,20,0R,RPT_FLD,22,1,2,2,C55,Secured Loans Report (For Metadata testing

2).DERIVED_5,C18,Cmt Bal (Subtotal)T,0,22,1,DERIVATION,37,=Sum(Fields!CUR_CMTMNT_BAL_AMT.Value)T,0,22,1,WHERE_CLAUSE,37,=Sum(Fields!CUR_CMTMNT_BAL_AMT.Value)R,RP_HS_FL,23,4,22,0R,RPT_FLD,24,12,2,2,C52,Secured Loans Report (For Metadata testing

2).RGN_NM,C6,RGN_NMT,0,24,1,DERIVATION,13,Straight moveR,RP_HS_FL,25,4,24,0R,FLD_SRC,26,24,18,0


HIWE PCC File


HIWE Report Reuse Rules .PCD


3. PCAFLDC.exe Scheduled Batch File

HIWEParmDaily.txt

-cHIWEWCOutputDailyFile.pcc-fHIWEWCOutputDailyFile.PCD-rHiweDailyPrscan.log-l-uirm_prod-prepadmin-REndeavorCA-tPRODUCTION-e0-*BMO_REPORT3.pcr-nHMDM Reporting-yCATEGORY

timeit PCAFLDC "PCAFParmDaily.txt" > "PCAFParmDailyCmdLE.err“tail /28 “HIWEDailyPrScan.log" > "PCAFImportReportSummary_Tail.log"


HIWEDailyPrScan.log

PCAF Import ReportControl file: HIWEWCOutputDailyFile.pccData file : HIWEWCOutputDailyFile.PCDParsing Reuse Rules File ==>BMO_REPORT3.pcr

HMDM Reporting - Populating ContainerDangling relationships:

NoneForcing database COMMITObject Reuse Testing Begins ...Perfect: ELEMENT Name: Unknown HIWE element...Adding: REL COL Name: BUC_CD...Perfect: RPT FLD Name: 2590 Privacy Law States - Counts and Listing.DERIVED_1....Importing...

Using container with name: HMDM Reporting id: 412682690.

0 seconds elapsed=============================================================== PCAF Import Report - summary===============================================================Control file: HIWEWCOutputDailyFile.pccData file : HIWEWCOutputDailyFile.PCD Entity RelateParsed: 43 295Fail to parse: 0 0Duplicate tag: 0 0To hookup: 43 295Dangling: 0 0Perfect: 43 291Updated: 0 0To add: 0 0Fail to add: 0 0------------------------------------------Total added: 0 0Container relates added: 0Container relates reused: 342Container relates skipped: 0Container relates fail: 0

4.PCAFImportReportSummary_Tail.log


4. HIWE Scan Sweeper Email


Parsing Problem- Stored Procedure RDL

<DataSets> <DataSet Name="dsetSecuredLoans"> <Query> <CommandType>StoredProcedure</CommandType> <CommandText>BPAI.PSECUREDLOANS</CommandText> <QueryParameters> <QueryParameter Name="PROFILEENDDATE"> <Value>=Parameters!PROFILEENDDATE.Value</Value> </QueryParameter> </QueryParameters> <DataSourceName>dsHIWE</DataSourceName> </Query> <Fields> <Field Name="LOAN_GRP_CD"> <rd:TypeName>System.Int32</rd:TypeName> <DataField>LOAN_GRP_CD</DataField> </Field> <Field Name="RGN_NM"> <rd:TypeName>System.String</rd:TypeName> <DataField>RGN_NM</DataField> </Field>


Sample Stored ProcedureCREATE PROCEDURE PSECUREDLOANS (IN PROFILEENDDATE DATE) SPECIFIC PSECUREDLOANS READS SQL DATA NOT DETERMINISTIC CALLED ON NULL INPUT LANGUAGE SQL DYNAMIC RESULT SETS 1========================================================================================================================================

P1: BEGIN

DECLARE currSQLState CHAR(5); DECLARE SQLCODE INTEGER DEFAULT 0; DECLARE SQLSTATE CHAR(5) DEFAULT '00000'; DECLARE DT_FROM_CURSOR DATE;

DECLARE EXIT HANDLER FOR SQLEXCEPTION ErrHandler: LOOP SET currSQLState = SQLSTATE; LEAVE ErrHandler; END LOOP;

IF (PROFILEENDDATE IS NULL) THEN-- SET PROFILEENDDATE = '12/31/9999'; SET PROFILEENDDATE = (SELECT CUR_DT FROM HMDM.CUR_DT_CNTL_D WHERE SRC_SYS_ID = 'AGGR'); END IF;

-- CALL BPAI.FIND_EOM_DT(PROFILEENDDATE);

BEGIN DECLARE C1 CURSOR WITH RETURN FOR select 1 as Loan_Grp_CD, b.rgn_nm, b.rgn_cd, a.acct_nbr, a.area_cd, a.buc_cd, a.acct_typ_cd, af.cur_bal_amt, lf.cur_cmtmnt_bal_amt, a.opn_dt, a.status_cd, a.prfl_end_dt from hmdm.cons_acct_d a inner join hmdm.buc_org_c b on a.buc_cd = b.buc_cd inner join hmdm.cons_acct_f af on a.sk_acct_id = af.sk_acct_id inner join hmdm.cons_ln_f lf on a.sk_acct_id = lf.sk_acct_id inner join hmdm.mrp_iw_prod_xref m on a.prod_id = m.prod_id where a.src_sys_id in ('XAM', 'XIM') and m.iw_prod_id in (700000000031, 700000000134) and a.status_cd <> '050' and a.prfl_start_dt <= PROFILEENDDATE and PROFILEENDDATE >= a.prfl_end_dt and a.prfl_actv_ind = 1 UNION…


Sample DAD File

What is a Document Access Definition (DAD) ?A DAD (Document Access Definition) file is an XML-formatted file that associates XML document structureto a DB2 database. It is used by the DB2 XML Extender to compose and decompose XML data into relationaldata. For more information on DAD files and their particular syntax, refer to the DB2 XML Extender Library


RDB to XML Mapping Editor


Walkthrough of Metadata Website



Wide Screen Shot With a Header

User Reports Generated


Analysis of Report Privacy Law States [2590]

1.2.3.4.


1. Which Application?


2. Fields in this Report?


Report Field TID Details

TID

TID


Table Column Details for TID


Partial Physical Table Listing


3. Sourcing?


4. What FileName?


Recommended Siteshttp://www-128.ibm.com/developerworks/websphere/techjournal/0111_lau/lau.html


Summary

A few words to review

If you can influence the Reporting Tool choice

Choose a parsing strategy early Generalize your architecture to be

repeatable for any reporting tool


Questions & AnswersMark Pahulje

Metadata Specialist

416 513-5636

[email protected]

BMO Financial Group

Technology and Operations

Information Management

120 Bloor Street East, 5th Floor

Toronto, Ontario

Canada M4W 3X1

www.bmo.com


Related Sessions

Session # Insert title of session related to your session topic

EJM01SN Enterprise Job Management Status and Plans


Exhibition Center

Related CA and Partner Technology Computer Associates

Booth 123 - Product X Booth 456 - Product Y

Partner Booth 998 – Company / Product X Booth 999 – Company / Services Y

Exhibition Center Tours Sign up at Information Desk Booth #xxx


CA Technology Services and EducationHear how CA’s learning solutions can help you meet your business objectives Visit CA Education in the Exhibition Center

in Booth xxx, visit ca.com/education or call us at 1-800-237-9273

Learn how CA Technology Services can help your business Visit the CA Technology Services stations in

the exhibition center or on the web at ca.com/services


Session Evaluation Form

After completing your session evaluation form ...

... place it in the basket at the back of the room.

Please left justify the session number