IDE 5.0_Basics_20061025

1

Using Informatica Data Explorer 5

Informatica Corporation, 2005-2006. All rights reserved.

Education Services

Version IDE-25102006

2

Agenda

• Overview of Informatica Data Explorer

• Importing Metadata and Accessing Source Data

• Column Profiling

• Data Rules

• Single Table Structural Analysis

• Cross Table Profiling

• Validating Table and Cross Table Analysis

• Normalization

• Repository

• Using the Repository Navigator

• Using Repository Reports

• Integration with PowerCenter

3

Introduction

4

Introduction Objectives

• Identify the components of the Informatica Data Explorer product suite

• Describe the Informatica Data Explorer process flow

Informatica Data Explorer Product Suite

IDE Source Profiler

IDEClient

Windows XP, 2000

IDEServer

Unix or Windows

UDB

Informix

Sybase

Oracle

COBOLPrograms

IDEFTM / XML

DDL,XML

& DTDs

IDE Repository

FlatFiles

IDEImport

Flat File

DDL

IDEProject

RepositoryNavigator

IDE ImportIMS

IDEImportVSAM

VSAM

IMS

Sequential

PDS

OS/390DB2

Unload

Command Files &

JCL

IDE Repository

PortsMSSQL

IDE Import

Relational

Via ODBC

6

FixedTargetMapping

RepositoryNavigator

Ports:InformaticaCWM

Sources

RDBMSODBC

FlatFile

* MainFrame

Connectors

Data Table n

Data Table 2

Data Table 1

Co

nten

t Pro

filin

g

Cross

Tab

le P

rofili

ng

Single Table profiling

IDE Profiling IDE Design

Consolidated Schema

Source DataKnowledge

Base

StructureContentQuality

Product Architecture

* The IMS and VSAM importers actually use a GUI (Source Profiler) to read a Copybook and generate a program to extract mainframe data into Flat Files for use by IDE

7

IDE Server Platform• Windows (2000/2003/XP)• Sun Solaris (7,8,9)• HP-UX (11 or later)•AIX (4.3, 5L)

IDE Server

Repository DBMS and

Server Platform

ODBC DriverConnectivity

Client

Workstation• Windows 2000• Windows XP

IDE Client

Project

Files

Workstation• Windows 2000/XP

Repository

Navigator

Workstation• Windows 2000/XP

FTM/XML

Data / Header

Files

•IBM DB2 UDB-7.2,8.1 •Informix 7.31,9.2,9.3 Informix 9•Microsoft SQL Server 7 and 2000)•Oracle 8i, 9i•Sybase 12 and 12.5

Relational Importers for:•IBM DB2 UDB-5.2, 6.1,7.1,7.2,8.1•Informix-7.24,7.31,9.1,9.2,9.3•Oracle -7.3,8,8i,9i•Sybase 10,11,12•ODBC(SQL Server, etc.)

Performs Actual Profiling

Profiling Results initial store

RepositoryCompleted Profiling

ResultsRepository does not

need to be on the same server as IDE

TCPIPConnectivity

ODBC/JDBCConnectivity

ODBC Driver (API 3.x conformance level 2)

Ports

XML format files

Technical Diagram

Flat File Importer for:•Fixed Length•Delimited•DB2 Unload

IDE Process Flow

IDE Data Profiling

IDE Data Prep / Import

Products

IDE Schema Development

• Data Extraction • Cleansing• Transformation

Specifications

FTM / XML Metadata Mapping

DB Load

Messaging

Target DB

Target DesignOR

Message

IDE Repository and NavigatorMetadata Management

OR

OR

Data• Relational• Flat Files• VSAM• IMS• ODBC

Documented Metadata

9

Introduction Review

• The Informatica Data Explorer product line: • Informatica Data Explorer

• Importer for Flat Files• DDL Generators• Source Profiler• Repository Navigator• Repository

• Import for Relational Databases• Import for VSAM• Import for IMS• FTM

10

Introduction Review (cont.)

• The IDE Process Flow consists of five major processes:• Data Preparation and Import

• Data Profiling

• Schema Validation and Development

• Metadata Development

• Metadata Management

11

Lesson 1

Importing Metadata and Accessing Source Data

12

Lesson 1 Objectives

• Explain what an Informatica Data Explorer Project is, and how it is used

• Create and setup Informatica Data Explorer Projects

• Define the term “metadata” as used by Informatica Software

• Explain the importance of metadata in Data Profiling using Informatica Data Explorer

13

Lesson 1 Objectives (cont.)

• Explain what source data are, and the ways in which they may be imported into Informatica Data Explorer

• Explain what the Informatica Data Explorer Flat File Importer does

• Describe the format of an Informatica Data Explorer Flat File, including the minimum requirements for Informatica Data Explorer to use it to access source data

14

Case Study Description

• The Customer Order system is a mainframe application accessed through a CICS user interface

• It was developed 10 years ago

• The Employee Identification system is an Oracle database created 2 years ago

• Business users are sure they know the data

• Senior executives suspect the quality of the data is bad

IDE Source Profiler

IDEClient

Windows 98, NT or 2000

IDEServer

Unix or NT

UDB

Informix

Sybase

Oracle

COBOLPrograms

IDEFTM / XML

DDL, IDE XML& DTDs

IDE Repository

FlatFiles

IDEImport

Flat File

DDL

IDEProject

RepositoryNavigator

IDE ImportIMS

IDE ImportVSAM

VSAM

IMS

Sequential

PDS

OS/390DB2

Unload

Command Files &

JCL

IDE Repository

PortsMSSQL

IDE ImportRelational

Via ODBC

Informatica Data Explorer Project

16

Informatica Data Explorer Projects

• The persistent data store used by IDE

• A Project is a UNIX or Windows NT container (directory, folder etc.)

• Projects contain:• Metadata

• Data

• Profiling as well as Mapping information

• Projects are opened and closed by the IDE Server

17

What is Metadata?

• Informatica Data Explorer defines metadata as:• Data that describes data

• Information about the characteristics of source data

• In Informatica Data Explorer, metadata is information that will create: • Schemas

• Tables

• Columns

• Other objects

18

Why Import Metadata?

• Must be imported into an Informatica Data Explorer Project before any subsequent tasks or activities can be started

• Informatica Data Explorer needs? to know the names of the Columns in order to store Data Profiling results

• Informatica Data Explorer needs to know how to interpret the source data (Fixed vs. Delimited)

• Provides basis for automated quality assessments in data profiling

IDE Data Sources

IDE Source Profiler

IDEClient

Windows 98, NT or 2000

IDEServer

Unix or NT

UDB

Informix

Sybase

Oracle

COBOLPrograms

IDEFTM / XML

DDL, IDE XML& DTDs

IDE Repository

FlatFiles

IDEImport

Flat File

DDL

IDEProject

RepositoryNavigator

IDE ImportIMS

IDE ImportVSAM

OS/390

Command Files &

JCL

IDE Repository

PortsMSSQL

IDE ImportRelational

Via ODBC

VSAM

IMS

Sequential

PDS

DB2Unload

20

IDE Flat Files

• Consist of two components

• Header File• Contains metadata describing contents of a data file

• Data file• Data in either delimited or fixed column format as well as

DB2 Load format

21

IDE Flat Files (cont.)

• Header and Data files may be • Separate files or

• Combined into one file

• A header file should not contain duplicate column names (IDE will automatically re-name them)

• IDE Flat Files may not contain Arrays (repeating groups or occurs)

Informatica Data Explorer Flat File Components

header:file=empinfo.dat

attribute:EMPIDdata_type=INTEGERnull_rule=NOT NULLmin_value=1000max_value=9999

attribute:LAST_NAMEdata_type=CHAR(20)null_rule=NOT NULL

attribute:FIRST_NAMEdata_type=CHAR(20)null_rule=NOT NULL

attribute:GENDERdata_type=CHAR(1)null_rule=NOT NULL

attribute:DEPTIDdata_type=CHAR(4)null_rule=NOT NULLmin_value=100

149,Francis,Lynn,3,200,MIS,Database Administrator,"120 Co

249,Venkatachalam,Nagarajan,3,200,MIS,Project Leader,"300289,Kim,Suk,3,200,MIS,Staff Consultant,"4040 N Fairfax Dr216,Masood,Airaj,,200,MIS,MIS Analyst,"300 N Wakefield Dr134,Swenson,Allison,F,200,MIS,Database Administrator,"900 164,Park,Allison,F,200,MIS,Database Analyst,"PO BOX 1471"323,Blaskiewicz,Allison,F,200,MIS,Technical Specialist,"3255,Barbles,Amy,F,100,Sales,Sales Executive,"4019 Rice Bl273,Karneh,Anna,1,200,MIS,Sr Prog Analyst,"12601 Fair Lak

Header File Data File

The data file example shows the associated comma delimited file to which this header file refers

The header file example shows some of the documented information that can be loaded into Informatica Data Explorer

23

Header and Data Files

• Header and data can be in one file

• We recommend that two files be made if created manually

• The more information that is provided in the header file, the more automatic comparisons Informatica Data Explorer can make

24

Login

25

Open Project

26

Import Metadata

27

Lab Exercises 1.1–1.6

28

Lesson 1 Review

• An Informatica Data Explorer Project is:• The persistent data store used by Informatica Data

Explorer

• Used to organize and partition the work effort

• Metadata describes the data source and is used by Informatica Data Explorer to access the source data

• A structure that contains: • Metadata

• Data

• Profiling and Mapping information

29

Lesson 1 Review (cont.)

• Informatica Data Explorer can import data from:• Relational Databases

• Oracle 7.3, 8, 8i, 9i or 10g

• Informix 7, 9.1, or 9.2

• Sybase 10, 11, 12 or 12.5

• IBM DB2 UDB 5.2, 6.1 7.1 or 7.2

• Microsoft SQL Server 7 and 2000 (using an ODBC driver)

• Flat Files• Delimited Format

• Fixed Length Format

• DB2 Load Format

30


• Informatica Data Explorer Flat Files must be:• ASCII or EBCDIC character format (no binary data)

• Binary data is supported via the DB2 Load Utility format

• Informatica Data Explorer Flat File may not contain:• Arrays (repeating groups or occurs)

• Duplicate column names

31


• Informatica Data Explorer Flat Files must have a header file along with the data file

• Additional information on data preparation is available in the Using Informatica Data Explorer Source Profiler course and the documentation

32

Lesson 2

Column Profiling

33

Lesson 2 Objectives

• Explain what Column Profiling is, and why it should be performed.

• Execute the Column Profile function of Informatica Data Explorer.

• Navigate and review the results of Column Profiling.

• Explain informational Tags.

• Describe when and how to apply informational Tags to Informatica Data Explorer objects.

34

What is Column Profiling?

• A process of discovering physical characteristics of each column in a file

• Comparing documented Metadata against Metadata inferred from the data source

• Column Profiling is done against data in the form of • ASCII flat files

• DB2 Load Utility files

• RDBMS tables

35

Why Profile Columns?

• Not all database metadata and documentation are accurate pictures of the data source

• Documented descriptions of data elements may be inconsistent with the way the element is actually used

• Informatica Data Explorer Column Profiling builds a description of a column (its metadata) based on the data it contains

36

Column Lists

• The results of Column Profiling are stored with the Columns in a Table

• Column List viewers can be opened from the Navigation Tree

• Column List viewers provide information about Documented and Inferred Metadata• Documented Metadata are supplied from the header

file or source table

• Inferred Metadata are those that Informatica Data Explorer determined from examining the data.

37

Column Profiling

38

Column Viewer

39


40

Drill Down

• Allows you to perform ad hoc drill downs through data presented in the Informatica Data Explorer viewers.

• Used to interrogate any data sources that can be accessed via an ODBC connection or Informatica Data Explorer Importer.

• Searches are issued against the selected data, and rows are returned for the specified search.

Drill Downs

41

Column Details

• Lists of properties about a Column that have been inferred by Informatica Data Explorer

• Columns can have several potential sets of characteristics

• The potential sets of characteristics are dependent on the physical view that is chosen

42

Drill Down

43

Drill Down Results

44


45

Column Value Pairs

• Informatica Data Explorer will store per Column• Up to 16,000 distinct values

• These are the most frequently occurring values from the set of all values that were observed during the Column Profile execution

• The frequency with which each value was observed

• Informatica Data Explorer will calculate• % Distribution for each distinct value based on the frequency

divided by the total rows profiled

46

Value Pair Review

• Issues to evaluate during Column Value Pair analysis:• Are the values/range of values correct?

• Is the data type correct?

• Is there a pattern or format to the data for this Column? Do all of the values match this pattern/format?

• Is there a difference in case for alpha characters? Are some values mixed case are others all upper or lower case? Is this an issue?

• Are there different representations (different abbreviations/misspellings) of the same data?

• Are there duplicate values in a field that should be unique?

47

Sorting Viewers

• It is possible to sort any of the tables displayed in Informatica Data Explorer

• By clicking on the column header, the results will be sorted in ascending order. Double-clicking again will sort the list in descending order

Value Pair Review

48

Sort Order

• Sorting is based on the character codes of the values in the data:• Spaces sort to the top of an ascending sort. When the caret

(^) symbol is displayed, the sort is based on the actual “space” character not the caret (^).

• Special characters (i.e. #, &, ‘)

• Nulls

• Numbers

• Alpha characters

49


50

Tags

• Informatica Data Explorer Tags come in various forms, depending on the type of information you want to convey:• Notes – general text

• Action Items – things that need to be done

• Rules – business rules defining nature of object

• Transformations – requirements to change the data to fit the object

51

Tags (cont.)

• Think of Tags as high-tech Post-Its™ that you can attach to many types of objects in an Informatica Data Explorer Project

• Note: All of the pull down menu items in Tags can be configured through server configuration files

52

Action Tag

53

Note Tag

54

Rule Tag

55


56

Content Presentation

• Constant Analysis

• Empty Column Analysis

• Inferred Data Type Analysis

• Null Rule Analysis

• Source Data Type Analysis

• Unique Analysis

• Frequency Analysis

• Pattern Analysis

• Domain Analysis

57

Content Presentations

58

Content Presentation (Continued)

59

Constant Analysis

60

Lesson 2 Review

• Column Profiling is about the analysis of column content and format

• Column Profiling scans data files and stores the resulting profile information in an Informatica Data Explorer Project

• Column Profiling information can be viewed by opening an Column List for a Table

61

Lesson 2 Review

• The results of Column Profiling are stored with an Column

• The results of Column Profiling include:• Primary and Alternate Data Types• Null Rules• Minimum/Maximum Value ranges• Value Pairs• Patterns

• Tags can be added to Columns or Tables to convey additional information or instructions about the Column

62

Lesson 3

Data Rules

63

Data Rules - Objectives

• What is a Data Rule?

• Using Data Rules in Informatica Data Explorer

• How to test for Data Rules

• Execute Data Rules tasks

• When to apply Data Rules in the data discovery process.

64

Define Data Rules

• What is a Business Rule? • Business Rule: describe the main characteristic of the data

• What is a Data Rule?• Data Rule is a constraint written against one or more

Tables that is used to find incorrect data.

• Can be viewed as business rules for data

65

Define Data Rules (cont.)

• Data Rules are often embedded in application programs

• The Informatica Data Explorer Practitioner can discover, document and test Data Rules against the initial source.

66

Using Data Rules in Informatica Data Explorer

• Data Rules is the process of using Informatica Data Explorer to determine if the externally proposed data relationships are fully supported by the source data.

• Discover if the source data supports the relationships and business needs.

• Data Rules are tested against the initial source, stored and then can be re-run after the data has been cleansed or moved.

67

Business Rules and Data Rules

• Employees with 2 or more years of service are paid 3 weeks vacation.

• Fulltime employees are assigned to a salary band.

• Employees in Dept C – salaries cannot be greater than $40,000.

• Department number contained in the employee record must correspond to an existing Department number.

• Does the Column contain a particular string of characters?

68

Business Rules and Data Rules (cont.)

• Does one Column include the full contents of another Column?

• In an address, is there a line of blanks followed by a line of non-blanks?

• Are all three fields of a key null?

• Is the date Column in the wrong format?

• Does the Column contain the right type of data for this type of record?

69

Create and Execute Data Rules

• Data Rules can be created from two locations:• Rules Tag

• Drill down

• Execute Data Rules from the Rules Tag viewer or Data Rules Management.

70

Drill Down

71

New Rule Tag

72


73

When to Apply Data Rules

• Tightly coupled to Drill Down

• Data Rules can be executed against different sources.

• Data Rules can be applied at any point in time during the data discovery process.

• Data Rules can be saved and re-run• After the data load as occurred or• A feed is supplied or• Data has changed for any reason

74

Complex Data Rules

RULE LoanTypeAmtTerm

SELECT "Loan_ID","Loan_Type","Loan_Amt","Loan_Term"

FROM <Use Table in Data Source>

WHERE (UPPER(LOAN_TYPE) = 'AUTO' and

(LOAN_AMT not between 3000 and 50000 or

LOAN_TERM not between 12 and 60)) or

(UPPER(LOAN_TYPE) = 'REAL' and

(LOAN_AMT not between 10000 and 500000 or

LOAN_TERM not between 36 and 360)) or

LOAN_TYPE is null or LOAN_AMT is null or LOAN_TERM is null

75

Data Rule Management

76

Lesson 3 Review

• Data Rules can be created on Columns that we think are volatile.

• Data Rules can be created, saved and ran on different data sources.

• Data Rules can be created from two locations:• Rules tab• Drill down

• Execute Data Rules from the Rules Tag viewer or Data Rules Management.

77

Lesson 4

Single Table Structural Analysis

78

Lesson 4 Objectives

• Explain what Table Structural Profiling is, and why it should be performed

• Define the term “Functional Dependency” as used by Informatica Data Explorer, and explain the significance

• Contrast a Single-Column Determinant to a Multiple-Column (or compound) Determinant as used by Informatica Data Explorer

79


• Define the terms “Inferred Dependencies” and “Model Dependencies” as used by Informatica Data Explorer

• Explain why and when an Inferred Dependency should be added to the set of Model Dependencies

• Define the term “Sample Data” as used by Informatica Data Explorer, and explain the use of Sample Data in Dependency Profiling

• Understand when and how to apply Informational Tags in Dependency Profiling

80

What is Table Structural Profiling?

• A process that discovers the interrelationships between columns in your source data

• Is performed against samples of data that you have imported into Informatica Data Explorer

• It identifies Columns that determine the value of other Columns

81

Why Profile Table Structure?

• Functional Dependencies determine the structure of a data model and/or database design

• Functional Dependencies can be equated to an elementary form of Business Rule

• Dependencies between data items suggest organization of data storage that is both natural and efficient

82

Why Profile Table Structure? (cont.)

• Quickly validate expected Dependencies (Keys)

• If data does not conform to expected or required dependency rules, you most likely have a data integrity problem

83

IDEServer

RDBMS

Flat FilesDB2 LU

What is Sample Data?

• Sample Data is actual data that you import into an Informatica Data Explorer Table either from:• Downloaded flat files or • Directly from a relational database

• Sample Data is a subset of the data in the source database:• Multiple data samples can be loaded into

Informatica Data Explorer• Each data sample is stored in the Project

• Sample Data is associated with a particular Table

84

Why Import Sample Data?

• Sample data is used in Table Structural Profiling to examine relationships of all columns of a given record

Source

Data

Column Profiling(stores results only)

ImportSampleData

Data Sample #1

Table Structural Profiling

(examines entire records)

Data Sample #2

85

A value of EMPNO always determines the same value of ENAME throughout the sample data

EMPNO ENAME

EMPNO

123456789012345789

ENAME

John DoeJane Smith

Eduardo SanchezJane SmithJohn Doe

Eduardo Sanchez

Functional Dependencies

• An Column is functionally dependent on other Columns that determine its value

86

Functional Dependencies (cont.)

• A Functional Dependency is written as:• A B

• ‘A’ is the Determinant Column

• ‘B’ is the Dependent Column

• The statement is ALWAYS read left to right• ‘A functionally determines B’, or

• “If I know a value for A, I can determine the value for B” or

• For each distinct value of ‘A’ there can only be one value of ‘B’

87

Functional Dependencies (cont.)

• The determinant side can be compound:• A + B C

• ‘A’ and ‘B’ together are the Determinant Column

• ‘C’ is the Dependent Column

• The determinant side can be Null:• Ø C

• Nothing is the Determinant Column

• ‘C’ is the Dependent Column

• ‘C’ has only one value, or one value and nulls, in the whole sample

88

Reviewing Inferred Dependencies

• You must review the set of Inferred Dependencies

• The Dependencies inferred by Informatica Data Explorer exist implicitly in the data

• You must make decisions as to which of the Inferred Dependencies explicitly represent the current use of the data

• The review process is to determine• Which dependencies should be included in the set of

dependencies from which the Normalized Schema will be generated

89

Sample Data

90

Exercises 4.1 - 4.4

91

Adding an Inferred Dependency to the Model

• Inferred Dependencies added to the model establish the Tables (tables) that will be created in Normalization• Normalization breaks a single Table (table) into multiple

Tables (tables)

• For example, “Employee” Table in the source system represents two Tables (Employee and Department) once the dependencies are created and the model is normalized

92

Adding an Inferred Dependency to the Model (cont.)

• Columns that do not participate as a Dependant are automatically included in the Primary Key• Informatica Data Explorer considers all Columns as part of

the key until a relationship is established

• Dependency Profiling is an iterative process

93

Exercises 4.5 - 4.6

94

Dependency Subject Area

• Inferred Dependencies• The set of dependencies that are inferred from a sample

of data for a Table

• Table Dependencies• A subset of the Model Dependencies that are wholly

contained in a Table

95

Dependency Subject Area (cont.)

• Model Dependencies• The set of dependencies that you determine fit into your

design and are supported by the data• Model Dependencies are associated at the schema level• Model Dependencies are the set of all dependencies

across all Tables• Model Dependencies are used to create the normalized

schema

96

Dependencies

97

Inferred Dependencies

98

Key Dependencies

99

Model Dependencies

100

Filter Dependencies

101

Add Dependencies to Model or Filter

102

When to Add an Inferred Dependency

• Review each Inferred Dependency and add to model only those that can have a explicit reason for existing• Is the application enforcing the dependency?

• Is the user/business enforcing the dependency?

• Is some outside source enforcing the dependency?

103

Types of Dependencies

• True• The dependency is true for 100% of the data analyzed• Example: Every time a unique value is known for

EMPID, additional information is available (i.e. Employee Name, Address, Phone, etc.)

• Gray• The dependency is almost, but not quite 100% true for

the data analyzed• One row causes the violation

104

Types of Dependencies (cont.)

• Unsupported

• Two or more rows in the sample data do not support the dependency

• Unknown

• The dependency has not yet been validated against the sample data (Basis dependencies validation appear as Unknown)

105

• Questions to Ask:• What caused the dependency to be gray?• Should another sample be imported for verification?

• Review each Inferred Gray Dependency and add to model only those that can have a explicit reason for existing:• Is the application supposed to be enforcing the

dependency?• Is the user/business supposed to be enforcing the

dependency?• Is some outside source supposed to be enforcing the

dependency?

When to Add an Inferred Gray Dependency

106

Lab Exercise 4.7

107

Tagging Dependencies

• You cannot tag an Inferred or Model Dependency

• You add Tags to the Column that is causing the problem

108

Compound Determinants

• Two or more Columns that uniquely identify the Dependent Column

• This often represents a M to 1 relationship in the data

• This happens quite often in older file-based systems

109

Lab Exercise 4.8 – 4.9

110

Lesson 4 Review

• Importing Sample Data stores the data inside an Informatica Data Explorer Project

• Sample Data is used as input to Dependency Profiling

• You must import Sample Data before you can perform the Profile Dependencies task using Informatica Data Explorer• Data samples are imported using the Import Sample

Data feature• Data samples can be retained from doing a Drill Down

or executing a Data Rule

111


• Dependency Profiling finds the relationships between Columns in the same source file or table

• All Inferred Dependencies are associated with sets of Sample Data

• Table Dependencies are dependencies that have been added to the model, and are associated with a specific Table

112


• Model Dependencies are the set of dependencies from all Tables in the schema

• Only Model Dependencies are used as input to the generation of a Normalized Schema

• All Dependencies inferred by Informatica Data Explorer exist implicitly in the data

113


• You will find many Inferred Dependencies that have no meaning in context of the application or business use of the data

• These are Implicit Dependencies that have no explicit meaning

• Dependency Profiling is an iterative process

114

Lesson 5

Cross Table Profiling

115

Lesson 5 Objectives

• Explain what Cross Table Profiling is, and why it should be performed

• Execute the Cross Table Profiling function in Informatica Data Explorer

• Navigate and review the results of Cross Table Profiling

• Define the terms “Synonym” and “Homonym” as used in Informatica Data Explorer

116


• Understand what data is used for Cross Table Profiling, and how potential Synonyms are identified

• Describe why and when a Synonym should be created

• Create a Synonym

• Understand the significance of creating Synonyms

117

Cross Table Profiling

• The process that identifies similarity between the values in other columns

• Performed using the value sets associated with the Column objects inside Informatica Data Explorer• These are the Value Frequency

Lists that were created by Column Profiling

118

Why Profile Redundancies?

• To uncover Columns that actually represent the same business facts

• Informatica Data Explorer can uncover two types of redundancies: • Synonyms

• Redundant data that you would like to eliminate through the creation of Synonyms

• Redundant data that is intended to improve database performance

• Homonyms• Data that looks redundant but actually represents quite different

business facts (Homonyms)

119

Comparing Value Sets

ABC

BCDE

Value SetOverlap

Value Set1

ValueOverlap

Value Set2

120

Inferred Redundancies

121

Exercise 5.1 - 5.2

122

• Two or more Columns having the same business meaning

• Comparing common values between columns can identify candidate Synonyms

SP_NOValueSet

EMPIDValueSet

28%overlap

Synonyms

123

Effect of Synonyms

• If the Primary Keys of two Tables are synonyms, they will collapse into a single Table in the Normalized Schema

TransactionID (PK)

ProductID

ProductName

InventorySupplier

TransactionID

TransactionID (PK)

ProductID

PruductName

SupplierName

SupplierAddr

TransactionID (PK)

SupplierName

SupplierAddress

124

Effect of Synonyms (cont.)

• If two Columns that are synonyms represent a parent-child relationship, they will result in two Columns in two Tables with one Column participating in a Primary Key and the other in the corresponding Foreign Key

OrderNumber

ProductID

ProductName

Order Payment

PaymentID

OrderID

CheckNumber

OrderNumber (PK)

ProductID

ProductName

OrderNumber

PaymentID

PaymentID

OrderNumber (FK)

CheckNumber

125

Homonyms Defined

• Two or more Columns having the same name yet different business meanings

70%overlap

SHIPPING_STATEValueSet

STATEValueSet

126

Making Synonyms

127

Synonyms

128

Exercise 5.3 - 5.4

129

Lesson 5 Review

• Cross Table Profiling is about data integration between sets of data

• Cross Table Profiling comprises 2 activities• Comparing value lists

• Use Foreign Key or Join analysis to compare value lists greater than 16,000

• Assigning Synonyms

• Rule of Thumb• Be conservative about making Synonyms

• You can always come back after you’ve normalized the schema and make more

130

Lesson 5 Review

• You can not make intra-table Synonyms, only inter-table

• You must have built Value Lists either during the Profile Columns task, or during the Import Sample Data task, before you can perform Cross Table Profiling

• Creation of Synonyms participates in Normalization

131

Lesson 6

Validating Table and Cross Table Analysis

132

• Understand how Validation differs from Cross Table Profiling

• Define and discuss the term Referential Integrity

• Explain various methods of validation and how it can be use

• Execute Validation tasks

• View Validation results

Lesson 6 Objectives

133

• Validation can be used to:• Define the exact overlap characteristics of two redundant

Columns• Validate a single or multi-Column foreign key• Validate that the keys of two tables do not overlap (Vertical

Merge)• Validate single or multiple Column keys (Validate Keys)• Validate a Join• Validate against reference table• Validate against Domain values

• Execute Validation from the Single Table Structural Analysis and Cross Table Structural Analysis

Validation

134

Referential Integrity

• Example A: An Order File contains an OrderID that uniquely identifies each customer order. There should be no OrderID values in the Order or Detail file that do not exist in the other.

Example A

135

Referential Integrity (cont.)

• Example B: An Order file may have OrderID values that do not exist in the Payment file (outstanding payments or unbilled customers). The Payment file should not have any OrderID values that do not occur in the Order file.

Example B

136

• Validation compares sets of Columns between two relations to discover the quality of the overlap.

• Validation exhaustively tests all the data.

• Cross Table Profiling discovers potential overlap between Columns.

• Cross Table Profiling estimates overlap.

• Results of Validation– sets of statistics about the overlap and non-overlapping values

Validation and Cross Table Profiling

137

• To understand the exact overlap:• Execute Validation from the Cross Table Profiling

• create a relationship (Primary Key / Foreign Key, Join, …) between the two Columns and choose Validate

Profile Redundant Columns

138

Exercise 6.1-6.2

139

• Validate a Single or Multi-Column Foreign Key

• Primary use – test the Referential Integrity of primary and foreign key relationships.

• Each row in a child table must reference a row in the parent table.

• Every order detail record must reference an order.

• Information discovered can be used to help write logic to perform the data integration.

Foreign Key Analysis

140

Foreign Key Analysis Results

141

Parents Without Children

142

Exercise 6.3-6.5

143

• Primary use – when two similar systems are merged together.

• Company A merges with Company B: payroll master records are merged.

• It is expected that all rows in the parent and child tables are orphans.• Employees of Company A are not on Company B’s payroll

master file.

• Employees of Company B are not on Company A’s payroll master file.

Vertical Merge Analysis

144

Vertical Merge

145

Vertical Merge Analysis Results

146

Exercise 6.6-6.8

147

• Primary use – validate keys in a single Table

• Validation looks at the table and checks to make sure that every row is unique.

• Use this feature to find any duplicate rows for keys discovered in Single Table Structural Analysis.

Validate Key Analysis

148

Validate Key

149

Validate Key Analysis Results

150

New Alternate Key

151

Validate Alternate Key

152

Exercise 6.9 - 6.12

153

Lesson 7

Normalization

154

Lesson 7 Objectives

• Explain what Normalization is and when it should be performed

• Execute the Normalization function of Informatica Data Explorer

• Navigate and review the results of Normalization

155


• Describe what an Column Trace is, and how it is used

• Understand how to modify the Normalized Schema by making changes to the Source Schema

• Explain the iterative nature of Normalization

156

Normalization

• A process that transforms an initial schema into a schema with greater integrity

• A process of transforming Source Schema into a:• Non-redundant

• Anomaly-free

• Third Normal Form model

• Normalization is based upon:• Dependencies added to the model in Single Table Structural

Analysis and

• Synonyms made in Cross Table Structural Analysis

157

Why Normalize?

• A Third Normal Form (3NF) schema has no: • Redundant Columns other than Foreign Keys

• Columns that are only partially dependant on the key

• Transitive Dependencies

• The Normalized Schema provides a checkpoint for the completeness and accuracy of the decisions you made during the profiling tasks

158

Exercise 7.1

Normalized Schema v. Source Schema

custord

ORDER_NO: char(4)ITEM_NO: char(6)

ORDER_DATE: datetimeSHIPDT: datetimePO_NUM: smallintLAST_NAME: varchar(10)FIRST_NAME: varchar(11)CNAME: varchar(36)CON_TTL: varchar(27)SHIPPING_STREET: varchar(40)SHIPPING_CITY: varchar(20)SHIPPING_STATE: char(2)SHIPPING_ZIP: varchar(10)PHONENUM: varchar(12)SP_NO: smallintQUANTITY: smallintITEM_DSC: varchar(25)SUPID: smallintUNIT_COST: moneyTAX_RATE: decimal(5,4)BILL_CODE: char(10)

empinfo

EMPID: smallint

LAST_NAME: varchar(17)FIRST_NAME: varchar(12)GENDER: char(1)DEPTID: smallintDEPTNM: varchar(14)TITLE: varchar(30)STREET: varchar(40)CITY: varchar(15)STATE: varchar(3)ZIP: varchar(10)PHONE: varchar(14)

All_Constant_Attributes

BILL_CODE: char(10)TAX_RATE: decimal(5,4)

ITEM_NO

ITEM_NO: char(6)

SUPID: smallintITEM_DSC: varchar(25)

ITEM_NO_ORDER_DATE

ITEM_NO: char(6)ORDER_DATE: datetime

UNIT_COST: money

ORDER_NO

ORDER_NO: char(4)

PHONENUM: varchar(12)SHIPPING_ZIP: varchar(10)SHIPPING_STATE: char(2)SHIPPING_CITY: varchar(20)SHIPPING_STREET: varchar(40)CON_TTL: varchar(27)CNAME: varchar(36)FIRST_NAME: varchar(11)LAST_NAME: varchar(10)PO_NUM: smallint

ITEM_NO_ORDER_NO

ITEM_NO: char(6)ORDER_NO: char(4)ORDER_DATE: datetimeEmployeeID: smallintDEPTID: smallint

QUANTITY: smallintSHIPDT: datetime

DEPTID

DEPTID: smallint

DEPTNM: varchar(14)

EmployeeID

EmployeeID: smallintDEPTID: smallint

PHONE: varchar(14)ZIP: varchar(10)STATE: varchar(3)CITY: varchar(15)STREET: varchar(40)TITLE: varchar(30)GENDER: char(1)FIRST_NAME: varchar(12)LAST_NAME: varchar(17)

160

Normalized Schema Anomalies

• Observable normalization anomalies may include:• Unexpected Tables

• Duplicate Tables

• Tables with strange/unexpected keys

• Columns in the wrong locations

161

Column Tracing

• Allows you to find the origin of an Column in another schema

• Used to determine the Source Model Dependencies and Synonyms (or the lack thereof) which may be causing the anomaly

162

Schema Locking

• The existence of a Normalized Schema causes Informatica Data Explorer to lock various objects in the Source Schema

• In order to modify Dependencies in the Source Schema, you must remove the Normalized Schema

163

Re-Normalizing

• In order to change the Normalized Schema, you must Remove the Normalized Schema

• Modify the Source Schema, then Re-run Normalization

• The next exercises:• Remove a dependency

• Add another Table

• Renormalize schema

• Review the new Normalized Schema

164

Lab Exercises 7.2 – 7.3

165

New Normalized Schema

All_Constant_Attributes

BILL_CODE: char(10)TAX_RATE: decimal(5,4)

ITEM_NO

ITEM_NO: char(6)

SUPID: smallintITEM_DSC: varchar(25)

SHIPPING_ZIP

SHIPPING_ZIP: varchar(10)

SHIPPING_STATE: char(2)SHIPPING_CITY: varchar(20)SHIPPING_STREET: varchar(40)

ORDER_NO

ORDER_NO: char(4)

PHONENUM: varchar(12)SHIPPING_ZIP: varchar(10)CON_TTL: varchar(27)CNAME: varchar(36)FIRST_NAME: varchar(11)LAST_NAME: varchar(10)PO_NUM: smallint

ITEM_NO_ORDER_NO

ITEM_NO: char(6)ORDER_NO: char(4)

UNIT_COST: moneyQUANTITY: smallintEmployeeID: smallintSHIPDT: datetimeORDER_DATE: datetime

DEPTID

DEPTID: smallint

DEPTNM: varchar(14)

EmployeeID

EmployeeID: smallint

PHONE: varchar(14)ZIP: varchar(10)STATE: varchar(3)CITY: varchar(15)STREET: varchar(40)TITLE: varchar(30)DEPTID: smallintGENDER: char(1)FIRST_NAME: varchar(12)LAST_NAME: varchar(17)

166

Lesson 7 Review

• Normalization is a 100% automated process

• The only inputs to the normalization process are• Dependencies added to the Model

• Column Synonyms

• Refinement of the Normalized Schema is an iterative process

167


• The Normalized Schema is most often used as a basis for• Baseline view

• Review for anomalies

• Comparison to business requirements

• Staging Area

• The Normalized Schema is not a business model

168


• Normalized Schema anomalies stem from either:• Dependencies added to the model

• Dependencies not added to the model

• Incorrect (or unmade) Synonyms

• You can Normalize the Source Schema as soon as you have added dependencies to the model during Single Table Structural Analysis• Actually, you can do it any time but it will just make

a copy of your existing schema if you have not added any dependencies.

169


• If you have not established inter-relational Synonyms, you will get duplicate Tables and/or Columns in the Normalized Schema• Duplicate Tables will appear in the Normalized Model

with an extension, such as:• EmployeeID

• EmployeeID_1

• Suggestions:• Make only one change at a time and then renormalize

• Often making one change in the Source Schema can result in several changes in the Normalized Schema

170

Lesson 8

Exporting to the Repository

171

Lesson 8 Objective

• Export Projects to the IDE Repository

172

What is the Repository?

• A series of relational database tables that store the results from the Informatica Data Explorer Product Suite

173

Repository Export

• The Repository Export dialog box enables you to export an IDE catalog to the Repository

• The Repository Export dialog box provides the ability to limit some of the data that is exported to the Repository

• Once in the Repository, the Catalog becomes available to a variety of DBMS tools, such as SQL, report generators, and so on

• All schemas in the Catalog will be exported to the Repository

IDE Repository Architecture

UNIX or Windows NT

IDEServer

ODBCDrivers

Windows XP, 2000

IDEClient

Client Server

Project

RepositoryRDBMS

UNIX or Windows NT

175

Exporting to Repository

176

Lab Exercise 8.1

177

Lesson 8 Review

• You control what information from a Project is included in the Export process

• The more you export, the longer the process will take

• Information exported to the IDE Repository becomes available to:• Informatica Data Explorer Repository Navigator

• Report Writing tools

• SQL tools

178

Lesson 9

Using the Repository Navigator

179

Lesson 9 Objectives

• Understand use of the Repository Navigator

• Access the IDE Repository and browse its contents using the Navigator

• Explain Tags

• Understand how to share information among departments

180

IDE Repository Navigator

• A browser for the contents of the IDE Repository

• Can be used by anyone in your enterprise

Repository~~~~

KnowledgeAbout

Corporate Systems

Structure

Content

Quality

181

IDE Repository Architecture

UNIX or Windows NT

UNIX or Windows NT

IDEServer

ODBCDrivers

Windows XP, 2000

IDEClient

ODBCDrivers

IDESourceProfiler

Client Server

IDEFTM/XML

Project

IDE Repository

RDBMS

RepositoryNavigator

182

Schema Viewer

• The Schema Viewer functions similar to the Navigation Tree in Informatica Data Explorer• You expand/contract objects

• You use a right-click of the mouse to view properties

• The Schema Viewer provides users with the ability to query profiling information for Tables and Columns (Properties, Tags, Sample Data, Value Frequency Lists) within each schema

183

Exercise 9.1-9.3

184

The Link Viewer

• The Link Viewer shows links between any two schemas in the current project.

• Link Viewer uses:

• View Links between Columns

• Find information on compatibility problems

• Access Tags associated with Links

Link Viewer

185

Table Viewer

• Provides SQL access to the IDE Repository

• Has several pre-built SQL queries

• Allows you to run your own custom queries

186

Exercise 9.4-9.6

187

Lesson 9 Review

• The IDE Repository provides:• Rapid access to source data knowledge

• Team collaboration

• Enhanced communication

• Flexible ad hoc reporting

188

Lesson 10

Repository Reports

189

Lesson 10 Objectives

• Understand what IDE Repository Reports are

• Demonstrate how to use Repository Reports

• Create a report using a Crystal Reports template

• Export a report using Crystal Reports

190

What are Repository Reports

• IDE Repository Reports are a series of reports to provide specific management information from the IDE Repository.

• Reports are written with Crystal Reports.

191

Why use Crystal Reports?

• Provides a user interface to guide the design of reports that are stored in a relational database

• Can export data to other programs such as Excel, Word or HTML pages

• Provides the flexibility to create custom or ad hoc reports. The user is not limited to the reports provided in the Informatica Data Explorer Product Suite

• Accesses the IDE Repository through an ODBC connection

192

Report Templates

• A series of reports are provided as an easy means of obtaining documentation from the IDE Repository

• The Report Templates can be modified to meet individual needs

193

List of Reports

Column Profile - By File Column Profiling results sorted by FileColumn Profile - By Field Column Profiling results sorted by FieldNull Rule Exceptions List of Attributes with Null, Zero or BlanksValue Frequency Value Frequency Lists for AttributesSupported Relationships Inferred Dependencies for each Data SampleModel Relationships Dependencies that have been added to the ModelOverlapping Data Redundancy Profiling Overlap ReportNotes Note Tag ReportAction Items Action Item Tag ReportRules Rule Tag ReportTransformations Transformation Tag ReportAttribute Links Reports Links between Attributes

194

Selection Criteria

• Allow users to select values for certain fields within the templates

• Limit the amount of data reported from the IDE Repository

• Each Template provides selection on ProjectName and SchemaName at a minimum

195

Exercises 10.1 – 10.4

196

Exporting Reports

• Crystal Reports provides an option to export report data into other file formats

• Useful for sharing data with individuals that do not have access to Crystal Reports

197

Exercises 10.5

198

Lesson 10 Review

• IDE Repository Reports provide reporting capability from the IDE Repository

• Additional reports can be created to meet business needs

199

Lesson 11

Integration with PowerCenter

200

PowerCenter Integration

• Informatica Data Explorer has the ability to share metadata with PowerCenter. This allows the business users to share knowledge that was found during the data discovery process with the PowerCenter developers.

• Objects that can be shared are: • Source and target schemas

• Filters

• Expressions (transformation tags in IDE).

201

Create a Transformation Tag

202

Transformation Tag

203

Set Physical Properties

204

Export to Repository

205

Open Fixed Target Mapping (FTM)

206

Open Your Project

207

Export to PowerCenter

208

Import Object into PowerCenter

209

Open Customer in Source Analyzer

210

Open a new Transform

211

Open Ports Tab

212

Informatica Resources

213

Informatica – The Data Integration Company

Informatica provides data integration tools for both batch and real-time applications:

Data Migration Data Synchronization

Data Warehousing Data Hubs

Business Activity Monitoring

214

• Founded in 1993

• Leader in enterprise solution products

• Headquarters in Redwood City, CA

• Public company since April 1999 (INFA)

• 2000+ customers, including over 80% of Fortune 100

• Strategic partnerships with IBM Global Services, HP, Accenture, SAP, and many others

• Technology partnership with Composite Software for Enterprise Information Integration (EII) – real-time federated views and reporting across multiple data sources

• Worldwide distribution

Informatica – Company Information

215

Informatica Affiliations

216

Informatica Resources

www.informatica.com – provides information (under Services) on:• Professional Services• Education Services

my.informatica.com – customers and contractual partners can sign up to access:• Technical Support• Product documentation (under Tools – online documentation)• Velocity Methodology (under Services)• Knowledgebase• Mapping templates

devnet.informatica.com – sign up for Informatica Developers Network• Discussion forums• Web seminars• Technical papers

217

Documents

IDE 5.0_Basics_20061025