Exploring Techniques of Data Quality and Profiling

1

iWay DQC and iDP

Kam WongSolutions Architect

Exploring Techniques of Data Quality and Profiling

April 20, 2012

What Is Data Profiling? What Are Some of Data Profiling Techniques? How To Monitor Your Data?

What Is Data Profiling?

Data profiling is about knowing your data

It discovers relationship between data elements, whether they are in the same data source or across multiple, heterogeneous data sources.

It performs statistical analysis against individual columns (as in relational database) discovering such things as the number of null values, patterns, whether the data matches the expected data type and so on.

2

What Are Some Of Data Profiling Techniques?

3

Profiling – Technical Basic Analysis

Minimums Maximums Averages Counts Etc.

Patterns / Masking Domain Extremes Quantities Frequency Analysis Foreign Key Analysis Charting Grouping / Aggregate Drilldown / Interactive Displays

How To Monitor Your Data?

4

View Profiles Compare data

quality over time – trend analysis

Monitor data quality index based on business rules

iWay Data Quality Management Life-Cycle

5

6

Demonstration

Thank-You

7