47
F M E & D a t a Validatio n With Mark Stoakes, George Crowe, and David Ellerbeck

Data Validation Victories: Tips for Better Data Quality

Embed Size (px)

Citation preview

Page 1: Data Validation Victories: Tips for Better Data Quality

F M E& D a t a

Validatio

nWith Mark Stoakes, George Crowe, and David Ellerbeck

Page 2: Data Validation Victories: Tips for Better Data Quality

Made here

S A F E S O F T W A R E

Page 3: Data Validation Victories: Tips for Better Data Quality

Colonial Pipeline Company

Page 4: Data Validation Victories: Tips for Better Data Quality

Colonial Pipeline Company –

Business Overview & Safety Share

• Interstate common carrier of refined

petroleum products

• Over 5,500 miles of pipeline stretching

from Houston to New York

• Headquartered in Alpharetta, GA

• Transports approximately

100 million gallons per day: – Gasoline

– Home heating oil

– Diesel fuel

– Commercial jet fuel

– Military fuels

Page 5: Data Validation Victories: Tips for Better Data Quality

Staffing Constraints…

Over 5,500 Miles

of Linear Assets

~ 215 Facilities

Supported by over 20 Engineering &

Design Contracting Firms

Infrastructure Managed with

~75 Engineers & Project Mgrs.

Asset Data Managed Utilizing

6 CAD/GIS SME’s

(1 Mgr., 3 GIS, 2 CAD)

Page 6: Data Validation Victories: Tips for Better Data Quality

CAD/GIS Team Project Manager Contractor

Color Legend:

Standards Validation w/i the AutoCAD®

Environment

CAD/GIS Team audits

the drawing files using

AutoCAD.

Contractor self-audits the

As-Built drawings and

submits them to a

secured server.

Beginning of

Process

CAD/GIS Team compiles

an audit report.PM approves As-Built

Data to be submitted to

the CAD/GIS Team

Page 7: Data Validation Victories: Tips for Better Data Quality

Standards Validation w/i the AutoCAD®

Environment

Page 8: Data Validation Victories: Tips for Better Data Quality

Standards Validation w/i the AutoCAD®

Environment

CAD Team audits the

drawing files using

AutoCAD.

Contractor self-audits the

As-Built drawings and

submits them to a

secured server.

Beginning of

Process

CAD Team compiles an

audit report and takes

appropriate action.

PM approves As-Built

Data to be submitted to

the CAD Team

Page 9: Data Validation Victories: Tips for Better Data Quality

Standards Validation w/i the FME

Environment

FUTURE DEPLOYMENT

FME Validator auto-runs and

generates an audit report which

is delivered to the Contractor

and Corporate personnel

PM approves

As-Built Data to be

submitted to the

CAD Team

Contractor uploads

the As-Built drawings

to a secured server.

CAD Team executes

the validation tool on

the drawing files.

CAD Team reviews

the audit report and

takes appropriate

action.

Contractor uploads

the As-Built drawings

to a secured server.

If audit results are

acceptable the CAD

Team takes custody

of the drawings.

If audit results are not

acceptable the

Contractor corrects

and resubmits the

drawings.

Page 10: Data Validation Victories: Tips for Better Data Quality

Standards Validation outside of the

AutoCAD® Environment

• Automation– External of AutoCAD

• Scalability– PC based and/or

Server based

• Cost– Time Reduction

• Customization– Pertinent to Industry or

Corporate specific standards

– Evolve with business processes

– Error logging, correction, and

notification… Our solution? …

FME

Page 11: Data Validation Victories: Tips for Better Data Quality

Reading CAD (DWG) Files

Robust DWG reading capabilities• Obvious: geometry (2D and 3D), layers, line types,

etc…

• Blocks

• Block Attributes

• Text

• Extended entity data

• Paper and model space

• Insertions

Extract

Page 12: Data Validation Victories: Tips for Better Data Quality

Manipulation of CAD

Objects and Entities

• Works just like other data types– Spatial Data

• Manipulation• Comparison• Replacement

– Tabular Data• Logical Tests• Data Integration• Queries, aggregation, formulas• Aggregation and reporting

Transform

Page 13: Data Validation Victories: Tips for Better Data Quality

Write to almost any

supported data type

• Back to CAD

• To Database

• To GIS

• To …

Load

Page 14: Data Validation Victories: Tips for Better Data Quality

Our Approach

StandardsBased

Well Documented CAD Standard

Repeatability

Same results every time

Emphasis on Automation

Minimize human intervention

Reporting Orientation

Identify and report

Page 15: Data Validation Victories: Tips for Better Data Quality

What Does it Mean in FME?

Out of the box transformers

Replicated logical tests

“Custom” transformers (sub-routines)

Standardized reporting approaches

Let’s look at

it…

Page 16: Data Validation Victories: Tips for Better Data Quality

Known Limitations of FME for CAD

Data Validation…

• Writing to multiple Layouts

within a single CAD file

• Plot Parameters are not

supported

– FME is not intended to be

used for hardcopy output

Your Imagination!

Page 18: Data Validation Victories: Tips for Better Data Quality

Questions?

Page 19: Data Validation Victories: Tips for Better Data Quality

Compliance Validation QA/QC

- Trust & Verify -

Page 20: Data Validation Victories: Tips for Better Data Quality

The spectrum of compliance

• Schema / Data Model

• Attributes

• CAD / GIS 2D Geometry

• 3D Geometry

• Topology

• Networks

• …

Single Item Validation

GeometryValidator

Comparative Validation

SpatialRelator

Page 21: Data Validation Victories: Tips for Better Data Quality

Improving Data Compliancy Using

FME

• City of Kitchener, near Toronto.

• Centralized GIS in ESRI SDE

environment.

• Maintain ~400 GIS layers

• User of FME for last ~8 years.

David van Riel- GIS Technologist -

Full Presentation

Page 22: Data Validation Victories: Tips for Better Data Quality
Page 23: Data Validation Victories: Tips for Better Data Quality
Page 24: Data Validation Victories: Tips for Better Data Quality

CAD Standards Checker

Key transformer:

• FeatureMerger

CAD Standard

Page 25: Data Validation Victories: Tips for Better Data Quality

CAD Standards Checker

Key transformer:• StatisticsCalculator

Error Reports

Page 26: Data Validation Victories: Tips for Better Data Quality

Attribute Checker

Page 27: Data Validation Victories: Tips for Better Data Quality

Attribute Checker

Key transformer:

• Joiner

Page 28: Data Validation Victories: Tips for Better Data Quality

Attribute Checker

Page 29: Data Validation Victories: Tips for Better Data Quality

Topology Checker

12 rules:• node edge snap

• node end snap

• floating node

• duplicate node

• line end snap

• line edge snap

• crossed line

• line end node

• duplicate line

• floating service

• one line two nodes

• both line ends snap

Page 30: Data Validation Victories: Tips for Better Data Quality

Topology Checker

Key transformers:

• Joiner

• NeighbourFinder

Page 31: Data Validation Victories: Tips for Better Data Quality
Page 33: Data Validation Victories: Tips for Better Data Quality
Page 34: Data Validation Victories: Tips for Better Data Quality

Validation processes

• At CCMEO, FME is used for :

– Feature translation

– Feature creation

– Feature validation

Catalog

Page 35: Data Validation Victories: Tips for Better Data Quality
Page 36: Data Validation Victories: Tips for Better Data Quality

Catalog based validations

• Spatial relations validation

• Domain attribute validation

• Proximity validation

• Minimal dimension validation

• Segmentation validation

• Data clipping validation

Page 37: Data Validation Victories: Tips for Better Data Quality

Catalog Based Validations

- Example -

Spatial relations validation

– Based on Egenhofer-Clementini DE-9IM

masks

Page 38: Data Validation Victories: Tips for Better Data Quality

Catalog based validations

- Examples -• Spatial relations

• Attribute value

Page 39: Data Validation Victories: Tips for Better Data Quality
Page 40: Data Validation Victories: Tips for Better Data Quality
Page 41: Data Validation Victories: Tips for Better Data Quality

Catalog validation within FME

• Real program example:

Page 42: Data Validation Victories: Tips for Better Data Quality

Automation

Page 45: Data Validation Victories: Tips for Better Data Quality

Questions?