Embed Size (px)
Processes undertaken in large data migration projectsSimon McCabeChief Technology Officer, IMGS
Discussion Context
High level case study of processes undertaken in large electricity and water data migrations run by IMGS over the last 5 years
High level data volumes:
Project 1:
60+ primary features
100+ secondary features
Approx. 28 million records
Project 2:
40+ primary features
Approx. 1.2 million records
Discussion Context
Topics covered:
Understanding the legacy system and its data
Designing a migration process
Tools to use (FME…)
Reporting and Reconciliation processes
Data Cleansing
Cutover\Deployment considerations
How FME played its part
Presentation will cover the full migration lifecycle
Analysis Design TestBuild Deploy
Step 1: Understanding legacy system
Analysis Phase
Step 1: Understanding legacy system
Forensic review of current data sets (what needs migration)
Review\define all source features (objects)
What is their makeup?
database tables, CSV, customised file formats, etc..
what are the field characteristics (definition, sparsely populated etc.)
how do we link graphics to attributes?
during migration will a features attribute or graphical data distinguish the feature (or a combination of both)?
What are they related too in the system?
other features/objects
how are they related to each other
should we create relationships during migration
Collate statistics
Before you leave the analysis phase audit the data in terms of volume.
Step 1: Understanding legacy system
Methodology: Workshops and Documentation!
Data Audit
What data is used by the legacy system and the users in the
customers department
External Interfaces?
What legacy system data is utilised by others (outside the
customers department)
Analysis Design TestBuild Deploy
Step 1: Understanding legacy system
• FME assists in the data audit (volumetric\statistics)
• FME helps you discuss the data in terms of what is in the system vs what the user thinks is in there
• Use FME to migrated (a simple migration) graphical data to staging tables
• Migrate style information, level information, size information (widths and heights) etc.. – building up a tabular picture of data
• Utilize Microsoft Excel\Word to provide a human readable view on the above to allow dialogue\discussion
Step 2: Design the migration process
Design Phase
Step 2: Design the migration process
What data is missing or unusable? E.g. do all records have a geometry?
What data is stored in a non-standard format? E.g. are dates in a column all recorded in the same format?
What data values give conflicting information?
What data is missing important relationship linkages?
What data is incorrect or out of date?
What data records are duplicated?
Common Design considerations [Data Quality]
Step 2: Design the migration process
If geometries are missing – should they be fixed in the source system or via migration (can depend on time and source system processes)
If geometries exist with no attributes, should they be migrated or abandoned?
How do we report on inconsistencies in data i.e. field dates might have a combination of DD-MM-YY or YY-MM-DD etc..
Common Design considerations [Data Cleansing]
If fixed during migration will this affect the source system
External Interfaces
(systems get out of sync…)
Step 2: Design the migration process
How you address\fix the points in the last 2 slides can depend on the overall team size.
Fix in the source system
Fix during migration
Don’t fix and migrate as is (if possible)
Or abandon the migration of the problematic feature instance
A key requirement from the migration however will be to report on all issues and allow dialogue on their resolution.
Common Design considerations [Team Size]
Step 2: Design the migration process
Walk before you run
Design\Sketch migration workflows before starting migration process (Microsoft Visio etc..)
Ensure all logical scenarios are covered
Document what is an abandon event vs what is an issue event
Abandon events = feature instance is not migrated
Issue event = an issue was encountered and needs to be reviewed (missing geometry, invalid date format etc..)
Design logging processes for developers to follow (legacy system references, event type and error description)
Common Design considerations [Workflows]
Step 2: Design the migration process
Start up and Shut down scripts (TCL or Python)
Log start time, end time, features read/written and run status to migration_log table
Assists in flagging long running scripts (an thus targeted performance updated)
Write errors/issues to ERROR_LOG table
Excel reports then designed to read error log table and provide periodic reports
FME transformers to validate geometries, dates etc.
OGC Validator
Geometry Validator
Duplicate Remover (if required)
Periodic reviews of reports feed back into FME processes and validation\logging requirements
Common Design considerations [Logging]
Logging
Tra
nsf
orm
ers
OGC Validator
start-up script
Duplicate Remover
GeometryValidator
Python
TCL
ERROR_LOG table
MIGRATION_LOG table
Step 2: Design the migration process
It is impossible to manually test all feature instances, all attributes, all relationships etc.. after they migrate to the destination system
If possible (preferable and advised):
Have a separate team (to the migration team) develop reconciliation reports
Feature Reconciliation List source features and their counts along side the migrated
destination features and their counts
Report and explain any discrepancies
Attribute Reconciliation Per source feature, list the source attribute along side their
destination attributes – all counts must be the same
Automate, Automate, Automate….
Reconciliation reports should easily identify/flag all issues and provide input into further dialogue and issue management.
Common Design considerations [Reconciliation]
Step 2: Design the migration process
Reconciliation Reports
Flag long running scripts
Report on issues Feature Reconciliation
Analyse source and destination feature counts side by side
& Report
Attribute Reconciliation
Analyse source and destination attribute counts side by side
Reconciliation Reports form an integral part of quality control, assurance and testing
Analysis Design TestBuild Deploy
Step 2: Design the migration process
• Decide on FME Desktop Vs FME Server
• Prototype designs where applicable
• Design process to be run with minimal intervention i.e. one play button to go from start to finish
• Ensure logging and reporting is designed into the solution and embedded within build FME scripts
Step 3 & 4: Build and Test
Build & Test Phase
Step 3 & 4: Build and Test
• FME is very powerful, and provides many ways to migrate data – however developers need to regularly review\critique their workflows
• Items to keep an eye on:
• Excessive use of SQLExecutor (can add hours to a migration script). Examples in scripts from previous projects can reduce scripts by 50%.
• Review all database calls i.e. any necessary SQLExecutor calls or queries used to read data – ensure indexes are applied to datasets (FME can only read\link the data if it is setup correctly in the database)
• If you have multiple destination (writer) connections – the first will write almost immediately, the others will be cached (and slow the migration down)
• Try to have only one writer per workspace
• Examples in scripts from previous projects can reduce scripts by 10 hours
• Use the power of the database
• Pre process data where possible
• Cache sequences (hours can be saved here)
Analysis Design TestBuild Deploy
Step 3 & 4: Build and Test
• Prototype where possible
• Design workflows provide the logical interpretation of the migration requirement BUT should not be used as the blueprint for the FME workspace design
• Use the principles of the workflow
• Might be faster to write database functions/procedures to pre-process some of the data
• Utilizing the power of the database
• Removes overly complex custom transformers from FME
• Simplifying the workspace
• Reducing overall time to process the data
• Example of the above can be seen with inexperienced developers or newbies to FME
• Once script in the past read a large dataset (1.5 million records) it had 2/3 SQLExecutors in the workspace which made approximately 4.6 million database calls during the migration (of one feature!)
Analysis Design TestBuild Deploy
Step 3 & 4: Build and Test
• Use the reconciliation process to quality stamp/approve and test your builds.
• At the start of build expect data quality issues (bugs and also source data cleansing activities)
• At the end of build and test data quality should disappear (bug fixing, source system remedies)
• Report analysis is critical moving from build to test to cutover.
• Issues should reduce
• All team members should be aware of open issues
Analysis Design TestBuild Deploy
Step 3 & 4: Build and Test
• The reconciliation report timings will feed times into the cutover\deployment window planning
• Should be reviewed regularly
• Additional reading on the subject of performance improvement could be:
• FME Performance and Profiling
• Turbocharging FME: How to Improve the Performance of Your FME Workspaces
• Performance Tuning FME
Analysis Design TestBuild Deploy
Step 3 & 4: Build and Test
Useful transformers used in the migration process
• Workspacerunner
• SchemaMapper
• AttributeCreator
• Tester
• TestFilter
• Aggregator
• SQLCreator
• SQLExecutor
• Counter
• GeometryRemover
• NullAttributeMapper
• Sampler
• TimeStamper
• StatisticsCalculator
• PointOnAreaOverlayer
• VertexCreator
• FeatureMerger
Analysis Design TestBuild Deploy
Step 5: Deploy
Deployment Phase
Step 5: Deploy
• At this stage everything is fully tested and OK to deploy.
• Cut over window• Timings of scripts can vary from the development server to
the test server to the production server (depending on spec’s of those machines)
• Run dress rehearsals into the final destination environment before cutover to get a view on expected timings.
• Once the final migration has run• Review the Feature reconciliation reports from this migration
to the last – all should be the same (or better)
• Review the Attribute reconciliation reports from this migration to the last – all should be the same (or better)
Analysis Design TestBuild Deploy
