Embed Size (px)
Citation preview
Experian Pandora Data Migration Survival Guide (Part 1 of 2) - 1
Data Migration Survival Guide (Part 1 of 2) Experian Pandora
Created by Dylan Jones Editor Data Migration Pro.com
Contents 1. Introduction 032. The Unique Challenges of a Data Migration Leader 043. Common Data Migration Assumptions Leaders Must Address 074. Leadership Planning 115. What Next? 156. Do You Need Help? 15
Copyright Notice© Experian
This document contains information, proprietary to Experian, which is protected by international copyright law. The information contained herein may not be disclosed to third parties, copied or duplicated, in whole or in part, without the prior written consent of Experian.
Experian Pandora Data Migration Survival Guide (Part 1 of 2) - 3
1. Introduction
Data Migrations pose a very real problem for project leaders.
A migration can be a complex, costly and deeply confusing experience for the uninitiated. Leaders new to the discipline will be bombarded with a cacophony of industry jargon and activities that they are unlikely to ever come across in other business or IT circles.
There are many reasons why data migration projects fail but ultimately the buck stops with the project leader. Yes, a poor methodology, inappropriate software or other best-practice failures can have a major impact on the outcome but more often than not it is a lack of committed, experienced, leadership that leads to project failure.
The problem for the newly appointed data migration leader of course is that they lack the relevant experience. Data migrations are often a rare event for the typical organisation and there are few leadership career tracks or training schedules provided for leaders.
There is hope however. Recent research points to the need for key best-practices to be adopted in order to improve the chances of success of your data migration. This guide, part 1 of a 2 part document, outlines many of these best-practices and draws on two decades of experience to help data migration leaders make the right choice at the right time.
Experian Pandora 4 - Data Migration Survival Guide (Part 1 of 2)
Challenge #1: Data Migration is like ‘A Solar Eclipse of the Project Management World’
At a recent event, one of the speakers commented that data migration projects were likened to a solar eclipse. Most project leaders may only see one in their entire lifetime but when they do it is a truly memorable experience.
The problem is exarcebated of course because not only does the project leader often have no experience of a large, complex data migration but their supporting team may have limited expertise also.
Challenge #2: Companies Struggle to Retain Specialist Knowledge and Skills
When your data migration is complete you will soon find all your key experts dispersing to the four corners of the organisation. That’s if you’re lucky, within one year there may be a fraction of the team still employed within the same organisation.
Data migration skills retention is a big problem for organisations and again this is due to the fact that data migration is not really seen as a discipline in its own right. Very few companies build a centre of excellence for data migration or even document the techniques and best-practices which they found beneficial on a project.
Another challenge is that particularly with the source (legacy) systems there is often a lack of expertise. Many of the legacy experts or developers may have moved on through attrition over the years leaving a major hole in the skills required to deliver the project.
Challenge #3: The Data Migration Knowledge Gap
As mentioned earlier, for most companies, data migrations are not a common event and data migration leaders can soon experience gaps emerging between perceived wisdom and what is really required to deliver a successful project.
Common gaps can occur around:
Which methodology should we adopt for the migration? A lot of companies mistakenly believe that a basic data integration or standard IT methodology will suffice.
What software tools are required? Many leaders will simply use whatever licenses are available but fall into the trap that their project strategy may require additional functionality that isn’t supported with the current tools at their disposal.
What project management approach should we adopt? Data migration projects are actually better suited to an Agile delivery approach as opposed to the classic waterfall model yet most leaders will opt for the latter - without understanding the implications.
What execution strategy must be deployed? Big-bang, parallel, phased, synchronised? What do these options mean for leaders, particularly those who are unfamiliar with these terms and the techniques needed to deliver them.
What is the knowledge gap of the source and target systems? What are the data quality levels? Where is the data residing? Which is the best source of data? All challenges that need to be resolved.
So, lots of questions and often not many answers await the budding project leader.
2. The Unique Challenges of a Data Migration Leader
Experian Pandora Data Migration Survival Guide (Part 1 of 2) - 5
Challenge #4: Acting as a Buffer Between Business, IT (and the Rest of the World!)
As a data migration leader you’re going to find yourself in a no-mans land between different groups who all have conflicting demands.
This can feel quite alien to those project leaders who typically get a lot of buy-in and support on their projects.
Here are some of the conflicts you may have to resolve:
Systems Integrators want to make a profit and will be openly resistant to tackle things like data quality issues or anything that falls outside of their remit
Business users and legacy experts want to be left alone and get on with their day job, they may react badly to being told to give up their time to supprt the project
Target technology providers want to get the project completed rapidly so they can get paid
Your internal IT team may want greater control of the project and see it purely in technical terms when it is in fact a business-driven exercise
You will find yourself in the middle, orchestrating these relationships and if this is your first data migration project it can be a daunting prospect.
Challenge #5: Leaders Struggle to Get Accurate, Timely Facts
One of the big challenges facing project leaders is knowing which facts to ask for and also getting credible responses back.
On one project I discovered that the project leader had failed to ask a number of key questions around data quality management in the legacy system. As a result, the
project went ahead to the live phase and failed dreadfully.
As a project leader you need to have a solid understanding of best-practice data migration methodology so that you can request and receive the right information throughout the project. So many leaders hit problems late in the project because they’ve been driving blind the whole time, basing their decisions on guesswork and assumptions.
Experian Pandora 6 - Data Migration Survival Guide (Part 1 of 2)
Notes: Data Migration Challenges Please use this section to document any challenges that you are facing right now, or you believe will be present during the data migration.
Experian Pandora Data Migration Survival Guide (Part 1 of 2) - 7
3. Common Data Migration Assumptions Leaders Must Address
“The way to keep yourself from making assumptions is to ask questions.” - Don Miguel Ruiz
Your success as a data migration leader hinges on managing and eliminating assumptions. To do this you need to ask (the right) questions constantly. If left unchecked, assumptions will more than likely cause the failure of your data migration project (it’s a statistical fact).
Why are mistaken assumptions so rife on data migration projects?
Assumptions are created by inexperience and oversimplification of data migration. If you remember, data migration skills are rare in most organisations and the task of data migration is often viewed as overly simple or quite basic in delivery.
Your role as a leader is to hope for the best but plan for the worst.
A great example of this was illustrated by one of my members on Data Migration pro who discovered 6 months after their healthcare migration was completed that they had lots of missing information in their patient records database.
I asked “Who was responsible for testing that all of the target data reconciled correctly with the legacy data?”
The project leader said it was probably the hospitals responsibility but they thought the vendor would do enough as part of their system testing – clearly there were some assumptions that hadn’t been validated.
Whenever you hear a vendor, team member or business user giving you facts, check their validity, what’s the source? Is it proven? Can they back it up.
If there are assumptions you need to work them through and get some second opinions.
Assumption #1: ‘We don’t need a data quality strategy’
This assumption is often borne from fear. Fear of delays, fear of cost and fear of that unknown monster that lies beneath all legacy systems – bad data.
This is one of the biggest failure points for projects and industry studies by Bloor Research have categorically proven that if you ignore data quality then not only will you suffer delays and outright failure but the target functionality will also be severely impaired.
The economics of ignoring data quality on data migration projects just don’t stack up. What you may save in short-term cost-cutting of tools and staffing will push you much deeper into the red when your project spins out of the control, suffers massive delays and fails to deliver what the business ultimately needs - a fully operational system with great quality data.
Personally, I’ve witnessed £50,000,000 migration projects fail to migrate a single item of data simply because data quality best-practices were ignored.
Assumption #2: ‘Data Migration is a technical initiative’
Another popular assumption you’ll meet is that data migration is a technical endeavour and should therefore either be outsourced completely to a technology-focused integrator or delivered entirely by an internal IT team.
Data migration has a strong undercurrent of technology and technical requirements but make no mistake, data migration is a business initiative. It demands central command and control from the business and total business engagement throughout.
Not convinced? The number one cause of migration failure as cited by recent studies by Bloor Research was lack of business engagement. The next time your stakeholders or systems integrator demand an IT-centric approach to data migration be mindful of the need to have the business fully involved, it’s their data and their future performance that is at stake.
Experian Pandora 8 - Data Migration Survival Guide (Part 1 of 2)
Assumption #3: ‘We need perfect quality data’
This may seem counterintuitive for many of you I’m sure as we’ve already outlined the need for a data quality strategy earlier.
However, the problem is that unlike data quality management programs that can run indefinitely, data migration projects are time-boxed.
As a data migration leader, tough decisions need to be taken to balance the trade off between time, resources, budget, quality.
The key is to make these decisions with assurance and all the facts to hand. Too many leaders assume that they don’t have the resources or budget to resolve data quality so they ignore it completely and hope for the best. Others focus their energies on improving the wrong type of data. This is why business engagement is so critical, they need to provide detailed feedback over what type of data is really needed and what quality levels have to be in place.
If you’re going to migrate sub-standard quality data you need to know exactly what impact that will have on the target system.
For example if your new engineering support system routes staff via GPS tracking to the nearest fault then chances are your post code and address data needs to be faultless.
However, if you have local engineering teams who know ‘their patch’ inside out and manage their own call-out rotas then you may not need 100% complete postal address information because the engineers can find the required location with minimal information.
The key here is that even if you don’t need perfect data quality you still need a perfect data quality management process because you absolutely need to know the true measure of all your data.
Assumption #4: ‘We can assess data quality with a sample of data’
When you’re assessing data quality you need to be examining full volumes of data. Your goal as a project leader should be to deliver a zero-defect migration at the point of execution. To do this there can be no surprises. Only assessing a sample of data leaves your plan wide open to delays and even outright failure as the project struggles to cope with the defects that slip through the net due to being missed via the sampling tactic.
There is simply no excuse for a sampling approach. Modern data migration and data quality technology can cope with full volumes so use them.
Don’t forget that you can’t measure data quality in isolation during a one-time process, it needs to be continual right up to the point of migration. Only then can you be sure you’ll get a zero defect migration which should be the absolute requirement for all migrations.
Assumption #5: ‘Our project is different, we don’t need best-practice approaches’
A lot of people create various excuses as to why they don’t need a data migration methodology.
They say that their supplier ‘has it covered’ or their data volumes are too small or it’s a trivial project.
Whatever the reason, so many projects fall into this trap and 18 months into a 6 month project they’re still trying to pick up the pieces.
I recommend you adopt a robust data migration methodology that has been proven to deliver successful projects. Ask for proof. Seek out past organisations who have followed the same approach.
Experian Pandora Data Migration Survival Guide (Part 1 of 2) - 9
In one instance, I was introduced to an organisation who refused to use data migration specialists on the project because they felt the ‘strategy was locked down’ and therefore any additional expertise was wasteful. The project was a catastrophe, running well over the forecasted budget and timescales.
Assumption #6: ‘We can juse use in-house technology to cut costs’
As a leader you need to really watch for this assumption.
Your IT team may have a rag-bag collection of tools and skills that whilst useful on everyday data analysis and coding projects will fail to scale and deliver on big, complex data migrations.
A classic example is the project team I visited that were trying to migrate their entire Oracle landscape on to Teradata using PL/SQL scripts, it turned into a nightmare for the project leader.
They soon lost a grip on configuration management, auditing, data quality rules, mapping rules, data lineage - everything required to successfully deliver a data migration.
As a leader you’ll be pushed to keep the costs down but consider this, if your project slips by 30% - what does that cost look like?
Now what about 50%? 80% I’ve seen projects slip by 300% because the team were trying to code everything with SQL and ill-matched internal tools.
It’s a false economy to cobble a migration architecture together using antiquated technology. When you introduce things like data quality, profiling, archiving and cleansing into the requirements you definitely need to be using the right tool for the job.
Assumption #7: ‘We can’t start designing the migration until the target system is ready’
There is often a hesitation to get started with your data migration project until the target system is fully defined. The viewpoint being that all of the data structures need to be tied down and specified before you can begin the mapping design process.
This is an incorrect assumption because there are a whole stack of activities that need to take place before you even start thinking about mapping source to target systems together. What’s more, you don’t need the target fully defined to start, you just need to know the broad functions that are required.
All data migration projects involve the movement not just of data but of business functions. By understanding the target functions you don’t need the detailed interface and target schema specs to get started. In a lot of situations the target vendor won’t even give you a schema because it’s their intellectual property!
Instead, document those functions required in the target system and then use that as a benchmark to validate and source your legacy data.
Can you find the information required? Does it have the right levels of quality? These are all problems you need to solve well in advance of the target system being ready.
If you hear the “no-target, no-migration” line being discussed then push back immediately and present a case for getting started.
Experian Pandora 10 - Data Migration Survival Guide (Part 1 of 2)
Summary Tips on Managing Assumptions
Here are some tips that you should find useful when managing data migration assumptions:
Counter assumptions with accurate facts gained through intelligence gathering
Seek expert help (independent if possible)
Understand the implications and risk of every assumption
Perform a Pre-Migration Impact Assessment before the data migration starts or as early in the project lifecycle as you are able
Notes: : Data Migration Assumptions Please use this section to document any assumptions that are found on your project and how you intend to address them:
Experian Pandora Data Migration Survival Guide (Part 1 of 2) - 11
4. Leadership Planning
“Give me six hours to chop down a tree and I will spend the first four sharpening the axe.” - Abraham Lincoln
The quote above outlines precisely what you should be doing as a data migration leader. The more effort you put into planning the migration from the outset, the easier it will be after the project really gets into overdrive.
Your biggest challenge as a data migration leader will be dealing with overwhelm. Migrations are not like other data projects in my experience due the dependencies they have to deal with and the tight deadlines you need to hit before you start really impacting the wider target system implementation. If you miss your planned go-live date then you really are heading for a world of pain as a data migration leader:
Suppliers will be on your back for more money
Sponsors will be on your back for quick results and good news stories
Business users will be on your back for a go-live status
The financial controller will want to know why you’re burning through her budget
With meticulous planning you can really go into a migration with the confidence that a lot of the hard work has been taken care of. Things like the programme office, the deliverables and artefacts list, the outline roadmap, the skills profiles, the technology requirements – try and get all of that firmed up right at the start.
Tip: Use the Data Migration Project Checklist to help you identify some of the key planning items required:
http://checklist.x88.com/data-migration-checklist/
How to Deal with the Data Migration Knowledge Gap
The following diagram outlines the typical problem facing leaders at the start of a data migration project. Far too many assumptions and a severe lack of knowledge.
The gap between assumptions and knowledge typically gets closed somewhere towards the end of the project, where it’s too late to make any real difference to the outcome. What follows is the all too familar increase in cost and delays. The final end date can finish some considerable time after the forecasted end date.
How can project leaders reduce the knowledge gap?
The key to success on a data migration is to accelerate and improve your ‘knowledge gathering engine’. You want to be finding those big ‘showstopper’ issues for example, right at the outset, ideally before the project has even commenced.
I therefore recommend that as a leader you demand for the introduction of a Pre-Migration Impact Assessment. I would go so far to say that this technique will guarantee your survival more than anything else on a data migration project.
The goal of a pre-migration impact assessment is to replace assumptions with knowledge, before the migration formally starts.
For a data migration leader this is a godsend. I’ve seen the look of astonishment on executive, senior sponsors and migration leaders faces when I deliver the results of a pre-migration impact assessment and believe me, they are eternally grateful when they can finally visualise the pitfalls they were about to fall into, well in advance of the project delivery date.
The diagram below illustrates how the knowledge gap is closed much earlier using a Pre-Migration Impact Assessment.
Assumptions
Knowledge
Unexpected Costs & Delays
Start Forecasted End
Actual End
Experian Pandora 12 - Data Migration Survival Guide (Part 1 of 2)
For the leader, a Pre-Migration Impact Assessment is a career saver because it determines such things as:
The skills and technologies required on the project
The likely danger areas that will impact strategy
The first draft outline plan for launching the migration
The cost estimates for more accurate forecasting
In one personal example, I carried out a Pre-Migration Impact Assessment (PMIA) for a large telecoms company. As a result they discovered that they were actually dealing with a Geographic Data migration, not just a relational data migration. They were then able to reflect that in their costing plan and migration execution strategy.
Without the PMIA they would never have understood those implications.
In one organisation they discovered that their legacy systems lacked up to 30% accuracy in their equipment data, compounded by 40% incompleteness in other parts of their data. This accumulation of data defects rendered their original data migration strategy totally flawed.
In another example, a company found that they had a large, unexpected, volume of duplicate equipment parts which required specialist software they hadn’t budgeted for. If the leader had commenced the project without this knowledge he would have had to go cap-in-hand to the finance officer for more cash – at the build phase of the project. No amount
of excuses gets away from the fact that you look bad as a leader whenever you demand more unplanned cash.
A key benefit of a PMIA is that you create an outline plan because essentially you’re performing a data migration simulation. You get a feel for what activities, resources and sequencing will be required when the project really gets motoring.
Armed with these kind of facts you get a much clearer picture of the likely costs involved.
How do you deliver a Pre-Migration Impact Assessment?
The diagram below outlines a classic Pre-Migration Impact Assessment process.
The goal is to create the PMIA in rapid time, this should not a long drawn out exercise. You are aiming to find vital intelligence about how the project should be structured and delivered.
Here are the stages of a PMIA explained:
High-Level Function Discovery : Because the PMIA is often initiated before the target has been fully specified a useful exercise is to understand what business functions are currently used in the legacy environment and planned for the target environment. This can often help you observe obvious gaps, even before data analysis.
Define scope and scale of data sources: Based on the functional analysis you can prioritise which entities (tables) to extract and which records are prioritised. Ideally, you should be using advanced data
1
2
Assumptions
Knowledge
Start Forecasted/Actual End
Pre-Migration Impact Assessment
Time
High Level Function Discovery
Scope & Scale
Rapid Landscape
AnalysisMigration
SimulationGaps & Issues
Security &
Access
Rescouring Facts
Artifacts
Experian Pandora Data Migration Survival Guide (Part 1 of 2) - 13
profiling and discovery tools to support this process to ensure that the right data is identified.
Data access and security approval : Some of the data may require specialist security measures (such as encryption), a PMIA activity will ensure that all of the security and access issues are resolved well before the project formally starts.
Rapid Landscape Analysis : This is typically performed with specialist data discovery, profiling and quality software because the aim is to analyse huge volumes of data in an extremely short space of time. You will also need to actually improve some of the data (typically offline, away from the production environment) to get a feel for what the costs and activities required for improving the data later in the migration will be, again specialist tools are advisable. The aim here is to identity what data is needed, how it should be connected, what quality it possesses and what likely issues will be faced during migration.
Migration Simulation : At this stage we have a better grip on what data will be required and how it connects so we can start to prototype what will happen when we link these disparate datasets together and start trying to move them into a format and structure that will support those earlier target business functions we outlined.
Resourcing Facts : As a result of the earlier activities we will start to identify the types of software required during the migration, the volumes of data and issues present, what type of data access and interfaces will be required - all key information for allocating resources later in the project.
Project Artifacts : A PMIA is basically a cutdown version of a full migration project so you can start to populate many of the artifacts that will be used later in the main project. For example, you will be able to create detailed data quality assessments,
mapping documents, stakeholder maps, interface specifications and many more resources, all from the PMIA activity. This is a great help for the Data Migration Leader who wants to get their project off to a flying start.
Gaps and Issues : Perhaps the single most important reason for running a PMIA is to find those gaps between the source and target systems that will prevent the likely success of your migration project. Many initiatives are canned because of assumptions about data, interfaces, access to expertise and other variables that soon turn into big gaps in the project. Use your PMIA as a tool to find these gaps and issues before the project really starts to ramp up.
Important: Use of Specialist Software
If possible you should always leverage advanced data discovery, profiling and data quality assessment software, particularly during the PMIA activity. It will save you literally months of effort and can be used to deliver all of the later data discovery, mapping and data quality functions as well.
What is Delivered by the Pre-Migration Impact Assessment?
Skills register: Types of data and issues found plus architectures required for migration e.g.
Customer address cleansing requires different skills to product equipment
“Big Data” volumes in a 24/7 environment? Low volumes from multiple systems?
Stakeholder Map: Who is responsible for IT? Business? Security? Downstream?
Security framework: Which data will require special security restrictions?
Outline Roadmap - Milestones, core activities, deliverables, technologies, resources
3
4
5
6
7
8
Experian Pandora 14 - Data Migration Survival Guide (Part 1 of 2)
Data discovery documentation - relationships and mappings, domain analysis, value distributions, quality metrics, outliers and issue reports
Impact assessment and risk register: Will it work? What are the risks?
Notes: Data Migration PlanningPlease use this section to document any planning changes you wish to implement.
Experian Pandora Data Migration Survival Guide (Part 1 of 2) - 15
5. What Next?
In the next edition of your Data Migration Channel subscription you will receive Part 2 of the Data Migration Survival Guide where we will cover:
Tips for setting up the Project Office
Tools and techniques for Landscape Analysis
Components of a Data Quality Capability for Data Migration
Agile Design and Build Tactics
Orchestrating the Go-Live Migration Strategy
Testing and Assurance
Archival and Decommissioning
Ongoing Data Quality Assurance
6. Do You Need Help?
If you have any questions about the topics raised or how to apply next generation data quality and data migration solutions on your project then please contact us:
E: [email protected] T: 0800 197 7920
About Experian Data Quality Experian Data Quality has built up exceptional market coverage assisting customers with their unique data quality challenges. We provide a comprehensive toolkit for data quality projects combining our market leading software with a vast scope of reference data assets and services. Our mission is to put our customers in a position to make the right decisions from accurate and reliable data. The size and scope of data management projects varies considerably but the common factor in all ventures is unlocking operational efficiency and improving customer engagement. We see the potential of data. Whether it’s in enabling ambulances to be sent to the exact location of an emergency or attributing charitable donations to the people who need it the most - data accuracy makes all the difference to service provision.
Dylan Jones Editor, Data Quality Pro
Dylan Jones is the Editor and Founder of Data Quality Pro and Data Migration Pro. He has over 20 years’ experience of delivering complex data quality and data-driven initiatives. Dylan is a keynote speaker, author and regular publisher of expert insights on data quality related topics.
Experian Pandora 16 - Data Migration Survival Guide (Part 1 of 2)
Experian Data Quality George West House, 2-3 Clapham Common Northside, London, SW4 0QL.T 0800 197 7920 | E [email protected] | W www.qas.co.uk/dataquality
© Experian, 2013. All rights reserved
The word “EXPERIAN” and the graphical device are trade marks of Experian and/or its associated companies and may be registered in the EU, USA and other countries. The graphical device is a registered Community design in the EU.
Experian Ltd is authorised and regulated by the Financial Conduct Authority. Experian Ltd is registered in England and Wales under company registration number 653331. Registered office address: Landmark House, Experian Way,
Intelligent interactions.Every time.
Intelligent interactions.Every time.
© Experian, 2015. All rights reserved
The word “EXPERIAN” and the graphical device are trade marks of Experian and/or its associated companies and may be registered in the EU, USA and other countries. The graphical device is a registered Community design in the EU.
Experian Australia Pty LtdLevel 6, 549 St Kilda RoadMelbourne, VIC 3004, Australia
T (61) 3 8699 0100 | F (61) 3 9923 6280 E [email protected] | W edq.com/au
