Expedia Teradata Migration The Good, the Bad and the Ugly…

a candid view into a migration project

Mar 19 2015

2

Introductions

David Jorgensen – Director of Software Development

Barb Nicholson – Engineering Manager

Gengmei Zhao – Engineering Manager

Todd Nielson – Sr Technical Lead

Daniel Tehan – Teradata Solutions Architect

TUG - Seattle

3

4

5

6

The Good • Retire DB2 by end of 2015 • Move Booking Data Sets • Minimize Impacts

The Bad • Minimal time for other business

efforts

The Ugly • Maintaining DB2 and Teradata in

parallel

7

Scope of the work TUG - Seattle

Current services supported DB2

3 MS SQL Servers

Windows file server

Informatica on Linux

Home-grown scheduler

Lab support – multiply by 4

ETL Tools

Home-grown platform

Informatica

Current languages leveraged DB2 SQL

SQL Server Transact SQL

Informatica

Windows CMD scripting

Windows VB scripting

Perl

Linux shell scripting

8

Scope of the work TUG - Seattle

GOAL: Migrate our data marts to Teradata

Timeline: 1 year

Number of tables to migrate Landing Zone 1500 Atomic Data Store 400 Data Mart 130

Number of ETL processes to replace Landing Zone 1500 Atomic Data Store 450 Data Mart 60

Other problems we’d like to address Reduce the number RDBMS services supported (2) Reduce the number of ETL tools (2) and languages (7) needed for dev/support

9

Scope of the work

GOAL: Migrate our data marts to Teradata

Timeline: 1 year

Number of tables to migrate Landing Zone 1500 Atomic Data Store 400 Data Mart 130

Number of ETL processes to replace Landing Zone 1500 Atomic Data Store 450 Data Mart 60

Other problems we’d like to address Reduce the number RDBMS services supported (2) Reduce the number of ETL tools (2) and languages (7) needed for dev/support

TUG - Seattle

10

Scope of the work

GOAL: Migrate our data marts to Teradata

Timeline: 1 year

Number of tables to migrate Landing Zone 1500 Atomic Data Store 400 Data Mart 130

Number of ETL processes to replace Landing Zone 1500 Atomic Data Store 450 Data Mart 60

Other problems we’d like to address Reduce the number RDBMS services supported (2) Reduce the number of ETL tools (2) and languages (7) needed for dev/support

TUG - Seattle

11

Scope of the work

GOAL: Migrate our data marts to Teradata

Timeline: 1 year

Number of tables to migrate Atomic Data Store 400 Data Mart 130

Number of ETL processes to replace Atomic Data Store 450 Data Mart 60

TUG - Seattle

Other problems we’d like to address Reduce the number RDBMS services supported (2) Reduce the number of ETL tools (2) and languages (7) needed for dev/support

12

Scope of the work

GOAL: Migrate our data marts to Teradata

Timeline: 1 year

Number of tables to migrate Atomic Data Store 400 Data Mart 130

Number of ETL processes to replace Atomic Data Store 450 Data Mart 60

TUG - Seattle

13

What next?

Drop A Anchors Aweigh…

Full steam ahead!!!

14

After the migration

Rather than “Anchors Aweigh!”, it should be “Weigh Anchors!”

What’s left to do

Move Landing Zone to Teradata

Migrate all processing to Teradata TPT and stored procedures/macros

Benefits

Fewer systems to manage

Simplified engineering skill set; easier to enhance applications

Easier to automate testing

When? Q1 / 2016

TUG - Seattle

Transform & Load

15

Current State – Booking Data Sets

DB2

Atomic Data Store

Data Marts

Landing Data Downstream Systems & Access Layer

Sources SQL Server

Landing Zone (LZ)

Hadoop

Lodging Cubes

MS Analysis

Services

TUG - Seattle

Transform & Load

16

Milestone 0 & 1: Sync of Data Marts to Teradata

DB2

Atomic Data Store

Data Marts

Landing Data Downstream Systems & Access Layer

Sources SQL Server

Landing Zone (LZ)

Hadoop

Lodging Cubes

MS Analysis

Services

TUG - Seattle

Teradata

DM

17

Milestone 1: Sync of Data Marts to Teradata

The Good:

» Data Marts available on Teradata

» Great team effort

» Adopted agile process

» Established documentation standards

The Bad:

» Learning curve:

• Database architecture changes

• Access and credentials

The Ugly:

» Frequent errors in sync processes

» Break/Fix during sync, new OPS processes

TUG - Seattle

Dev

18

Production

Milestone 2: Snapshot of ADS and DM to Lab

Teradata

DM ADS

DB2

ADS

DM

Test Teradata

DM ADS

TUG - Seattle

Description To Do Complete Status

Step 1 DB2 to TDProd 413 Done

Step 2 TDProd to TDLab 413 Done

Step 3 Create Snapshot Views for DEV environment

243 Done

Step 4 Create Snapshot Views for TEST environment

243 Done

19

Milestone 2: Snapshot of ADS and DM to Lab

The Good:

» Identified the full list of inventory

» Transferred 10TB of data

» Global efforts

The Bad:

» “Lift and shift” decision

» DB2 capacity limitations

» TPT file import challenges

The Ugly:

» Tables with data were accidentally wiped out

TUG - Seattle

Transform & Load

20

Milestone 3: Landing Zone -> TD ADS

DB2

Atomic Data Store

Data Marts

Landing Data Downstream Systems & Access Layer

Sources SQL Server

Landing Zone (LZ)

Hadoop

Lodging Cubes

MS Analysis

Services

TUG - Seattle

Teradata

DM

ADS

Transform & Load

21

Milestone 4: ADS -> DM processing in parallel Teradata

DB2

Atomic Data Store

Data Marts

Landing Data Downstream Systems & Access Layer

Sources SQL Server

Landing Zone (LZ)

Hadoop

Lodging Cubes

MS Analysis

Services

TUG - Seattle

Teradata

DM

ADS

Transform & Load

22

Milestone 5: Downstream Processing

Landing Data Downstream Systems & Access Layer

Sources SQL Server

Landing Zone (LZ)

Hadoop

Lodging Cubes

MS Analysis

Services

TUG - Seattle

Teradata

ADS

DB2

Atomic Data Store

Data Marts

DM

Transform & Load

23

Milestone 6: Source -> TD Landing Zone

Landing Data Downstream Systems & Access Layer

Sources

Hadoop

Lodging Cubes

MS Analysis

Services

TUG - Seattle

Teradata

SQL Server Landing

Zone (LZ)

ADS

DM LZ

Transform & Load

24

Final State

Downstream Systems & Access Layer

Sources

Hadoop

Lodging Cubes

MS Analysis

Services

TUG - Seattle

Teradata

ADS

DM LZ

25

TUG - Seattle

26

Addendum

1 footnote 2 footnote Source: Source

The Good, the Bad and the Ugly: http://word.legionwithin.com/wp-content/uploads/2013/04/16268_the_good_the_bad_and_the_ugly-220x140.jpg

Vector from Despicable Me:

http://st.houzz.com/fimgs/61f198a2038b6a11_5587-w251-h251-b0-p0---prints-and-posters.jpg

Morris Mini:

http://www.toywonders.com/productcart/pc/catalog/5301_2-1959-MORRIS-MINI-MINOR-SALOON-112-Sun-Star.jpg

USS Ronald Reagan: https://upload.wikimedia.org/wikipedia/commons/3/33/US_Navy_080728-N-5961C-005_The_Nimitz-class_aircraft_carrier_USS_Ronald_Reagan_%28CVN_76%29_drops_her_anchor_to_secure_the_ship.jpg