3501 How Nike Minimized System Downtime With the Near Zero Downtime Method Through ALM

]Orange County Convention CenterOrlando, Florida | May 15-18, 2011

Near Zero Downtime@ Nike

Courtney Hofmann NikeMarc Hopkins Nike

Mariusz Debowski SAP

Real Experience. Real Advantage.

[

2

Learning Points

? Understand how Nike was able to leverage Near Zero DownTime (NZDT) to reduce the Business Downtime for itsUpgrade and Unicode Conversion project.

? Understand the key differences between the NZDTmethodology and a standard Upgrade & Unicode Conversionapproach.

? Understand the challenges & benefits introduced by the NearZero Down Time methodology.


[ Agenda1. Near Zero Down Time Overview

2. Nike Upgrade Project - Overview

3. NZDT Project Schedule

4. NZDT Project Execution

5. NZDT Challenges

6. NZDT Learning's

3


[ Costs of downtime vs. cost of availability

? Downtime costs are usually not linear over thetime of system outage

? They depend on the business impact? For longer downtimes costs can increase

progressively? Example: when SCM processes are stuck longer

than 3h the whole production will be stuck andcause even higher costs

Availability Cost Curve

Cos

t

95% 98% 99,5% 99,9%

Availability

(Source: Gartner Group)

Costs

? Redundant components? Disaster recovery site? System management tools? IT staff? Capacity planning? Guaranteed SLA? Proactive services

downtime

cost

s

4


[ Business Continuity RoadmapCurrent Areas of Investment

SPs EhPs Upgrade

Any system

NW hub

App specific

SAPehpi (for EHPs &SP Stacks only)

SAPup

nZDMEP/PI

NZDT (ERP)

Standardtools

Customerprojects

XXL Customers

EnBW: ~30min Business downtime

SBSA: ~ few hoursEhP5 TCO lab:3h tool downtime,10h tool runtime

New Iterative Shadow Upgradeready for pilot end of 2011

Extension to Banking Servicescurrently in evaluation

Further optimizationand improvements

5


[ Downtime Minimization

X00s customersrunning global systems in7x24x365 mode

X000s customersrunning mission criticalapplications

X0000s customerswith less demand on businesscontinuity

Need forBusinessContinuity

Need forminimalOutages

Need forlowest TCO &simplicity

Prio 1 Demand

Near Zero Downtime /Iterative Shadow Upg

SAP ServicesBusiness Downtime Optimization

SAP tools easy to use andappropriate documentation

SAP is constantly working on supporting our customers in the effort tominimize their system downtimes

Further optimization of the existing tools for software updates

Sharing with the customers the collected experience and best practices

Creation of new tools and service offerings for customers operating their systems in 24x7 mode

6


[ 10 steps to Near Zero DowntimeHigh level illustration

Delta replay

Recording

PRDR/3 4.6C nUC

hostA

clone

downtime

Prepare + Upgrade Uptime

Upgradeand Unicode conversion

+ Transports+ Add Ons+ Pre-Testing

Last Delta replay

Final-Validation

Infrastructureadjust. -interfaces

Upgrade + UC Downtime Post ProcessingDeltaTransfer

Infrastructure

PRDR/3 4.6C nUC

hostB

PRDECC 6.03 UC

hostB

hostA

Validation

2

3

45

6

7 8

9

10

21

1. Users are working on PRD system2. Copy of PRD system & start recording of transaction data3. Data from Production live transactions are recorded4. Upgrade and Unicode conversion on the Copy to ERP 6.035. Finish EhP implementation and replay transactions from Production to Copy6. Users shutdown on Production system and finish replaying transactions7. Start reconciliation source to target system8. Get signoff that both systems are reconciled9. GO decision10. Start up users on Copy (new PRD)

DO

WN

TIM

E

7


[ Near Zero DowntimeApplicability use cases

The method can be used for numerous downtime events? Release Upgrade

?Enhancement Package implementation?Support Package Stack installation?Unicode conversion?Customers release mass transports

and non-SAP events?OS/DB migration?OS patches?DB patches?DB reorgs

and any combination of events above

Delta replay

Recording

PRDERP 6.02

hostAclone

downtime

Prepare + Upgrade Uptime

Implement EhP 4 (and others)

+ Transports+ Add Ons+ Pre-Testing

Last Delta replay

Final-Validation

Infrastructureadjust. -interfaces

EhP installation downtime Post ProcessingDeltaTransfer

Infrastructure

PRDERP 6.02

hostB

PRDERP 6.04

hostB

hostA

Validation

? 8


[

Near Zero Downtime @ Nike

1st combined Upgrade & Unicode Conversion using NZDT

9


[ The Upgrade A New SAP Landscape

10

Opportunity for Dramatic Technical Changes

? SAP Upgrade from 4.6C to ERP 6.0? New Oracle Version - 10.2.0.4? New Hardware HP Superdome Virtual Partitions with shared CPUs? New Storage Frame EMC VMax? New Network Infrastructure? Unicode conversion performs a full database reorg of all tables & indexes? Oracle Index Key Compression? Complete Technical RE-DO of existing system


[

? Three parallel upgrades/Unicode Conversions were in scope (25TB, 12TB, ~1 TB)? First Upgrade & Conversion took 16 days (18TB)? The challenge was to reduce it to 7 days? Reduced the standard upgrade & Unicode conversion to 4 days? System continued to grow at 500GB / month causing downtime to increase (25TB)? 4+ days was still too long for the business

Why Nike Chose NZDT

How much downtime can we afford?

11

Building Blocks of the Standard ERP Upgrade






5. NZDT Challenges

6. NZDT Learning's

12


[ Nikes ERP System Landscape

PRA320

PRD300

Config. Development

PTE300

PTA320

TransportPath

110110 120120

168168

ER1

100100

Security

130130

PTF330

PRF330

0.5TB9 TB21 TB

Single shared code &configuration base

Single developmentenvironment is source of alltransports

All 3 systems follow the sameupgrade path

13

Landscape Details


[ Nike ERP Upgrade - Release ScopeBeforeBefore

SAP R3 4.6c 10 year old system

16 languages implemented via MDMPTechnology

Old hardware nearing end of life

1695 Modifications 182 Clones 13 Special Developments 459 SPDD / 2521 SPAU Historically grown enhancements

Language specific Interfaces

Twin Upgrade and Unicode ConversionTarget

6+ days of Business Downtime

AfterAfter

SAP ERP 6.0

Unicode Fully reorganized database Standardized & optimized database layout

New Hardware/Infrastructure Footprint

870 Modifications 81 Clones 7 Special Developments 87 SPDD / 1395 SPAU Consolidated redesigned enhancements

in key areas

XML based UTF8 Interfaces in SAP

Near Zero Down Time

Weekend Outage window

BasisPlatform

NikeCustom

Solutions

UpgradeProcess

14


[ Standard Upgrade vs. Near Zero DowntimeStandard UpgradeStandard Upgrade

Major upgrade phases executed duringdowntime? PRD: 69 hours

Unicode Conversion executed duringProduction System downtime? PRD: 96 hours

4000 Transports imported duringdowntime? PRD: 20 hours

NZDTNZDT

Major Upgrade changes imported whileProduction systems were available

Unicode Conversion executed whileProduction systems were available

4000 Transports imported while Productionsystems were available

Upgrade

UnicodeConv.

PostUpgrade

Tasks

15


[ Nikes ERP Upgrade Project History

2011201020092008

InitialUpgrade

Effort(Dev & QA)

UpgradeProof ofConcept Upgrade

DevelopmentTUUC*

Methodology

NZDTProof ofConcept

NZDTProject

UpgradeQA Systems

GoLive!

*TUUC: Twin Upgrade & Unicode Conversion

16


[ Nike ERP Upgrade High-Level Project Requirements

? Minimum impact to productive operations due to upgrade

? Business Downtime as low as possible

? Allow flexibility to reschedule upgrade within a short timeframe

? One-step Upgrade procedure

? In case of problems the upgrade must be completely reset-able(also possible using snapshot technology established at Nike)

Nik

e R

equi

rem

ents

SAPs Near-Zero-Downtime Technology

Next StepNext Step

17






5. NZDT Challenges

6. NZDT Learning's

18


[ Nike NZDT Project Schedule20102009

? Code Remediation? QAST System Test? SIT Integration Test? UAT User Acceptance Test? Dry Run? Go-Live

? NZDT POC Proof of Concept? NZDT SAP Analysis? QAST Build (NZDT without delta)? NZDT Full execution 1? NZDT Full execution 2? NZDT Dry-Run Build

Legend: Duration Milestone

2011

J JA O JF M M J A S N DDNO F M

Upgrade Go-Live with 3parallelsystems.

Total number of builds is 15 (5build cycles with 3 systems each).4 build cycles need completeNZDT procedure.

19






5. NZDT Challenges

6. NZDT Learning's

20


[ Stepping Through Nikes Upgrade with NZDT

21


[ Cutover Timeline Uptime & Downtime

13-D

ec

14-D

ec

15-D

ec

16-D

ec

17-D

ec

18-D

ec

19-D

ec

20-D

ec

21-D

ec

22-D

ec

23-D

ec

24-D

ec

25-D

ec

26-D

ec

27-D

ec

28-D

ec

29-D

ec

30-D

ec

31-D

ec

1-Ja

n

2-Ja

n

PREPARE, UpgradeUptime Clone NZDT Recording Window Down-time

CloneUpgrade

Down-time,SPUMG from UAT

UnicodeConversion

UnicodeConversion

Tran

spor

ts

Dat

a C

onve

rsio

ns

SUM

G

Del

ta R

epla

y

Dat

a C

onve

rsio

ns

Test

ing

ProductionR/3 4.6C

NZDT CloneUC Source

UC TargetFuture ERP 6.0

22


[ Cutover Timeline The Downtime Window12/31 1/1 1/21/3

RampDown

Final Replay, HW Reconfiguration, Final Build Steps,Upgrade Conversions, Manual Config, FI Audit Snapshots

FI Ramp Up,Link Testing

Control Test

Timeline Callouts

? Ramp Down includes system lockout, FI Audit Snapshots, Delta Queue Cleanouts.? Final Delta Replay: Good performance and no unexplainable differences? Upgrade Conversions: Good performance? Total downtime duration 79 hrs

(from system lockout until Go Decision)

? Hardware reconfiguration required 8 hours? Not using NZDT would have resulted in at least

an extra 6 days of downtime!

23


[ NZDT Validation

SAP2007 /

? Technical Reconciliation? Initial plan:

Highly complex automated comparisonbetween NZDT Upgrade andnon-NZDT Upgrade system (Tech Rec 1)

? Summer / Fall 2010:SAP developed direct 4.6c to ERP 6.0reconciliation covering (Tech Rec 2)? Release changes? Unicode ConversionVia Table & Field level comparisons? Enabled ability to validate build during Cutover downtime

? Business Testing? Unicode Conversion? Unix Scripts for row-counts

0

50

100

150

200

250

300

350

Time in hrs

Tech Rec 1 Tech Rec 2






5. NZDT Challenges

6. NZDT Learning's

25


[ Challenges Project

Issue Description Resolution

NZDT Performance Impactto PRD

NZDT recording puts additional load onproduction systems

Multiple recording tests in Production

NZDT Recording FreezeTriggers

Frozen tables cause dumps in Production 24 x 7 monitoring by SAP & NikeDaily review meeting

Large amount of extrahardware

Multiple copies required of each Productionsystem

Virtualization and Cloning technique

Highly complex process Recording, mapping and replay technologycreates a highly complex process

Partnership with SAP enabled accessto expert resources

Security Access Unique NZDT process did not allow for theuse of Fire Fighter tool

Limited number of core teammembers with build access

Language Remediation forUnicode Conversion

On-going difficulties to establish ownershipof language remediation tasks

Small expert team performedremediation during Cutover

26






5. NZDT Challenges

6. NZDT Learning's

27


[ Nike Learnings Overall project

? Small cohesive core team with technical expertise used iterative execution& refinements for complete NZDT solution.

? Excellent Cooperation between Nike and SAP teams

? Perseverance of entire team through extreme hours and complexity toachieve objectives.

? Co-location of project team kept everyone on the same page regardingstrategy, current activities, planned activities, issues, and status

28


[ Nike Learnings Architecture

? Virtualization and Cloning technology set the stage to enable futureflexibility, rapid provisioning of ERP QA systems

? Initiate discussions with infrastructure partners early in the process

? Use of Oracle Flashback to instantly restore to a specified point in timewas vital (was utilized in every build)

? Plan for enough hardware to allow flexibility and ability to adapt tochanges in requirements

29


[

]? Thank you for participating.SESSION CODE:3501

Please remember to complete and return yourevaluation form following this session.

For ongoing education in this area of focus, visit www.asug.com.


[ Appendix

31


[ Nike ERP Landscape before Upgrade

PRA320

PRD300

Config. Development

PSQ300

PSA320

ProdSupport

110 120

168

DE1

100

Security

Prod Support

130

PSF330

PRF330

ER1

100

PTA320

110 120

130

PTE300

PTF330

12/10Upgrade

168

SecurityConfig. Development

32


[ Upgrade Prepare PhasePRA

320PRD

300

110 120

168

DE1

100

Prod Support

130

PRF330

ER1

100

PTA320

110 120

130

PTE300

PTF330

12/10Upgrade

168


33


[ Delta Recording Upgrade & Unicode Conv.PRA

320PRD

300

PRD300

PRA320

110 120

168

DE1

100

Prod Support

130

PRF330

PRF330

ER1

100

PTA320

110 120

130

PTE300

PTF330

12/10Upgrade

168


Upgrade & Unicode Conversion

DeltaRecording

34


[ Delta ReplayPRA

320PRD

300

PRD300

PRA320

110 120

168

DE1

100

Prod Support

130

PRF330

PRF330

ER1

100

PTA320

110 120

130

PTE300

PTF330

12/10Upgrade

168


DeltaRecording Delta

Replay

35


[ Go Live

PRD300

PRA320

PRF330

ER1

100

PTA320

110 120

130

PTE300

PTF330

12/10Upgrade

168


36


[ Near Zero DowntimeCustomers cases

Customers successful with Near Zero Downtime

Airbus EMEA, IS-Aerospace & Defense, Aircrafts Spare Parts ManagementRelease upgrade from SAP R/3 4.6C to SAP ERP 6.0First successful implementation of the NZDT method in June 2009System size 2 TBDatabase OracleAchieved technical downtime 4 hrsAchieved business downtime 6 hrs

Nestl EMEA, Nutrition and Food IndustryRelease upgrade from SAP R/3 4.7 to SAP ERP 6.03Successful GoLives of 9 ERP systems with the NZDT method in Spring 2010Total system size 80 TB (all 9 systems)Achieved technical downtime 2 hrsAchieved business downtime 16-20 hrs

Customer B EMEA, Diary ProductsRelease upgrade from SAP R/3 4.7 to SAP ERP 6.03 combined with Unicode conversionSuccessful GoLive of the ERP system with NZDT in October 2010Total system size 2 TBDatabase OracleTechnical downtime 3:30 HrsBusiness downtime 12 hrs

? 37



Customers with running Near Zero Downtime projects

Nike USA, IS-Foot & ApparelRelease upgrade from SAP R/3 4.6C to SAP ERP 6.03 combined with Unicode conversionSuccessful GoLive of 3 ERP systems with NZDT in January 2011Total system size 40 TBDatabase OracleTechnical downtime 18 HrsBusiness downtime 79 hrs

Customer D EMEA, Automobile and Industrial Equipment and Consumer GoodsCustomers release including implementation of EhP4Successful GoLive of 5 ERP system with NZDT in November 2010NZDT will be used on a regular basisTechnical downtime 3 HrsBusiness downtime 20 Hrs

Customer E USA, High-Tech IndustryRelease upgrade from SAP R/3 4.7 to SAP ERP 6.03 combined with Unicode conversionPlanned GoLive of the ERP system with NZDT in August 2011Database OracleTechnical downtime ? HrsBusiness downtime 15 hrs

? 38



Customers with running Near Zero Downtime projects

Customer F EMEA, Consumer ProductsRelease upgrade from SAP R/3 4.7 to SAP ERP 6.04 combined with Unicode conversionSLO projectSuccessful GoLive of the ERP system with NZDT in February 2011Database OracleTechnical downtime 19 HrsBusiness downtime 48 hrs

Customer G EMEA, Household ProductsCustomers release including implementation of EhP4Successful GoLive of ERP systems with NZDT in January 2011NZDT will be used on a regular basisTechnical downtime 4 HrsBusiness downtime 20 Hrs

? 39


[

]? Thank you for participating.SESSION CODE:3501

Please remember to complete and return yourevaluation form following this session.

For ongoing education in this area of focus, visit www.asug.com.

Documents

3501 How Nike Minimized System Downtime With the Near Zero Downtime Method Through ALM