Journey of a bi stack to the cloud UKOUG2017vanpupi.stepi.net/.../2017/12/Journey_of_a_bi_stack_to_the_cloud_20… · The journey of a bi-stack to the cloud VM1 VM2 VM3 AZURE West-Europe

THE JOURNEY OF A BI-STACK TO THE CLOUDPieter Van Puymbroeck

2

Dad Oracle DBA Sailor Musician working for Exitas Oracle Platinum Partner in BELGIUM

http://vanpupi.stepi.net

@vanpupi

http://vanpupi.stepi.net

About Exitas

Quality - Passion - Personality

• Team of 25 dba’s• Consultancy, managed services,…• http://www.exitas.be

The journey of a bi-stack to the cloud

3MembershipTiers• OracleACEDirector• OracleACE• OracleACEAssociate

bit.ly/OracleACEProgram

500+TechnicalExpertsHelpingPeersGlobally

Connect:

Nominateyourselforsomeoneyouknow:acenomination.oracle.com

@oracleace

Facebook.com/oracleaces

[email protected]

mailto:[email protected]

Agenda

• The project• Cloud semantics• The design• Tests• Provisioning• Initial load• Lessons learned• Q/A


Little red ridinghood and the big bad GDPR: how does it impact you

The project dates from January 2017 (The new content is added, where applicable)

Safe Harbour statement

And the message is still valid ;-)

The project

…


The project


How it started …

• Request for assistance for a move to the cloud?• it’s about 55TB prod data

• Prddb1 : 34TB• Prddb2 : 16TB• Prddb3 : 5TB

• Agile: “iterative incremental process”• oh by the way … non-prod is ~ 360TB … we think …


How it started …


Fact check


Regions


Location: Amsterdam

Azure


It’s all about avoiding limits

Disks


StandardDisks (HDD) PremiumDisks(SSD)

ConsistencyofprovisionedperformancePerformanceisnotprovisioned

This is what was available during the project:

Jan’17

Disks


And this is available nowadays:

Networking

Howto: https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-create-vm-accelerated-networking


Jan’17

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-create-vm-accelerated-networking

Accelerated networking (ACN)

Accelerated networking:• General available • Recommended





Availability sets

• https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-manage-availability

• VMsdeployedondifferentracks(=FaultDomain(FD))anddifferentphysicalservers(=UpdateDomains(UD))

• CfrVMwareAnti-AffinityRules


https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-manage-availability



Images


A D F G

H L N

Jan’17

Images


The current vm’s: Linux: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes

Windows: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes

Choosing a VM

• MS defined categories• Own requirements • VM = predefined set of resources

• # vCPUs• # vRAM• standard vs premium storage os/data disks• Maximum disk throughput• # data disks• Max disk throughput• Network bandwidth


Choosing a VM


1. Workload- Cpurequirements- Memory requirements- Diskrequirements

3.VMSize- CPU- RAM- #datadisks- Diskthroughput

2. Storage- S:faststorage- VMscanusepremiumstorage- VMsusesolid statedrives(SSD)

Standard_DS15_v2

Choosing a VM


Requirements:- High workload- High #disks- ACN needed- High throughput

—> DS15_V2—>Hard coupling between memory and cpu

Topology - current


DC1 DC2

BACKUPSHARED

BIDBDEDICATED

SAPBIBODEDICATED

DC1

DC2DC1ESXStretchedCluster

ESXClusterPROD ESXClusterNON-PROD

RHS– 8nodesspread over2DCs

VM VM VM VM

VM VM VM VM

DataguardVM

VM VM

Topology - To be: Requirements

•high network throughput ~ 825 MB/s—> RMAN Backup – CIFS share (Mos-note: 444809.1)

•High # number of disks —> Oracle database requires 34 TB

•Accelerated networking only available on DS15_v2 —> Backup Server

•Currently DS15_v2 only available and West Europe —> with accelerated networking


Topology - To be


VM1

VM2

VM3AZUREWest-Europe

PRODUCTION/STANDBY

DEVDB1

UATDB1 TESTDB1

REFRESH(Non-Prod)

Snapshots

DEDICATEDBACKUP

AZUREWest-Europe

Dataguard

ACCDB1

NFSServerArchiveData

DBPRD1

DBVM DBVM

DBPRD2

DBPRD3

DBDG1

DBDG2

DBDG3

Folder perDB

Folder perDB

Folder perDB

DEVDB2

UATDB2 TESTDB2

ACCDB2

DEVDB3

UATDB3 TESTDB3

ACCDB3

Testing

Network tests• Simple ping + TCP tests

Local disk performance - 2 tests:• 1 storage account + 34 disks• 34 storage accounts + 1 disk

Layout:• 1 volume group • XFS striped lvm


Testing

Network tests - normal network


0.75

2.77

Testing

Network tests - ACN


0.5

1,41

Testing

Test case 1 : Windows 2 windows


0100200300400500600700

1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829303132333435363738

MB/sec

Counter SampleNumber

SMBServerWriteMB/sec

0

200

400

600

800

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

MB/sec

Counter SampleCounter

NetworkAdapterReceivedMB/sec

Testing

Test case 2a : Linux 2 windows - Non Accelerated Networking


Testing

Test case 2b : Linux 2 windows - Accelerated Networking


0

50

100

150

200

250

300

350

400

450

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

MB/sec

Counter SampleNumber

SMBServerWriteMB/sec

NoAcceleratedNetworkingforLinux WithAcceleratedNetworkingforLinux

Testing


VM1

VM2

VM3AZUREWest-Europe

PRODUCTION/STANDBY

DEVDB1

UATDB1 TESTDB1

REFRESH(Non-Prod)

Snapshots

DEDICATEDBACKUP

AZUREWest-Europe

Dataguard

ACCDB1

NFSServerArchiveData

DBPRD1

DBVM DBVM

DBPRD2

DBPRD3

DBDG1

DBDG2

DBDG3

Folder perDB

Folder perDB

Folder perDB

DEVDB2

UATDB2 TESTDB2

ACCDB2

DEVDB3

UATDB3 TESTDB3

ACCDB3

Testing


0

50

100

150

200

250

300

350

400

450

500

1 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 52 56 60 64 68 72 76 80 88 96 112 128

PhysicalI/O- ReadMB/s

upd0% upd10%

upd20%

upd30%

Testing


0

100

200

300

400

500

600

1 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 52 56 60 64 68 72 76 80 88 96 112 128

PhysicalI/O- ReadMB/s

upd0% upd10%

upd20%

upd30%

Testing

Backups• Prod db

• Protected by Dataguard• Bi-weekly full backup

• Week 1 -> Big PRDDB1 backup • Week 2 -> Medium PRDDB2 / Small PRDDB3 backup

• Retention• 1st week is on premium storage available via share• Week 2 -> 4: cfr azure offload


Testing

- 3 backup servers in AS + ACN, but in Separate AS than the db servers- 1 share per server- rman - 4 channels per share - tuning parameters needed: - sql 'alter system set "_backup_disk_bufcnt"=20 scope=memory'; - sql 'alter system set "_backup_disk_bufsz"=1048576 scope=memory'; - sql 'alter system set "_backup_file_bufcnt"=20 scope=memory'; - sql 'alter system set "_backup_file_bufsz"=1048576 scope=memory'; - CONFIGURE DEVICE TYPE DISK PARALLELISM 4; - CONFIGURE CHANNEL DEVICE TYPE DISK MAXPIECESIZE 20G MAXOPENFILES=40;

- Maximum backup speed achieved was the vm limit (450MB/s) , so throttling will be needed.- Restore speed is about 400MB/s per oracle server


Expect the unexpected

Managed disks





• Managed disks are now globally available. • Simple management• Snapshots possible

—> Recommended!

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/managed-disks-overview


https://docs.microsoft.com/en-us/azure/virtual-machines/windows/managed-disks-overview

Provisioning


Treat them as cattle, not as pets

Provisioning


FullprovisioningScenario RefreshScenario

VirtualMachine Exists: sourceVM+db(standby)New: targetVMnon-production

Exists:sourceVM+db (standby)Exists:targetVM

Storage Exists:sourcestorageNew:targetstoragenewVM

Exists:sourcestorageExists:target storagetargetVM

Data Exists:source dbNew:targetdb

Exists:sourcedbNew:targetdb

1.STDBYDBCOMPONENTS2.ACCDBCOMPONENTS3.ACCDBPRE-REFRESHSCRIPTS4.SHUTDOWNACCDB5.UNMOUNTACCFILESYSTEMS6.SHUTDOWNSTDBYDB7.SNAPSHOTSTDBYDISKS8.DETACHACCAZUREDISKS9.SNAPSHOT->AZUREDISK10.ATTTACHACCAZUREDISKS11.MOUNTACCFILESYSTEMS12.STARTU/RECOVERACCDB13.ACCDBPOST-REFRESHSCRIPTS

Provisioning


VGACCDB1

D1

1

ORA

CLESTAN

DBY

1

D1

1

D1

2 2

D1

2

sda

1DGDB1

DB

D1

40

D1

40 40

2

sdb

sdal

......

40

ORACLEACCEPTAN

CE1

sda

1 ACCDB1

DB2

sdb

sdal

...

40

VGDGDB1

Initial Dataload


Initial data load

• AZ copy

• express route

• No snowball / truck /… (yet)


And we needed it …

Initial data load

So we created it ourselves


Initial data load

• Lowest Lead Time for production cut-over Oracle Databases

• Phased approach to spread workload• Tested and validated through dry-runs to assure

production move• One-off approach : good enough solution/cost


Initial data load

Our solution? Dataguard!


Initial data load

• Full Production backup – Physical TransferWhen ? Migration Day – 6What ? 50 TB of three databasesHow ? Disk copy and Courier Transfer

• Incremental Production backup – Physical TransferWhen ? Migration Day-2What ? 15 TB of incremental changes of 5 days How ? Disk copy and Courier Transfer

• Incremental Production backup – Network TransferWhen ? Migration DayWhat ? 6 TB of incremental changes of 5 daysHow ? ExpressRoute Network Transfer


Initial data load


Azure


It’s all about avoiding limits

Initial data load

LIMITS:• Azure BLOBS have a throughput limit of 60 Mbytes / sec• Azure Storage Accounts : # : 250 per subscription• Max ingress per storage account (EU) : 5 Gbps (GRS/ZRS), 10 Gbps (LRS)• Max egress per storage account (EU) : 10 Gbps (GRS/ZRS), 15 Gbps (LRS)• Standard Storage Account : 20.000 IOPS, 800MB/sec• Premium Storage disk P20 : 2300 IOPS, 150 MB/sec• Premium Storage disk P30 : 5000 IOPS, 200 MB/sec• VM DS15_V2 : Disk : 64.000 IOPS, 640 MB/sec , Network : 15 Gbps• VM GS5 : 80.000 IOPS / 2000 MB/sec, , Network : 32 Gbps


Lessons learned

• Get people out of their comfortzones!—> old procedures, lots of manual work.

• People get uncomfortable by change.—> No decent service catalog so every exceptional situation is executed.

• Trust but verify the facts


Lessons learned

• Check! What?! Double check

• Fail early and correct.

• Involve the vendor

• Ask for previews, document and Test them!


Lessons learned

Expect the unexpected• project manager got fired• oracle changed the licensing rules

!!! But we delivered !!!


Would I do it again?



Mail: [email protected] Twitter: @vanpupi

mailto:[email protected]

Documents

Journey of a bi stack to the cloud UKOUG2017vanpupi.stepi.net/.../2017/12/Journey_of_a_bi_stack_to_the_cloud_20… · The journey of a bi-stack to the cloud VM1 VM2 VM3 AZURE West-Europe