60
1 SQLBits XIV - The Big Backup Theory I know, I know, who schedules a 75 minute session on Backups!?! In your shoes my reaction would be

SQLBits XIV - The Big Backup Theory

Embed Size (px)

Citation preview

Page 1: SQLBits XIV - The Big Backup Theory

1 SQLBits XIV - The Big Backup Theory

I know, I know, who schedules a 75 minute session on

Backups!?!

In your shoes my reaction would be…

Page 2: SQLBits XIV - The Big Backup Theory

2 SQLBits XIV - The Big Backup Theory

“Oh what fresh hell is this?”

The Dumpling Paradox S1 E7

Page 3: SQLBits XIV - The Big Backup Theory

The Big Backup Theory

Richard Douglas

Page 4: SQLBits XIV - The Big Backup Theory

6 SQLBits XIV - The Big Backup Theory

The goal of this

presentation

The Cooper-Hofstadter Polarization - S1 E9

Page 5: SQLBits XIV - The Big Backup Theory

7 SQLBits XIV - The Big Backup Theory

I do work for a vendor, there may be some

subliminal messages in this presentation

Page 6: SQLBits XIV - The Big Backup Theory

8 SQLBits XIV - The Big Backup Theory

Don’t have a “Backup Strategy”

Page 7: SQLBits XIV - The Big Backup Theory

9 SQLBits XIV - The Big Backup Theory

Agenda

• Why do we need backups?

• Backups available in SQL Server

• Understanding the Transaction Log

• Backup Strategy vs. Restore Strategy

• How to make your backups faster

• How to make your restores faster

Page 8: SQLBits XIV - The Big Backup Theory

10 SQLBits XIV - The Big Backup Theory

Have a “Restore Strategy”

Page 9: SQLBits XIV - The Big Backup Theory

11 SQLBits XIV - The Big Backup Theory

If we haven’t met yet…

• Richard Douglas

• Sales Engineer / SQL Evangelist

• SQL Server MCITPro

• SQL User Group Leader

• Blog: http://SQL.RichardDouglas.co.uk

• Twitter: @SQLRich

• Email: [email protected]

• Slides: http://www.Slideshare.net/SQLRich

Page 10: SQLBits XIV - The Big Backup Theory

12 SQLBits XIV - The Big Backup Theory

Seriously, think about it.

A “Restore Strategy” makes so much sense!

Page 11: SQLBits XIV - The Big Backup Theory

13 SQLBits XIV - The Big Backup Theory

This time Sheldon is wrong…Backups are Essential!

The Cooper-Hofstadter Polarization S1 E9

Page 12: SQLBits XIV - The Big Backup Theory

14 SQLBits XIV - The Big Backup Theory

Why do we need backups?

• Protect valuable data

• Provides the potential to return to a “point in time”

• Confusion between Disaster Recovery and High Availability

• Backup misconceptionsImage backups are sufficient

Snapshots are sufficient

Differential backups are incremental

System databases only need to be backed up once a week

Only need to keep my last backup

Buffer pool usage

Page 13: SQLBits XIV - The Big Backup Theory

15 SQLBits XIV - The Big Backup Theory

Have a “Restore Strategy”

Page 14: SQLBits XIV - The Big Backup Theory

16 SQLBits XIV - The Big Backup Theory

Recovery Models• Simple

• Full

• Bulk Logged

Data

FileLog File

SQL

Database

Automated administration Manual Administration

Note: Recovery model will be defaulted to the one in the Model database

Page 15: SQLBits XIV - The Big Backup Theory

17 SQLBits XIV - The Big Backup Theory

Backup Types• Full

• Differential

• Log

• File / Filegroup

• Tail of the log

• Database Snapshot

Page 16: SQLBits XIV - The Big Backup Theory

18 SQLBits XIV - The Big Backup Theory

Have a “Restore Strategy”

Page 17: SQLBits XIV - The Big Backup Theory

20 SQLBits XIV - The Big Backup Theory

Transaction Log Internals

• The Transaction Log is segmented into pieces calledVirtual Log Files (VLF’s)

• There will always be at least one “active” VLF

• The number of VLF’s is determined by the size at creation / growth time.– Under 64MB 4VLF’s

– >=64 and > 1GB 8 VLF’s

– >= 1GB 16 VLF’s

• But wait, there’s more. VLF’s aren’t the smallest segment of the Transaction Log …

Active VLF

Inactive VLF

Inactive VLF

Inactive VLF

Inactive VLF

Page 18: SQLBits XIV - The Big Backup Theory

21 SQLBits XIV - The Big Backup Theory

Log Blocks

• Each VLF will have a number of “Log Blocks” inside it

• These blocks vary in size:– Anywhere between 512bytes and 60KB

• A new block is created when:– The data inside the block hits the 60KB limit

– A commit or abort occurs within a transaction

• Each log block contains a parity bit of 64 or 128 which is flipped every time it is used.

• Inside each log block we have…

Page 19: SQLBits XIV - The Big Backup Theory

22 SQLBits XIV - The Big Backup Theory

Explaining the transaction log is a bit like nesting dolls

Page 20: SQLBits XIV - The Big Backup Theory

23 SQLBits XIV - The Big Backup Theory

Log Records

• Log records are identified by Log Sequence Numbers or LSN’s– VLF Sequence nbr : Log block nbr : Log record nbr

• Each data page contains the LSN of the last record that made changes to that page.

• Log records are written in the order they occur.i.e. SQL Server will not wait for a transaction to finish before writing away another concurrent transaction.

• Log records often store before and after images of changes.

• They contain a list of pages that this change has affected.

• Log records are stored in the buffer pool (log cache) until flushed to diskLook for WAITLOG wait type.

This will become clear when we

cover the restore process

Page 21: SQLBits XIV - The Big Backup Theory

24 SQLBits XIV - The Big Backup Theory

Checkpoint

• Data pages are marked as “dirty” when they are changed.

• The checkpoint process writes all dirty pages to disk and marks them as cleared.

• Checkpoint is an optimisation process, it reduces the amount of writes to disk to keep data up to date.

Page 22: SQLBits XIV - The Big Backup Theory

26 SQLBits XIV - The Big Backup Theory

Have a “Restore Strategy”

Page 23: SQLBits XIV - The Big Backup Theory

27 SQLBits XIV - The Big Backup Theory

Restore Strategy main considerations

• Time taken to restore dataaka Recovery Time Objective (RTO)

• Data loss acceptabilityaka Recovery Point Objective (RPO)

• Cost associated with system unavailable

Backup Strategy main considerations

• Time taken to perform a backup

• Cost associated with longer maintenance window

• Cost associated with storing backup files

Backup Strategy vs Restore Strategy

Page 24: SQLBits XIV - The Big Backup Theory

28 SQLBits XIV - The Big Backup Theory

Showing you why you need to have a “Restore

Strategy”

Page 25: SQLBits XIV - The Big Backup Theory

29 SQLBits XIV - The Big Backup Theory

Full backup00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

M

T

W

T

F

S

S

Page 26: SQLBits XIV - The Big Backup Theory

30 SQLBits XIV - The Big Backup Theory

Full backup00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

M

T

W

T

F

S

S

Page 27: SQLBits XIV - The Big Backup Theory

31 SQLBits XIV - The Big Backup Theory

Full backup00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

M

T

W

T

F

S

S

Page 28: SQLBits XIV - The Big Backup Theory

32 SQLBits XIV - The Big Backup Theory

Differential00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

M

T

W

T

F

S

S

Page 29: SQLBits XIV - The Big Backup Theory

33 SQLBits XIV - The Big Backup Theory

Differential00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

M

T

W

T

F

S

S

Page 30: SQLBits XIV - The Big Backup Theory

34 SQLBits XIV - The Big Backup Theory

Differential00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

M

T

W

T

F

S

S

Page 31: SQLBits XIV - The Big Backup Theory

35 SQLBits XIV - The Big Backup Theory

Transaction Log Backup00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

M

T

W

T

F

S

S

Page 32: SQLBits XIV - The Big Backup Theory

36 SQLBits XIV - The Big Backup Theory

Transaction Log Backup00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

00

01

02

03

04

05

06

07

08

09

10

11

12

13

14

15

16

17

18

19

20

21

22

23

M

T

W

T

F

S

S

Page 33: SQLBits XIV - The Big Backup Theory

37 SQLBits XIV - The Big Backup Theory

Figured out it’s safer to have a

“Restore Strategy” yet?

Page 34: SQLBits XIV - The Big Backup Theory

38 SQLBits XIV - The Big Backup Theory

Optimizing backups

Server Architecture

Database Architecture

Backup Options

Page 35: SQLBits XIV - The Big Backup Theory

39 SQLBits XIV - The Big Backup Theory

Have a “Restore Strategy”

Page 36: SQLBits XIV - The Big Backup Theory

40 SQLBits XIV - The Big Backup Theory

Server Architecture

Page 37: SQLBits XIV - The Big Backup Theory

41 SQLBits XIV - The Big Backup Theory

What computer do you have? And please don’t say “a white one”.

The Peanut Reaction – S1 E16

Page 38: SQLBits XIV - The Big Backup Theory

42 SQLBits XIV - The Big Backup Theory

• IO Subsystem– RAID Levels

– Disk Partition Alignment

– Dedicated drives

• Network cards– Add more NICs

– Use different subnets

– Use different switches

Optimizing backups – Server Architecture

The DBA team start negotiations with the Infrastructure team

http://www.flickr.com/photos/jakesutton/859193891/in/photostream

Page 39: SQLBits XIV - The Big Backup Theory

43 SQLBits XIV - The Big Backup Theory

Have a “Restore Strategy”

Page 40: SQLBits XIV - The Big Backup Theory

44 SQLBits XIV - The Big Backup Theory

Optimizing backups – Database Architecture• Table design

– See Natural Born Killers webcast November 2012 (http://dell.to/15RwEtz)

– Blogged at http://bit.ly/GHuUrH

– Space that a table uses is not completely released after you use a DELETE statement to delete data from the table in SQL Server: http://support.microsoft.com/kb/913399

• Index Design – See indexing webcasts October 2012 (http://dell.to/19urm8s)

• File placement

• Compression

Page 41: SQLBits XIV - The Big Backup Theory

45 SQLBits XIV - The Big Backup Theory

End of subliminal messages

Page 42: SQLBits XIV - The Big Backup Theory

46 SQLBits XIV - The Big Backup Theory

Actually, I wouldn’t mind

a caramel macchiato

Page 43: SQLBits XIV - The Big Backup Theory

47 SQLBits XIV - The Big Backup Theory

Optimizing Backups

• Compression

• BufferCount

• MaxTransferSize

• BlockSize

• Multiple backup devices

Other Backup Options of note

• Init / NoInit

• Format / NoFormat

• Copy Only

• Checksum

• Continue After Error

Backup Options

For a full list of all the backup options refer to Books Online (BOL)http://msdn.microsoft.com/en-us/library/ms186865.aspx

Page 44: SQLBits XIV - The Big Backup Theory

48 SQLBits XIV - The Big Backup Theory

Backup Options Continued

• Memory used is accessed from outside of the bufferpool

• You can calculate memory usage:Buffercount * MaxTransferSizeSetting these values too high may cause out of memory conditions

Buffercount calculations for Full / Diff backups:

• 2000– (NumberofBackupDevices * GetSuggestedIoDepth)

+ NumberofBackupDevices + (DatabaseDeviceCount)

• 2005 and above– (NumberofBackupDevices * GetSuggestedIoDepth)

+ NumberofBackupDevices + (2*DatabaseDeviceCount)

Page 45: SQLBits XIV - The Big Backup Theory

49 SQLBits XIV - The Big Backup Theory

Calculating BufferCount

• GetSuggestedIODepth– Disk = 3

– Tape = 3

– VDI = 1 default and 4 max based on device spanning needs http://bit.ly/1i69GUG

• DatabaseDeviceCount = The number distinct drives the database files reside on.

• I think the formula for 2008+ is wrong. Waiting for CSS to return my email.The following seems to work:

(NumberofBackupDevices * GetSuggestedIoDepth [4]) + NumberofBackupDevices + (2*DatabaseDeviceCount)

Page 46: SQLBits XIV - The Big Backup Theory

50 SQLBits XIV - The Big Backup Theory

Demo

Page 48: SQLBits XIV - The Big Backup Theory

52 SQLBits XIV - The Big Backup Theory

Restore Problems• Broken Transaction Log Chain

– Clearing the transaction log (using WITH TRUNCATE ONLY or WITH NO LOG)

– Changing the database to SIMPLE recovery model (checkpoint will clear log))

– Reverting from a database snapshot

– Deleting a log file in the chain

– Backups to NUL devices

Page 49: SQLBits XIV - The Big Backup Theory

53 SQLBits XIV - The Big Backup Theory

Backups are like

Schrodinger’s cat

Page 50: SQLBits XIV - The Big Backup Theory

54 SQLBits XIV - The Big Backup Theory

Restore Myths• Downgrading a backup

• No need to test backups

• Restore time will be the same as the backup time

• Table restores

Page 51: SQLBits XIV - The Big Backup Theory

55 SQLBits XIV - The Big Backup Theory

The Restore ProcessA restore operation can have up to four phases:

1. File creation and initialization

2. Data and/or log copy

3. REDO phase of recovery

4. UNDO phase of recovery

Page 52: SQLBits XIV - The Big Backup Theory

56 SQLBits XIV - The Big Backup Theory

The Restore Process File creation and initialization

• In the beginning there was file creation.

• Then there was the zeroing of the files.

• If the database already exists overwrite using the “WITH REPLACE” clause,do not drop and restore.

• Use “Instant File Initialization” (IFI) if possible.Remember the log file still needs to be zeroed out.

• There is a security risk.

• TF 3004 will provide more information about IFI

• TF 1806 will disable IFI

Page 53: SQLBits XIV - The Big Backup Theory

57 SQLBits XIV - The Big Backup Theory

The Restore ProcessData and/or log copy

• In this stage SQL Server:

– Reads the data from the backup file(s)

– Writes the data to the database files

• Having multiple data files and backup files can improve I/O throughput.

• Using native or third party compression reduces the amount of data to be read (and potentially copied across the network).

Compression vs DeDuplication

• Whilst deduplication does reduce storage costs it will be rehydrated before copying back across the network and so will not aid the speed of a restore.

Page 54: SQLBits XIV - The Big Backup Theory

58 SQLBits XIV - The Big Backup Theory

The Restore ProcessREDO phase of recovery (1)

• The database being recovered has it’s boot page (page 9) checked.

• This page will provide the details of the most recent checkpoint.

• Checkpoint log records are examined to see if there is a list of uncommitted transactions

• Recovery will start at the oldest uncommitted transaction for the most recent checkpoint

Page 55: SQLBits XIV - The Big Backup Theory

59 SQLBits XIV - The Big Backup Theory

The Restore ProcessREDO phase of recovery (2)

• Each log record is tied to an LSN and contains page numbers of records it has altered

• Each data page’s page header contains the last LSN which made a change to it

• The redo phase compares the two LSN’s

• The redo phase uses something akin to the following formula:

IF PageHeaderLSN < LogRecordLSN

BEGIN

Replay Log Record –Record has not been updated

END

ELSE

DoNothing –Record has this change and possibly some others

Page 56: SQLBits XIV - The Big Backup Theory

60 SQLBits XIV - The Big Backup Theory

The Restore ProcessREDO phase of recovery (3)

Finding the end of the log

There are two ways:

1. If the log has not been wrapped then it checks for the occurrence of 512 byte block full of zeros.

2. If it has been wrapped then it reads through all the active VLF’s in the order of their sequence number and reads through each log block until it finds one with the wrong flipped parity bit

Page 57: SQLBits XIV - The Big Backup Theory

61 SQLBits XIV - The Big Backup Theory

The Restore ProcessUNDO phase of recovery

• The redo phase keeps track of uncommitted transactions. If there are uncommitted transactions during recovery then the undo phase kicks in

• As mentioned in the redo section the data page page header contains an LSN. This is compared to the Log Record LSN

IF PageHeaderLSN >= LogRecordLSN

BEGIN

Undo Log Record –Record requires rollback

Create a “compensation” log record

Update PageHeader LSN with compensation log record LSN

END

Page 58: SQLBits XIV - The Big Backup Theory

63 SQLBits XIV - The Big Backup Theory

Further ways to optimize restores• Using “Instant File Initialization”

• Using the correct restore strategy

• Server Architecture

• Database Architecture

• Backup compression

• Restore options– Buffercount

– MaxTransferSize

Page 59: SQLBits XIV - The Big Backup Theory

65 Global MarketingThe Big Backup Theory

Any questions?

Page 60: SQLBits XIV - The Big Backup Theory

Thank you for attendingRichard [email protected]@SQLRich