Geek Sync: Field Medic’s Guide to Database Mirroring

Field Medic’s Guide to Database Mirroring August 18, 2015

Hi, I’m Kendra Little

What you’ll learn today

Rescue scenarios

Why mirroring matters Special setup for mirrors

Monitoring mirroring 3

12

4

Field medics need to know…

Rescue scenarios

Why mirroring matters Special setup for mirrors

Monitoring mirroring 3

12

4

Mirroring: Your friend since 2008 Became supported in SQL Server 2005 SP1

An important performance enhancement called ‘Log Stream Compression’ was added in SQL Server 2008

Works in Standard Edition (synchronous only) •  “Basic Availability Groups” are a planned feature for SQL

Server 2016 that we believe will be in Standard Edition, but…

(continued) Mirroring is much easier to manage than Availability Groups

• No Windows Failover Cluster required • Much simpler to document • Management can all be done using SSMS/TSQL

Mirroring has perks, too! • Automatic page repair • Doesn’t require a domain / can be used with multiple domains

(using certificates)

Mirroring is ‘deprecated’ … but this doesn’t mean it’s dead

It will be removed in a future version of SQL Server • What version that is hasn’t been disclosed. Features may stay

‘deprecated’ for many many versions without being removed.

Fully supported in SQL Server 2012 and 2014

Well known, widely documented, established feature • Supported by many vendors • Typically stable

Not ending a TSQL Statement in a “;”

Also deprecated…

Mirroring modes Choose between safety and performance

“High Safety” mode • Synchronous • Automatic failover is optional • Witness – use a witness if you’re using automatic failover. (Can

be Express Edition)

“High performance” mode • Asynchronous • Enterprise Edition only • No automatic failover • Do not use a witness with this mode

An example: Async

Database Mirroring

Problems mirroring doesn’t solve

Databases are mirrored individually, and fail over individually • No distributed transaction support (not in AGs prior to 2016,

feature is planned at date of this presentation) •  If you want them to fail over as a group, you have to write

custom code and make sure it happens

You must keep SQL Agent Jobs and logins in sync and manage where they run

Mirroring lots of databases on one instance can be difficult due to thread requirements (Also true for AGs)

Mirroring is the right choice when…

One mirror is enough

Individual database failover is enough (not a group)

You aren’t ready to manage the complexity of a cluster

And you may not have many other choices if you’re using SQL Server 2008 or SQL Server 2008R2

Demo: automatic page repair in action

This database is mirrored

High safety with no witness – no automatic failover

The view from the GUI

We change to high performance mode

Let’s corrupt a page

dbo.CriticalTable has only one data page

The table has a single row where the ‘val’ column is quoting a famous sloth meme

We’re going to corrupt that page and see if mirroring can fix the issue

Here’s the offset of the page in the data file

We have to shut down the SQL Instance this time to edit the page

Open the data file with a hex editor

CTRL+G, set offset as dec relative to begin

Type over the values to corrupt

Save and click YES

Start the SQL Server back up

Try reading the page

Try again

Verify what happened

The fine print Automatic page repair only kicked in when corruption was detected

• Maximum of 100 rows per database • Rows in sys.dm_db_mirroring_auto_page_repair are cleared

on instance restart

You still need • alerts and monitoring •  to set page_verification to CHECKSUM for the database •  to run CHECKDB (to detect corruption where reads haven’t

been happening) •  to investigate root cause when this happens

But isn’t this much better than repairing the page yourself?

This feature is also in AlwaysOn Availability Groups. Mirrors are just easier to manage!

Field medics need to know…

Rescue scenarios

Why mirroring matters Special setup for mirrors

Monitoring mirroring 3

12

4

Setting up logins You can script these all out from SQL Server Management Studio

If you allow SQL Authentication, this can be complex though • You have to look up the passwords yourself, and you don’t

want to save them in a script in plain text • The SID of the login matters so that the login will map up with

the database users, and the SID doesn’t script automatically

Robert Davis has published a free script that helps you automate this and solve both of these problems: http://www.sqlsoldier.com/wp/sqlserver/transferring-logins-to-a-database-mirror

‘Orphaned users’ If you do have a login where the SID wasn’t transferred over to the mirror, the users associated with the login will be ‘orphans’ – and they won’t work

You can detect and fix this without recreating the login if it happens using sp_change_users_login:

http://msdn.microsoft.com/en-us/library/ms174378(SQL.90).aspx

CLR Assemblies: Trustworthy? CLR Assemblies are part of the database and are mirrored

But they may not work when you first fail over without extra steps • The TRUSTWORTHY database property isn’t automatically

restored with the database. If you need this, it must be enabled the first time you fail over

• Not all CLR assemblies require this-- you can see your current setting in the sys.databases DMV

Setting up SQL Server Agent jobs

The easy way to script out all jobs in SSMS:

View: Object Explorer Details

In the Object Explorer Details pane, you highlight all jobs and script them out in a single script

But beware… •  If job steps use log files, make sure the same path exists

where you’re installing it • Make sure you enable/disable the right jobs on the mirror

Making jobs work auto-magically after failover

You need custom code to help your jobs out: •  If an automatic failover occurs, the right jobs need to run

automatically, too • You need to accomplish this without having jobs fail all the time

if they’re running against the mirror– because that will destroy monitoring for failed jobs

Each job needs to check “Am I the principal?” and act accordingly

This goes for your application jobs as well as maintenance

Special setup: Certificates If you’re using certificates that are signed in the master database, these need to be set up on the mirror also

Mirroring doesn’t replace the need to securely back up those certificates in a place where you’re sure you won’t use them!

Field medics need to know…

Rescue scenarios

Why mirroring matters Special setup for mirrors

Monitoring mirroring 3

12

4

Split brain in High Safety Mode Split brain is as bad as it sounds: everything shuts down

Let’s say you’re using high safety mode with a principal, a mirror, and a witness

•  If the principal can’t see either the mirror or the witness, it can’t know if the database is online at the witness. It will take the database offline.

•  If the mirror can’t see the principal or the witness, it freaks out too, and puts the database offline.

The database is offline everywhere.

We’ve lost quorum!

Constant communication between the principal, mirror, and witness is critical

Split brain The database

could be online somewhere else. I’d better shut it

down!

Derp derp derp derp

The database could be online

somewhere else. I’d better shut it

down!

Suspended mirror ‘Suspended’ status can happen after things go wrong

• Forced service • SQL Server bugs in some versions (Examples:

http://support.microsoft.com/kb/2403218 , http://support.microsoft.com/kb/975681 )

‘Suspended’ means that data is not being transferred from the principal to the mirror

• The log will grow on the principal • You’re no longer protected by mirroring

To fix, on either partner run: ALTER DATABASE YourMirrorDB SET PARTNER RESUME;

Suspended mirror I’ll save all the changes in my

transaction logs for when you come back!

What’s going on?

Disconnected witness We’ve seen this if there have been network interruptions to the witness, or problems in DNS

Mirroring may keep running (depending on whether or not the communication between the primary and the mirror is interrupted)

If you have a disconnected witness, you may fix this by restarting the endpoints on the witness

To prevent unplanned failovers, I prefer to: • Switch to high performance mode and remove the witness • Repair the witness • Add the witness back and switch to high safety mode

Disconnected witness I sure hope there’s no network

interruptions…

Me too!

Derp derp derp derp

Forcing up the mirror Example: you’re using asynchronous mirroring (“high performance” mode). This doesn’t have automatic failover.

The principal server is lost!

You need to bring things up on the mirror. To do this, you run: •  ALTER DATABASE YourMirrorDB SET PARTNER

FORCE_SERVICE_ALLOW_DATA_LOSS;

Discussion: • What do you do to identify how much data was lost? • What other steps need to be done immediately if you have to

do this?

Mirroring and Virtualization Technically, it’s supported by VMware:

“AlwaysOn Availability Groups and database mirroring can be used in combination with vSphere vMotion, vSphere HA, and DRS to maximize SQL Server availability.”

… SQL Server on VMware Availability and Recovery Options

http://www.vmware.com/files/pdf/solutions/SQL_Server_on_VMware-Availability_and_Recovery_Options.pdf

But it’s not all that simple Communication is critical

Network interruptions can cause big problems: • Unplanned failovers in mirroring • Loss of quorum

Even planned movements of virtual machines can cause packet loss and communication drops (and we’re not even talking about storage migration) – this can cause failovers

And you need to make sure that all components in mirroring are never on the same host

Your biggest takeaways Monitoring is critical

•  If you have automatic failover, you need to make sure the witness and mirror are healthy

•  If you ignore failures, the likelihood of split brain taking everything down goes up

Placement of the witness is important •  If you’re using multiple datacenters, the witness should be in

the primary datacenter

Field medics need to know…

Rescue scenarios

Why mirroring matters Special setup for mirrors

Monitoring mirroring 3

12

4

sys.database_mirroring This DMV helps you quickly see which databases are mirrored

For each database that’s mirrored, you can see: • Mirroring mode • Who’s the partner and what their state is • Who’s the witness and what their state is • Endpoint address

Database Mirroring Monitor Job This is created automatically if you use SQL Server Management Studio to set up mirroring

• Runs every one minute by default • Calls the built in sp_dbmmonitorupdate procedure (which you

can use in your own code, if you want)

This job allows you to see the current mirroring status by using: • The Database Mirroring Monitor tool • The sp_dbmmonitorresults procedure

Launching the monitor Right click on the database name in SSMS

Snoop around under ‘Tasks’

Performance counters Per database mirrored

Tracking and baselining these is helpful in case you need to troubleshoot performance problems

Some highlights from the full list: •  Log Send Queue KB •  Redo Queue KB •  Pages sent / sec •  Redo bytes / sec •  Transaction Delay

http://msdn.microsoft.com/en-us/library/ms189931.aspx

Thresholds and alerts • Set in the Database Mirroring Monitor (hard to see) • These are set per database mirrored • You must set up alerts (with Database Mail and operators) to

get notified, and it’s a lot to tackle if you mirror multiple dbs

It’s better to buy monitoring You can build your own, but:

• Complex and time consuming to set up alerts for state changes and performance alerts yourself

• You’ll get lots of duplicate alerts (especially if you’re mirroring multiple databases)

• The Database Mirroring Monitor shows a limited amount of history… troubleshooting past failovers can be tricky

The medic’s advice for mirroring

Become a mirroring pro Create a test environment

• Run through setup multiple times • Practice planned failover, switching modes

Implement and test monitoring • Notification of state changes (pausing, failovers) • Create a delay by performing multiple index rebuilds on the

primary – do you get notifications of latency? • Shut the witness down – do you get a notification

Make it fail • Practice forcing service on a mirror • Practice recovering afterwards

Here’s How …

Idera Can Help You.

www.idera.com