VMWare Support for NovaBACKUP - NovaStor · Backups/Restores can be done via NovaBACKUP DC (HiBack) or NovaBACKUP. Features Possibility to backup VMs by connecting to ESX, ESXi or

VMware support for NovaBACKUP DC &

NovaBACKUP

Author: HFR

Creation date: 29.01.2010

Last modified: 28.01.2011

Version: 8.5 (Matching HiBack version)

VMware support for NovaBACKUP DC & NovaBACKUP ................................................................... 1

Description ......................................................................................................................................... 1

Features .............................................................................................................................................. 2

Supported environments and conditions .............................................................................................. 2

Main program: vmware.exe ................................................................................................................ 2

Command: vmware list ................................................................................................................... 3

Command: vmware ping ................................................................................................................. 3

Command: vmware createdba ......................................................................................................... 4

Command: vmware backup ............................................................................................................. 5

Command: vmware restore ............................................................................................................. 7

Files.................................................................................................................................................. 10

[vmware.ini] - VMware INI file .................................................................................................... 10

[<vmhost>.ini] - VM host INI file ................................................................................................. 11

[<vmware>.dba] - vmware.exe DBA file ...................................................................................... 11

Settings priorities .......................................................................................................................... 11

[config.vddk] - VDDK Configuration File .................................................................................... 12

[vmware.log] - VMWARE Logfile ............................................................................................... 12

Upgrading VM hardware version and enabling Changed Block Tracking ......................................... 13

Notes .................................................................................................................................................... 15

Research ................................................................................................. Error! Bookmark not defined.

DUMP Virtual Disk ............................................................................ Error! Bookmark not defined.

Change ID handling ............................................................................ Error! Bookmark not defined.

File System Support ............................................................................ Error! Bookmark not defined.

Limitations on Mounting Virtual Disks ............................................... Error! Bookmark not defined.

Limitations on Changed Block Tracking ............................................. Error! Bookmark not defined.

Enabling Changed Block Tracking ..................................................... Error! Bookmark not defined.

To enable Changed Block Tracking with the VMware vSphere Client .......... Error! Bookmark not

defined. To enable Changed Block Tracking and back up with the VMware vSphere API . Error! Bookmark

not defined. Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for

holding off I/O in the frozen virtual machine. ..................................... Error! Bookmark not defined.

DiskChangeInfo QueryChangedDiskAreas ............................................. Error! Bookmark not defined.

DiskChangeInfo QueryChangedDiskAreas ............................................. Error! Bookmark not defined.

Description

The main goal to be reached with the VMware support is to allow to do a backup and restore of whole

Virtual Machines (VMs) in its CURRENT state.

Current state means: Older snapshots aren't available after a restore of the VM anymore.

Backups/Restores can be done via NovaBACKUP DC (HiBack) or NovaBACKUP.

Features

Possibility to backup VMs by connecting to ESX, ESXi or vCenter servers.

Backups can be done

- full

- full - used blocks only (needs vSphere)

- incremental (needs vSphere)

- differential (needs vSphere)

The backup can be done even the VM is powered on. In this case VMs, running an appropriate Windows

Version and using actual VMware tools, may be backed up using VSS to allow to have application

consistent backups for VSS enabled apps vs. crash consistent backups.

HiBack will read/write the data of the virtual disks via pipe. NovaBACKUP may use a temp directory to

buffer the data.

vmware.exe, together with its DLLs and configuration files, can be used stand alone from any Windows

machine which can reach an ESX, ESXi or vCenter server via network.

Data for backup and restore can be retrieved either via network (SSL encryption is possible), via SAN, if

the storage is connected to the machine which is running vmware.exe, or via hot add, if vmware.exe is

running in a VM and is backing up another VM.

Supported environments and conditions

ESX, ESXi, vCenter servers up to version 4.1

Backup client (vmware.exe) running on Windows systems (even in a VM)

Works with any ESXi that is not standalone (Foundation and above) (VDDK restriction)

Incremental and differential backup is using the Changed Block Tracking (CBT) feature of vSphere

CBT support has to be enabled manually per VM

CBT will slow down the VM a measurable amount

CBT needs a VM which hardware version is at least version 7.

The hardware version of a VM can be upgraded by the user.

Hot add (hotadd) backup is only possible, if the VM to be backed up only has only virtual SCSI disks

Main program: vmware.exe

This is the main program.

It handles the

- inventory,

- backup,

- restore and

- dba file creation

of and for the VMs.

[vmware.exe] logs information about the backup process in its own [vmware.log] log file.

The [vmware.log] log file is necessary to allow to support incremental and differential backups.

In case of success, vmware.exe returns with exit code 0.

In case of an error the exit code is not 0.

Error messages will be written to STDERR.

Command: vmware list

This creates a list of the VMs on the given host. The list is returned on STDOUT.

vmware list

--host <hostname/IP>

[--user <username>]

[--password <password>]

[--locale <en/de>]

[--filter powerState:poweredOn|poweredOff]

--host: VMware host

--user: user name to access the VMware host

Can be set via <hostname>.ini file: Option 'VMUSER'

--password: password to access the VMware host

Can be set via <hostname>.ini file: Option 'VMPASSWORD'

--locale: Select the locale of messages returned from the VMware host

EN: English

DE: German

--filter: Restricts the list to contain only VMs which where powered

on or powered off

Returns on STDOUT: <Name of VM 1>

...

<Name of VM n>

Command: vmware ping

This allows to do a connection test.

In case of success, vmware.exe returns with exit code 0.

vmware ping


[--user <username>]


[--locale <en/de>]

--host: VMware host






EN: English

DE: German

Command: vmware createdba

This allows to create dba files for all VMs existing on a specific VM host.

For all VMs dba files for full, incremental and differential backups will be created in the following

directory structure:

<vmware.exe home directory>\<VM host name>\<name of the VM>\

<VM host name>_<name of the VM>_full.dba

<VM host name>_<name of the VM>_inc.dba

<VM host name>_<name of the VM>_diff.dba

vmware createdba


[--user <username>]


[--locale <en/de>]

[--filter powerState:poweredOn|poweredOff]

[--dbadir <directory>]

[--indirectfile <indirect file name>]

--host: VMware host






EN: English

DE: German

--filter: Restricts the list to contain only VMs which where powered

on or powered off

--dbadir: alternative directory to create the dba files in

--indirectfile: if given, this file will contain the references to the

created dba file names

Command: vmware backup

At first a short description what has to be done in case of a backup.

Full backup:

1. Create a snapshot of the VM addressed by its name

2. Retrieve and store the VM specifications [vm_spec.xml]

3. For all virtual disks:

If available: Get CBT information using ChangeID "*"

Get blocks of the virtual disk (using CBT information if applicable)

Store Blocks

If applicable:

Store ChangeID into [vmware.log]

Store CBT information

4. Remove snapshot

Incremental/Differential backup:

1. Create a snapshot of the VM addressed by its name

2. Retrieve and store the VM specifications [vm_spec.xml]

3. For all virtual disks using its [hw-key]:

Store new ChangeID into [vmware.log]

Get CBT information

Get blocks of the virtual disk using ChangeID from previous backup

retrieved from [vmware.log]

Store CBT information

Store Blocks

4. Remove snapshot

vmware backup

[--host <hostname/IP>]

[--user <username>]


[--backuptype <F/D/I>]

[--compression <RLC/ZLIB1-ZLIB9>]

[--dbafile <dba file name>]

[--locale <en/de>]

[--logfile <vmware-logfile>]

[--tempdir <tempdir>]

[--transportmodes <[san][:][nbdssl][:][nbd]>]

[--stamp <unix time stamp/$<Environment variable>>]

[--vmname <VMNAME>]

--file <pipe|file>

--host: VMware host

Can be set via DBA file: Option 'HI_VMHOST'





--backuptype: optional

F[ULL]: full backup [default]

D[IFF]: differential backup

I[NC]: incremental backup

Can be set via DBA file: Option 'HI_TYPE'

--compression: Compress the backup

RLC: Use run length compression

ZLIB1: Use zlib compression (lowest compression but fastest)

...

ZLIB9: Use zlib compression (best compression but slowest)

Can be set via [vmware.ini] file: Option 'compress'

--dbafile: Path and name of the <vmware.dba> file to read

Via the <vmware.dba> file most of the necessary parameters can be

specified


EN: English

DE: German

--logfile: Path and file name of the <log file name> file

Can be used to specify an alternative path and name of the log

file to be used.

If not given vmware.exe creates and uses the file [vmware.log] in

the directory of vmware.exe.

--tempdir: optional temporary directory

Uses default OS temp directory if not given

--transportmodes: Select the transport modes which should be tried to

retrieve the data from the VM virtual disks. Multiple transport

modes has to be separated by a colon.

Example: --transportmodes=san:nbdssl:nbd

Example: --transportmodes=nbd

san: Virtual disks where accessed via SAN if possible

ndbssl: Virtual disks where accessed over network (with SSL)

nbd: Virtual disks where accessed over network

Default: san:nbdssl:nbd

--stamp: Unix time stamp to be added to the <vmware.log>

Example: --stamp 001278660412

Example: --stamp $STAMP

In the second example, the stamp will be extracted from an

environment variable named 'STAMP'.

--vmname: Name of the VM to be backed up

Can be set via DBA file: Option 'HI_VMNAME'

--file : Backup destination file name or pipe name

If the destination is a file, vmware.exe will check if a file

with the given name already exists. In this case, vmware.exe

won't continue.

Command: vmware restore

The restore of a VM can be done either forwards or backwards. If you work forwards, the restore might

write some sectors more than once. If you work backwards, you must keep track of which sectors were

restored so as to avoid restoring them again from older data.

But: It is always necessary to create the VM using the latest stored VM specifications. Such implies a

little bit to use the backwards method.

Restore of a full backup:

1. Retrieve latest full backup

2. Create a VM using the restored VM specifications [vm_spec.xml]

3. Disable CBT if applicable

4. Create a snapshot


Restore virtual disk (using CBT information if applicable) into the snapshot

6. Enable CBT if applicable

7. Revert to snapshot

8. Remove snapshot

Restore VMs having differential or incremental backups

1. Retrieve latest incremental or differential backup

2. Create a VM using the restored VM specifications [vm_spec.xml]

3. Disable CBT if applicable

4. Create a snapshot


Restore virtual disk (using CBT information if applicable) into the snapshot

Cache information about already restored blocks

6. For all earlier backups (beginning with the newest)

6a. Restore incremental or full backup

6b. For all virtual disks:

Restore virtual disk (using CBT information if applicable) into the snapshot. Use

cache to ignore blocks already restored to the virtual disk.

Update cache information about already restored blocks

7. Enable CBT if applicable

8. Revert to snapshot

9. Remove snapshot

6a implies, that it is necessary to restore more then one backup. It has to be decided how such

should be handled. Either the user has to restore each incremental or the corresponding full

backup by hand or all restores will be done automatically.

In case of doing it automatically, an interaction with the data movers HiBack and NovaBACKUP has to

be designed to allow to retrieve different backups via the external module [vmware.exe].

Problem with working backwards:

Due to the need to cache the already restored blocks it is necessary to keep the cache until the whole VM

has been restored.

vmware restore

[--host <hostname/IP>]

[--user <username>]


[--compression <RLC/ZLIB1-ZLIB9>]

[--dbafile <dba file name>]

[--locale <en/de>]

[--tempdir <tempdir>]

[--transportmodes <[san][:][nbdssl][:][nbd]>]

[--stamp <unix time stamp/$<Environment variable>>]

[--vmname <VMNAME>]

--file <pipe|file>

--host: VMware host

Can be set via DBA file: Option 'HI_VMHOST'





--compression: Specify the compression mode used for the backup

RLC: Use run length compression

ZLIB1: Use zlib compression (lowest compression but fastest)

...

ZLIB9: Use zlib compression (best compression but slowest)

Can be set via [vmware.ini] file: Option 'compress'

ATTENTION: The compression mode chosen here has to be the same as

used for the backup. If not, the restore will fail with an

appropriate message.

--dbafile: Path and name of the <vmware.dba> file to read

Via the <vmware.dba> file most of the necessary parameters can be

specified


EN: English

DE: German

--tempdir: optional temporary directory

Uses default OS temp directory if not given

--transportmodes: Select the transport modes which should be tried to

retrieve the data from the VM virtual disks. Multiple transport

modes has to be separated by a colon.

Example: --transportmodes=san:hotadd:nbdssl:nbd

Example: --transportmodes=nbd

san: Virtual disks where accessed via SAN if possible

hotadd: If vmware.exe runs inside of a VM the backup can be made

without using the network. Only VMs with virtual SCSI disks

can be backed up. The overhead on the ESX(i) hosts is

higher.

ndbssl: Virtual disks where accessed over network (with SSL)

nbd: Virtual disks where accessed over network

Default: san:hotadd:nbdssl:nbd

--stamp: Unix time stamp to be added to the VM name in case the a VM

with the given name already exists

Example: --stamp 001278660412

Example: --stamp $STAMP

In the second example, the stamp will be extracted from an

environment variable named 'STAMP'.

--vmname: (New) name for the restored VM

Can be set via DBA file: Option 'HI_VMNAME'

--file : Source file name or pipe name

Name of the backup file (or pipe) the restore should be done

from.

ATTENTION

The restore process has to be done in the following described manner:

1. Start with the latest (newest) incremental backup (if exists)

2. Continue with the next older incremental backup until you reach either a differential or a full

backup (if applicable)

3. Restore the latest (newest) differential backup (if exists)

4. Restore the full backup.

Sample scenarios to restore the VM to the latest backed up state

Scenario 1 - Full, differential and incremental backups

1. Full backup (oldest)

2. Incremental backup


4. Differential backup


6. Incremental backup (newest)

-> Restore 6., 5., 4. then 1.

Scenario 2 - Full and incremental backups




4. Incremental backup (newest)

-> Restore 4., 3., 2. then 1.

Scenario 3 - Full and differential backups


2. Differential backup

3. Differential backup (newest)

-> Restore 3. then 1.

Scenario 4 - Only full backups


2. Full backup

3. Full backup (newest)

-> Restore only 3. (or an older version)

vmware.exe will create a temporary directory for the VM in the temp directory. This will be removed,

when restoring a full backup. It contains essential information about already restored blocks when

restoring incremental and differential backups so the contents of this directory has to be kept till the full

backup has been restored.

Files

The following files can (or where) be used by vmware.exe.

[vmware.ini] - VMware INI file

This file can either be used to set some default settings.

This ini file resists in the same directory as vmware.exe.

Values starting with --, i.e., <--vmname>, corresponds to the vmware(.exe) command line parameters.

[vmware.ini]

[server=<--host>]

[user=<--user>]

[password=<--password>]

[debug=<0|1>]

[buffers=2]

[compress=<--compression>]

Description:

compression: Allowed compression settings

-1|rlc : RLC

0 : No compression

1-9|zl1-9|zlib1-zlib9 : ZLib1 (lowest) to ZLib9 (best compression)

debug: 1 enables further debug output shown on STDOUT.

buffers: Number of internal buffers. Has to be at least 2.

[<vmhost>.ini] - VM host INI file

This file contains the login information for the VM host.

This ini file resists in the same directory as vmware.exe.


[<vmhost>.ini]

VMUSER=<--user>

VMPASSWORD=<--password>

VMTRANSPORTMODES=<--transportmodes>

[VMHOST=<VM host name or IP>]

Description:

VMTRANSPORTMODES: See option <--transportmodes> in the 'backup' section.

Remarks: VMHOST is optional and is not used by vmware.exe.

[<vmware>.dba] - vmware.exe DBA file

This file can be used together with HiBack. There is no need to use it with NovaBACKUP.


HI_AGENT=vmware

HI_AGPATH=<path to VMWARE>

HI_VMHOST=<--host>

HI_VMNAME=<--vmname>

HI_TYPE=<--backuptype>

HI_COMM=<--file>

HI_VMTRANSPORTMODES=<--transportmodes>

HI_VMLOG=<--logfile>

Description: HI_AGENT: Agent to be called from HiBack. vmware[.exe] in this case.

HI_AGPATH: Full path to the agent

Other options: See description in section 'backup'

Allowed Values: HI_TYPE: FULL/DIFF/INC

Settings priorities

Settings given via ini files and/or command line will be used with the following priorities.

1. Command Line: Highest priority

2. DBA file

3. <vmware.ini> file

4. <VM host>.ini file: Lowest priority

[config.vddk] - VDDK Configuration File

Needed for the actual API to allow to support retrieving blocks of virtual Disk via SAN attached VM

storages.

vmware.exe will use this file. [config.vddk]

tmpDirectory = "<TempDirectoryForLogging>"

vixDiskLib.transport.LogLevel = "<logLevel>" (default is 6)

vixDiskLib.nfc.RequestTimeoutMs = "<NfcRequestTimeoutInMilliseconds>"

vixDiskLib.nfc.ReadTimeoutMs = "<NfcReadTimeoutInMilliseconds>"

vixDiskLib.nfc.WriteTimeoutMs = "<NfcWriteTimeoutInMilliseconds>"

[config.vddk]

vixDiskLib.transport.LogLevel = "6"

<logLevel>

0: Quiet

1: Panic

2: Error

3: Warning

4: Informational

5: Verbose

6: Trivia

[vmware.log] - VMWARE Logfile

The path and name of this ini file may be given via command line option 'logfile'.

It contains information about the stored virtual machines. [vmware.exe] will read this to retrieve the

appropriate ChangeIDs for a VM. Stamp VM name Backup Type Change IDs

F/I/D

111 Test F

3001|52 c9 02 62 91 68 18 3a-00 f0 4f c4 00 96 7b 47/144:

2000|52 2e 60 6d d6 be 89 17-5f a1 80 d1 8c 95 cb 70/146:

2001|52 07 a6 c0 2f d1 ad b0-89 04 72 e7 3f 4d d2 ee/178

112 Test 2 F 2000|52 07 a6 c0 2f d1 ad b0-89 04 72 e7 3f 4d d2 ee/31

113 Test 2 I 2000|52 07 a6 c0 2f d1 ad b0-89 04 72 e7 3f 4d d2 ee/32:

3000|52 2e 60 6d d6 be 89 17-5f a1 80 d1 8c 95 cb 70/1

114 Test3 F 2000|52 07 a6 c0 2f d1 ad b0-89 04 72 e7 3f 4d d2 ee/33:

3000|52 2e 60 6d d6 be 89 17-5f a1 80 d1 8c 95 cb 70/2

115 Test3 I 2000|52 07 a6 c0 2f d1 ad b0-89 04 72 e7 3f 4d d2 ee/34:

3000|52 2e 60 6d d6 be 89 17-5f a1 80 d1 8c 95 cb 70/1

116 Test3 D

2000|52 07 a6 c0 2f d1 ad b0-89 04 72 e7 3f 4d d2 ee/35:

3000|52 2e 60 6d d6 be 89 17-5f a1 80 d1 8c 95 cb 70/2:

3001|52 c9 02 62 91 68 18 3a-00 f0 4f c4 00 96 7b 47/1

The value 2000, 3000, 3001 in this example corresponds to the [HW-KEY] of the virtual disk.

The value 52 c9 02 62 91 68 18 3a-00 f0 4f c4 00 96 7b 47/144, e.g., is a change ID.

[vmware.log] - Stored as comma separated file 111,Test,F,"3001|52 c9 02 62 91 68 18 3a-00 f0 4f c4 00 96 7b 47/144:2000|52 2e 60 6d d6 be 89 17-5f a1

80 d1 8c 95 cb 70/146:2001|52 07 a6 c0 2f d1 ad b0-89 04 72 e7 3f 4d d2 ee/178"

112,"Test 2",F,"2000|52 07 a6 c0 2f d1 ad b0-89 04 72 e7 3f 4d d2 ee/31"

2000|52 07 a6 c0 2f d1 ad b0-89 04 72 e7 3f 4d d2 ee/32:

3000|52 2e 60 6d d6 be 89 17-5f a1 80 d1 8c 95 cb 70/1"

113,"Test 2",I,"2000|52 07 a6 c0 2f d1 ad b0-89 04 72 e7 3f 4d d2 ee/32:

3000|52 2e 60 6d d6 be 89 17-5f a1 80 d1 8c 95 cb 70/1"

114,Test3,F,"2000|52 07 a6 c0 2f d1 ad b0-89 04 72 e7 3f 4d d2 ee/33:

3000|52 2e 60 6d d6 be 89 17-5f a1 80 d1 8c 95 cb 70/2"

115,Test3,I,"2000|52 07 a6 c0 2f d1 ad b0-89 04 72 e7 3f 4d d2 ee/34:

3000|52 2e 60 6d d6 be 89 17-5f a1 80 d1 8c 95 cb 70/1"

116,Test3,D,"2000|52 07 a6 c0 2f d1 ad b0-89 04 72 e7 3f 4d d2 ee/35:

3000|52 2e 60 6d d6 be 89 17-5f a1 80 d1 8c 95 cb 70/2:

3001|52 c9 02 62 91 68 18 3a-00 f0 4f c4 00 96 7b 47/1"

Full backup A new "F"ull backup entry will be appended to [vmware.log] which contains the new change IDs

necessary for further incremental or differential backups.

Incremental backup 1. The last entry with a backup type "D" or "F" (in that order), which also matches the VM name, will be

searched.

2. The HW keys of the VM virtual disks will than be compared to the HW keys retrieved from

[vmware.log].

3. The change ID, which corresponds to the matching HW key, will than be used to request the

differences occurred in the virtual disks since than.

4. A new "I"ncremental backup entry will be appended to [vmware.log] which contains the new change

IDs necessary for further incremental or differential backups.

Differential backup 1. The last entry with a backup type "F", which also matches the VM name, will be searched.

2. - 4.: Same steps as described in 'Incremental backup'.

Upgrading VM hardware version and enabling Changed Block Tracking

Per default, the tracking of changed blocks (CBT) isn't enabled for a VM as CBT will slow down the

execution of a VM.

1. To enable CBT you have to install the latest VMware tool into the VM.

2. Also you have to update the VM to hardware version 7

3. At least some configuration parameters have to be added to the VM configuration.

The main parameter is "ctkEnabled" with the value "true".

But for each disk the changes should be tracked, CBT has to be enabled separately.

E.g., for the disk device scsi:0:1 you have to add a parameter "scsi0:1.ctkEnabled" with the value "true".

If CBT isn't enabled at all or for a specific device, the device will be backed up in full mode every time.

Incremental and differential backup of such a disk isn't possible as the information about changed blocks

aren't available.

Notes

The BIOS (NVRAM) of the Virtual Machine will NOT be backed up. This implies, that it won't be restored. If the boot order had been changed in the BIOS, the boot order has to be corrected manually after a restore.

Troubleshoot

SAN mode restores may be slower than NBD transport mode

Problem

When attempting restoration of a VMware virtual machine from disk backup in SAN environments, the

restore job slows down with the events in VMware client GUI displaying “clear lazy zero”. VMware

instead of zeroing a VMDK as it is created, sets a lazy zero bit on each block of the VMDK and clears it

later. This results in the performance of restores with SAN transport mode to be slower than in NBD

transport mode.

The message “Clear Lazy Zero” is repeated every second on the vSphere console.

Solution

To resolve this issue, perform restores directly to ESX server instead of vCenter.

Documents

VMWare Support for NovaBACKUP - NovaStor · Backups/Restores can be done via NovaBACKUP DC (HiBack) or NovaBACKUP. Features Possibility to backup VMs by connecting to ESX, ESXi or