1 | P a g e · 5 | P a g e Debugging UDP Protocol problem To trouble shoot the UDP Protocol problem we need to enable the debug as mentioned below C:\Program Files\CA\ARCserve Unified

$Page 1: 1 | P a g e · 5 | P a g e Debugging UDP Protocol problem To trouble shoot the UDP Protocol problem we need to enable the debug as mentioned below C:\Program Files\CA\ARCserve Unified$
1 | P a g e

2 | P a g e

This Documentation, which includes embedded help systems and electronically distributed materials, (hereinafter referred to as the “Documentation”) is for your informational purposes only and is subject to change or withdrawal by CA at any time. This Documentation may not be copied, transferred, reproduced, disclosed, modified or duplicated, in whole or in part, without the prior written consent of CA. This Documentation is confidential and proprietary information of CA and may not be disclosed by you or used for any purpose other than as may be permitted in (i) a separate agreement between you and CA governing your use of the CA software to which the Documentation relates; or (ii) a separate confidentiality agreement between you and CA. Notwithstanding the foregoing, if you are a licensed user of the software product(s) addressed in the Documentation, you may print or otherwise make available a reasonable number of copies of the Documentation for internal use by you and your employees in connection with that software, provided that all CA copyright notices and legends are affixed to each reproduced copy. The right to print or otherwise make available copies of the Documentation is limited to the period during which the applicable license for such software remains in full force and effect. Should the license terminate for any reason, it is your responsibility to certify in writing to CA that all copies and partial copies of the Documentation have been returned to CA or destroyed. TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENTATION “AS IS” WITHOUT WARRANTY OF ANY KIND, INCLUDING WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. IN NO EVENT WILL CA BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY LOSS OR DAMAGE, DIRECT OR INDIRECT, FROM THE USE OF THIS DOCUMENTATION, INCLUDING WITHOUT LIMITATION, LOST PROFITS, LOST INVESTMENT, BUSINESS INTERRUPTION, GOODWILL, OR LOST DATA, EVEN IF CA IS EXPRESSLY ADVISED IN ADVANCE OF THE POSSIBILITY OF SUCH LOSS OR DAMAGE. The use of any software product referenced in the Documentation is governed by the applicable license agreement and such license agreement is not modified in any way by the terms of this notice. The manufacturer of this Documentation is CA. Provided with “Restricted Rights.” Use, duplication or disclosure by the United States Government is subject to the restrictions set forth in FAR Sections 12.212, 52.227-14, and 52.227-19(c)(1) - (2) and DFARS Section 252.227-7014(b)(3), as applicable, or their successors. Copyright © 2013 CA. All rights reserved. All trademarks, trade names, service marks, and logos referenced herein belong to

their respective companies.

3 | P a g e

Table of Contents

UDP Installation Setup log 4

UDP Uninstallation Setup log 4

UDP Update log 4

Debugging UDP Protocol problem 5

Debugging UDP RPSWebServices problem 5

Debugging UDP VCMUPgrade Problem 5

Debugging UDP –WEBUI Problem 5

Debugging UDP Management Console 6

Debugging UDP Agentless issues 7

Debugging UDP Virtual Standby issues 7

ASBU-UDP Lite Integration 8

Debug for UDP replication 9

Debugging UDP Agent 11

Debugging UDP Agent Web Service 13

Debugging UDP agent Web UI 14

Advanced Format – Configuration Switches 15

Data Deduplication Logs 16

Deduplication Format – Pairing Logs between Client and Server 17

Debug Logs related to Data Store 18

BMR Limitations 19

Debugging BMR 19

Reading BMR Logs 21

Boot Failure Issues 22

Debug logs for Installation failure on the UDP Linux Agent Server 26

Logs for Backup/Restore Failures on the UDP Linux Agent Server 26

Logs on the UDP Linux Node (Agentless) 26

Logs for BMR related issues on the UDP Linux Agent Server 27

Logs on the BMR Target Node (Linux) 27

Enabling Debug on Job logs 27

On UDP Linux Agent Server

On UDP Linux Node (Agentless)

Debugging Linux Agent / Agentless communication issues 28

On UDP Linux Agent Server

On UDP Linux Node (Agentless)

Log for UI (Linux) 28

Log for Web Service (Linux) 28

Extracting logs from the *.db files 28

Debug Settings for File Copy 29

4 | P a g e

UDP Installation Setup log

1. Root Location: %windir%\Temp\arcserve\Setup\UDP\Install_yyyy_mm_dd_time

2. Open the file setup.log from the root location

3. Find the last line to get the last installation log dir. The last line looks like the following:

C:\Windows\Temp\arcserve\Setup\UDP\Install_XXXX_XX_XX_XX_XX_XX\

It means all the log files of the last installation are under this folder

4. C:\Windows\Temp\arcserve\Setup\UDP\Install_ XXXX_XX_XX_XX_XX_XX\

Open and read ProdWiz.log in this folder. ProdWiz.log is the log entrance for this time

installation.

Below logs will also help us while investigating setup issues

%windir%\inf\setupapi.dev.log

%windir%\setupact.log

%windir%\windowsupdate.log

%windir%\logs\cbs\cbs.log

From command prompt type ‘wmic qfe >c:\updates.txt’ and get the updates.txt file

UDP Uninstallation Setup log

1. Root Location: %windir%\Temp\arcserve\Setup\UDP\Uninstall\

2. Open the file Uninstall-History.log from the root location

3. Find the last line to get the last installation log dir. The last line looks like the following:

C:\Windows\Temp\ARCserve\Setup\UDP\Uninstall\Uninstall_XXXX_XX_XX_XX_XX_XX\

It means all the log files of the last uninstallation are under this folder

:\Windows\Temp\ARCserve\Setup\UDP\Uninstall\Uninstall_XXXX_XX_XX_XX_XX_XX\

4. Open and read Uninstall.log in this folder. Uninstall.log is the log entrance for this time

uninstallation

UDP Update log

1. Root Location: %windir%\Temp\arcserve\Setup\UDP\Update\

2. Open the root location

3. Order the folders in the root location by time

4. Open the folder with the latest time. The folder name format is like CA_YYYY-MM-DD_hh-

mm-ss, for example: CA_2014-05-21_14-52-09

5. Open the file SPInstaller.log in this folder. SPInstaller.log is the log entrance for this time

Update installation.

5 | P a g e

Debugging UDP Protocol problem

To trouble shoot the UDP Protocol problem we need to enable the debug as mentioned below

C:\Program Files\CA\ARCserve Unified Data Protection\Management\Configuration\

Edit the file name log4j-changeProtocol. Look for the following line

log4j.category.com.ca=INFO, logout and Change the “INFO” to “DEBUG”

Refer to the Log file name - ChangeProtocol.log which is located at the below path

(C:\Program Files\CA\arcserve Unified Data Protection\Management\log

Debugging UDP RPSWebServices problem

To trouble shoot the UDP RPSWebServices problem we need to enable the debug as mentioned

below


Edit the file name log4j-rpswebservice, look for the line

log4j.category.com.ca.arcflash=INFO, logout and Change the “INFO” to “DEBUG”

Add the following line log4j.category.com.ca.arcflash.listener.manager=DEBUG

Refer to the Log file name – RPSWebService.log which is located at the below path

Log file location : (C:\Program Files\CA\arcserve Unified Data Protection\Management\log

Debugging UDP VCMUPgrade Problem


Edit the file name log4j-vcmupgrade. Look for the following line

log4j.category.com.ca=INFO, logout and change “INFO” to “DEBUG”

Refer to the Log file name – vcmupgrade.log which is located at the below path


Debugging UDP –WEBUI Problem


Edit the file name log4j-webui. Look for the following line

log4j.category.com.ca.arcflash=INFO, logout and change the “INFO” to “DEBUG”

Refer to the log file name WebUI.log which is located at the below path


6 | P a g e

Debugging UDP Management Console

1. Using a text editing application, open the following file on the CA ARCserve Central Host-Based VM

Backup server:

C:\Program Files\CA\ARCserve Unified Data Protection\Management\Configuration\ log4j-

ARCAPP.properties

2. In the properties file, locate the following parameter:

log4j.category.com.ca.arcserve.edge=INFO

Change INFO to DEBUG.

Note: You do not need to restart the service.

The log level changes to DEBUG.

(Optional) Modify the following parameters as needed.

log4j.appender.logout.MaxFileSize=5120KB

log4j.appender.logout.MaxBackupIndex=10

Note: The values that are listed in these parameters are the default values.

The following diagram illustrates the location of the properties file and the parameters that you change

to configure the debugging mode.

Logs to Refer:

C:\Program Files\CA\ARCserve Unified Data Protection\Management\logs

ARCAPP.log

tomcat_####.log

7 | P a g e

Debugging UDP Agentless issues

Troubleshooting UDP Host Based Backup issues, you need to enable debug on the UDP Proxy Node.

Set the value for the Debug DWORD under the following registry key to 1.

HKEY_LOCAL_MACHINE\SOFTWARE\CA\ARCserve Unified Data Protection\Engine

DWORD value: “Debug” and set as 1

Logs to Refer(Agentless of VMware Virtual Machines)

C:\Program Files\CA\ARCserve Unified Data Protection\Engine\Logs

HBBU-ESX-20140603-110007-508-1935-11800-W2008R2

Logs to Refer(Agentless of MS Hyper-V Virtual Machines)


HBBU-Hyperv-20140603-134505-257-1940-11268-W2008DC

On the MS Hyper-V Host:

C:\Windows\arcserve UDP Host-Based VM Backup\5.0.1823\Log

Debugging UDP Virtual Standby issues

Troubleshooting UDP Virtual Standby issues, you need to enable debug on the UDP Monitor Node and

the source node. For UDP Agentless as source, enable this debug on the UDP Proxy Node.

If RPS is used as destination for the source, then RPS would server as the convertor and the debug needs

to be enabled on the RPS node.

Set the value for the Debug DWORD under the following registry key to

1.HKEY_LOCAL_MACHINE\SOFTWARE\CA\ARCserve Unified Data Protection\Engine

DWORD value: “Debug” and set as 1

Logs to Refer:

i. Where source is Standalone UDP Agent


VSB_##########

ii. Where source is Agentless Backup


Eg of Log Name -> [email protected]_S0000000270

[email protected]_S0000000270

mailto:[email protected]_S0000000270



8 | P a g e

iii. Where source is RPS


VSB_########

ASBU-UDP Lite Integration

On the UDP Agent Node or UDP Proxy Node or UDP RPS Node, where the ARCserve Backup Client Agent

for Windows is installed, incorporate the following:

HKEY_LOCAL_MACHINE\SOFTWARE\ComputerAssociates\CA ARCServe Backup\ClientAgent\Parameters

Debug set value as 1

TruncLog set value as 0

Logs to Refer:

For Browse issues refer universal agent log,

C:\Program Files\CA\SharedComponents\ARCserve Backup\UniAgent\Univag.log

For backup and restore issues refer the ARCserve Backup Client agent for windows logs

C:\Program Files\CA\ARCserve Backup Client Agent for Windows\LOG

In RPS server side, all D2DNativeFacade.dll related logs will be under

C:\Program Files\CA\arcserve Unified Data Protection\Engine\Logs\D2DNF.log

Other dependent dll logs refer RPS side logs folder and also agent side below folder

C:\Program Files\CA\SharedComponents\ARCserve Backup\UniAgent\Log

9 | P a g e

Debug for UDP replication

RPS replication involves different modules during the replication process which may be

split into three parts

1) Work flow which controls the whole job progress:

a. RPSReplication.exe

b. CID2DPlugin.dll

2) Replicate disk data:

a. AFStor.dll

b. GDDClient.dll

c. GDDServerDll.dll (The data store has to be stopped and started again so that

the GDDServer.exe process restarts

3) Communication between RPS servers:

rps_trc.dll

All of the above modules are located under “C:\Program Files\CA\ARCserve Unified Data

Protection\Engine\BIN”

Registry location to enable Debug:

HKEY_LOCAL_MACHINE\SOFTWARE\CA\ARCserve Unified Data Protection\Engine\debug

Depending on which module seems to be causing the issue debug should be enabled accordingly. The

“Debug” key does not exit by default, we would have to create a key and name it as “Debug” and then

create a Dword value with the name of the module and set the log level

Debug Levels:

0 Error

1 Warning

2 Information (default log level)

3 Detailed

4 Debug log

10 | P a g e

Below screenshot has debug enabled for RPSReplication.exe, AFStor.dll and GDDClient.dll

Log location:

The logs are located under the logs folder in the engine install directory.

C:\Program Files\CA\arcserve Unified Data Protection\Engine\Logs

Log Name:

Source 1. RPSRep(Out)*.log

Target 1. RPSRep(In)*.log

2. GDDServer*.log

Jumpstart logs

1. RPSJumpstart(Out)*.log 2. RPSJumpstart(In)*.log 3. GDDServer*.log

Naming convention:

RPSRep (In/Out)-<yyyymmdd>-<hhmmss>-<ms>-<JobId>-<node name>.log

(i.e) RPSRep (In/Out)-<20140526>-<013038>-<350>-<85>-<machine name>.log

11 | P a g e

Debugging UDP Agent

Log level of each module can be controlled by registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\CA\ARCserve Unified Data

Protection\Engine\Debug\ModuleName:DWORD

How to read debug log?

In arcserve UDP r5.0, the debug log was re-organized by job or process. For each particular job, the

debug log was consolidated in one single file. For example: \

Backup Job: backup-<yyyymmdd>-<hhmmss>-<ms>-<job id>-<process id>.log

Restore Job: restore-<yyyymmdd>-<hhmmss>-<ms>-<job id>-<process id>.log

On RPS Server, the merge log is named as “Merge(<Agent Node Name>-<RPS Server Name>-<Data

Store Name>-R)-<yyyymmdd-hhmmss-ms>-<Job id>.log”

On UDP Agent, the merge log is named as “Merge(<Agent Node Name>-D)-<yyyymmdd-hhmmss-ms>-

<Job id>.log”

MountingDriver.Log

Each piece of debug log was appended with the module name and build number.

How to adjust debug log level using registry?

Note: - To reset debug log level by registry, you need to restart the service and the specified application.

The debug log level is 2 (Information) by default, but you can reset log level by registry or by command

line. More higher level means more output of debug log.

Go to HKEY_LOCAL_MACHINE\SOFTWARE\CA\ARCserve Unified Data Protection\Engine\Debug.

Create a DWORD with the name of the module and set the required Debug Level as the value.

12 | P a g e

Check the module name in the Backup log and enable debug as per the requirement.

If you want to enable more debugging on AFBackend.exe, then create a DWORD for that

module.

How to adjust debug log level using command line?

Note: - When you set debug log level by cmd line you don’t have to restart application, it works

immediately.

cmdutil.exe /log /setlevel:<the new level> /pid:<the process id>

Type “cmdutil.exe /log /?” to see the details.

Activity log and Job History

• In r5.0, the activity log and job history are saved in SQLite DB instead of “Activity.log”

The SQLite database file are under “..\Engine\Database\”

13 | P a g e

D2D.ActivityLog- the database to save activity log of UDP agent

D2D.JobHistory - the database to save Job history of UDP agent

• RPS.ActivityLog - the database to save activity log of RPS server ( only exists on RPS server )

RPS.JobHistory - the database to save job history of RPS server ( only exists on RPS Server )

• A set of command lines to create / dump / shrink SQLite database…. Type “cmdutil.exe /db /?”

for the details.

Debugging arcserve UDP agent Web Service

Go to C:\Program Files\CA\arcserve Unified Data Protection\Engine\Configuration\

Open the file "log4j-webservice.properties" and find line "log4j.category.com.ca.arcflash=INFO,

logout".

To enable debug, change the value “INFO” to “DEBUG” and save the file.

Other available values are,

https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html

o ALL The ALL has the lowest possible rank and is intended to turn on all logging.

o DEBUG designates fine-grained informational events that are most useful to debug an application.

o ERROR designates error events that might still allow the application to continue running.

o FATAL designates very severe error events that will presumably lead the application to abort.

o INFO designates informational messages that highlight the progress of the application at coarse-

grained level.

o OFF The OFF has the highest possible rank and is intended to turn off logging.

o TRACE designates finer-grained informational events than the DEBUG

o WARN The WARN level designates potentially harmful situations.

Restart the arcserve UDP Web Service.












14 | P a g e

Logs Location:


webservice.log

Tomcat Logs Location:

C:\Program Files\CA\arcserve Unified Data Protection\Engine\TOMCAT\logs

tomcat_<process id>.log

Debugging UDP agent Web UI

Enable debugging for arcserve UDP agent Web Service also.

Go to C:\Program Files\CA\arcserve Unified Data Protection\Engine\Configuration\

Open the file "log4j-webui.properties" and find line "log4j.category.com.ca.arcflash=INFO,

logout".

To enable debug, change the value “INFO” to “DEBUG” and save the file.

Other available values are,


o ALL The ALL has the lowest possible rank and is intended to turn on all logging.

o DEBUG designates fine-grained informational events that are most useful to debug an application.

o ERROR designates error events that might still allow the application to continue running.

o FATAL designates very severe error events that will presumably lead the application to abort.

o INFO designates informational messages that highlight the progress of the application at coarse-

grained level.

o OFF The OFF has the highest possible rank and is intended to turn off logging.

o TRACE designates finer-grained informational events than the DEBUG

o WARN The WARN level designates potentially harmful situations.

Restart the arcserve UDP Web Service.

Logs Location:


webservice.log

WebUI.log

Tomcat Logs Location:

C:\Program Files\CA\arcserve Unified Data Protection\Engine\TOMCAT\logs












15 | P a g e

Advanced Format – Configuration Switches

Configure AFStor.ini for standalone agent

DisablePreRead=0

[“0” means pre-reading is enabled. In restore-like job, pre-reading could optimize

performance. The basic idea of pre-reading is that it could proactively read out more

data and cache in memory based on the statistics and analysis on previous reading

callings of the job. E.g. if the job is to sequentially read data, it will read big chunk to

cache in memory in advance. If it is to randomly read data, it will intelligently terminate

pre-reading.]

DisableReservingSpaceForMerge=0

[“0” means enable reserving space for merge job. When merge job happens, it requires

some temporary swapping space to rebuild data slice file. In case that disk is full, merge

job could not continue if it could not find the temporary swapping space. To solve this, it

could reserve the space when specify the backup destination.

The size of reserved space is identical to slice size (i.e. SliceSizeInMB).

- If multiple UDP agents back up to the same shared folder, they will share the same reserved space.

- If backup destination is detected to support sparse, it won’t reserve space. The reason is that it could leverage spare mode to continue merge job in case that disk is full.]

EnableSparse=1

[“1” means that merge job will leverage sparse mode to free disk space.

If destination supports spare and the obsolete data percentage in data file is smaller

than 50%, merge job will leverage sparse mode to free disk space. With this way, freeing

disk space is very quick.

In case of “EnableSparse = 0”, merge job will rebuild data file to free disk space only

when obsolete data percentage is bigger than 50%. It will never leverage sparse mode

to free disk space.]

SliceMergeThresholdInPercentage=50

[“50” means that merge job will trigger data rebuilding to free disk space when obsolete

data percentage in slice file is bigger than 50%. When obsolete data percentage is

smaller than 50%, it only leverages spare mode to quickly free disk space.]

SliceSizeInMB=512

[“512” means that data slice size is 512MB. It is corresponding to the source data size.

16 | P a g e

After compression, the size of every data slice file might be different, which depend on

the compression percentage.]

Configure registry for non-deduplication datastore(Need to restart datastore to make it take

effect)

Data Deduplication Logs

Log Level

Control log level follows the same way as other modules:

Logs of GDDClient.dll

Like other modules, GDDClient log has been consolidated into log file of every job.

For example:

Backup-20140518-220009-504-6-384.log:

[2014/05/18 22:01:27:613 02 00384 11024 ] LoadGDDClient successfully.

{AFBackend.exe::AFStor.dll(1893.0)}

[2014/05/18 22:01:27:613 01 00384 11024 ] GDDInitServer, No WAN

IP,{AFBackend.exe::GDDClient.dll(1893.0)}

[2014/05/18 22:01:27:738 01 00384 11024 ] calling g_pfnLoadCfgFile()...

{AFBackend.exe::GDDClient.dll(1893.0)}

[2014/05/18 22:01:27:738 01 00384 11024 ] calling g_pfnInitlizeTransport()...


17 | P a g e

Logs of GDDServerDll.dll

Each role of GDDServerDll has a separated log file named after:

GDDServer-[DatastoreName]-[RoleType].log

Deduplication Format – Pairing Logs between Client and Server

Each role serves multiple clients and prints logs for all of them. So the key step for trouble shooting is to

filter out corresponding server logs for specific client (job). Take a backup job and hash role for example:

Find logs of establishing connection in job log and locate its GUID

Backup-20140518-220009-504-6-384.log:

[2014/05/18 22:01:28:957 01 00384 11024 ] connecting Hash server [0], [lijle01-2k8-1

:5002] {AFBackend.exe::GDDClient.dll(1893.0)}

[2014/05/18 22:01:28:957 01 00384 11024 ] trying to connect [155.35.64.80:5002]


[2014/05/18 22:01:28:957 00 00384 11024 ] SSL: caller settings, enable:0, flag:1

{AFBackend.exe::rps_trc.dll(1893.0)}

[2014/05/18 22:01:28:957 00 00384 11024 ] Connection GUID:fcaa3f9f-964b-45e7-

a919-5795b72b02b4 {AFBackend.exe::rps_trc.dll(1893.0)}

[2014/05/18 22:01:28:957 00 00384 11024 ] socket=2176, connected to

0x9b234050@5002. local: 0x9b234050 @ 56118 {AFBackend.exe::rps_trc.dll(1893.0)}

[2014/05/18 22:01:29:801 02 00384 11024 ] CClientCmdMngr::Init:Connect to [lijle01-

2k8-1:5002] successfully. {AFBackend.exe::GDDClient.dll(1893.0)}

Search corresponding logs on server roles based on connection GUID

GDDServer-1893-GDD-HashRole.log:

[2014/05/18 22:01:28:973 00 09300 09576 ] session=20, new session

{GDDServer.exe::rps_trc.dll(1893.0)}

[2014/05/18 22:01:28:973 00 09300 09576 ] socket=1548, GUID:fcaa3f9f-964b-45e7-a919-

5795b72b02b4 {GDDServer.exe::rps_trc.dll(1893.0)}

[2014/05/18 22:01:30:988 01 09300 09704 ] ConID:113fbcbd, Backup job ID[6], Client

Name[lijle01-2k8-1], Client Time[2014-5-18 22:1:30]

{GDDServer.exe::GDDServerDll.dll(1893.0)}

[2014/05/18 22:01:40:832 01 09300 09704 ] ConID:113fbcbd, Hash Time: search [0.04],

insert [0.00], flush [4.94] Seconds. {GDDServer.exe::GDDServerDll.dll(1893.0)}

[2014/05/18 22:01:42:598 00 09300 07080 ] socket=1548, closed by peer gracely when

18 | P a g e

receiving data. rc=0 {GDDServer.exe::rps_trc.dll(1893.0)}

[2014/05/18 22:01:42:895 02 09300 10156 ] ConID:113fbcbd, session 20 has been reset or

closed gracefully. from 155.35.64.80 [2], [Closed_by_peer]

{GDDServer.exe::GDDServerDll.dll(1893.0)}

Important Information:-

1. GUID can be used to find the log when role accepts the connection.

2. A server side session number 20 is assigned. It can be used to track how and when the

connection is closed.

3. ConID 113fbcbd is assigned and can be used to filter activities on this connection.

4. The first log on this connection provides critical info like job type, job ID and client name

for you to further cope with client side logs. Client time is quite useful when system

times are different between client and server.

Debug Logs related to Data Store

• File Name:

DataStoreService.log

RPSWebService.log

GDDServer_<DSName>_DataRole.log

GDDServer_<DSName>_IndexRole.log

GDDServer_<DSName>_HashRole.log

Log Path: <UDP_HOME>\Engine\Logs

Debug setting to enable DataStoreService.Log in debug

In the registry path: HKEY_LOCAL_MACHINE\SOFTWARE\CA\ARCserve Unified Data

Protection\Engine\debug

Add DWORD value “DataStoreInstService.exe”.

Available Debug values are:

0 Error

1 Warning

2 Information (default log level)

3 Detailed

4 Debug log

19 | P a g e

Data Store Threshold Settings

Modify Threshold Settings

https://support.ca.com/cadocs/0/CA%20ARCserve%20Unified%20Data%20Protection%20Version%205

%200%2000-ENU/Bookshelf_Files/HTML/Solutions%20Guide/caudp_mod_ds_threshold.htm

Fix the Data Store Threshold Warning and Error Messages

https://support.ca.com/cadocs/0/CA%20ARCserve%20Unified%20Data%20Protection%20Version%205

%200%2000-ENU/Bookshelf_Files/HTML/Solutions%20Guide/caudp_fix_ds_threshold_msg.htm

BMR Limitations:

DO NOT support restore multiple dynamic disks

DO NOT support restore from “Current State” of VSB VMs converted from HBBU Backups

DO NOT suggest to directly use shared folder for BMR when the backup destination is on

datastore.

Debugging BMR

• DO NOT collect BMR logs just when data restore is completed, instead you should collect logs at

the last reboot screen.

• Once the BMR Restore is completed we will get the below screen. Do not collect the logs @

this point.

https://support.ca.com/cadocs/0/CA%20ARCserve%20Unified%20Data%20Protection%20Version%205%200%2000-ENU/Bookshelf_Files/HTML/Solutions%20Guide/caudp_mod_ds_threshold.htm

https://support.ca.com/cadocs/0/CA%20ARCserve%20Unified%20Data%20Protection%20Version%205%200%2000-ENU/Bookshelf_Files/HTML/Solutions%20Guide/caudp_mod_ds_threshold.htm

https://support.ca.com/cadocs/0/CA%20ARCserve%20Unified%20Data%20Protection%20Version%205%200%2000-ENU/Bookshelf_Files/HTML/Solutions%20Guide/caudp_fix_ds_threshold_msg.htm

https://support.ca.com/cadocs/0/CA%20ARCserve%20Unified%20Data%20Protection%20Version%205%200%2000-ENU/Bookshelf_Files/HTML/Solutions%20Guide/caudp_fix_ds_threshold_msg.htm

20 | P a g e

• After data restore completed, BMR still do lots of things to make machine bootable. This

information is critical for looking into boot failure issues. We can collect the logs after getting to

the below screen.

• If missed to collect BMR logs, you can also boot machine with BMR image find logs under

“[C:\]Windows\Temp\BMRLog_0000]”

21 | P a g e

Reading BMR Logs

Search for the “-!-”sign from file “DrMain.log”, the search result list the most critical information /

working flow of BMR. That would help to locate BMR issue quickly.

22 | P a g e

Boot Failure Issues

Black Screen Error

Normally black screen error occurs before loading winload.exe. Possibly caused by…

Boot from wrong disk ( user error )

No active partition

Active partition corrupted

BCD/Bootmgr corrupted

Incorrect MBR (Mast Boot Record) or VBR (Volume Boot Record)

Troubleshooting:

• Confirm BIOS settings to see if boot disk is correct.

Use utility “diskpart.exe” to check if system partition is active.

Run “chkdsk” for system partition to see if errors reported.

Use “bcdedit.exe” to verify if BCD is correct.

Use BMR internal command “!export” to export the MBR (the first sector of disk) and VBR (

23 | P a g e

the first sector of volume) for further investigation.

Use diskpart.exe to set / view active partition

Use BCDEdit.exe to set / view BCD settings

24 | P a g e

If the BCD file is incorrect, the “device”, “osdevice” is “unknown” (Refer the above screen shot)

Data corruption issue

Normally show BSOD error 0xC000021a, 0xC0000225 ….

0xC000021a: This means that an error has occurred in a critical user-mode subsystem.

0xC0000225: Windows failed to load because of a critical system driver is missing, or corrupt.

This kind of issue is because some of windows critical files got corrupted. The data corruption might

occur during backup, merge, replication, RHA Sync, BMR restore, bad network adapter ( like E1000 )…..

Troubleshooting: • Run “chkdsk” for boot volume (C:\) to see if error reported. • Run BMR internal command “!FileValidate” to catch the possible corrupted files. • If this is a VM, change network adapter and try again.

!FileValidate

Type !FileValidate to catch possible corrupted files

25 | P a g e

• This command will catch possible corrupted files, collect system dump, event log… Hence please

redirect the output to a remote shared folder.

• It is using embed check sum to validate file and works well for windows 2008 and above system,

might not be effective for windows 2003.

• It might catch out lots of corrupted files, Just pay more attention on files under

\Windows\System32 ( the .dll, .exe, .sys)

BSOD Error 0x0000007B

The 7B error is always a nightmare for us. It means Windows operating system has lost access to the

system partition during startup.

Possibly caused by

The disk driver ( SCSI, RAID, Fiber Chanel) does not work for target device.

The particular version of this driver does not work on target system.

The driver exits but device is missing on target system.

Other 3rd divers might also cause this error, such as “sptd.sys” for virtual CD/DVD driver.

Troubleshooting:

Difficult to investigate this kind of issue, have to do case by case.

If user have same kind of server with same version of Windows installed, that would be helpful.

We can compare the installed drivers to find root cause.

Environment Error

Some of boot failure issue might also caused by environment error, such as…

The Hypervisor ( ESX Server, Hyper-V, Xen…) does not support the specified OS.

Incorrect VM settings. For example, created a Windows 2008 VM but actually restore a

Windows 2003 to it.

Boot fail, but directly run into windows recovery mode.

26 | P a g e

The possible reasons are…

The BCD is incorrect, it Windows cannot find winload.exe / winload.efi

The winload.exe is corrupted or missing.

AD server boot failure

Sometimes AD server boot fail after BMR, this is because some of AD files corrupted. In case of this, you

can boot into AD Recovery Mode to restore AD firstly.

NOTE:

In r5.0, if any of AD, System or WMI write is in bad status, the backup job will fail.

Debug logs for Installation failure on the UDP Linux Agent Server

/tmp/d2dsetup-yyyy-mm-dd_hh:mm:ss.log /tmp/d2dsysinfo.log

Logs for Backup/Restore Failures on the UDP Linux Agent Server

/opt/CA/d2dserver/data/ActivityLog.db /opt/CA/d2dserver/data/JobHistory.db /opt/CA/d2dserver/data/ARCserveLinuxD2D.db /opt/CA/d2dserver/logs/

Logs on the UDP Linux Node (Agentless)

1) UDP logs:

/tmp/logs/

2) System related logs:

lspci –n > /$path/lspci.log cat /var/log/message > /$path/message.log dmesg > /$path/dmesg.log ls /dev/* > /$path/dev.log sfdisk –l > /$path/sfdisk.log parted -l > /$path/parted.log uname –a > /$path/uname.log cat /etc/issue > /$path/issue.log cat /boot/grub/menu.lst > /$path/menu.lst

where $path denotes any absolute path.

27 | P a g e

Logs for BMR related issues on the UDP Linux Agent Server

/opt/CA/d2dserver/data/ActivityLog.db /opt/CA/d2dserver/data/JobHistory.db /opt/CA/d2dserver/data/ARCserveLinuxD2D.db /opt/CA/d2dserver/logs

Logs on the BMR Target Node (Linux)

Do not reboot the server at end of BMR and login into the Live CD (Hit “Enter” key to enter the shell prompt.

1) UDP logs:

/tmp/logs/

2) System related logs:

lspci –n > /$path/lspci.log cat /var/log/message > /$path/message.log dmesg > /$path/dmesg.log ls /dev/* > /$path/dev.log sfdisk –l > /$path/sfdisk.log

where $path denotes any absolute path.

Additional Debug if required can be enabled:

Enabling Debug on Job logs:

On UDP Linux Agent Server:

export CRE_LOG_DEBUG_LEVEL=X (where X can be any value between 3 and 6. Default is 3)

Location of the log: /opt/CA/d2dserver/logs/logs/d2d-job-$jobid.log (additional debug

logging)

Once the issue is reproduced, we can remove debug using the “unset” command.

e.g.: unset CRE_LOG_DEBUG_LEVEL

On UDP Linux Node (Agentless):

Append “export CRE_LOG_DEBUG_LEVEL=X”(where X can be any value between 3 and 6. Default is 3) to the file /root/.bashrc and rerun the job

Location of the log: /tmp/logs/d2d-job-$jobid.log (additional debug logging)

where $jobid denotes the actual Job ID. Once the issue is reproduced, edit the /root/.bashrc file and remove the line "export CRE_LOG_DEBUG_LEVEL=X”

28 | P a g e

Debugging Linux Agent / Agentless communication issues:

On the UDP Linux Agent Server

export D2DEALIB_DEBUG=1

Location of log: /opt/CA/d2dserver/logs/ealib.log

Once the issue is reproduced, we can remove debug using the “unset” command.

On the UDP Linux Node (Agentless)

Append the following two lines to the file /root/.bashrc and rerun the job

export D2DEALIB_DEBUG=1 export D2DSVR_HOME=/tmp/

Location of the log: /tmp/logs/ealib.log

To disable the debug, please remove the above added lines from the /root/.bashrc file.

NOTE: The “ealib.log” is created in the above locations only when we enable debug.

Log for UI:

Modify file /opt/CA/d2dserver/configfiles/ log4j-webui.properties:

Change “log4j.category.com.ca=INFO, logout” to “log4j.category.com.ca=DEBUG, logout”

Restart D2D Service to effect the change: /opt/CA/d2dserver/bin/d2dserver restart

Location of the log: /opt/CA/d2dserver/logs/webui.log (additional debug logging)

Log for Web Service:

Modify file /opt/CA/d2dserver/configfiles/log4j-webservice.properties

Change “log4j.category.com.ca=INFO, logout” to “log4j.category.com.ca=DEBUG, logout”

Restart D2D Service to effect the change: /opt/CA/d2dserver/bin/d2dserver restart

Location of the log: /opt/CA/d2dserver/logs/webservice.log (additional debug logging)

Extracting logs from the *.db files

For example, let’s say we have a CAD2DL issue, xxxxx.

Create a folder called “xxxxx” under /tmp on the linux machine (Absolute Path: /tmp/ xxxxx) Copy the ActivityLog.db, JobHistory.db and ARCserveLinuxD2D.db to “/tmp/ xxxxx”

Absolute path for the customer’s *.db files would be:

/tmp/xxxxx /ActivityLog.db

/tmp/xxxxx /JobHistory.db

/tmp/xxxxx /ARCserveLinuxD2D.db

29 | P a g e

Go to the /opt/CA/d2dserver folder and execute the below command:

source /opt/CA/d2dserver/bin/setenv

/opt/CA/d2dserver/sbin/sqlite3 /tmp/xxxxx/ActivityLog.db “select * from ActivityLog;” >

/tmp/xxxxx/Activity.Log

/tmp/xxxxx/Activity.Log is a text file which can be read using any editor or notepad.


/opt/CA/d2dserver/sbin/sqlite3 /tmp/xxxxx/JobHistory.db “select * from JobHistory;” >

/tmp/xxxxx/JobHistory.Log

/tmp/xxxxx/JobHistory.Log is a text file which can be read using any editor or notepad.


/opt/CA/d2dserver/sbin/sqlite3 /tmp/xxxxx/ARCserveLinuxD2D.db “select * from

ARCserveLinuxD2D;” > /tmp/xxxxx/ARCserveLinuxD2D.Log

/tmp/xxxxx/ARCserveLinuxD2D.Log is a text file which can be read using any editor or notepad.

Debug Settings for File Copy

Debug Amazon S3

C:\Program Files\CA\arcserve Unified Data Protection\Engine\BIN\CCI\Config\ log4j-Amazon.properties

Open this file and find line " log4j.category.com.ca.AmazonLibrary=DEBUG, logout ". Available log levels are INFO DEBUG WARN ERROR FATAL

Debug Azure

C:\Program Files\CA\arcserve Unified Data Protection\Engine\BIN\CCI\Config\ log4j-Azure.properties

Open this file and find line log4j.category.com.ca.AzureLibrary=DEBUG, logout". Available log levels are INFO DEBUG WARN ERROR FATAL

CCI Logs - Collect the following logs folders

C:\Program Files\CA\arcserve Unified Data Protection\Engine\BIN\CCI\Log folder

30 | P a g e

Amazon - C:\Program Files\CA\arcserve Unified Data

Protection\Engine\BIN\CCI\Plugin\Amazon\Log folder

Azure - C:\Program Files\CA\arcserve Unified Data Protection\Engine\BIN\CCI\Plugin\Azure\Log

folder

Files under UDP Agent Logs folder

AFArchive.log - Debug log for archive job creation and job monitor.

ArchivePolicy-XXXX.log - Debug log for filecopy common modules like policy parsing,

catalog.

FileCopy-XXXX.log - Debug log for file copy.

WebService.Log

It is better to collect the UDP Agent Logs folder.

Which CCI Logs to refer?

C:\Program Files\CA\arcserve Unified Data Protection\Engine\BIN\CCI\Log - Refer GLLog0.log

Amazon - C:\Program Files\CA\arcserve Unified Data Protection\Engine\BIN\CCI\Plugin\Amazon\Log -

Refer Amazon.Log

Azure - C:\Program Files\CA\arcserve Unified Data Protection\Engine\BIN\CCI\Plugin\Azure\Log - Refer

Azure.Log

Documents

1 | P a g e · 5 | P a g e Debugging UDP Protocol problem To trouble shoot the UDP Protocol problem we need to enable the debug as mentioned below C:\Program Files\CA\ARCserve Unified