45
1 Planning & Executing your Upgrade to NBU 7.6- 1729 Suresh Suseendran, Lead Architect Planning & Executing your Upgrade to NBU 7.6 Praveen Vunnava, Lead Architect

Planning & Executing your Upgrade to NBU 7.6- 1729

  • Upload
    others

  • View
    6

  • Download
    0

Embed Size (px)

Citation preview

1

Planning & Executing your Upgrade to NBU 7.6- 1729 Suresh Suseendran, Lead Architect

Planning & Executing your Upgrade to NBU 7.6

Praveen Vunnava, Lead Architect

SYMANTEC VISION 2014

Agenda

Planning & Executing your Upgrade to NBU 7.6 2

Top 5 Reasons for Upgrading to NBU 7.6 1

Pre-Upgrade Steps/ Best Practices 2

Upgrade Best Practices 3

Post Upgrade Best Practices 4

Upgrade Tools/Scripts 5

SYMANTEC VISION 2014

Top 5 Reasons for Upgrading to NBU 7.6

3

SYMANTEC VISION 2014

• Superior Integration with VMware environments • Next Generation Deduplication Architecture

• Storage Life Cycle Windows and Targeted Auto Image

Replication

• Intelligent Policy Framework for Oracle

• Replication Director for VMware, Application, and Block Array Support

4

Top 5 Reasons for Upgrading to NBU 7.6

SYMANTEC VISION 2014

Pre-Upgrade Steps/Best Practices

5

Maintenance windows are too

narrow

The Customer Experience with NetBackup Upgrades

Tasks Customers Perform

1. Installation, configuration and upgrades

2. Planned and unplanned maintenance

3. Reduce and eliminate risk

4. Troubleshooting

5. Define and manage standard builds

6. Obtain and deploy license / manage compliance

7. Server to storage connectivity (path mgmt)

8. Performance tuning post upgrade

9. Application level protection

10. Implement new features post upgrade

11. Planning for all activities

Install & upgrade takes too long

Can’t collect product inventory to meet

business needs

Limited tools available to identify and mitigate

risks

No time to wait on hold with Support

SYMANTEC VISION 2014

What is Symantec Operations Readiness Tools?

• SORT is a website designed to:

– Support Symantec enterprise products

– Automate and simplify some of the most time-consuming administrator tasks

• Use SORT to:

– Determine if systems are ready for product installation or upgrade

• Generate checklists

• Generate system specific custom reports

• Media, Master, and client systems

– Get a list of deployed hot fixes (and EEB’s) on a set of systems

– Identifies if any EEB related fixes have been addressed or not

– Determine fixed-in state with the NetBackup Hot Fix and EEB Release Auditor

– Quickly link to NetBackup documents, Support, SymConnect, etc

– View Future Platform and Feature Plans for early notification of planned changes to supported items

• Trouble-free installations/ upgrades

• Product and license key inventory

• Advanced support-based searches

• Extended error code solutions

Improve Operations Efficiency

• Real-time patch notifications

• Risk assessments and mitigation pre-upgrade (EEB’s)

• Actionable server error reports

• Pre-Upgrade checks to validate readiness of the environment

Manage Risks

Improving the Total Customer Experience with SORT

Enhance the overall product experience for Symantec Enterprise users in three key areas

• Server and environmental specific recommendations

• Tune and optimize customer environment (Shared memory tunable verification & recommendations)

Align to Best Practices

SYMANTEC VISION 2014

Pre-Upgrade Steps / Procedure : SORT – Data Collector

SYMANTEC VISION 2014

Pre-Upgrade Steps / Procedure : SORT – Data Collector

SYMANTEC VISION 2014

Pre-Upgrade Steps / Procedure : SORT – Data Collector

SYMANTEC VISION 2014

Pre-Upgrade Best Practices

• SORT should be used as the starting point

– Generate an installation check list from Symantec Operations Readiness Tool (SORT).

– (https://sort.symantec.com/checklist/install/)

• Review Upgrade guide (http://www.symantec.com/docs/DOC6449)

• Review Upgrade Portal (http://www.symantec.com/docs/TECH74584 )

• NetBackup Catalog Consistency Utility (NBCC) (http://www.symantec.com/docs/TECH156730)

SYMANTEC VISION 2014

Pre-Upgrade Best Practices – Pre-checks

• Perform catalog consistency check to validate the catalog and NBDB

• Perform sanity check on EMM

• NBDB unload may be required to fix some EMM inconsistencies: Open a support case to perform this (if required)

• Fix any configuration related issues

• Fix any dedupe disk pool / ost related issues

SYMANTEC VISION 2014

Pre-Upgrade Best Practices – Catalog conversion

• Catalog size (Please review the Upgrade Guide for guidelines on time estimation)

– Catalog size is important : catalog migration is part of upgrade process

– Database rebuild: NBU 7.6 utilizes version 12.0.1 of Sybase SQL Anywhere database

– Modify server.conf to increase the “-ch” setting (change it to at least half the physical memory of the master) so that conversion process would complete faster. Please modify it back after the upgrade is complete

Note: Time required to upgrade from 7.5.x is longer than performing upgrades from pre-7.5 versions since the database (NBDB) is small prior to NBU 7.5

SYMANTEC VISION 2014

Pre-Upgrade Best Practices

• Perform consistent backup of NetBackup catalog for any rollback or recovery purpose.

• Perform consistent backup of OS for any rollback or recovery purpose.

• If NetBackup binaries are under OS drive & RAID(1) control & protect the drive by breaking the mirror , preserve one drive in case of any roll back required.

• Verify enough space is available for both catalog conversion and also policy information (class folders/directories) : Upgrade Guide

• Run NBSU on master & Media servers to gather configuration info & other related information about the master / media servers.

SYMANTEC VISION 2014

Pre-Upgrade Best Practices

• Perform a nbdb online backup to save emm DB in a separate location (preferably on a different server) other than the same volume as catalog.

• Verify and make copies of the following prior to Upgrade:

– Configuration touch files

– Custom scripts

– Any other miscellaneous customization performed to NBU configuration files

• Validate & download all respective patches that are required from OS perspective to support NetBackup 7.6.0.x version – utilize SORT for this.

SYMANTEC VISION 2014

Pre-Upgrade Best Practices

• Cleanup all NetBackup configuration files and remove unwanted entries

• In case of clustered master server , validate the compatibilities of VCS & OS binaries & backup all configuration files etc.

• Recommended to have a minimum of 20 to 30% free space available on the following file systems “ NetBackup binary , catalog , log volumes , MSDP ” .

• Enable logging for daemons like EMM , nbpem , nbstserv , bpdbm , bprd in a proactive manner to collect and analyze logs in case of any failures.

• Make sure the emm.conf , server.conf file entries are having appropriate values to support the environment.

SYMANTEC VISION 2014

Pre-Upgrade Best Practices – NetBackup Appliances

• In case of Symantec appliances validate and confirm the following :

• Make sure IPMI configuration is enabled and accessible

• Minimum of 20 to 30 % of disk space should be available for Catalog & MSDP volume.

• Perform overall health check of the appliance hardware components to find out for any kind of hardware issues and address any kind of hardware issues by opening support cases accordingly.

Symantec Confidential: Internal Use Only

Pre-Upgrade Planning - Catalog DB Sybase rebuild

• NBU 7.6 utilizes version 12.0.1. During the upgrade, .db files are migrated to the same version for better performance.

• Requires about 7.5 minutes per GB (12 minutes for BMR_DATA.db).

– total Size of EMM DB multiply by 7.5

– BMR DB multiply by 12.

• BMR_DATA.db 7.5+ file size can become large

– Check the size of the file and if greater than 1GB, requires to perform a pre-install purge. TECH211811 will detail the steps.

– Failure to purge wont cause any issues but rebuild will take longer times. Post 7.6 upgrade, the purge is built in

• DO NOT loose patience and kill this process, or DB recovery will be needed.

19

Symantec Confidential: Internal Use Only

Pre-Upgrade Planning - MSDP conversion

• The MSDP metadata and reference management changes in NetBackup 7.6.0.1 require that the existing database records be converted to the new format during an upgrade installation.

• Installer will ask if conversion should be started automatically or manually. As a best practice, it is recommended to let the installer start the process.

• SPAD/SPOOLD will not run until the conversion is complete, and the conversion process will start them when it is done.

20

Symantec Confidential: Internal Use Only

NetBackup Pre Upgrade Tools

• Migration_check.pl/.exe

– SORT data-collectors are similar

– Performs image counts in best available way, and understands what to count.

– Can be run during and/or after the upgrade to check results

• Install_prep.pl/.exe

– Record current state of policies/SLP’s/DSSU schedules

– Deactivate so that scheduled tasks won’t happen

– Reactivate once upgrade is complete

• Parallel_cat_import.pl/.exe

– Performs the “Guided Method” many-at-once client import

21

Symantec Confidential: Internal Use Only

NetBackup Pre Upgrade Tools : MSDP

• pddeobjectcount.pl

– Performs MSDP object image counts in best available way, and provides an estimation of the overall MSDP upgrade time frame.

– Should be run before the upgrade.

– Detailed information of how to execute and interpret the results of the tool is explained in the technote : “http://www.symantec.com/business/support/index?page=content&id=TECH213728”

22

SYMANTEC VISION 2014

Upgrade Best Practices

23

SYMANTEC VISION 2014

Upgrade Best Practices

• Perform the NetBackup Upgrade in the following order:

– NBU OpsCenter server

– NBU master server

– NBU media servers

– NBU clients

24

SYMANTEC VISION 2014

Upgrade Best Practices : Installer Pre-Checks

• The installer now checks for certain conditions that are known to cause issues, and will prevent the install from starting

– Does enough free space exist in the data directory to actually rebuild the .db files?

– Does enough free space exist in the MSDP pool to perform the conversion?

– Do links exist in the NetBackup directory structure on Windows?

• Updates to the latest version (must have internet connection for this to work) – EEB’s, any known fixes as well any identified upgrade issues so far.

Symantec Confidential: Internal Use Only

Upgrade Best Practices

• Run a hot catalog backup.

• Validate the performance of the Catalog file system and make sure it is not fragmented and yielding best performance for read & write I/O.

• Stop all customized or third-party scripts.

• Disable OpsCenter data collection for this master server.

• Disable all storage lifecycle policies (SLPs).

• Deactivate all NetBackup policies.

• Deactivate all disk staging storage units for all pre-NetBackup 7.5.x environments.

26

Symantec Confidential: Internal Use Only

Upgrade Best Practices

• Perform any cluster-specific tasks

• For clustered NBU Master servers, take the following NetBackup resources offline:

– MSCS clusters: Take all of the NetBackup group resources offline except for the disk, the virtual name, and the virtual IP address.

– VCS clusters: Take the NetBackup resource offline.

27

Symantec Confidential: Internal Use Only

Upgrade Best Practices – MSDP/NBU Appliances

• Do not stop or restart services on a MSDP media server or appliance during the MSDP data conversion. Please wait for the conversion to complete

• In case appliance related Upgrade please make note of these :

– Rollback after upgrade is complete is not supported

– Upgrade takes a long time – plan about 2 hours+ Mostly due to time for ISO install

– After rollback unable to attempt upgrade again for 3+ hours

28

SYMANTEC VISION 2014

Post Upgrade Best Practices

29

Symantec Confidential: Internal Use Only

Post Upgrade Best Practices

• Run a NBCC and validate the consistency of the catalog , to validate the sanity of EMM.

• Once the catalog migration and the MSDP conversion finish, reactivate the following in the order as shown:

– All disk staging storage units.

– All NetBackup policies.

– All storage lifecycle policies (SLPs).

– OpsCenter data collection for this master server.

• Perform test backups & restores for few of the policies/critical clients.

30

Symantec Confidential: Internal Use Only

Post Upgrade Best Practices

• Perform a full backup of the catalog file system & file system hosting NBU binaries.

• In case of a clustered master server , fail over the services between nodes and verify the functionality & performance.

• Update parameters in configuration files for MSDP / EMM to get optimized performance.

• Configure SLP windows to enable scheduling of duplication or replication jobs.

31

Symantec Confidential: Internal Use Only

Post Upgrade Best Practices

• If you upgraded a master server that allowed nonroot users to administer NetBackup, you must reconfigure the permissions and the group. The default permissions and group on the newly installed files allow only a root user to perform NetBackup administration.

• If you made changes to NetBackup scripts before the upgrade, apply those changes to the new, upgraded versions of the scripts.

• Upgrade any add-on products (such as NetBackup language packages) on all upgraded clients. All add-on products should be at the same version as the NetBackup client.

32

Symantec Confidential: Internal Use Only

Post Upgrade Best Practices

• If any touch files were leveraged prior to the upgrade, some of the touch file parameters have been migrated to Master Server Host Properties (For example: SLP parameters)

– Please review Upgrade Guide on all of the different parameters

• Verify the changes to any commands to make sure that your custom scripts are not affected : Upgrade Guide

33

SYMANTEC VISION 2014

Upgrade Tools/Scripts

34

Symantec Confidential: Internal Use Only

Upgrade Tools/Scripts

• Pre Upgrade Scripts / Tools :

Migration_check.pl/.exe

Install_prep.pl/.exe

Parallel_cat_import.pl/.exe

pddeobjectcount.pl

• Post Upgrade Scripts / Tools :

• To start the conversion of MSDP database manually use this script : “/usr/openv/pdde/pdconfigure/scripts/installers/PDDE_convert.sh”

• Excute this script to cleanup storage artifacts from the conversion process : “/usr/openv/pdde/pdcr/bin/stconv –cleanup”

35

SYMANTEC VISION 2014

NBU Upgrade Use Cases

36

SYMANTEC VISION 2014

Use Case 1 – NBU Appliances Architecture

37

SYMANTEC VISION 2014

Use Case 1 – Pre-Upgrade Steps

• Generate the SORT checklist for upgrade

– (https://sort.symantec.com/checklist/install/)

– Verify compatibility of all the components identified in the checklist report generated from SORT website – NBU master, media and clients

– Identify any Late Breaking News and EEB’s that are required as part of the upgrade

– Verify the space requirements for upgrade process to complete successfully http://www.symantec.com/docs/DOC6449

– Identify the catalog size so that approximate time estimate can be made for catalog conversion as part of the NBU master server upgrade on NBU 5230 appliance

• 7.5 minutes per GB of EMM database

• 12 minutes per GB of BMR database

– Identify any client platforms that might be impacted as part of the upgrade

38

SYMANTEC VISION 2014

Use Case 1 – Pre-Upgrade Steps

• Run “pddeobjectcount.pl” script on all of the NBU 5230 media server appliances

– Provides the time estimate required to convert the MSDP database to new refDB format

– Include an additional hour to the estimate to be conservative

• Run NBCC on both master server appliances

• Perform sanity check on EMM database of both master server appliances – NBDB consistency check

• Fix any issues that were identified as part of both NBCC and NBDB consistency checks

• Perform a successful backup of OpsCenter database

39

SYMANTEC VISION 2014

Use Case 1 – Pre-Upgrade Steps

• Cleanup all NetBackup configuration files and remove unwanted entries (touch files)

• Create copies of any customs scripts that were implemented

• Stop NBU and create copies of NBDB (copy it to a remote server) on both NBU 5230 master server appliances

• Disable OpsCenter data collection for both NBU 5230 master server appliances

• Modify server.conf file on both master server appliances to change “-ch” setting to half the physical memory of the NBU 5230 appliance to help speed up the catalog conversion during upgrade of the master server

• Identify any nbrb.conf settings prior to upgrade to make sure that they are addressed after the upgrade is complete : nbrb.conf settings are migrated into EMM database

40

SYMANTEC VISION 2014

Use Case 1 – Pre-Upgrade Steps

• Delete any unwanted policies, storage unites, entries in EMM database (for example: de-commissioned media servers etc..)

• Run a catalog backup job on both master server appliances at both data centers (A & B)

• Deactivate the following components (Install_prep.pl):

– NBU policies including SLPs

– Any custom script execution

– Any cron jobs defined for NBU 5230 appliance maintenance tasks

– Identify any CRON or 3rd party scheduled jobs and deactivate them

41

SYMANTEC VISION 2014

Use Case 1 – Upgrade

• Upgrade the environment in the following order:

– NBU OpsCenter server

– NBU 5230 master server appliance at Data Center B

– NBU 5230 master server appliance at Data Center A

– NBU 5230 media server appliance at remote site A

– NBU 5230 media server appliance at remote site B

– NBU 5230 media server appliances at Data Center A

– NBU 5230 media server appliances at Data Center B

– NBU clients at remote sites & data centers

42

SYMANTEC VISION 2014

Use Case 1 – Post Upgrade Activities

• Run catalog consistency check

• Activate the following components:

– All of the policies including SLPs

– Any custom script execution

– Any CRON jobs defined for NBU 5230 appliance maintenance tasks

– Any CRON or 3rd party scheduled jobs

– OpsCenter data collection on the master server appliances

• Run catalog backup

• Run acceptance testing

– Backups, Restores of a few select clients & data types

– Verify and test SLP/optimized duplication

43

SYMANTEC VISION 2014

Use Case 1 – Post Upgrade Activities

• Update server.conf on NBU 5230 master server appliances to change “-ch” setting back to optimal value

• Update custom scripts to accommodate for changes to any NBU commands & settings with the upgrade (if any)

• Update parameters in configuration files for MSDP / EMM to get optimized performance (SLP parameters, SEGK size, thresholds, high/low water marks, max i/o limits on MSDP storage pools etc..)

• Make sure any changes to nbrb.conf are updated in EMM database after the upgrade (nbrbutil –listsettings)

• Implement new features like targeted AIR, SLP windows for better efficiencies across the entire architecture

44

Thank you!

Copyright © 2014 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

45

Suresh Suseendran, Lead Architect ([email protected])

Praveen Vunnava, Lead Architect

([email protected])