CDF DB Taking Stock ‘10

1

By Anil Garg – Database Services By Anil Garg – Database Services

Aug 18, 2010Aug 18, 2010

2

Infrastructure

Capacity Planning

Backup & Recovery

Accomplishments

DB Availability Metrics

What’s Next

Concerns

Q & A

CDF DB Taking Stock ‘10 - Outline

3

CDF DB Taking Stock ‘10 – Infrastructure - DatabasesType Machine Usage OS TypeOracle v10.2.0.4

bzora2 CDF online Prd SunOS 5.10 64bit

SPARC T5220 1165 MHz32GB

b0dau36 CDF online Dev /Int SunOS 5.10 64bit

SunE450, 4x400MHz, 4Gb

fcdfora3 CDF offline Prd SunOS 5.10 64bit

Sun M4000, 4x2.4MHz, 32G

fcdfora1 CDF offline Dev/Int SunOS 5.10 64bit

SunE4500, 8x400MHz, 2.5G

fcdfora7 CDF offline Prd Replica

RHEL 5.5 64bit SENECA8x2.66GHz 16Gb

fcdfora5 CDF offline Replica Test

RHEL 5.5 32bit DELL 2650, 4x2.8GHz, 2.0G

4

CDF DB Taking Stock ‘10 – Infrastructure - DatabasesType Machine Usage OS TypeOracle v10.2.0.4

bzora1 Future CDF online Dev/int

SunOS 5.10 64bit

SunFire V440, 4x1.2GHz, 16Gb

fcdforaN Future CDF offline Replica Test

RHEL 5.5 64bit SENECA8x2.66GHz 16Gb

5

Report for: Space Usage for Database cdfonprd Period: Apr 2010 To June 2010

SLA breaches: None Incidents/Near-misses: None.

Current Database Space Usage Allocated (GB) 871 Used (GB) 680 Allocated Free Space (GB) 192  This Quarter EST: Next Quarter Available on

ServerBegin 650G 680GB bzora2

428GBEnd 680GB 710GB

Based upon current usage trending, we are NOT expecting any storage need in next 1 year.

CDF DB Taking Stock ‘10 – Disk Capacity Planning

Graph/Trending:

Used

AllocatedAllocated

Used

6

Report for: Space Usage for Database cdfofpr2 Period: Apr 2010 To June 2010

SLA breaches: None Incidents/Near-misses: None.

Current Database Space Usage Allocated (GB) 764 Used (GB) 484 Allocated Free Space (GB) 280 This Quarter EST: Next Quarter Available on

ServerBegin 450GB 484GB fcdfora3

472GBEnd 484GB 518GB

Based upon current usage trending, we are NOT expecting any storage need in next 1 year.

D0 DB Taking Stock ‘10 – Disk Capacity Planning

Graph/Trending:

Used

Allocated

Used

Allocated

7

Report for: Space Usage for Database cdfstrm1 Period: Apr 2010 To Jun 2010

SLA breaches: None Incidents/Near-misses: None.

Current Database Space Usage Allocated (GB) 497 Used (GB) 324 Allocated Free Space (GB) 172  This Quarter EST: Next Quarter Available on

ServerBegin 303GB 324GB fcdfora7

1.3TEnd 324GB 345GB

Based upon current usage trending, we are NOT expecting any storage need in next 1 year.

D0 DB Taking Stock ‘10 – Disk Capacity Planning

Graph/Trending:

Used

Allocated

8

Graph/Trending cdfonprd - Growth 91GB per yr / cdfofpr2 – 96GB/ per yr

CDF DB Taking Stock ‘10 – Yearly Charts

cdfonprd cdfofpr2 *

UsedAllocated

* New Agent was Implemented in Sep ’09Note: Not enough Data for cdfstrm1 since migrated to new host in Jul ‘10 . However, growth is 80GB/Year

9

CDF DB Taking Stock ‘10 – Backups Details

Database Frequency Local Copies Export

Archive Retention Dcache Size Free

cdfonprd Daily 2*No 5Daily - Dcache 1.7T 440G cdfofpr2 Daily 2No 5Daily - Dcache 900G 436G cdfstrm1 Daily 2No 7No 560G 1.2T

10

Migrated cdfofpr2 database to fcdfora3 Migrated cdfstrm1 to fcdfora7 Migrated cdfonprd to bzora2 Implemented backup for cdfstrm1 Deployed SAM Schema cuts as needed Moved the backups to HITACHI SMS100 from

NEXSAN for cdfonprd database OS and DB Security Patches up-to-date Kerberos Proof of Concept Migrated and upgraded the Database Monitoring tool

CDF DB Taking Stock ‘10 - Accomplishments

11

Time Period Jun 01, 2009 to Jul 21, 2010 CDT cdfonprd:

Overall Availability(%): 99.46 Total Downtime (hours): 0.33

Total Blackout Time (hours): 5.65

cdfofpr2: Overall Availability (%): 99.79

Total Downtime (hours): 0Total Blackout Time (hours): 3.4 Note: 655 minutes downtime due to Power Outage.

Not reported by OEM since OEM was down due to Power Outage.

cdfstrm1 :Overall Availability (%): 100% ( Due to failover to cdfofwrite service) Note: 201 minutes downtime due to Power Outage.

Not reported by OEM since OEM was down due to Power Outage.

CDF DB Taking Stock ‘10 – DB Availability MetricsAvailability State

12

Upgrade Oracle Databases to 11g ( Support Expiry Jul ‘11)

Migrate off of 3par SAN storage to another vendor in FY11 . ( Since maintenance cost on 3par will be too high in FY 2012)

Co-ordinate testing Oracle Kerberos for SAM App Upgrade the DB Monitoring Agent Move b0dau36 to bzora1 Move fcdfora1 to fcdfora4 Replace fcdfora5 to new hardware

CDF DB Taking Stock ‘10 - Next

13

Schedule to switch to Ntuples for migration of PRE-SCALE data.

Need to get the name of current application owners for CDF online and offline.

Offline Application Owners ( To be reviewed) http://www-css.fnal.gov/dsg/internal/databs_appl/

applications_CDFoffline.html

Online Application Owners ( To be reviewed) http://www-css.fnal.gov/dsg/internal/databs_appl/applications_CDFonline.html

CDF SAM ReplicationPros and Cons :http://www-css.fnal.gov/dsg/internal/ora_repl/

CDF_SAM_REPLICATION.htm

CDF DB Taking Stock ‘10 – Concerns

14

Q & A

CDF DB Taking Stock ‘10 – Q & A