Upload
emma-fletcher
View
217
Download
1
Embed Size (px)
Citation preview
1
CDF Taking Stock ‘08
By Anil Kumar By Anil Kumar
CD/LSCS/DBI/DBACD/LSCS/DBI/DBA
July 16, 2008July 16, 2008
2
Infrastructure
Space Usage & Capacity Planning
Backup & Recovery
Accomplishments
What’s Next
Concerns
Q & A
CDF Taking Stock ’08 - Outline
3
Machine Usage Type OS Oraclebzora1 Prd - Online SunFire V440, 4x1.2GHz, 16Gb 5.9 64bit 10.2.0.3 64bit b0dau36 Dev/Int - Online SunE450, 4x400MHz, 4Gb 5.9 64bit 10.2.0.3 64bit
fcdfora4 Int/Prd - Off Sun-Fire V880 8x900MHz, 32Gb 5.9 64bit 10.2.0.3 64bit fcdfora1 Dev/Int - Off SunE4500, 8x400MHz, 2.5Gb 5.9 64bit 10.2.0.3 64bit
fcdfora6 Prd Replica Dell PE6650, 8x2GHz, 8Gb RHEL3 10.2.0.3 fcdfra5 Dev Replica Dell2650, 4x2.7GHz, 2Gb RHEL3 10.2.0.3
CDF Taking Stock ’08 - Infrastructure
4
CDF Taking Stock ’08 – Space Usage
5
CDF Taking Stock ’08 – SAM Space Usage
SAM Space Usage
020000400006000080000
1000006/
9/20
02
10/9
/200
2
2/9/
2003
6/9/
2003
10/9
/200
3
2/9/
2004
6/9/
2004
10/9
/200
4
2/9/
2005
6/9/
2005
10/9
/200
5
2/9/
2006
6/9/
2006
10/9
/200
6
2/9/
2007
6/9/
2007
10/9
/200
7
2/9/
2008
6/9/
2008
Date
Size
(MB
)
Series1
6
ON-LINE Growth: 80-90GB /Year
Next two years CDF online expected growth 170G. 145G is available.
OFF-LINE
Growth: 50-60GB/Year Next two years CDF offline
expected growth is 120G. 612G available.
Note: SAM growth : 25-30G
CDF On-line Space Usage
0100000200000300000400000500000
12/1
9/2
000
6/1
9/2
001
12/1
9/2
001
6/1
9/2
002
12/1
9/2
002
6/1
9/2
003
12/1
9/2
003
6/1
9/2
004
12/1
9/2
004
6/1
9/2
005
12/1
9/2
005
6/1
9/2
006
12/1
9/2
006
6/1
9/2
007
12/1
9/2
007
6/1
9/2
008
Date
Siz
e(M
B)
Series1
CDF Taking Stock ’08 – Capacity Planning
CDF Offline Space Usage
050000
100000150000200000250000300000
12/2
4/2
000
6/2
4/2
001
12/2
4/2
001
6/2
4/2
002
12/2
4/2
002
6/2
4/2
003
12/2
4/2
003
6/2
4/2
004
12/2
4/2
004
6/2
4/2
005
12/2
4/2
005
6/2
4/2
006
12/2
4/2
006
6/2
4/2
007
12/2
4/2
007
6/2
4/2
008
Date
Siz
e(M
B)
Series1
7
Applic. Machine Capacity Used Rman Capacity Rman Used
CDF Online bzora1 800G 655G 939G 500G*
CDF Offline fcdfora4 1.3T 688G 3.5T 3.0T**
CDF Replica fcdfora6 1.2T 466GNo RMAN implementation
* One Copy of RMAN and archives (24GB in size on average) ** Two Copies of RMAN on NAS and the NAS is shared with other systems.
CDF Taking Stock ’08 – Backup Space Used
8
cdfonprd Daily, 5 days of archives. Allocated 939GB, Used 300GB Back-up time -> 2 Hrs RMAN backup to SAN. Also One local copy. Backups to dcache daily. No Export.
cdfofprd Daily, 4 days of archives Allocated 3.5TB, Used 2.8TB (2 copies, 342GB each), Back-up time: 4Hrs
15Mins Both copies of rman to NAS. No local copy. Backup to dcache daily.
Daily Export of SAM and Offline Apps (e.g. Books, Filecatalog and Frontier)
cdfreplica No RMAN Backup and No Export since it is replica. In case of failure, replication will be reinstantiated.
CDF Taking Stock ’08 – Backups Details
9
CDF Online Production 100% uptime CDF Offline Production 99+% uptime. ( 150 Minutes maintenance, not down but
dealing with index corruption for CachedFiles table during cleanup activity ) CDF Replica 100% uptime.
Upgraded the Oracle databases to version 10.2.0.3
OS and DB Security Patches up-to-date
Tuning of CDF online databases. (Redo tuning, Backup to SAN, API Tuning) Performed targeted monitoring/benchmark tests.
Move RMAN to NON-LOCAL Disks – Decrease Contention – Permanent Tests with SUN -> Suspected Disk Contention. New bigger disks & Distribute I/O Report on many untuned queries, API modifications to tune queries.
Deployment of tuned slow control code is pending.
Proof of product for ASO ( Advanced Security Option)
Deployed Six SAM Schama cuts
CDF Taking Stock ’08 - Accomplishments
10
Disk Capacity Upgrade on cdf online prd database. This will also help to distribute the I/O across multiple mount points and hence tuning of I/O Replacing 12 146GB with 12 300GB, ~900GB additional space, 136G dedicated
space for archives. – Planned for August Implementation.
Upgrade the Oracle databases to version 10.2.0.4 OR 11g
ASO ( Advanced Security Option) Deployment. Db on fcdfora5 was used for proof of product. More tests will be performed on b0dau36 and fcdfora1.
May consider moving cdfofdev/cdfofint databases to 3PAR storage since 2-3 incidences of block corruption on existing disks.
CDF Taking Stock ’08 - Next
11
EXPEDITE the MIGRATION of PRE-SCALE Schema to Production. Risk: Production data is hosted on dev system. No Backup. Risk of losing data in case
of failure.
Disks on b0dau36 is getting old, hard to find, not supported by SUN. Need to consider getting new storage for b0dau36.
Deputy CDF DB Liaison ?
Enforce the Integration tests before deploying the schema/APIs in production.
Single point of failure for SAM and DFC applications.
There are still externally authenticated accounts on cdfonline. Need to be converted to password.
Some of CDF application Data Models are not in designer.
CDF Taking Stock ’08 - Concerns
12
Q & A
CDF Taking Stock ’08 – Q & A