EMC IT Best Practices for Oracle: Enable Reduced Infrastructure Costs, Improved Performance, and Increased Availability

EMC PERSPECTIVE

Table of Contents

Executive summary .......................................................................................................................3

Enterprise-wide infrastructure consolidation .................................................................................4

Economies of scale with global instance ........................................................................................5

Maximizing application uptime in global instance .........................................................................6

Multi-million dollar cost savings driven by grid computing .............................................................7

Cost efficiency and flexibility through virtualization .......................................................................8

Data warehouse consolidation increases competitive advantage ....................................................9

Enterprise Flash drive performance gains ....................................................................................10

Flexible storage tiering and consolidation ...................................................................................11

Application retirement and archiving drive down storage growth ..................................................12

Multi-level IT initiatives have significant impact ..........................................................................13

Sharing best practices in the Oracle IT community .......................................................................14

2

Executive summaryAt EMC® Corporation’s manufacturing facilities in the U.S., Ireland, and Brazil, EMC information stor-

age systems are assembled, software is loaded, and integrated solutions are packaged for distribu-

tion 24 hours a day. In a single quarter, EMC processes tens of thousands of orders generated by its

400 sales offices and thousands of partners, and ships products to customers in 60 countries world-

wide. This high-throttle, unending cycle of manufacturing, ordering, and shipping generates expo-

nential volumes of data for EMC to manage, store, and protect.

While EMC information infrastructure solutions are designed to help customers grow and operate

efficiently, EMC shares the same strategic objectives for its own global IT environment. In the last five

years, it has been more challenging to meet these goals as EMC has experienced explosive growth of

its revenue, product line, customer base, and an IT infrastructure spanning five global data centers.

EMC’s Oracle implementation, which is one of Oracle’s largest customer environments in the world

and spans 800 databases, has encountered 20-30 percent data growth annually. With this expan-

sion, there has been significant strain on application performance, scalability, and availability. Rapid

data growth has also made it challenging to keep up with increasing demand for data center space

and energy resources.

EMC was faced with spending $120 million to build a new data center or $26 million for a data center

upgrade. EMC IT decided it could manage growth more successfully and cost effectively by deploying

an efficient IT infrastructure strategy that included consolidation, virtualization, best-in-class backup

and replication, as well as data deduplication. This strategy has provided not only cost reductions

but an infrastructure optimized for high data availability and top performance.

“Last year, EMC publicly committed to significant cost reductions as a way to increase its competitive-

ness and profitability,” says Tony Pagliarulo, EMC’s Vice President of Application Development.

“With IT as one of EMC’s fastest growing cost centers, IT efficiency was targeted as a major priority.

Our distributed Oracle environment was identified as an especially ripe area for cost reduction. By

consolidating our Oracle environments onto the most efficient, green, and scalable infrastructure

possible, we’ve been able to help make EMC more lean, flexible, and competitive.”

In fact, Oracle recently recognized EMC’s IT efficiency accomplishments with its “Enable the

Eco-Enterprise” award. EMC was cited for significant Oracle database consolidation and VMware®

server virtualization, saving the company millions of dollars in data center hardware, management,

and power and cooling.

EMC’s newly consolidated Oracle infrastructure is the result of many different enterprise-wide IT

initiatives focused on:

• Improving availability and quality of business-critical data and processes

• Reducing capital and operational IT expenditures

• Slowing rampant server and storage growth

• Increasing IT responsiveness and agility

This EMC Perspective highlights several of these projects including:

• Oracle global single instance

• Oracle application business continuity

• Oracle grid computing

• Virtualization

• Oracle data warehouse consolidation

• Business intelligence

• Enterprise Flash drives

• Automated tiering

• Data archiving

• Application retirement

3

Enterprise-wide infrastructure consolidationIn the last five years, EMC has grown from an $8.23 billion to a nearly $15 billion company, and

acquired more than 40 firms. In addition, the company has continued its rapid transition from a

primarily storage business to one with substantially increased focus on software and services.

To manage this dramatic transition, EMC was continually adding new custom applications, often

each with their own software, storage, and server infrastructure. One of its fastest-growing and most

widespread infrastructures supported Oracle, a powerful engine of advanced databases and applica-

tions that touches nearly every facet of the business. EMC’s 40,000-plus employees access Oracle to

manage software development, manufacturing, pricing, sales, ordering, service, financial reporting,

E-Business analysis, and other critical functions.

Without a unified growth management strategy for Oracle, data and business process redundancy

was on the rise, IT capital expenditures and administration costs were increasing, and high perfor-

mance and availability were becoming more difficult to maintain.

To reverse this trend, EMC IT executed a multi-pronged approach to consolidate its Oracle environ-

ment. EMC deployed global instances of its Oracle 11.03 enterprise planning resource (ERP) and

E-Business Suite 11i customer relationship management (CRM) environments. The company applied

Oracle database grid computing using Oracle Real Application Clusters (RAC). In addition, VMware

ESX® Server virtualization software was implemented to consolidate physical Oracle servers to more

cost-effective virtual machines that leverage EMC’s storage and software solutions.

Today, EMC’s new IT infrastructure supports 170 Oracle application modules. Oracle E-Business

Suite 11i and Oracle 11.03, along with Oracle Grid 11g database architectures, utilize the breadth

of EMC’s industry-leading technology stack of information management solutions and VMware

platforms.

For optimal efficiency, EMC IT has deployed a tiered storage strategy based on an application

rationalization methodology. EMC IT allocates the most critical CRM and ERP production data to

its high-end Symmetrix® DMX™ and Symmetrix V-Max™ storage; file sharing data such as business

forms to EMC Celerra® networked-attached storage; less critical test and development data to

mid-tier EMC CLARiiON® networked storage; and backups to EMC Disk Library. The overall EMC

storage infrastructure is 6.5 petabytes.

EMC uses advanced software, including EMC SRDF®, EMC TimeFinder®, EMC PowerPath®, VMware,

and Informatica, to manage and protect the end-to-end lifecycle of its Oracle information

infrastructure.

4

Economies of scale with global instanceEMC adopted Oracle global instance to consolidate multiple Oracle application modules onto a single

software platform (see Figure 1). With a global instance of the Oracle E-Business Suite, for example,

EMC has 35 applications sharing the same database infrastructure. In addition, EMC operates a

separate global instance of its 11.03 Oracle ERP environment.

Figure 1: Global Instance of E-Business Suite key statistics

E-Business Suite Key Statistics

One of the largest deployments of

Oracle E-Business Suite in the world

Business StatisticsQuotes 2,263,231Orders 674,458Service Requests 1,391,325Projects 110,512IB Instances 59,937,589Material Transactions 12,305,357Parties 9,429,011Relationships 10,356,648Time Cards 7,336,539

System StatisticsDB Size 7TBNumber of Rows 8.8 BillionIOPS 15KInterconnect Traffic 3,000–4,000 Blocks/Sec

Daily VolumeDatabase Transactions 754/Sec 22 Million/DayArchive Log 1.2 TBConc. Jobs 70KWorkflow Events 250KWorkflow Rates 20,193,476

• Very high usage penetration within the enterprise. – 50,000+ Named Users – 4,000+ Concurrent Users at peak periods• 70+ Application Tiers—Dell/Linux• Two-node RAC Architecture—224 cores• Oracle Database 10 R2, Solaris 10• Extremely dense module usage – Largest users of Service, Install Base and Contracts, Oracle Sales, Configurator – One of the largest users of iStore Projects TCA, Order Management, etc.• Application Continuity—keeping 11 available during maintenance

With global instance, EMC has reduced hardware infrastructure costs and eliminated the complexity

of managing multiple versions of Oracle software. The result has been improved operational efficiency

across dozens of financial, sales, and support processes combined with increased automation and

integration of cross-enterprise functions.

For example, by standardizing financial processes globally with Oracle Financials, EMC can more

quickly establish a complete global view of financial information each quarter. Consequently, EMC

has reduced financial close timeframes from 10–12 days to three-to-four days, providing managers

with faster access to financial data, and enabling better-informed planning and decision making.

By deploying Oracle Projects, EMC has transitioned from fixed-fee to time-and-materials engagements

to improve profit margins of projects handled by its technical services organization. EMC now can

more easily and quickly locate and deploy technical support skill sets across the company, which has

reduced its dependency on third-party support vendors and increased professional services revenues.

When EMC acquires a company, the average time required to integrate the new firm’s technical

services into EMC’s Oracle ERP solution has been slashed from six-to-nine months to less than three

months in the global instance environment. This enables EMC to more rapidly sell new services, book

revenue, and analyze and track project costs.

Global instance also has enabled EMC IT to reduce costs associated with managing Oracle applica-

tions and the supporting IT infrastructure. Capital expenditures for new IT assets, such as servers

and storage for new Oracle applications, have decreased by 50 percent since the database and IT

infrastructure can be easily shared across projects. Oracle licensing fees are lower since the global

instance does not have to be replicated to support new projects.

Standardized processes for multiple Oracle applications, tighter integration, and a single system to

optimize also translate into more efficient and agile software administration. For example, EMC IT can

respond more quickly to changing business requirements due to quicker applications development,

simplified patching, and consolidated backup and disaster recovery processes.

5

6

Maximizing application uptime in global instanceIn a global instance environment, the business impact of maintenance-driven application downtime

for Oracle E-Business Suite was growing as the number of users increased and their productivity and

decision-making were hampered when critical processes were not available. Downtime was caused

by frequent maintenance and upgrades that were needed to ensure Oracle E-Business Suite deliv-

ered top performance and reliability for EMC’s 24x7 global operations.

In response, EMC IT developed the Application Continuance Tool (ACT) to enable most production

processes to occur during maintenance. Using EMC replication software, EMC IT quickly creates

copies of Oracle application environments to facilitate maintenance activities, avoiding the time

and complexity of moving data over corporate networks. Primary and replicated production data are

stored on an EMC Symmetrix DMX-3 system and EMC Celerra NSX gateway. To facilitate replication

and management of this data, EMC uses TimeFinder, EMC Ionix™ ControlCenter®, Celerra Replicator,

PowerPath, and SRDF software.

Since deploying ACT, EMC has reduced maintenance downtime incidents during a quarter from a

total of 30–40 hours to two-to-three hours. ACT enables users to continue working for 88 percent

of the time during off-line operations, resulting in increased satisfaction for EMC internal customers.

Since EMC IT has the freedom and flexibility to perform maintenance more often, they can work more

efficiently and be more responsive to changing Oracle requirements. The Oracle Applications Users

Group (OAUG) awarded EMC IT the “Innovator of the year” award in 2008 for ACT.

7

Multi-million dollar cost savings driven by grid computingAnother critical consolidation strategy that produced competitive advantages and savings for EMC is

Oracle grid computing. Rather than using servers and storage dedicated to a single Oracle database

and associated application modules, EMC manages multiple Oracle databases and applications

within a consolidated grid computing architecture of Oracle Real Application Clusters (RACs).

Before grid computing, EMC’s Oracle 10g database environment comprised 26 applications, nine

Oracle software versions, 45 stand-alone database servers, and 51 databases supporting separate

development, test, and production environments. The new 10g RAC and grid computing environment

supports 32 applications, one Oracle software version, four database servers, and six databases.

One 10g RAC supports development and test and another 10g RAC supports production.

With Oracle grid computing, EMC has gained more flexibility, scalability, and higher utilization

because new applications can be launched into the Oracle RAC cluster environment, leveraging the

shared information infrastructure. Provisioning databases for new environments now takes days

instead of weeks.

In 18 months, EMC has saved $3 million due to cost avoidance associated with Oracle replacement

servers and storage, as well as reduced cost savings for IT staff who now manage a single, consistent

version of Oracle software. As EMC expands grid computing to other database architectures, such as

Oracle 11g, an additional $7 million of savings is expected over the next 12–18 months.

With a consolidated grid architecture, EMC has decreased power usage and floor capacity require-

ments. In addition, availability and performance have improved across the environment because of

standardized and published service levels.

Cost efficiency and flexibility through virtualizationVirtualization is a major initiative to help manage the growth of EMC’s Oracle server infrastructure,

reduce costs, and increase flexibility. Using VMware ESX Server, EMC IT has virtualized several of its

Oracle business intelligence application environments.

To date, EMC has 375 virtual machines running Oracle Database, Oracle Fusion Middleware/Weblogic,

Hyperion, and Oracle E-Business Suite R12. Being able to divide physical servers into virtual

machines has increased overall server utilization and productivity. Over a period of 12 months,

EMC IT has calculated that virtualization has reduced costs by 82 percent due to decreased server

expenditures, power and cooling resources, rack space, and overall data center space (see Figure 2).

Figure 2: Virtualization cost savings

Oracle Products (Annual Cost)

0

$200,000

$400,000

$600,000

$800,000

$1,000,000

Virtualized Non-Virtualized Savings

Oracle Products

EMC IT is planning to implement an additional 216 virtual machines by February 2010. With a 1:1

ratio, EMC IT estimates that virtualization will have enabled it to avoid the cost of purchasing and

maintaining hundreds of additional physical servers.

Because EMC IT can generally provision a new virtual machine in one day, versus a two-to-three

month window for a physical server, it is able to deploy new Oracle projects more quickly.

“The reduction in time to implement a virtual server versus a physical server increases our IT agility

and saves months in our project lifecycles,” says Pagliarulo. “As project delivery timelines shrink

because of virtualization and grid computing, we’re going to see substantial gains from a business

perspective.”

8

Data warehouse consolidation increases competitive advantageGrid computing and virtualization set the stage for EMC to consolidate three major Oracle data

warehouse environments into one. This new data warehouse includes detailed sales data that can

be organized by customer type, solution, services, industry, geographic location, timeframe, and

other parameters. EMC employees use this data to make critical decisions about product develop-

ment, pricing, channel strategies, geographic focus, and distribution, and ultimately improve EMC’s

competitive position.

Before the data warehouses were integrated, there was significant data redundancy across EMC’s

finance, customer service, sales, technical support, and corporate quality organizations. This

resulted in increased production processing loads, increased risk of data discrepancies, and incon-

sistent business rules. It took considerable time for users to retrieve data from different sources, and

there was often no way to verify that the data retrieved was the most current. The absence of central-

ized, readily available data was affecting users’ ability to analyze information and make decisions.

As the data warehouses grew, it became increasingly costly to manage and operate them at consis-

tent service levels. There was significant infrastructure redundancy with multiple servers, storage,

and databases often hosting overlapping data. Because the architectures differed, high availability,

disaster recovery, and information lifecycle management practices also could not be consistently

applied.

EMC IT’s solution was to rationalize and standardize its business intelligence assets. EMC consoli-

dated three copies of its installed Oracle database to one; four copies of the customer database to

one; and four copies of the production database to one. The server environment also was migrated

from high-end Sun and Fujistu servers to lower-cost Dell servers running Linux, consolidating from

21 database servers to 11 servers. EMC IT also completely virtualized its Oracle application servers

in this environment with VMware software, and migrated the consolidated data warehouse and

database servers to the Oracle grid.

From a storage perspective, EMC employed the use of a tiered storage strategy, which uses Enterprise

Flash drives, Fibre Channel drives and SATA drives on EMC Symmetrix VMAX™. This new approach

resulted in significant performance increases for critical business intelligence functions including:

• 10x performance gains for one-day processing

• 180 percent improvement in batch job performance

• Data mart loading times reduced by 50 percent

• 200 percent improvement in dashboard rendering

This has given EMC a greater competitive advantage because users can analyze, forecast, and

respond more quickly and accurately to rapidly changing business trends and customer

requirements.

This centralized data warehouse is also more cost effective and easier to manage. Consolidating

the data warehouse infrastructure onto the Oracle grid enabled moving to a lower-cost Linux server

platform, and decreased storage capacity from 90-100 to 20-30 terabytes.

In addition, this new centralized approach to EMC’s data warehouse has greatly simplified backup

and recovery operations. Leveraging EMC TimeFinder, a clone copy of the production data warehouse

is taken everyday in the morning. Oracle RMAN database backup is then executed off of this new

backup copy instead of production to an EMC Disk Library (EDL). Performing the Oracle backup off of

a point-in-time copy of production reduced backup times from 16 hours to 4 hours.

With decreased storage capacities, smaller more modular servers, and fewer instances of Oracle to

manage, EMC IT has decreased management resources for application support by 30 percent.

9

10

Enterprise Flash drive performance gainsIncreased performance and transaction loads of the consolidated Oracle data warehouse led

to EMC IT’s decision to move this environment from Symmetrix DMX to Symmetrix V-Max high-

performance Enterprise Flash drives (EFDs). In a cutover lasting only 110 hours, 10 terabytes of data

and 1,700 users were migrated to the Symmetrix V-Max environment with Dell Linux servers running

Oracle 11.1.0.7 RAC.

Compared with Fibre Channel, the EFDs delivered significant performance improvements (see Figure 3):

• Execution time for near-realtime data reduced from three hours to 56 minutes—a 67 percent

improvement

• 62 percent reduction in commissions batch processing time

• 77 percent reduction in rendering of business Intelligence (OBIEE) dashboards and drilldowns

• Average 62 percent reduction in processing time for main finance batch jobs

• Dramatic reductions in feeds to Essbase

Figure 3: Selected EFD improvements

Selected EFD improvements

050

100150200250300350400

Overall DS-Core FinFact Margin Cur SMF MV’s BookBill

GDW

IDW

In addition, there was a 10:1 consolidation of Fibre Channel drives per EFD. With significantly higher

performance and dramatically fewer drives, the EFDs proved to be 88 percent more cost effective than

Fibre Channel drives. The EFDs performed Oracle reports replays three times faster than Fibre Channel.

“To meet our service-level objectives, we need to deliver critical business intelligence to tens of

thousands of users, very rapidly and reliably, 24 hours a day,” says Ramesh Razdan, EMC’s Director,

Enterprise Technology Services. “Having the right information at the right time is critical to making

the right decisions. EMC relies on this information to continually adjust sales, order management,

distribution, and other operations to ensure we meet customer and overall market expectations.”

11

Flexible storage tiering and consolidationEMC IT is leveraging the tiering functionality of Symmetrix V-Max, EMC’s storage platform purpose-built

for the virtualized environment, to further reduce cost and maximize performance. Oracle database

grid information is non-disruptively migrated across multiple tiers of storage, enhancing information

lifecycle management (ILM) and data mobility (see Figure 4).

Figure 4: Tiered Symmetrix V-Max data warehouse infrastructure

EMC IT—Consolidated Oracle Optimized Warehouse Using Symmetrix V-Max

FC, SATA (R19)REVP PASP DICP

NRT Prod

REVP PAST DICT

NRT Test

REVD PASD

NRT Dev

PRODUCTION

Global Data Warehouse

Backup Copy

Gold Copy

TEST

DEV

FC, SATA (R5)

FC, SATA (R5)

EFD (R5)FC, SATA (R19)

FC, SATA (R5)

SATA (R5)

SATA (R5)

SATA (R5)

Flash Drives—Objects with most stringent response time

Fibre Drives—Objects from current & recent few quarters

SATA Drives—Historical data

Dynamic SRDF

BCV/Clone

BCV/Clone

EMC Symmetrix V-Max512 GB Cache400 GB Flash Drives300 GB—15K RPM Fibre Drives1 TB—7200 RPM SATA Drives

BCV/Clone

Flash Drives

Primary Site

DR

Global Data Warehouse

Perf

FC, SATA (R19)

FC, SATA (R19)

BCV/Clone

Secondary Site

For example, EMC IT uses Symmetrix V-Max virtual LUN migrations to move data across EFDs, Fibre

Channel drives, and SATA drives assigned with varying levels of RAID protection. The virtual LUN

migrations have proved to be easy to manage, efficient, and reliable because the target is treated

as an internal mirror of the logical device and is assigned its own RAID protection and storage tier.

With the flexibility of virtual LUN migrations, Symmetrix V-Max is facilitating EMC’s ILM strategies,

such as moving entire Oracle databases, table spaces, partitions, or ASM disk groups among storage

tiers during production operations. EMC also can more easily adjust service levels and performance

requirements according to the criticality of application data.

“The flexible and automated tiering provided by V-Max makes it a seamless process,” states Razdan.

“We assign our Flash drives to high-transaction data; slightly aged production data to Fibre Channel;

and SATA for inactive or less-used data—and do it all very efficiently in a single frame. This will help

us achieve even more consolidation and cost savings.”

12

Application retirement and archiving drive down storage growthData archiving and application retirement was identified as another major opportunity for EMC IT to

reduce costs and slow data growth.

EMC’s Oracle Applications 11i database had been growing at a rate of 160 gigabytes per month with

the majority of the growth coming from customer service, direct express, commissions, projects, and

contracts. Side effects were slow performance for users and high infrastructure costs as new storage,

CPU, and memory were continuously added. Other issues included high maintenance and resource

costs for performance tuning, development, storage and Unix administration, and growing backup

windows.

At this rate, EMC IT forecasted that in three years the 11i database would be 12 terabytes—a size

that would not be manageable—and 3.6 petabytes of storage would be needed in five years.

To slow growth, EMC IT is archiving and purging inactive data and reorganizing data through func-

tions such as backup and database compression, as well as decommissioning obsolete or inactive

applications. EMC calculates that nearly 300 terabytes of storage is being saved as a result.

In the next phase, EMC is focused on reducing the size and growth rate of non-production environ-

ments through additional archiving. EMC also is preparing to retire old read-only applications, as

well as reclaim storage and hardware used by retired applications, which would improve ERP and

CRM performance and reduce maintenance downtime.

EMC’s analysis shows that these changes would avoid the purchase of 1.2 petabytes of new storage

over the next five years based on a conservative projected annual growth rate of 30 percent.

As part of this latest phase, EMC IT engaged in a 10-day proof of concept (POC) of an Informatica data

archiving and application retirement solution with the assistance of EMC Solutions and Informatica

Professional Services.

First, the POC focused on archiving the contracts and project accounting modules in the Oracle 11i

E-Business Suite, which is projected to save 15 terabytes of storage in the first year and 75 terabytes

within three years. The POC also focused on subsetting—the process of creating smaller databases

from larger, more complex databases. By subsetting EMC’s three largest Oracle 11i modules—

Configurator, Install Base, and Customer—EMC IT forecasts it could avoid the purchase of 300–400

terabytes over five years.

The engagement also involved retiring 20 IT read-only applications and archiving retired application

data to EMC Centera® content-addressed storage. By reclaiming storage hardware, EMC IT expects to

eliminate the need to purchase 100 terabytes of storage over five years.

13

Multi-level IT initiatives have significant impactEven with these initiatives, Oracle databases will continue to grow as EMC expands into new markets,

automates new functions, and accumulates more data with every transaction and process.

EMC IT is committed to aggressively evaluating and pursuing the latest technologies and strategies

that will help it to continually improve IT efficiency and flexibility, and take cost out of the infrastruc-

ture across all facets of its Oracle environment⎯including databases, application modules, servers,

storage, and networks.

Due to a combination of storage tiering, virtualization, and sophisticated database consolidation,

EMC IT will be able to continue driving substantial savings and return on investment from these

initiatives.

EMC CorporationHopkintonMassachusetts01748-91031-508-435-1000In North America 1-866-464-7381www.EMC.com

Sharing best practices in the Oracle IT communityEMC is committed to documenting best practices and sharing lessons learned in the global IT

community via its EMC Proven™ Solutions testing program.

One of EMC Proven’s most featured application environments is Oracle, which is deployed by

thousands of EMC customers worldwide. Many of EMC’s Oracle customers regularly access EMC

Proven publications to learn about proven strategies that reduce costs and improve IT efficiency.

Here are a few examples of recent EMC Proven Oracle white papers:

• Application Retirement

• Data Archive

• Subsetting

EMC2, EMC, EMC Proven, Celerra, Centera, CLARiiON, ControlCenter, DMX, Ionix, SRDF, Symmetrix, TimeFinder, V-Max, and where information lives are registered trademarks or trademarks of EMC Corporation in the United States and other countries. VMware and ESX are registered trademarks or trademarks of VMware, Inc. in the United States and other jurisdictions. All other trademarks used herein are the property of their respective owners. © Copyright 2010 EMC Corporation. All rights reserved. Published in the USA. 01/10 EMC Perspective H6867