SAP Solutions on VMware Infrastructure – White Paper ... · location, with an OS migration and a database upgrade. The team originally estimated a project The team originally estimated

SAP Solutions on VMware® Infrastructure: Reducing Infrastructure Costs

at Thales Rail Signaling Solutions, Inc.

Case Study: Migrating from UNIX to Windows

May 2009

VMware Inc. Running SAP on VMware® Infrastructure

Case Study – Success Story ii

Contents

Executive Summary.......................................................................................................1

The Business Need.........................................................................................................1

Hosting Solutions Considered.......................................................................................................................................................................1

The Business Solution...................................................................................................2

Key Solution Drivers .............................................................................................................................................................................................2 Risks Involved ...........................................................................................................................................................................................................3 Solution Migration ................................................................................................................................................................................................4 Design Considerations .......................................................................................................................................................................................4 Solution Performance .........................................................................................................................................................................................6

Primary R3 (SAP 4.6c) Central Instance Server (with Oracle DB) .......................................................................................................6 Primary Production VMware ESX Host (Physical Server) ......................................................................................................................7

Key Project Success Factors.............................................................................................................................................................................7 Lessons Learned..................................................................................................................................................................................................11

Outcome Summary .................................................................................................... 11

Authors and Key Partners ......................................................................................... 11

Author........................................................................................................................................................................................................................11 Contributing Authors .......................................................................................................................................................................................11 Vendors .....................................................................................................................................................................................................................12

Contact Information................................................................................................... 12


Case Study – Success Story 1

Executive Summary Thales Rail Signalling Solutions Inc. (TRSS) is a world leader and pioneer in critical railway applications, designing, supplying and installing advanced signaling systems to control railway traffic with maximum safety. After the sale of the rail signaling business from Alcatel to Thales in 2006, TRSS initially worked with an external provider for hosting SAP solutions. We (TRSS) chose this provider based on competitive pricing, expertise, and the ability to migrate our data within three to four months. For two years, this external provider hosted our SAP R3 4.6c, BW 7.0, and Portal 6 application solutions running on an HP-Unix/Oracle platform stack.

In 2008, the external provider TRSS was using informed us they were no longer providing hosting services and that we would have to look for an alternative SAP hosting solution. After considering several hosting options, we decided to bring the environment in-house, moving to the Microsoft Windows Server 2003 Datacenter x64 Edition operating system and the Oracle 10 Database running on VMware® Infrastructure.

After a successful migration to this in-house solution, we realized the following benefits:

• A 40 percent cost savings

• Up to four times performance improvement

• Higher availability - 100 percent availability since go live (six months as of this writing)

• Full redundancy

• Cost-effective disaster recovery options These benefits were achieved on systems running VMware Infrastructure, Windows 2003 64-bit operating systems, Oracle 10 Databases, and the SAP R3 4.6c application.

The Business Need The key requirements enforced by the migration to the in-house solution were:

• It had to be competitively-priced.

• It needed to include a disaster recovery solution - something not included with the previous external hosting contract.

• Due to privacy and security concerns, hosting needed to be located in Canada and be SAS-70 compliant (or equivalent).

Hosting Solutions Considered

During our project planning phase, we evaluated several hosting options. In addition to costs, we also considered criteria such as risk, support availability, and platform stability. The following table outlines our evaluation of the advantages and disadvantages of each option.



Hosting Solution Advantages Disadvantages

Hosting with External Provider:

Frequently not based in Canada

All had much higher costs than the existing hosting provider

In-House Solutions:

Staying with HP Unix (on standalone servers)

Lower risk (OS migration not required)

Very good industry reputation for SAP

OS stability (viruses, security)

Best option to migrate within a short timeframe

No in-house expertise with HP Unix

Higher cost compared to other solutions

Low flexibility (high cost to add additional servers)

High cost for disaster recovery

Moving to 64-bit Windows and Oracle DB (on standalone servers)

In-house expertise

Lower cost than HP Unix

Accepted industry solution

Perceived OS instability (viruses, security)

Lower flexibility (high cost to add additional servers)

Moving to 64-bit Windows and Oracle DB on VMware Infrastructure

In-house expertise

Significantly lower costs than any other solution

Minimal hardware (physical servers)

Full server redundancy

Lowest cost disaster recovery (minimal hardware)

Sharing of hardware resources (CPU/memory)

High flexibility – low cost to add additional servers

Perceived lack of maturity running SAP and Oracle on VMware Infrastructure

Uncertainty around sizing (limited availability of guidelines)

Oracle support statements conditional

The Business Solution After careful consideration, we decided to bring the hosting solution in-house using VMware Infrastructure and the Windows operating system.

Key Solution Drivers

Key drivers for this decision were:

• Cost effectiveness.

• Extensive in-house expertise with Windows and Oracle databases.



• External expertise

- Illumiti had extensive experience with SAP migrations.

- Syscom Consulting had extensive experience with VMware and IT/Server Infrastructure. • Executive support from Thales’ senior management.

• Disaster recovery and redundancy options were available.

• Centralized management and monitoring tools.

• Flexibility, with options to quickly add new virtual machines.

• Minimal costs to add software and hardware.

• VMware Snapshot feature allowing for fast rollbacks.

• Ability to virtualize the network within VMware Infrastructure (supports VLAN tagging of the virtual NICs).

Risks Involved

To lower project risk, we decided that Oracle Database would remain as the database platform, regardless of the end solution. As a result, we would upgrade all Oracle databases to Oracle 10G.

Moving from UNIX to a Windows operating system was also a potential risk due to perceived stability and performance issues. We recognized a few factors that would mitigate this risk:

• Windows product maturity.

• Internal and external expert resources were readily available to support the platform.

• Solid patch management, change control and virus protection processes were already in place for other Windows-based solutions at TRSS.

• Proven successful cases of moving from UNIX to Windows at other companies.

• SAP Zero Administration Memory Management capabilities under Windows. Running SAP 4.6C on VMware Infrastructure under Windows also added risk. A phrase we heard often was “It should work.” Adding to our concern was the fact that few SAP customers we knew had implemented VMware Infrastructure in their production environments at that time.

Oracle support seemed conditional on proving that any problem reported was not related to VMware Infrastructure (see Oracle note 249212.1). This was a concern as we were looking for a stable and reliable platform.

The keys to mitigation of these risks were:

• Assurance from VMware personnel that they would support us during the transition and that they had other customers running similar environments.

• Our own extensive and positive experience running Windows on VMware Infrastructure.

• VMware Infrastructure provided full redundancy and disaster recovery at a much lower cost than any other solution.



• The added flexibility of adding virtual machines in the future.

• This environment was supported by SAP for future SAP versions and we were planning an upgrade to the latest version of SAP (ECC6).

Solution Migration

The project involved migrating software and data from the hosted servers to new servers at our location, with an OS migration and a database upgrade. The team originally estimated a project completion timeline of three months.

To mitigate the identified risks, we decided that the first step in the migration plan would be the creation of a sandbox environment. This allowed for the migration of the SAP environment into a standalone environment where end users could test to ensure the effectiveness of the system migration.

We also determined that the best approach for the migration was to perform a full test in the target production environment (VMware Infrastructure environment installed on production hardware). This test, required by the SAP Migration Check, was crucial for validating the final production environment.

Design Considerations

Choosing the number of VMware ESX physical hosts:

• We chose two physical VMware® ESX hosts using the latest x64 hardware from IBM (model X3850 M2). The key was to ensure the hardware supported enough RAM and CPU on each host for current needs, while accommodating future growth and providing the ability for a single ESX host to run all our SAP instances efficiently.

Determining CPU resource allocation for each server:

• Choices included assigning 1, 2, or 4 virtual processor cores to a single virtual machine. In the end, we assigned 2 CPUs for each production server based on VMware best practice guidelines for SAP solutions, which advise starting with smaller-sized virtual machines and adding vCPUs as required.

• A single processor was not sufficient for servers running a database, as this would cause a bottleneck for backups. As a result, all database virtual machines were assigned a minimum of 2 virtual CPUs.

SAN LUN and Disk Configuration:

Configuration setup included:

• LUN locking

• Proper LUN and disk configuration

• Virtual vs. direct-to-disk LUNs



We discovered that we needed more assigned LUNs due to performance issues as a result of LUN locking. For this project, we classified LUNs into two types under VMware Infrastructure:

• “Virtual LUN” – uses what VMware calls “Virtual Machine File Systems” (VMFS); assigned to multiple virtual machines.

• “Direct LUN” – uses what VMware calls a “Raw Device Mapping” (RDM), or could use a VMFS with a single LUN and single virtual machine assignment.

For a complete review of best practices for running SAP solutions on VMware Infrastructure, see the VMware white paper titled Best Practice Guidelines for SAP Solutions on VMware Infrastructure here: http://www.vmware.com/partners/alliances/technology/sap-whitepapers.html.

The following table lists the LUN configuration we used for a typical server:

Drive Purpose SAN LUN # Direc t/Vir tual VMFS/RDM

C OS 1 Virtual VMFS

D Oracle DB / BIN 2 Direct RDM

E Oracle Archives 1 Virtual VMFS

F Swap File 1 Virtual VMFS

H Oracle Redo Logs1 3 Direct RDM

I Oracle Redo Logs2 4 Direct RDM

J Oracle Redo Logs3 5 Direct RDM

K Oracle Redo Logs4 6 Direct RDM

Note there are a maximum of 256 LUNs per physical VMware Infrastructure host. In general,in our opinion it would be ideal to have a Direct LUN for high I/O use and a Virtual LUN for lower I/O use.

SAP 4.6c is NOT available for Microsoft Windows x64:

• 64-bit SAP 4.6c is only available on Windows Itanium (not Windows x64)

• VMware Infrastructure is not supported on Windows Itanium (only Windows x64)

Running SAP 32-bit Kernel on 64-bit Windows OS with 64-bit Oracle:

• As a 32-bit program cannot call a 64-bit program directly, we had to ensure the SAP database tools and Oracle client were 32-bit and at the correct version levels.

• Understanding memory utilization of the 32-bit SAP kernel running in a 64-bit Windows OS was challenging. However, there were advantages over a strictly 32-bit Windows, Oracle, and SAP install.



- Under Windows x64, 32-bit programs automatically get the maximum 4GB memory assigned. (Under Windows 32-bit, this would have been a maximum of 3GB with the /3G switch in the boot.ini file.) Note that the /3G switch is not needed in Windows x64.

- 64-bit Oracle had access to memory above the 4GB 32-bit limit and, as a result, we used 64-bit Oracle with the 32-bit Oracle client.

• It was more difficult determining how to set up the SAP and Oracle parameters as a result of

the mixed 32-bit and 64-bit environment.

Making the export work:

• Sizing the export files correctly was critical. We found that under SAP 4.6c, the export files cannot be too large so they were set at 250MB each.

• SAP 4.6c had to be imported into 32-bit Oracle 9 and then upgraded to 64-bit Oracle 10. Attempts to go directly to 64-bit Oracle 10G failed.

Solution Performance

One of the key requirements for the new solution was adequate performance. User testing showed up to a fourfold increase in performance compared to the previous hosted solution. The following section lays out settings and performance statistics for each of the virtual machines:

Primary R3 (SAP 4.6c) Central Instance Server (with Oracle DB)

• This virtual machine showed the most consistent load and did not exhibit any signs of significant bottlenecks. Select performance details of this server are discussed here as a reference example.

• The virtual machine is allocated two vCPUs and 16 GB RAM

• CPU Performance

- Business hours

• Utilization averages less than 10 percent with spikes to 25 percent

• CPU Ready reaches maximum of 2,000 ms, while CPU Used is 60,000 ms

=> CPU Ready/CPU Used = 3.3 percent

- Non-business hours (during backup)

• Utilization averages 75 percent from 8pm to midnight

• CPU Ready reaches maximum of 12,000 ms, while CPU used is 200,000 ms

=> CPU Ready/CPU Used = 6 percent

These results indicate that allocating two vCPUs is the proper choice for this virtual machine to meet the 75 percent processor utilization needed during non-business hours. Note that allocating four vCPUs would have increased the contention between virtual machines for CPU resources and increased the CPU Ready Counter, so virtual machine response time could have



been impacted. The results also indicate that response time of the virtual machine is similar to the response time of a similarly-configured physical Windows server.

• Disk Performance

- Business hours

• Utilization is light, reaching maximum of 3 MBps throughput to virtual machine disks

• Allocation of 16 GB of memory to the virtual machine results in most database activity being cached in memory and little need to access disk

- Non-business hours

• Utilization is fairly consistent at 15 MBps from 8pm to midnight • Network Performance

- Business hours

• Negligible – averages 200 KBps with spikes to 3 MBps

- Non-business hours

• Utilization is fairly consistent 15 MBps from 8pm to midnight

Primary Production VMware ESX Host (Physical Server)

The four most significant production virtual machines were placed on one VMware Infrastructure host to isolate them from the Dev and QA virtual machines.

• CPU Utilization

- Business hours – normally 10 pecent to 25 percent

- Non-business hours – 40 percent from 8pm to midnight

• Disk Utilization

- Business hours –utilization is light, reaching maximum of 4 MBps throughput and an average of 2.5 MBps

- Non-business hours – 35 MBps throughput from 8pm to midnight

• Network Utilization

- Business hours

• Negligible, 100 KBps average.

Note: Communication between the virtual NICs of the four virtual machines is via memory, and it does not create load on the six physical NICs of the host. In situations of high ‘network’ communication loads between virtual machines, this can have a significant benefit.

• Non-business hours – 30 MBps throughput from 8pm to midnight

Key Project Success Factors

• The right consulting partners (Illumiti and Syscom Consulting)



• Good contacts and support from the vendor (VMware)

• Excellent Support from SAP and the hosting provider

• Choosing the right hardware (IBM Model x3850 M2)

• Choosing the right project manager. One who was:

- A detailed planner

- An excellent communicator

- Detail-oriented

• Obtain an SAP Certified Migration Consultant

• Highly qualified and flexible resources both internal and external

• Integration testing in the target production environment

• Documenting the build process accurately

• Effective communications across all involved teams and all levels, and not just within the team

• Solid patch management and virus protection process; we reduced the number of changes to minimum during project to help minimize risks

• Solid change management process

• Following best practices from SAP and VMware

- VMware SAP Best Practice Guide:

http://www.vmware.com/files/pdf/whitepaper_SAP_bestpractice_jan08.pdf

- SAP Service Market Place Installation Guides and Notes:

http://service.sap.com



0 1 2 3 4 5 6 7 8 9 10 11 12 13P WR A LARMWARN®

0 1 2 3 4 5 6 7 8 9 10 11 12 13P WR A LARMWARN®

Figure 1. High Level Landscape – Before and After



0 1 2 3 4 5 6 7 8 9 10 11 12 13PWR ALARMWARN®

0 1 2 3 4 5 6 7 8 9 10 11 12 13PWR ALARMWARN®

Figure 2. New Landscape – Detailed



Lessons Learned

• Register with SAP for a migration check early on in the process.

• Remind users of upcoming downtime repeatedly, using multiple methods of communication.

• Allow more time for sizing (sizing is critical).

• Do not underestimate the amount of basis work required to complete the project.

• Ensure that you engage a good migration consultant.

• Follow SAP and VMware best practices closely, particularly at the beginning of the project, and especially for VMware LUN Configuration.

Outcome Summary The project was a complete success. The team was able to execute the migration on time and within budget. The following benefits were realized:

• Expected

- Flexibility – on the fly re-allocation of system resources

- Significantly lower costs

- Improved security

- High availability (and redundancy)

- Much smaller footprint (two physical servers versus eleven physical servers)

• Unexpected

- The extent of the performance improvements achieved; performance testing showed a performance gain of up to four times original .

- No Issues encountered during integration testing!!

- Seamless “Go Live” launch!

- User feedback has been very positive, especially on performance.

- 100 percent availability (No down time – now six months after “Go Live”).

Authors and Key Partners

Author

Joe Moore: [email protected]

Thales Rail Signalling Solutions Inc.

Contributing Authors

Awni Koudmani: [email protected]



Illumiti Inc.

Syscom Consulting Inc.

Vendors

Two key vendors assisted with this migration:

• Illumiti Inc. is a North American systems integration and management consulting firm that is also an SAP Gold Level Channel Partner. Illumiti successfully managed the migration of our SAP environment to use 64-bit Windows and the Oracle Database on VMware Infrastructure. This included SAP R3 4.6c, BW 7.0, and Portal 6 migrationn.

• Syscom Consulting Inc. is an external IT provider. Syscom managed the installation and configuration of VMware Infrastructure, virtual machines, backup and related IT infrastructure.

Contact Information Thales Rail Signalling Solutions Inc.

1235 Ormont Drive

Toronto, Ontario

M9L 2W6

Tel: 416-742-3900

Illumiti Inc.

15 Allstate Parkway

Suite 600

Markham, Ontario

L3R 5B4

Tel: 905-415-4015

Syscom Consulting Inc.

1000-1090 W Georgia Street

Vancouver, B.C. V6E 3V7

Tel: 604.684.5344

Documents

SAP Solutions on VMware Infrastructure – White Paper ... · location, with an OS migration and a database upgrade. The team originally estimated a project The team originally estimated