04-2010_SAPinsider

7/31/2019 04-2010_SAPinsider

http://slidepdf.com/reader/full/04-2010sapinsider 1/4

Susanne Janssen , SAP AG an d Fayçal Had j ia t , EDF

column | Performance & Data Management Corner

Subscribe today. Visit sapinsider.wispubs.com.

How to Avoid the Most Common Load Testing MistakesSecrets to Success from EDF’s Massive Load Test Project

Susanne Janssen(susanne. [email protected]) has been amember o the SAP Per ormanceand Scalability team at SAP AGsince 1998. She specializes inper ormance methodologies andprocedures, as well as hardwarecapacity planning. In this role,she spent time at EDF during thefrst stage o its load test series.

For any organization that’s planning a new systemimplementation or upgrade, load tests are vital.These tests validate whether or not a system canhandle the expected uture load required to supportthe company’s evolving business needs. While there

are many resources that detail how to per orm loadtests, we’ve ound that certain aspects o load test-ing are too o ten overlooked, causing project delays,uncertain results, and increased costs.

Most people think that the most important criteriaor a success ul load test project is whether targetedkey per ormance indicators (KPIs) are met. But anothercrucial criterion is how well you handle the series osystem and coding optimization recommendationsthat the load test yields — a actor which can also in u-ence how well you meet your KPIs. I properly under-

stood and implemented, these recommendations arepower ul instruments to ensure solid, long-term per-ormance within the production system.

So how can you make sure that your companymeets its target KPIs during load testing — and, moreimportantly, once your system is in production? Werecommend a two-pronged strategy: Properly pre-pare your test system data and conduct extensiveunit or single user tests be ore you even begin yourull load tests, and understand how to sustainablyhandle the per ormance-optimizing recommenda-

tions that your load test will generate.To help illustrate the benefts o this dual approach,

we’ll ollow the example o one utilities company,Electricité de France (EDF), that has executedextremely success ul and productive load tests as aresult o ollowing this approach (see sidebar on thenext page). Throughout this article, you’ll fnd tips andbest practices rom EDF to help others ollow its lead.

See www.service.sap.com/performance and www.service.sap.com/tmcfor more information.

S t Up Y ur L ad T sts f r Succ ss:Plann n and Pr parat n T psTo get the best results rom your load test, planningand preparation are key. To start, you’ll need to builda team o business owners, SAP unctional experts,

SAP technology experts, and per ormance experts.This team will then be responsible or laying thegroundwork or the load tests by, or example:

Determining the scope o the tests

Determining which business processes need testing

Determining the timeline o the project, the KPIs,and whether additional experts will be needed

Managing interdependence with the go-live andunctional development cycle

Identi ying the type o data to load into the test

systemSetting up unit testing or single user testing

What ollows are key considerations or each phase.

Increase Testing Efficiency ThroughProgressive ScopingOne precondition or success ul load testing is defningwhat you want to test. To do this, you must frst deter-mine and set the scope o the test. This involves:

Inventorying available resources — includinghardware, so tware, teams, and support

Defning test objectives and monitoring indicators

Assessing what is required to achieve your objectives

Documenting the decision process or determin-ing which processes should be tested

In EDF’s case, the team needed to test many di er-ent aspects o per ormance and scalability to ensurethat the system could handle the fnal volume tar-gets. That’s why EDF decided to run a series o loadtests, each with a di erent scope.

Fayçal Hadjiat( aycal.hadjiat@ed . r) is project manager at EDF,where he’s worked or 10 years.He leads mass-market per ormanceinitiatives to ensure that EDF willmeet its goal o accommodating28 million business partners by2013. He is currently in theplanning phase or this project’snext major test stage, thetechnical in rastructure tests.

This article appeared in the Apr n May n Jun 20 0 issue o SAPinsider (http://sapinsider.wispubs.com )and appears here with permission rom the publisher, Wellesley In ormation Services (WIS).

http://sapinsider.wispubs.com/

mailto:[email protected]






















7/31/2019 04-2010_SAPinsider



The goal o the frst load test (conducted in 2009),which represented 10 million business partners ,was to identi y potential application bottlenecksor the target volume. This test ocused on thecompany’s most critical business processes andtested the majority o custom-developed coding.In addition, the tests helped to analyze the layouto the architecture to see i it could serve as a basisor the target architecture.

The goal o the second load test (to be conductedthis year), which represents 20 million business

partners , is to check the in rastructure’s e fciency— this includes testing how it interacts with SAP andnon-SAP systems, the height o its availability, andits ail-over limit, or example. EDF will also test aew unctional modifcations — including imple-mented recommendations rom the previous test

phase — to analyze their per ormance impact.

The goal o the third load test (to be conductedin 20 ), which represents 28 million business

partners , is to veri y whether the productionsystem will be able to handle the fnal target load.

When dealing with such large volumes o usersor with such a large-scale implementation, we

.

2.

3.

recommend splitting your load tests in such a waythat you can more e ectively address each set ochallenges. This approach allows you to progressivelyimplement improvement recommendations and thenvalidate them in the next round o tests.

Run Your Tests on Representative DataBecause the goal o load tests is to provide solid rec-ommendations or how to improve and optimize theproduction system, the load test system itsel shouldre ect the data layout o the production system asclosely as possible. In addition, matching the testingsystem and the production system will improve yourprecision when sizing the production system. Todetermine what kind o data needs to be loaded intoyour system, the project team will need to workclosely with the unctional experts, who will contrib-

ute valuable insight into the kind o data that the pro-duction system will contain.

EDF took great e orts to load real, representativemaster data and transactional data, and then “age”this data in the per ormance test system, so that itsdatabase would be similar in all aspects to the utureproduction system.

Focus on Testing Critical Business ProcessesAnother way to limit the complexity o your load testsis to ocus only on critical business processes. In some

cases, these processes will be clear — a bank wouldlikely want to ocus on account balancing calculation

jobs, or example. But in other cases, determiningthese processes might require urther analysis.

For instance, dominating EDF’s business load aremultiple batch processes that run at night, some owhich can be run in parallel and some o whichdepend on the completion o a previous job — that is,they’re on the “critical path.” To determine exactlywhich processes are on the critical path, EDF ana-lyzed its current production system to fnd the execu-

tion time o the processes in relation to the amounto data processed and the parallelization parameters(the number o parallel jobs or the time intervals, orexample). This analysis helped them identi y whichprocesses needed special attention at the load test-ing stage, as well as which processes would needspecial attention in the new system.

Run Unit or Single User Tests Before Load TestingOnce you have identifed which processes you wantto test, prepared the test system, and set up the

EDF’s Load Test Project: An Introduction to Its Size and ScopeIn the context o the deregulation o the energy market in Europe,Electricité de France (EDF) decided to replace its numerous utilities’ legacysystems with SAP or Utilities, using SAP ERP 6.0, SAP CRM 5.0, andSAP NetWeaver Business Warehouse 7.0.

This project was frst conceived in 2003. Its goal was (and is) to accom-modate 28 million business partners rom nearly 00 legacy systems on thenew SAP landscape by 20 3. As o December 2009, 5 million business partnershave already migrated. These migrations are done in steps involving 500,000to million business partners at a time. To ensure that its system in rastruc-ture can handle this high load, EDF decided to closely cooperate with SAPActive Global Support (which they engaged through a SAP MaxAttention

contract), as well as with their application partners and per ormance expertsrom EDF’s database, operating system, and storage providers.

With the goal o reacting exibly to any changes in technology or requiredbusiness unctions that might come up during the implementation, andmind ul o the large load that the in rastructure would need to handle, EDFdecided to per orm three cycles o tests, one each year, each with a di erentscope. EDF has fnished the frst test cycle, simulating the load o 0 millionbusiness partners (roughly one third o the target volume). In 20 0, EDF willsimulate the load o 20 million business partners; and in 20 , EDF will simu-late the load o 28 million business partners.




7/31/2019 04-2010_SAPinsider



scripts, you can begin the testing phase. To improvethe results o your load tests, SAP recommends run-ning initial tests, either as unit tests — that is, oneprocess step at a time — or as single user tests , inwhich one user or click stream is monitored or per-ormance KPIs prior to the load tests. Why? Well, oryour load test to produce meaning ul optimizationsuggestions, you’ll need to re-run your test cases sev-eral times to identi y the root cause o the per or-mance problem, and then validate the impact o thesuggested solution.

These initial tests reveal areas o improvement,many o which — such as minor changes to system set-tings, customizations, or coding — can be put intopractice quickly. Other recommendations may revealunctionality-related problems, architectural issues,or severe changes to the coding, which the company

can address and then validate during load testing.Let’s revisit our EDF example. Be ore EDF started

its “stage one” load tests, the individual process stepswere tested as separate unit tests. The team wasthen able to analyze and optimize the per ormanceo each process.

Then, using the data rom these unit tests, EDFcould determine the optimal system settings and theprecise combination o parallelization parameters —including number o jobs, processing interval, andpacket size — to use in the load tests to render the

highest amount o throughput or each process.Thanks to unit testing, EDF identifed several

bottlenecks be ore even reaching the load testingphase and was able to optimize its coding, thus cut-ting the batch night process duration in hal ( rom 30hours down to only 3). EDF reduced the duration othe batch night process even urther a ter the loadtesting phase (down to 9 hours), a eat that may nothave been possible without the changes made duringthe unit testing phase.

Us Y ur L ad T st R sults W s ly:Handl n R c mm ndat nsA ter your load test, the experts who ran the tests,monitored the system, and analyzed the processeswill generate a list o recommendations to improvethe system and optimize per ormance. A very simplerecommendation could be to increase the table bu erby 500MB, a parameter modifcation that can be doneon the y and will only impact available memory.More complex recommendations are those that a ectcustom coding or modifcations to the SAP standard

— these require urther testing once implemented. Tobest manage these recommendations, you’ll need toestablish an e ective procedure or evaluating theimpact o each recommendation, as well as its poten-tial requirements or urther coding and test e orts.This procedure must include methods or quali yingand prioritizing the recommendations, as well as amechanism to integrate resulting improvements into

the production rollout.

Prioritize Your RecommendationsBe ore you can implement the recommendationsrom the per ormance test team, you’ll need toquali y and prioritize them. An easy way to do thisis to set up a matrix (see Figure 1 on the nextpage). This matrix would help you easily see howyour recommendations are ranked in terms o easeo implementation, pervasiveness, unctional impact,additional research required, and so on.

4 NoTe!

S e f the i pr ve-e ts that a it

test ide tifies ighttake weeks r eve

ths t address,s their be efit wit be i ediate y

visib e d ri g thead test.

EDF’s Tips for Unit TestingTake time to present the algorithm and unctionality o each tested pro-gram to the entire team o per ormance experts. When everyone is on thesame page, the team can ocus on essential activities and won’t have toworry about redundancy.

The team that per orms the unit test should also do the load testing, sincethey best know the history and understand the interdependencies.

Involve unctional experts to review the business aspects o the load testscripts. A small unctional problem in the script can distort results.

In case o Z-coding, involve the development team so that they can ostercoding optimization. The team can quickly estimate the e ort to implementa recommendation into the coding and may point out interdependencies.

EDF’s Tips for Managing Load Test RecommendationsEstablish a test coordinator to manage prioritization and urther handlingo the recommendations.

Set up a small, additional test system to deal with recommendations thatwill take longer to implement — those that could a ect unctions andapplication coding, or example.

Since some tests will need to be reproduced several times to determine wheth-er the recommendations are being implemented success ully, it’s a good ideato have a system backup strategy in place so that the conditions concern-ing system settings and data structure or each test cycle are identical.

Allocate adequate time to implement recommendations — this includesleaving enough time or in-depth analysis o more complex recommenda-tions, which can be done in parallel to the testing.




7/31/2019 04-2010_SAPinsider



At EDF, the di erent experts (business application,system, storage, database, and SAP applicationexperts) sat in the same room so that everyonewas always on the same page. EDF also appointedan independent test coordinator, whose job was tomitigate a common agreement on the applicabilityo the recommendations issued by the di erentparties involved. This coordinator was empowered

to decide how critical recommendations should bedealt with, or example, by demanding additionaltests and validations.

Consolidate and Integrate RecommendationsSome o the recommendations will require additionalunctional development work be ore they can be imple-mented. The per ormance test team, there ore, mustwork closely with the application development team toclearly communicate what the recommendations entailand to test any completed development work. To acili-tate this and stay organized, the per ormance test team

should categorize the recommendations — as unc-tional or per ormance-related, or example.

A close relationship with the application develop-ment team can also help i you come across a prob-lem and cannot easily fnd a solution. In this case,you’ll need to run proo -o -concept tests to helpdetermine viable recommendations.

C nclus nThe true objective o any load test is to ensure thatyour production system will run smoothly at go-live.

As demonstrated by EDF’s success ul load testingstrategy, the keys to reaching this objective are two-old: Rigorously prepare or load tests by setting uptruly representative test scenarios on a production-like test system, and take great care to ensure that therecommendations obtained during the tests are notonly o high quality, but are sensibly addressed.

To learn more about SAP’s services, which can helpguide your company through success ul load tests,visit service.sap.com/tmc . n

Job # Problem Description and Impact Status Suggested Solution Timeline

Job 1 Runtime o the job too long.

Impact o data storage: very high.

Error message number: 23456

j Modi y standard coding — responsibility o partner A.

Evaluate impacts o tests planned or calendar week X:Runtime (- 0%) Error log fles written (-90%)

Done

Job 2 Parallelization not possible.

Impact on business need: high.

Error message number: 65432

l SAP suggests a custom development.

Business to decide i they want it.Contact person: Jane Doe

Within twoweeks

FigURe 1 p Exa p e f ade isi a d f w- patrix

EDF’s Tips for Qualifying and Prioritizing RecommendationsRank each recommendation by priority, complexity, and potential gain —this can be in terms o either per ormance or unctional aspects.

Set deadlines and keep track o the recommendations’ ulfllment status.

Decide where the recommendations should be implemented frst — or exam-ple, determine whether to implement them in the load test system immedi-ately or to test them in a separate system frst to mitigate potential risks.

Figure out i implementing the recommendations will require additionalvalidation tests.

EDF’s Tips for Integrating Recommendations into the OverallDevelopment Process

Make sure that the application development team allocates su fcient timein their schedules to implement coding modifcations.

Together with the person in charge o the development cycle, defne amethod or handing over recommendations to the per ormance test teamand or validating improved coding.

With the unctional experts, prepare a recommendation consolidation.This list should include in ormation on the potential impact o implement-ing your requirements. You can then pass this list on to development.

Work closely with the application development team to conduct proo -o -concept tests or problems or which there are no readily apparent solutions.

For quality assurance, include a per ormance expert in the developmentassessment.

...from

“A Custom Per ormance Test — Is It orYou?” by Susanne Janssen and Dr. UlrichMarquard ( SAP Professional Journal , May/June 2000, www.SAPpro.com )

Sizing SAP Systems by Susanne Janssenand Dr. Ulrich Marquard (SAP PRESS;www.sap-press.com )

AdditionalResources...


http://service.sap.com/tmc







Documents

04-2010_SAPinsider