TESTING AND OPTIMIZING WE APPLIATION’S PERFORMANE · Oracle Web Cache Microsoft Windows Server 32bit Oracle Web Cache 10.2.2 Oracle AS RedHat Linux ES4 U6 Oracle Application Server

TESTING AND OPTIMIZING WEB APPLICATION’S PERFORMANCE

AQA CASE STUDY

Intro to Load Testing2

Copyright ©2009 TEST4LOAD – Software Load Test Experts

What is Load Testing?

Load testing generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program's services concurrently.

As such, this testing is most relevant for multi-user systems, often one built using a client/server model, such as web applications.

Copyright ©2009 TEST4LOAD – Software Load Test Experts 3

Why Load Test Your Application?

High cost of mission-critical application failure

Simulate real-life workload conditions

Find boundaries of your system

Locate potential problems before your customers do

Reduce development cycle and time-to-market

Predict and reduce infrastructure costs


Load Testing Goals

Performance Ensure your application’s performance under basic

workload (baseline) Stress Find break points and boundaries of your applicationReliability How stable is your application under extended period

of loadScalability Determine how your application scales for the

workload growth


Load Testing and Product Life Cycle


• Define performance requirementsRequirements

• Design your system based on performance requirementsDesign

• Load test separate componentsDevelopment

• Load test a whole systemTesting

• Monitor your system’s performanceProduction

Load Testing Workflow

Plan Script Test Analyze Improve


AQA Case Study8


Background – AQA

The Assessment and Qualifications Alliance (AQA) is the largest of the three English exam boards.

Awarding 49% of full course GCSEs and 42% of A-levels nationally.

Processing over 3.5 million exams each year.

Providing online services to teachers, exams officers, examiners and students.


SUT – ERA (Enhanced Results Analysis)

ERA is a free on-line tool that gives teachers in UK an instant analysis of exam results.

ERA’s key features:

Grades reports by centre and subject

Cross-centre and years results comparison

Individual students grades analysis

Question by question analysis


The Challenge

Database size over 100 million records

10 million active records

70,000 users


22/08/2007 - Requests per 10 minutes

0

500

1000

1500

2000

2500

3000

00:00

01:00

02:00

03:00

04:00

05:00

06:00

07:00

08:00

09:00

10:00

11:00

12:00

13:00

14:00

15:00

16:00

17:00

18:00

19:00

20:00

21:00

22:00

23:00

14/08/2007 - Requests per 10 minutes

0

500

1000

1500

2000

2500

3000

00:00

01:00

02:00

03:00

04:00

05:00

06:00

07:00

08:00

09:00

10:00

11:00

12:00

13:00

14:00

15:00

16:00

17:00

18:00

19:00

20:00

21:00

22:00

23:00

Workload of less than

10 pages/min at usual

day

Pick of 300 pages/min

at results day

12

Deployment Architecture


13

Test Architecture


Server Configuration

Oracle Web Cache Microsoft Windows Server 32bit Oracle Web Cache 10.2.2

Oracle AS RedHat Linux ES4 U6 Oracle Application Server 10.3.3 ChartFX 6.2

Oracle RDBMS RedHat Linux AS4 U7 Oracle RDBMS 10.2.0.3

Identifying Test Scenarios

Pick 5 most common scenarios in your system

Don’t add scenarios which operate similar back-end flows

Identify problematic back-end flows ahead and make sure scenarios cover them.

14


Identifying Potential Bottlenecks

Search Usually sensitive to various input values.

Simple or not implemented paging can cause to slow response time

Search Results If paging is not implemented, result page can be very large

Back to search results page from item details can execute search again

External and 3rd Party Components 3rd party components can cause to deadlocks or poor performance

OSSO server can perform slower than the rest of the system

Export to Excel Usually returns large result set. Can cause to slow performance.


Execution Plan

Test the system with “light” input parameters first

Get through the “obvious” bottlenecks, constantly increasing the workload

Run nightly tests every night to ensure reliability and stability of the system

At level of 500 users, apply “heavy” input parameters

Test system for randomly distributed workload


17

10 Users – Conn. Pool Starvation


0%

20%

40%

60%

80%

100%

Connection Pool

Connections

Measurement Avg./Sum Max.

Page Time 1.6 sec 34 sec

HTTP 200 3,348

HTTP 304 2,190

HTTP 302 410

HTTP 500 48

18

100 Users – Apache Shows Its Limit


Measurement Avg./Sum Max.

Load Size 100

Page Time 5.6 sec 67 sec

Pages per Second 2.1 7.5

HTTP 503 8

19

500 Users – Slow DB & OSSO


Measurement Avg/Sum Max.

Load Size 1000

Page Time 24 sec 2.5 min

Pages per Second 7 19

HTTP 500 368

HTTP 503 157

1000 Users – Search & Large Content


Measurement Average/Sum

Max.

Load Size 1000

Page Time 1 min 7.5 min

HTTP 500 27

HTTP 503 626

21

Concluding 1000 Users 8 hrs Test


Measurement Average/Sum

Max.

Load Size 1000

Page Time 0.717 sec 6.169 sec

Load Issues Pyramid

10 – Conn. Starvation

100 – Apache Limit

500 – Slow DB Queries

1000 – Code, Search, Cache


Achieved Results

Supported concurrent sessions/users:

From 10 to 1000 (x100)

Reliability

From 90% failure to ~100% reliable system

Response Time at 1000 VU

From 1 min to less than a second

Pages per Minute

From 13 PPM to 166 PPM


Tips for Effective Load Testing24


Optimizing Search Mechanism

Use existing solutions.

Use free text search only as the last option.

Limit user input to reasonable values.

Implement paging on both search and result levels.

Improve “Back to search results” mechanism.


GZip Compression Effectiveness

Pros: Reduce size of large text pages by 80%-90%

Improve response time by 50%

Decrease throughput

Cons: Increased CPU utilization by 35%

Tip: Apply GZip selectively to text-based static resources and

large dynamic pages

Do not apply GZip to every response!


Database vs Web Servers Costs

Additional database setup and maintenance costs are significantly higher than of web server.

Make DB work easier by making web server work harder.

Memory costs less than CPU or new hardware – rely more on memory, but keep in mind that everything has its limits.


Developers and Testers Tips

Get to know your user.

Evaluate his system usage behavior or try to predict the common flow.

Know the “costly” operations of your system. Try to build the flow that omits recurrent usage of the operations.

Do not be reluctant to cache data, still don’t overuse the cache. Remember that shared data should be stored once for all users.


Project Manager Tips

Make sure all involved parties are available at the moment of test execution.

Share test findings among all project members.

Log all changes to the test environment. You will need to reapply them later to the production environment.

Make sure the final results meet the performance requirements and business user expectations.


Team Synergy

Synergy and cooperation between the following teams is crucial for the success of the test:

Database Team

Application/Development Team

System Team

Performance Team or Specialist


Nachum (Nick) Dimer

Owner & Chief Consultant

IL: +972.54.4600515

UK: +44.783.8673770

PL: +48.519.717875

[email protected]

www.test4load.com

Contact Info

Questions?

Documents

TESTING AND OPTIMIZING WE APPLIATION’S PERFORMANE · Oracle Web Cache Microsoft Windows Server 32bit Oracle Web Cache 10.2.2 Oracle AS RedHat Linux ES4 U6 Oracle Application Server