Combining Performance Testing And Modelling For Easy Jet.Com V 1.0

Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management

Combining Performance Testing and Modelling for easyJet.com


Danny Quilton, COO, Capacitas



Agenda

• Business Context

• easyJet’s Goals

• Challenges with Agile

• How can we Reduce Risk?

• Performance Testing

• Performance Modelling

• What you need

• Costs

• Benefits

© Capacitas 2002-2009 2



About Capacitas

• UK-based Capacity and Performance Management specialists

• Training and professional services

• Hardware-vendor independent

• Tools independent

© Capacitas 2002-2009 3



Business Context

© Capacitas 2002-2009 4



Business Context

• Capacity planning service www.easyJet.com

• Business critical service

• 98% of revenue earned through the e-commerce service

• Highly competitive market

• Frequent innovation by adopting agile development

• Monthly software releases

• Legislative changes

• Consistent growth; typically 20% p.a.

• Low-cost operating model

• Requirement to manage risk

© Capacitas 2002-2009 5



easyJet’s Goals

6© Capacitas 2002-2009



easyJet’s Goals

7© Capacitas 2002-2009



Agile Development Process

© Capacitas 2002-2009

Product Backlog

Sprint Planning Meeting

Sprint Backlog

Release

Sprint(15-30 days)

INOUT

SprintRetrospective

Meeting

Scrum(Daily

Meeting)

LESSONS LEARNED

Pre-release Activities(e.g. Training, Documentation)

Sprint Review

Product Backlog

Sprint Planning Meeting

Sprint Backlog

Release

Sprint(15-30 days)

INOUT

SprintRetrospective

Meeting

Scrum(Daily

Meeting)

LESSONS LEARNED

Pre-release Activities(e.g. Training, Documentation)

Sprint Review

8



Challenges Associated with Agile

• Frequent change introduces risk

• Window available for performance testing is typically 1-2 weeks

• Final production candidate code often not available at the start of testing

• Multiple code releases within the testing window

• Increased probability of functional and environmental defects

• Frequent rework of code necessitates modifications to test scripts

• Cumulative effect on capacity from 12 releases per annum

• Narrow time window to resolve performance defects

© Capacitas 2002-2009 9



Challenges Associated with Agile

© Capacitas 2002-2009 10



Typical Causes of Degraded Service Performance

© Capacitas 2002-2009 11



How Can We Reduce Risk?

Cause of degraded performance Mitigated by performance testing?

Mitigated by modelling?

Changes in user patterns No Yes

Poor performance of 3rd parties May be limited by environment

Yes

Physical capacity bottlenecks May be limited by size of the environment

Yes

Logical bottlenecks Yes If known

Software configuration changes Yes Yes, if tested

Increases in demand May be limited by size of

the environment

Yes

© Capacitas 2002-2009 12



Why Test?

• Performance testing is used to understand the software’s performance characteristics

• Do unknown time and growth related defects exist?

• Memory leaks

• Poor database design/indexing

• Do unknown concurrency related defects exist?

• Database locking issues

• Memory locking issues

• Race conditions

• Obtain model inputs

© Capacitas 2002-2009 13



Why Model?

• Modelling uses our understanding of the software to predict end to end service performance under different scenarios

© Capacitas 2002-2009 14



Scenario Modelling

© Capacitas 2002-2009 15



ModellingTesting

Application

Coverage of Testing and Modelling

InfrastructureBusiness ScenariosTechnical Scenarios

© Capacitas 2002-2009 16



High Level Process

© Capacitas 2002-2009 17



Risk Assessment

© Capacitas 2002-2009 18



Risk Assessment

• Assess the potential performance impact on different elements ofinfrastructure

• Plan tests to help mitigate that impact

• Focus on areas of highest risk due to limited timeframes

Web DB Other

Project X --- Medium Medium Low Test 1

Project Y --- Low High Medium Tests 2 & 3

Project Z --- Low Low Low None

Description ID

Risks

Risk Mitigation

© Capacitas 2002-2009 19



Performance Tests Employed

© Capacitas 2002-2009 20



Baseline Tests

• Used to determine the impact of the planned release

• Comparative performance between releases

• May be Load, Stress or Soak Tests

• Service times and response times are compared to each other

Database Server

Service Time

Web Server

Service Time

Test 1.1 Release X 752 1240

Test 1.1 Release Y 813 1192

% Difference 8% -4%

0

200

400

600

800

1000

1200

1400

Service Tim

e (ms)

Example service time comparison between two

fictional baseline tests

© Capacitas 2002-2009 21



Single Transaction Type Tests

• To determine the impact on the infrastructure of individual user transactions

• Every business process which is to be modelled will have a calculated service time for each hardware element

132

233

156

122

221

328

Milliseconds Web CPU per page

(fictional data)

Home Search Select

Options Login Purchase

© Capacitas 2002-2009 22



Modelling

© Capacitas 2002-2009

Performance Test Results

23



Performance Modelling

Business

Service

Component

Virtual Infrastructure

© Capacitas 2002-2009 24




Business

Service

Component


Aircraft

Home Page Search Booking

HTTP1 APP1 APP2 DB1

© Capacitas 2002-2009 25




Business

Service

Component


600

100 80 5

160 70 10 80

3.5 transactions/s

© Capacitas 2002-2009 26



• What infrastructure to include?

• Major contributor to the performance experienced by end-user

• Highly utilised

• Critical

• Load growth

• Frequently reviewed

Model Scope


© Capacitas 2002-2009 27



Model Scope

• What transactions to include

• Business critical

• Importance to revenue

• High-volume

• High component cost

ServiceHome Page Search BookingChange Password

© Capacitas 2002-2009 28



Demand Model Validation

• The model should be validated on an ongoing basis against actual values

• The first stage is to validate the demand part of the model to ensure that arrival rates are being calculated correctly

• Following this the true demand can be used to validate the utilisation calculations

600

100 80 5

610

80 80 6

© Capacitas 2002-2009 29



Capacity and Performance Model Validation

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Comparison of Actual Database Load to Predicted Database Load

Actual Database CPU Predicted Database CPU

© Capacitas 2002-2009 30



What you need – People

• Capacity and performance management skills:

• Demand forecasting

• Modelling

• Load testing

• A strong understanding of risk management

• A good understanding of the business model

• Flexibility

• Project management skills

© Capacitas 2002-2009 31



What you need – Process

• Interface with the release team

• Participation in the agile development ‘scrums’

• Access to developers to understand application functionality

• Interface with the test environment management team for test planning

• Interface to the business to access business forecasts

• Interface to operations to access production measurements

© Capacitas 2002-2009 32



What you need – Tools & Environment

• An appropriately sized test environment

• Production-like environment is not required

• Consider the ratio of capacity between presentation, business and data tiers

• Toolset

• Performance monitoring tools (production and test)

• Load generation tool

• Modelling tool

© Capacitas 2002-2009 33



What you need – Environment

© Capacitas 2002-2009 34



What you need – Data

• Software/hardware configuration (production and test)

• Performance measurements from production and test environments

• Benchmark data (Spec, TPC, etc.)

• Service times derived from performance test results

• SLAs

• Internal

• SLAs with third-parties

• Historical and forecast business demand

• Historical and forecast service demand

• Measured error of previous forecasts

© Capacitas 2002-2009 35



Historical Business Demand

y = 0.0461x - 1673.1R² = 0.9668

0

20

40

60

80

100

120

140

160

180

Mar-04

Jun-04

Sep-04

Dec-04

Mar-05

Jun-05

Sep-05

Dec-05

Mar-06

Jun-06

Sep-06

Dec-06

Mar-07

Jun-07

Sep-07

Dec-07

Mar-08

Jun-08

Sep-08

Dec-08

Mar-09

Num

ber of Owne

d Aircraft

Delivery Date

Fleet Plan to April 2009

GB Airways

acquisition

© Capacitas 2002-2009 36



Historical Service Demand

© Capacitas 2002-2009 37



Costs

© Capacitas 2002-2009 38



Benefits

“Capacitas give us the confidence in making the correct release decision”

James Mackay, Test Manager, easyJet

© Capacitas 2002-2009 39

Test Phase Status

Risk Assessment Complete

Performance Test Plan Complete

Targeted Performance and Stress Testing Complete

Regression Performance and Stress Testing Complete

All Defects: Closed Complete

Capacity Models Updated Complete

Capacity Plan updated and communicated Complete



Benefits

• Testing and modelling complement one another

• The combined approach provides effective risk-mitigation in the context of:

• Frequent change

• Narrow performance test window

• Business growth

• The approach enables us to rapidly answer business and technical‘what if’ scenarios

• Costs are limited through the use of a scaled-down performance test environment and low-cost modelling tools

• Provides performance assurance in a highly competitive market place

© Capacitas 2002-2009 40



Questions?

• Presentation will be available for download at www.capacitas.co.uk

• Please visit us at our stand for any further questions

© Capacitas 2002-2009 41

Documents

Combining Performance Testing And Modelling For Easy Jet.Com V 1.0