How Fast is Fast Enough?How Fast is Fast Enough?
© Copyright Performize-IT LTD.
Haim Yadid Performance expert, Performize-IT
Services
Consulting for R&D organizations• Problem analysis• product analysis
• Training• Java Performance Optimization
© Copyright Performize-IT LTD.How fast is fast enough?
““Fast” is More than a FeatureFast” is More than a Feature
© Copyright Performize-IT LTD.How fast is fast enough?
Speed is
© Copyright Performize-IT LTD
First and foremost, we believe that speed is more than a feature. Speed is the most
important feature. If your application is slow, people won’t use it.
Fred Wilson: Top tech VC from NYC with investments in companies such as Twitter, del.icio.us, Etsy, and FeedBurner.
How fast is fast enough?
Revenue Drop
Bing – A page that was 2 seconds slower resulted in a 4.3% drop in revenue/user.
• Google – A 400 millisecond delay caused a 0.59% drop in searches/user.
• Yahoo! – A 400 milliseconds slowdown resulted in a 5-9% drop in full-page traffic.
© Copyright Performize-IT LTD
Taken from Steve Sounders blog
How fast is fast enough?
Effect of improving performance
• Shopzilla – Speeding up their site by 5 seconds increased the conversion rate 7-12%, doubled the number of sessions from search engine marketing, and cut the number of required servers in half.
• Mozilla – Shaving 2.2 seconds off their landing pages increased download conversions by 15.4%, which they estimate will result in 60 million more Firefox downloads per year.
• Netflix – Adopting a single optimization, gzip compression, resulted in a 13-25% speedup and cut their outbound network traffic by 50%.
© Copyright Performize-IT LTDHow fast is fast enough?
What is Bad Performance ?
When user experience is bad• When customers are complaining• When application cannot scale• When applications crash• When applications become non responsive
© Copyright Performize-IT LTD.How fast is fast enough?
What is good performance
When customers are happy about performance • Customers are complaining on other things• Application is responsive• Application is stable• Application scales as needed
© Copyright Performize-IT LTD.How fast is fast enough?
Good Enough Performance
Performance behave according to requirements• Not the best performance Good enough
© Copyright Performize-IT LTDHow fast is fast enoughHow fast is fast enough?
Performance AssurancePerformance AssuranceActivities Activities
© Copyright Performize-IT LTD.
Analysis Design Development StabilizationHow fast is fast enough?
Example: Collaboration App.
A Google sites like collaboration application• Registered user will have access to a document repository (documents, presentations, spreadsheets)
• CRUD on all documents• Editing content online from the browser• Conversion from and to Microsoft office documents
© Copyright Performize-IT LTD.How fast is fast enough?
Capacity planning
The important dimensions for your application• Expected value for each dimension.• Design capacity for all system and for a single computing unit.
© Copyright Performize-IT LTD.
AnalysisHow fast is fast enough?
Example: Capacity planning
© Copyright Performize-IT LTD.
AreaArea Sub areaSub area Capacity Capacity
Users Number of users 1M
Logged in users 60K
Documents Max #docs per user 100
Total number of docs 10M
Avg doc size 500KB
Max doc size 10MB
# import & exports/ min 50
Workload Requests / sec 2000
AnalysisHow fast is fast enough?
Define Major Scenarios
What are the application most common scenarios?• What is the correct distribution of user scenarios• Based on real statistics • In not possible… your best guestimate• Refine your scenarios along product versions
© Copyright Performize-IT LTD.
AnalysisHow fast is fast enough?
Example: Major Scenarios
Scenarios for a regular user
Scenario nameScenario name DistributionDistribution DescriptionDescription
Login 10%
Create new doc 15%
Update document 50%
Delete document 5%
Share /Publish document
10%
Rename document
10%
© Copyright Performize-IT LTD.
AnalysisHow fast is fast enough?
Performance KPIs
Your performance contract• What is the maximum your user can tolerate for operations
• Usually (can vary between organizations) • 1 second for web interactive operations• 200ms for ajax operations • 5 secs for async operations • Several minutes for heavy data mining operations
© Copyright Performize-IT LTD.
AnalysisHow fast is fast enough?
Tolerance to regression
When you are developing new version• Adding new functionality• Changing architecture• Define what you expect to be the regression/ improvement from last version
© Copyright Performize-IT LTD.
AnalysisHow fast is fast enough?
Sizing
The amount of resources required to support the capacity planning
• Number of machines (Web servers, DB etc…)• Memory for machine• Amount of Disk space• Expected DB growth per day
© Copyright Performize-IT LTD.
DesignHow fast is fast enough?
Example: Sizing
Machine typeMachine type ParameterParameter ValueValue
Web container CPU Cores 8
Memory 4GB
Disk 40GB
DB server CPU Cores 16
Memory 16GB
Disk 4TB raid 5
© Copyright Performize-IT LTD.
DesignHow fast is fast enough?
System Safety Valves
• AKA Fuses• Protect application from crossing the capacity limit• Crucial to
• Keep system up and running• prevent customer misuse• prevent DoS attacks
• Example:• Do not enable to create than 10M more users• Do not let the 60001 user to login• Do not permit to create more then 10M document
© Copyright Performize-IT LTD.
DesignHow fast is fast enough?
UI related Safety Valves
Give bounds to certain UI behaviors• Crossing the bounds may have no sense or damage user experience
• Number of documents per folder may not exceed 100
• Number of chars per line of document may not exceed 2000
© Copyright Performize-IT LTD.
DesignHow fast is fast enough?
Risk Assessment
For new feature you must assess performance risk• High risk features require careful analysis• High risk features should be handled with care
• POC• Extensive testing
• How does a new feature affects the capacity planning
© Copyright Performize-IT LTD.
DesignHow fast is fast enough?
Example: Risk Assessment
FeatureFeature RiskRisk Measures to takeMeasures to take
Add tooltip to UI toolbar Low
Add indexing for all documents
HighPOC at early
stages
© Copyright Performize-IT LTD.
DesignHow fast is fast enough?
POC
For features with high performance risk • Test performance effect as soon as possible• Even before all functionality is done• Capture the risk nature with minimal implementation
© Copyright Performize-IT LTD.
Design DevelopmentHow fast is fast enough?
Automation
Development phase is the time for automation• Not later• Ability to populate the system• Reliable simulators• Load generation scripts• Automation should be • Fast • Reliable• Easy to use
© Copyright Performize-IT LTD.
DevelopmentHow fast is fast enough?
Code review
Delicate code areas should be reviewed • Concurrency should be handled with care• Premature optimisation should be avoided
© Copyright Performize-IT LTD.
DevelopmentHow fast is fast enough?
Component Tests
Test only one scenario/ component• Easier to isolate• May behave differently than the system test• Covering all components one by one may take long time
© Copyright Performize-IT LTD.
Development StabilizationHow fast is fast enough?
System Tests
Tests the system as whole• Configured according to capacity• Hard to isolate problem
© Copyright Performize-IT LTD.
StabilizationHow fast is fast enough?
The Performance Lab
A system which simulates real application• Simulate data feeds and virtual users
© Copyright Performize-IT LTD.
StabilizationHow fast is fast enough?
Populate your System
• Use automation modules to load your system with data • Create users• Create documents• Create document sharing
• Populate the system to its full capacity planning with a reasonable time
© Copyright Performize-IT LTD.
StabilizationHow fast is fast enough?
Challenge Your Backend
Mechanisms which do not require user interaction• Develop simulators • Run simulators according to capacity planning
© Copyright Performize-IT LTD.
StabilizationHow fast is fast enough?
Virtual Users
Running a performance test on real users is not practical
• Use a tool to • Record• Parameterize • Replay with large numbers of users
© Copyright Performize-IT LTD.
StabilizationHow fast is fast enough?
Load Tests
Running your application with accordance to your capacity planning
• This test when passes will give a good indication that your product will hold in production
• (Assuming that your capacity planning is accurate)
© Copyright Performize-IT LTD.
StabilizationHow fast is fast enough?
Stress Tests
Run your system under a massive load• 2 times or 3 times the capacity • May reveal concurrency problems such as deadlocks
• May show how system behaves under DoS attacks• Do not expect that system will behave correctly under those circumstances
© Copyright Performize-IT LTD.
StabilizationHow fast is fast enough?
Endurance Tests
Run the system under load for a long time• Measure application stability• Unveil resource leaks (memory threads etc.)• Deadlocks and other concurrency problems
© Copyright Performize-IT LTD.
StabilizationHow fast is fast enough?
Single User Test
Measure end to end performance • Include rendering time and network time• Single Users test should run under load
© Copyright Performize-IT LTD.
StabilizationHow fast is fast enough?
Performance Testing Dilemma
• Should start as soon as possible May have huge impact on code and design
• Cannot start too earlyConducting performance problem on a non
feature complete system is a waste of time
© Copyright Performize-IT LTD.How fast is fast enough?
Tip: Give High Priority
Performance problem should get a high priority from the R&D
• They tend to be complex• They tend to be nasty• They tend to insert uncertainty • It is always simpler to fix the color of the button!!!
© Copyright Performize-IT LTD.
The Button will wait
How fast is fast enough?
Allocate the best resources
Your best engineers• Your best hardware
© Copyright Performize-IT LTD.How fast is fast enough?