Performance Testing your Kuali Student Product

open source administration software for education

next generation student system

http://creativecommons.org/licenses/by/3.0/

open source administration software for education 2

Agenda

• Introductions• The Challenge• Approach• Performance Automaton Strategy• Picking a tool• Application Results• Demo• Next Steps• Lessons Learned• Practices that work for us



Introduction

• Kyle Campos: Performance Engineer for Kuali Student. 11 years experience testing and building automation frameworks, both functional and performance, for web, unix and mobile applications. QA Architect for several startups building processes, tooling and test automation frameworks.

• Ben Clark, QA Manager for Kuali Student. 12 years of experience in both the computer software and banking industries. Have lead performance testing efforts using off-the-shelf and home grown solutions. ASQ Certified Six Sigma Black Belt and Software Quality Engineer. Earned a B.A. from the University of Montana.



The Challenge: Performance Testing KS

• Experience• Management support• Implementor User Group support

• Lacked common definitions of the space

• Poorly defined requirements• Lacked institutional system

environment information• No Kuali Student profile information

(how many courses over a year, cycle info, user info, etc)

• No test machines for perf testing• Lacked tools for the job• No qualified resources

+ Forces for achieving success - Forces against achieving success



The Approach

Action Plan

•Hire experienced performance engineer•Utilize Functional Council and IUG to obtain requirements and data•Define the performance automation strategy•Utilize cloud technologies for test infrastructure•Pick a tool that meets the need

• Lacked common definitions of the space

• Poorly defined requirements• Lacked institutional system

environment information• No Kuali Student profile information

(how many courses over a year, cycle info, user info, etc)

• No test machines for perf testing• Lacked tools for the job• No qualified resources



Performance Automation Strategy

• Develop test suites based on critical features list

• Benchmark tags & measure performance gain/loss

• Add suites as features are developed

• Use usage targets provided by universities



Open source tools – lots out there!

Tool Description Pros Cons

JMeter A java desktop app designed to load test functional behavior and measure performance

• HTTP protocol Support• Documented• List Support

• Didn’t work with GWT/Ajax

httperf A software tool which is able to simulate load situations for web services, and calculates performance metrics against the results

• Familiarity• Robust• Extensible• HTTP protocol Support• Documented• List Support

No dynamic variable support

The Grinder Java load testing framework that can run a distributed test using many load injector machines

• Flexible• Extensible• Documented• List Support

Requires Java expertise

Tsung Built for high transaction Env• Built for high trans env• Extendable• HTTP protocol Support• Documented• Dynamic variable support

Not being used across other KS products

Homegrown Build a tool in-house to test performance

Total control • Expensive• Time consuming• Proprietary• Reinventing the wheel

http://jakarta.apache.org/jmeter/

http://www.hpl.hp.com/research/linux/httperf/

http://grinder.sourceforge.net/

http://tsung.erlang-projects.org/



Tool Choice - Tsung

• Tsung XML API is a homegrown Ruby framework that allows for more rapid load test development, maintenance and execution

• Load Framework is tagged and released as validated against milestone releases

• Can easily configure custom test suites and load profiles

Tsung + Tsung XML API (Ruby) = Load Automation Framework



Tool Choice - Tsung



Test Design

• Test layer doesn’t generate any XML, all XML generated by API calls

• Test layer environment agnostic (standalone/embedded)

• Tests are added to suites and given a probability to run

• API docs dynamically created through rdoc

• Test configured & launched by homegrown utility



EC2 Network Diagram



Application Results - Key Metrics

• Concurrent sessions– driven by launched sessions/min

• Mean transaction times– feature defined transactions– highest 10sec mean time

• Iterative growth changes (linear?)• System resources

– cpu, mem



Application Results

• Data collected into Numbers(excel) for calculations

• Published on wiki• Demo



Demo

• Driver Demo



What’s Next?

• Tuning guide documentation• Configuration



Example problem walkthrough

• During Create Proposal load test found 4 concurrent sessions– upped to 16-18 sessions by upping

thread pool– Way too many threads being consumed

per session• Several rounds of debugging through

configuration changes• Traced bottleneck to disabling of Rice

caching. Resolved after 5 months



Lessons Learned

• A proof of concept may not have really proved anything

• Tsung is great for flexible load sessions but difficult to manage concurrent sessions

• Your load tool must have runtime dynamic variable support to work with GWT (Tsung does)


Practices that work for us

• Know your goals and requirements• When in doubt, benchmark• Review the test approach with entire team• A coding effort needs a coder; hire the best• Pick a tool that’s within your resource

capability• Separate data, tests and tool• Utilize the cloud – it’s cheap and easy

d


Resources

• Automation testing home– https://wiki.kuali.org/display/KULSTG/

Automation+Testing• Load Test Framework tester’s guide

– https://wiki.kuali.org/display/KULSTG/Load+Testing+-+Tester%27s+Guide

• Load testing results– https://wiki.kuali.org/display/KULSTG/

Automation+Testing+Results• Contact

– Kyle Campos / Skype: kylecampos_kuali / [email protected]

– Ben Clark / Skype: benclark_kuali / [email protected]

d


Questions?

d


Timeline

Perf. B

ug D

etec

ted

Perf. B

ug F

ixed

12/09 01/10 02/10 03/10 04/10 05/10 06/10 07/10 08/10 12/1011/1010/1009/10

Hire

Com

plet

ed

Stra

tegy

in p

lace

Tool

Rev

iew

Code

Com

plet

e

Tool

Tes

ting

Bench

mar

k

Bench

mar

k

Bench

mar

k

Bench

mar

k

M4

M5M6

M7

M8

M9

Release 1.1

Release 1.0

Bench

mar

k

Bench

mar

k

M3

EC2

Impl

emen

ted

EC2

Use A

ppro

ved

Technology

Performance Testing your Kuali Student Product