What does QA mean to Sakai?

Megan May – Sakai FoundationAaron Zeckoski – CARET

Alan Berg – UVADavid Horwitz – Cape TownSeth Theriault- Columbia

Linda Place – U of M

2

Today’s Session

• What is QA? • Unit/integration testing• Static code review• Release/Branch management • QA server network• QA WG• Performance testing

3

What is quality assurance?

What is the role of the QA Director?

the set of planned and systematic actions necessary to provide appropriate confidence that a product or service will satisfy the requirements for quality.

Unit/Integration Tests

Aaron Zeckoski

5

Test Driven Development

• Create the Class/API -> Write the Test -> Program the Implementation -> Run the Test

• Forces you to use your own method (and hopefully check if it is intuitive to use)

• Requires an immediate check against the javadoc (API)

• Makes the developer think about how the method will work and what it does BEFORE they write any code

• http://en.wikipedia.org/wiki/Test-driven_development

6

Bug fix test cases

• When there is an issue created against your code you should create a test case to test for that specific failure before you fix it

• These steps– Create a test case which fails as indicated in the issue

from the issue tracker (JIRA)– Reference the JIRA URL in the comment for the test case– Run the test and make sure it fails– Fix the problem– Rerun the test to make sure it passes– Now you are safer from regressions

7

Tests vs. Code Review

• Static code review is valuable and helpful• Peer code review is more valuable• Neither is a substitute for unit testing and

integration testing though• Why?

– They do not provide for protection from regression– They do not supply the context that tests do

• Tests show how to use the code and provide a type of documentation

Static Code Review

Alan Berg

9

Static Code Review

Functional testing– Guarantees a minimum quality– Black box looks from the outside in– Does not cover all the code– Labor intensive– Does not totally scale as the code base scales

10

Static Code Review

• Static Code Reviews– Looks at the source code not a running instance– White box, looks from the inside out– Looks through all the code base once a day– Automated– Steps over commit boundaries– Information overload– Requires the right mentality from the community– Need to eat the bugs with little fuss

1111

TOUR

qa1-nl.sakaiproject.org/codereview/bug_dashboard/index_generic.html

1212

TOUR

qa1-nl.sakaiproject.org/codereview/bug_dashboard/index_generic.html

13

Source code

13

14

Evolutionary

14

• Using PMD, Findbugs, QJ Pro, Jlint and custom Perl code

• Tools getting better• Findbugs does not have many False positives• PMD has configurable rules• More shaping the information to avoid information

overload• Engaging in debate on how to deal with bugs• Need to change developers perceptions

15

Other

15

• Code coverage• Automatic removal of simple bug types script• Tracking of long term use of memory• Provisioning for performance testing• More to come

Branch/Release Management

17

Branch Management & Enlightened Self interest

• Why would we want to be branch managers?– first to go to 2.5.x – wanted a say in ensuring a high quality branch

for production– Its not actually a big overhead outside of the

release cycle

QA Server Network

Seth Theriault David Horwitz

19

QA Server Network

Amsterdam Boston UCambridge Cape TownColumbia IndianarSmart Georgia Tech

Charles Sturt U

Coming soon - server in Japan

20

Running a qa server

• What you need:– a computer – a network connection– a database– know how to compile and install Sakai

• Initial setup– Build server

21

Running a qa server

• Updating and maintenance– downloading source– build deploy– upgrade data (maybe)– usually once a week during release

QA WG

23

Collab https://collab.sakaiproject.org/portal

JIRA

Confluence

http://jira.sakaiproject.org/

http://confluence.sakaiproject.org/confluence/display/QA/Home

WG Tools

Sakai Performance Testing

Current model and limitationsDesired model and benefits

Dr. Linda M Place, University of Michigan

25

Talking Points

• Current model– Black box testing– Performance failure debugging and fixing

• Desired model– White box testing– White box testing with baseline load– Minimize production performance failures

26

Black Box Testing

• Black box testing– Running system with projected use-case scenarios

for acceptable throughput and response times• How will users experience application?

– Baseline comparison• Throughput and response times compared against

previous version test results• Looking for as good as or better results from new version

– Typically occurs at end of QA process

27

Qualities Being Tested

• Push application load against a system to identify quality of performance– Transaction response time for end users– Throughput (hits per second)– Reliability (percentage of failed

transactions)– Scalability (what happens when scale

increases significantly)– Capacity (upper limits of system)

28

Results

• Are metrics being met?– Measures of success

• Current Sakai test metrics– Baseline or better response times and throughput– New transactions do not disrupt baseline metrics– New transaction stress tests

• How many occurrences before system becomes unusable?

29

Failed Performance Debugging

• Designing isolated tests to address obvious performance problems in production code– Use tests to isolate performance cause– Use tests to verify fix before adding to

production instance

• Putting out fires

30

White Box Testing

• Pushing system to identify application, database, operating system, or network problems– Tune environment to identify and address

specific problems– Tests watched by developers, DBA, system and

network administrators, and performance engineers

• Typically occurs during code development

31

Little in Sakai Today

• Developers may design performance tests as part of development– Process not codified as required standard for

development– Most developers do not have a system of

sufficient scale to push code adequately– All components of production environment not

used and monitored• Developers; database, system & network

administrators; performance engineer

32

Value to Sakai

• Adding white box testing into Sakai QA process finds and fixes numerous performance problems before they reach formal QA process

• Engages Operations team early in development cycle rather than after performance problem affects user community

• Debugging easier prior to full integration with application suite

33

Want White Box Testing?

• Stay in room for next session– Sakai Community Performance Testing: A

Working Proof of Concept

• Participate in community model to improve your code prior to subjecting it to QA

• Don’t expect someone else to identify and fix your codes performance shortcomings

34

Questions