Quality Assurance Testing

Tony MackPlanetLab Developers Meeting

Overview

• How do we validate/test our software?o Developers test their own work, submit when they feel its

ready.o Does it build?o Developers have different testing methods/processes

(unit tests, manual tests, scripts, etc...)o end to end system tests

•  Did we test everything?o Backwards compatible?o Did new development break old functionality?

Objectives

• Extensible testing system (test harness) o Define a testing process (what are we testing? How do

we test it?)o Configurable (Adding/Modifying/Updating test process

should be easy)o Flexible (Test harness can run in different environments)o Automatedo Developers can integrate their tests into the system

• Accountingo Test results are logged. Can be reviewed at any time o Track long term debugging efforts/progress

Test Types

• New development• Bug fixes• Backwards compatibility

o Are we losing required functionality by upgrading/replacing/deprecating

o Does current plc software support new node software?o Does new plc software support old node software?

• Unit testso Fine grain module testing

• End to end testso Do previous operations still work (can we create a slice at

plc and log into it on a node)• Long term tests

o Bugs that arise over time• Stress tests

o What are the limits? How much of a beating can it take?

Current Test Harness • Create a vserver and setup the testing environment (Optional)• Download and install the nightly MyPLC (if build failed and no

nightly exists, abandon or test in a previously working environment)

• Configure the PLC and add relevant test data to dbo Site, Nodes/People/Slices at site, Slices on nodes, Keys for

person accounts, NodeGroups, etc...• Run the api uint test(check if all api methods operate as

expected and return what expected)• Attempt to boot selected nodes (using qemu)

o Attempt to boot a 4.2 and 4.1o If nodes come up in boot bootstate, check if NodeManager

is running and creating slices• Execute node test scripts (cpu scheduler, planetflow, ...)

To Do

• Long term tests• Upgrading a PLC thats already installed• Specific NodeManger tests (slice attributes and resource

management)• Multiple PLC's and federation• Hardware diversity found in real nodes