FunkLoadFunkLoad is a functional and load web tester

Table of Contents

1 Introducing FunkLoad1.1 What is FunkLoad ?1.2 Where to find FunkLoad ?1.3 Installation1.4 Examples1.5 Documentation1.6 Credits

2 Test runner2.1 Usage2.2 Examples2.3 Options

3 Benching3.1 Principle

PageTestCycleCycles

3.2 Bench runnerUsageExamplesOptions

3.3 Tips4 Bench report

4.1 Usage4.2 Examples4.3 Options

5 Test Recorder5.1 Recording a new FunkLoad test5.2 The fl-record command

UsageExamplesOptions

6 Credential server7 Monitor server8 The FunkLoadTestCase

8.1 Browser APIgetpostexistssetBasicAuthclearBasicAuthsetUserAgentaddHeaderclearHeadersdebugHeaders

8.2 XML RPC APIxmlrpc

8.3 Assertion helpers APIgetLastUrlgetLastBaseUrllistHrefgetBodyThe response object

8.4 Configuration file APIconf_getconf_getIntconf_getFloat

conf_getList8.5 Logging

logdlogi

8.6 Lipsum APILipsumgetWordgetUniqWordgetSubjectgetSentencegetParagraphgetMessagegetPhoneNumbergetAddress

8.7 Utilsxmlrpc_get_credentialxmlrpc_list_groupsxmlrpc_list_credentials

9 FunkLoadDocTest10 Other Test Cases

10.1 The ZopeTestCasezopeRestartzopePackZodbzopeFlushCachezopeAddExternalMethod

10.2 CPSTestCasecpsCreateSitecpsLogincpsLogoutcpsCreateGroupcpsVerifyGroupcpsCreateUsercpsVerifyUsercpsSetLocalRolecpsCreateSectioncpsCreateWorkspacecpsCreateDocumentcpsCreateNewsItemcpsChangeUiLanguagecpsListDocumentHrefcpsSearchDocId

11 Bug reporting

1 Introducing FunkLoad

1.1 What is FunkLoad ?FunkLoad is a functional and load web tester, written in Python, whose main use cases are:

Functional testing of web projects, and thus regression testing as well.Performance testing: by loading the web application and monitoring your servers it helps you topinpoint bottlenecks, giving a detailed report of performance measurement.Load testing tool to expose bugs that do not surface in cursory testing, like volume testing or longevitytesting.Stress testing tool to overwhelm the web application resources and test the application recoverability.Writing web agents by scripting any web repetitive task, like checking if a site is alive.

Main FunkLoad features are:

FunkLoad is free software distributed under the GNU GPL.

Functional test are pure Python scripts using the pyUnit framework like normal unit test. Python enablecomplex scenarios to handle real world applications.

Truly emulates a web browser (single-threaded) using an enhanced Richard Jones' webunit:

basic authentication supportcookies support

referrer supportfetching css, javascript and imagesemulating a browser cachefile upload and multipart/form-data submissionpost any kind of content type like application/xmlhttps supporthttps with ssl/tls by providing a private key and certificate (PEM formatted)http_proxy support

Advanced test runner with many command-line options:

set the target server urldisplay the fetched page in real time in your browserdebug mode to display http headerscheck performance of a single page (or set of pages) inside a testgreen/red color modeselect or exclude tests cases using a regexsupport normal pyUnit testsupport doctest from a plain text file or embedded in python docstring

Turn a functional test into a load test: just by invoking the bench runner you can identify scalability andperformance problems.

Detailed bench reports in ReST or HTML (and PDF via ps2pdf) containing:

the bench configurationtests, pages, requests stats and charts.the 5 slowest requests.monitoring one or many servers cpu usage, load average, memory/swap usage and networktraffic charts.an http error summary list

View a report example.

Differential reports to compare 2 bench reports giving a quick overview of scalability and velocitychanges. View a diff report example.

Easy test customization using a configuration file or command line options.

Easy test creation using embeded TCPWatch as proxy recorder, so you can use your web browser andproduce a FunkLoad test automatically, including file upload or any ajax call.

Provides web assertion helpers to check expected results in responses.

Provides helpers to retrieve contents in responses page using DOM.

Provides a funkload.CPSTestCase to ease Zope and Nuxeo CPS testing.

Easy to install (EasyInstall) and use, see examples in the demo folder.

Works fine with CPS or any Zope application, CherryPy, Nuxeo EP or any JBoss Seam JSF MyFacesapplications ...

1.2 Where to find FunkLoad ?Either:

Latest stable package using EasyInstall:

EasyInstall the latest snapshot

Bleeding edge svn sources:

See CHANGES file for information about distribution contents.

1.3 InstallationSee the INSTALL file for requirements and installation.

1.4 ExamplesSee the demo folder for test scripts and the report example.

For package installed with easy_install you need to run fl-install-demo to extract the demo examples.

1.5 DocumentationThis page is the main FunkLoad documentation, there are also:

CHANGES for information about distribution contents.INSTALL for requirement and installation.API documentation generated with epydoc.SLIDES introducing FunkLoad.

1.6 CreditsThanks to Frank Cohen's TestMaker framework, Richard Jones and all contributors.

2 Test runnerA FunkLoad test can be used like a standard unittest using a unittest.main() and a 'python MyFile.py'.

To ease testing FunkLoad come with an advanced test runner to override the static configuration file.

The loop-on-pages option enable to check response time of some specific pages inside a test without changing

the script, which make easy to tune a page in a complex context. Use the debug option to find the page

numbers.

Note that fl-run-test can be used to launch normal unittest.TestCase and (if you use python2.4) doctest in a

plain text file or embedded in a python docstring. The --debug option makes doctests verbose.

2.1 Usage

2.2 Examples

2.3 Options

3 BenchingThe same FunkLaod test can be turned into a load test, just by invoking the bench runner fl-run-bench.

3.1 PrincipleHere are some definitions used in bench mode:

CUs: Concurrent Users, which is the number of threads.STPS: Average of Successful Tests Per Second during a cycle.

SPPS: Average of Successfully Page Per Second during a cycle.RPS: Average Request Per Second, successfully or not.max[STPS|SPPS|RPS]: maximum of STPS|SPPS|RPS for a cycle.

Page

A page is an http get/post request with associated sub requests like redirects, images or links (css, js files).This is what users see as a single page.

Test

A test is made with 3 methods: setUp/test_name/tearDown. During the test_name method each get/postrequest is called a page.

Cycle

A cycle is a load of n concurrents test during a 'duration' period. Threads are launched every 'startupdelay'seconds, each thread executes test in a loop.

Once all threads have been started we start to record stats.

Only tests that end during the 'duration' period are taken into account for the test stats (in the representationbelow test like [---X are not take into account).

Only pages and requests that finish during the 'duration' are taken into account for the request and pagesstatistic

Before a cycle a setUpCycle method is called, after a cycle a tearDownCycle method is called, you can usethese methods to test differents server configuration for each cycle.

Cycles

FunkLoad can execute many cycles with different number of CUs, this way you can find easily the maximumnumber of users that your application can handle.

Running n cycles with the same CUs is a good way to see how the application handles a writing test over time.

Running n cycles with the same CUs with a reading test and a setUpCycle that change the applicationconfiguration will help you to find the right tuning.

3.2 Bench runner

Usage

Examples

Options

3.3 TipsHere are few remarks/advices to obtain workable metrics.

Since it uses significant CPU resources, make sure that performance limits are not hit by FunkLoadbefore your server's limit is reached. Check this by launching a bench from another host.

Having a cycle with one user gives a usefull reference.

A bench is composed of a benching test (or scenario) run many times. A good benching test should notbe too long so you have a higher testing rate (that is, more benching tests can come to their end).

The cycle duration for the benching test should be long enough. Around 5 times the duration of a singlebenching test is a value that is usually a safe bet. You can obtain this duration of a single benching testby running fl-run-test myfile.py MyTestCase.testSomething.

Rationale : Normally a cycle duration of a single benching test should be enough. But from the testingplatform side if there are more than one concurrent user, there are many threads to start and it takessome time. And on from the tested platform side it is common that a benching test will last longer andlonger as the server is used by more and more users.

You should use many cycles with the same step interval to produce readable charts(1:10:20:30:40:50:60 vs 1:10:100)

A benching test must have the same number of page and in the same order.

Use a Makefile to make reproductible bench.

There is no debug option while doing a bench (since this would be illegible with all the threads). So, if abench fails (that is using fl-run-bench), use fl-run-test -d to debug.

Using fl-record is very easy and very fast to create a scenario. But since it doesn't support HTTPS, thegood practise is to first record a scenario with fl-record on HTTP, and then change the url back to httpsin your FunkLoad test configuration file.

Always use description in post/get/xmlrpc, this improves the readability of the report.

4 Bench reportTo produce an HTML or ReST report you need to invoke the fl-build-report, you can easily produce PDF report

using Firefox 'Print To File' in PostScript then use the ps2pdf converter.

4.1 Usage

fl-build-report analyze a FunkLoad bench xml result file and output a report.

4.2 Examples

4.3 Options

5 Test Recorder

5.1 Recording a new FunkLoad testStarting with FunkLoad 1.3.0 you can use fl-record to record your navigator activity, this requires the

TCPWatch python proxy see INSTALL for information on how to install TCPWatch.

1. Start the recorder:

This will output something like this:

2. Setup your browser proxy and play your scenario

in Firefox: Edit > Preferencies > General; Connection Settings set localhost:8090 as yourHTTP proxy

Play your scenario using your navigator

Hit Ctrl-C to stop recording:

3. Replay you scenario:

You should see all the steps on your navigator.

4. Implement the dynamic part and assertion

Code the dynamic part like getting new url of a created documentAdd assertion using FunkLoad helpersUse a credential server if you want to make a bench with different users or simply don'twant to hard code your login/password.

Note that fl-record works fine with multi-part encoded form and file upload but will failed to record https

session.

Note that fl-record will handle automaticly JSF Myfaces token, which enable to easily record and play any

JBoss Seam application.

5.2 The fl-record command

Usage

Examples

Options

6 Credential serverIf you are writing a bench that requires to be logged with different users FunkLoad provides an xmlrpccredential server to serve login/pwd between the different threads.

It requires 2 files (like unix /etc/passwd and /etc/group) the password file have the following format:

The group file format is:

Setup a configuration file like in the demo/cmf folder, then start the credential server:

More options:

See the funkload-demo/cmf example for a credential configuration file.

7 Monitor serverIf you want to monitor a linux server health during the bench, you have to run a monitor xmlrpc server on thetarget server, this require to install the FunkLoad package.

On the server side you need to install the FunkLoad tool then launch the server using a configuration file(example in the demo/simple folder.):

On the bench host side setup your test configuration like this:

Then run the bench, the report will include server stats.

Note that you can monitor multiple hosts and that the monitor is linux specific.

8 The FunkLoadTestCaseFunkLoadTestCase extends the pyUnit unittest.TestCase with browser capabilities, configuration file helpersand assertions helpers. FunkLoad provides also some tools to generate random inputs and communicate withcredential servers.

Here is an overview of the api, you can find more on

8.1 Browser API

get

This emulates a browser http GET link. It will fetch the url, submits appropriate cookies, follow redirection,register new cookies, load css and javascript.

It also simulates a browser cache by not reloading a css, a javascript or an image twice.

Note that this is an emulation with some limitation:

It is single threaded (it loads images one after the other)It does not interpret javascriptSee trac tickets that starts with Browser: for other limitations

This method returns a webunit HTTPResponse.

Parameters:

url a valid urlparams a dico of parameters that going to be append to the url like url?key1=value1&...description is used on the bench report to describe the user actionok_codes is a list of http expected code like [200:301] if the http response is not in the list get willraise a test failure exception, if not provided assume that the default list is [200, 301, 302].

Note that if the url already contains encoded parameters you should not use the params parameter.

post

Same interface than the get() but it uses a http post method. You can upload a file by setting a params likethis:

You can even post any contents like this (available since 1.9.0):

exists

Return True if the http response code is 200, 301 or 302, and return False if http code is 404 or 503, othercodes will raise a test failure exception.

setBasicAuth

Next requests will use the http basic authentication.

clearBasicAuth

Remove basic auth credential set by setBasicAuth.

setUserAgent

New in 1.3.0. version.

Set a User-Agent http header for the next requests, the default browser behaviour is to use the agent defined in

the configuration file under [main] user_agent or to use the default FunkLoad/version string. Using this method

enable to change the user agent during a test case.

addHeader

New in 1.3.0. version.

Add an http header for the next requests.

clearHeaders

New in 1.3.0. version.

Remove all headers previously added by addHeader or setUserAgent, and remove the referer as well.

debugHeaders

New in 1.11.0. version.

Display headers sent to the server for the next get or post request.

8.2 XML RPC APIYou can test or bench xmlrpc services using the following API.

xmlrpc

Call the method_name at url using xmlrpclib. You can use the setBasicAuth method before to handle the http basic

authentication. Note that due to xmlrpclib limitation you can not use an http proxy.

Parameters:

url the url of the xmlrpc servermethod_name the name of the procedure to callparams a list of parameters to pass to the methoddescription is used on the bench report to describe the action

8.3 Assertion helpers APIFunkLoad uses the unittest assertion (assert_, assertEquals, fail, ...), but provides some methods to check the

http response. After fetching a page you can use the following methods.

getLastUrl

Return the last accessed page url taking care of redirects.

getLastBaseUrl

Return the <base /> href value of the last accessed page.

listHref

Return a list of href anchor url present in the last html response, filtering href using the url_pattern regex if

present and filtering the text content of the link with content_pattern if present.

getBody

Return the last response content.

The response object

The response returned by a get or post are webunit HTTPResponse object

getDOM return a SimpleDOM interface of the fetched html page, see the webunit SimpleDOM api instructionsfor details.

8.4 Configuration file APIA FunkLoadTestCase class uses a configuration file to setup variable configuration, like the base server url tobe tested, the test description, credential access, logging files and other test specific parameters. The testconfiguration file have the same name of the FunkLoadTestCase with a '.conf' extension. See documentedexamples in the demo folder (fl-install-demo).

conf_get

Return an entry from the configuration file. Note that the entry may be overriden by a command line option.

Parameters:

section the section in the configuration file.key the key.default a default value.

conf_getInt

Return an integer.

conf_getFloat

Return a float.

conf_getList

Additional parameter:

separator the default separator is a colon ':'.

Return a list

8.5 LoggingA FunkLoadTestCase store its results in an xml file (like request and test result) and put other log information

into a text log and/or output to the console.

logd

Debug log message

logi

Information log message

8.6 Lipsum APITo generate dummy document contents you can use the funkload.Lipsum api, this is a very simple "Loremipsum" generator.

You can see some examples by doing:

Lipsum

Parameters:

vocab a list of word, Lipsum provide 3 lists V_ASCII, V_DIAC, V_8859_15chars the list of char used to build an identifiersep some separators used in sentences like coma, question mark ...

getWord

Return a random word from the vocabulary.

getUniqWord

Generate a kind of uniq id.

getSubject

Return a subject of length word.

Parameters:

length the number of words in the subjectprefix a prefix to add at the beginning of a the subjectuniq add an uniq identifier in the subjectlength_min/max the words length is a random between min and max

getSentence

Return a sentence with some separators and and a ending point.

getParagraph

Return a paragraph of length sentences.

getMessage

Return a message with length Paragraphs.

getPhoneNumber

Return a random phone number.

Parameters:

lang can be fr or en_USformat can be short, medium or long

getAddress

Return a random address.

8.7 UtilsTo communicate with FunkLoad services like the credential server, there are some wrappers in the utilsmodule.

xmlrpc_get_credential

Return a tuple login, password of a user that belong to group if specified.

xmlrpc_list_groups

List groups name served by the credential server.

xmlrpc_list_credentials

List all login/password served by the credential server.

9 FunkLoadDocTestSince FunkLoad 1.5 you can use funkload easily from a doctest:

FunkLoadDocTest exposes the same API than The FunkLoadTestCase.

10 Other Test Cases

10.1 The ZopeTestCaseThis class extends the FunkLoadTestCase providing common Zope tasks.

zopeRestart

Stop and Start the Zope server.

Parameters:

zope_url the zope url.admin_id and admin_pwd the zope admin credential.time_out maximum time to wait until the zope server restart.

zopePackZodb

Pack a zodb database.

Parameters:

database the database to pack.days removing previous revision that are older than days ago

zopeFlushCache

Remove all objects from all ZODB in-memory caches.

zopeAddExternalMethod

Add an External method an run it.

10.2 CPSTestCaseThis class extends the ZopeTestCase providing common Nuxeo CPS tasks. You need to import theCPSTestCase that works with your CPS for example CPS338TestCAse or CPS340TestCase.

cpsCreateSite

Build a new CPS site.

Parameters:

admin_id and admin_pwd the zope admin credential.manager_id and manager_pwd the cps manager credential.zope_url the Zope server url [*].site_id the CPS site id.

[*] if the zope_url and site_id is not given we guess it using the server_url

cpsLogin

CPS log in.

cpsLogout

Logout the user logged in using cpsLogin.

cpsCreateGroup

Create a CPS group.

cpsVerifyGroup

Create a CPS group if not present.

cpsCreateUser

Create a CPS users.

cpsVerifyUser

Create a CPS users if not present.

cpsSetLocalRole

Grant role to name in url.

cpsCreateSection

cpsCreateWorkspace

cpsCreateDocument

Create a random document in the parent_url container.

cpsCreateNewsItem

Create a simple news in the parent_url container.

cpsChangeUiLanguage

Change the ui locale selection

cpsListDocumentHref

Return a clean list of document href that matches pattern in the previous page fetched.

cpsSearchDocId

Return the list of url that ends with doc_id, using catalog search.

11 Bug reportingThe list of open task and bugs are here: http://svn.nuxeo.org/trac/pub/report/12

If you want to report a bug or if you think that something is missing, send me an email bdelbosc _at_nuxeo.com.

Generated on: 2010-07-27 20:11 UTC.