Embed Size (px)
Citation preview
How to Use The Grinder Performance
Testing Tool
Introduction
In this article I’ll go over installing and setting up Grinder. I am not going to go into too many details since my aim is to provide information that will enable you to have Grinder up and running quickly. If you want more information, you can look at the rather thorough Grinder User Guide. At the end of this guide, you’ll know how to install and set up grinder, record a test, modify (and/or parameterize) it, start running it through the console, and record data from the test. Note: The instructions in this guide relate to a Windows environment. You can run Grinder in Linux; the set-up is not much different. The only differences will be in installation locations and shell scripts.
Downloading everything you need
First you need to download Grinder. You can get it from here. You should only need to download the binary distribution and not the source distribution (unless you’re planning on extending Grinder). Unzip the installation file to somewhere convenient. I extracted the file to /user/username/grinder. Make sure that your version of the JDK is greater than or equal to 1.4, otherwise Grinder won’t work!( http://java.com/en/download/installed.jsp) The second thing you need to install is Jython. Note that this is not absolutely necessary, but I found it useful to have around when I was trying to parameterize the test-scripts, and I needed to test something out first. I recommend not using the Jython that comes with Ubuntu. Instead, download and install Jython 2.2.1 from here (there are issues with Jython 2.5 and Grinder).
Setting up Grinder
For demonstration purposes, I’m going to show you how to set up Grinder with a local grinder.properties file, with a local console and a local agent running workers. To get an explanation as to what these terms mean, take a look at the picture below:
As you can see, you essentially have a console that sits on top of everything. It acts like a controller, pushing out scenarios and test cases (scripts) to agents that are running on load-injection machines. Each agent is in charge of one or more workers that inject load into the system you are testing. The workers also report data back to the console. The console aggregates all the data it receives from the workers and presents it to you. This is the ideal scenario that you would use for actual performance testing, however like I said before, I’m not going to concentrate on that – I’m just going to show you how to quickly set up Grinder and run a performance test on your local machine. If you want to look into distributing scenarios and tests across different agents, you can take a look here.
The first thing you want to do is set up a grinder.properties file. I put this file into the root directory of the checked out app (/users/username/grinder/). It’s not necessary to put it there; you can put it wherever you want. So open up your favorite text editor, and create the following file:
# Tell Grinder how many processes, threads and runs (iterations) we have in this scenariogrinder.processes = 1grinder.threads = 1grinder.runs = 1
# grinder.jvm.arguments sends arguments to the JVM. The reason I have the following value# is to prevent an error (Jython complains about caches) from popping up when you start# a test. Jython caches imports and so you need to let it know where it can store the imports.# Though it's not strictly necessary, you do need it if you are performing wildcard imports# (like java.util.*)grinder.jvm.classpath=build/classes;build/testgrinder.jvm.arguments=-Dpython.home=c:/jython2.5.2grinder.jvm.arguments = -Dpython.cachedir=/tmp/mycachegrinder.script = proxy-output.py
# Here you set up the arguments that tells the agent where the console lives.grinder.hostID = localhostgrinder.consoleHost = localhostgrinder.consolePort = 6372grinder.useConsole = truegrinder.reportToConsole.interval = 500
# sleepTime is a synonym for "think time". You don't want the load-generator hammering away at the# app one request after another with a uniform interval. sleepTime introduces time variations between# different requestsgrinder.initialSleepTime = 0grinder.sleepTimeFactor = 1grinder.sleepTimeVariation = 0.2
# These properties are used to configure the behavior of Grinder with respect to logginggrinder.logProcessStreams = truegrinder.reportTimesToConsole = truegrinder.debug.singleProcess = falsegrinder.useNanoTime = falsegrinder.logDirectory=logsgrinder.numberOfOldLogs=2grinder.logProcessStreams=false
If you want more information regarding the grinder.properties file, take a look at this page.
Alright, now that you have the grinder.properties created, you also need a few shell-scripts to easily start and stop the console, agent, and TCP Proxy. You will have to change the values of GRINDERPATH, GRINDERPROPERTIES, and JAVA_HOME to values that make sense on your system.
File setGrinderEnv.batset GRINDERPATH="C:\users\ijain\grinder\grinder-3.7.1"set GRINDERPROPERTIES="C:\users\ijain\grinder\grinder.properties"set CLASSPATH=%GRINDERPATH%\lib\grinder.jar;%CLASSPATH%
File setGrinderEnv.cmdset GRINDERPATH="C:\users\ijain\grinder\grinder-3.7.1"set GRINDERPROPERTIES="C:\users\ijain\grinder\grinder.properties"set CLASSPATH=%GRINDERPATH%\lib\grinder.jar;%CLASSPATH%
File startAgent.batcall "C:\users\ijain\grinder\setGrinderEnv.cmd"java -cp %CLASSPATH% net.grinder.Grinder %GRINDERPROPERTIES%
File startConsole.batcall "C:\users\ijain\grinder\setGrinderEnv.cmd"java -cp %CLASSPATH% net.grinder.Console
File startTCPProxy.batcall "C:\users\ijain\grinder\setGrinderEnv.cmd"java -cp %CLASSPATH% net.grinder.TCPProxy -console -http > proxy-output.py
Creating a Test
Before you start performance testing, you need to have a test. Grinder test scripts are written in the Jython language. You can write your tests from scratch if you wish (it can be useful if you’re trying to test things like XML-RPC), but more often than not you will be using Grinder’s proxy tool to record a usage scenario and generate a test script. The tool that Grinder comes with is called TCP Proxy. It sits between the browser and the server and records requests and responses that move between the browser and the server. By itself, it simply echoes whatever goes through it (in that sense, it makes a pretty good debugging tool), but we want it to generate Jython test scripts. Note: TCPProxy actually uses XSLT to translate the header information into a Jython script. This is pretty neat, actually because you can use XSLT to translate the code to Groovy, or even JRuby.
First, you need to start up TCP Proxy. Simply run startTCPProxy.bat and you should be good to go. A little window will pop-up letting you know that the proxy is running. There’s a textbox in the window that lets you insert comments too. Once you the proxy is started up, you need to configure your browser to use the proxy at localhost:8001. Now access the app you want to test, in your browser. Log in and do whatever it is you need to do (business card select Template, edit, save, login, add to cart, logout). Note: If you get an alert box that complains about security certificates, you can add an exception in Firefox by going to Edit->Preferences->Advanced->Encryption Tab->View Certificates->Add Exception. Put in the URL to your server and Firefox should grab the certificate and add an exception.
Once you’re done recording the test, press the Stop button in TCPProxy.
WARNING: When you’re running TCPProxy, make sure that you have no other tabs open!, do a clear cache and Also make sure that you have no plugins trying to phone home (through the browser). TCPProxy intercepts every communication that your browser makes so you don’t want to pollute your results with irrelevant requests (I think it’s possible to filter the data using the domain name, but I haven’t looked into how to do it yet)
Running the tests
Now we can actually start running the tests and recording data! The first thing you need to do is start up your app. Then, you need to start the Grinder console (click on startConsole.bat). After
that, click on startAgent.bat to start up the agent. You should see something like this in your shell:
Console will look like this and a console window will also open
Agent will look like this:
If you see this, it means that your agent started up, and was able to talk to the Grinder console. In the Grinder console, there are four icons in the top-left corner of the window. If you mouse-over the leftmost icon, you will see a tooltip that says “Start the worker processes”. Click that button. A dialog box will pop up saying that you haven’t selected a properties file and that the worker processes will run the script set in the agent’s properties file. This is fine, since you are running Grinder locally and not distributing a file. Click Ok. You should now see something like this in your shell:
If you check back into the Grinder console, you can see that it is grabbing data from the agent. Now just sit back and let the test run. Once the test is complete, hit the stop button and exit out of console. If you go to the directory from where you started up the agent, you will see three log files: data_<hostname>-N.log, error_<hostname>-N.log, and out_<hostname>-N.log. data_<hostname>-N.log is the file that contains all the data gathered (like mean response time, transactions per second, etc.). It is a comma-separated-values file and you can open it up in Excel or Openoffice Calc. error_<hostname>-N.log contains any errors encountered while running the test, and out_<hostname>-N.log contains logging data from the script file. You are now in a position to analyze data from the log files.
Analyzing Grinder-Generated Data
Grinder spits out raw data. So it’s not that user-friendly when compared to, say, WebLoad. This makes sense when you consider the fact that Grinder is geared towards developers. Grinder says “Here is the data I collected, do whatever you want with it”. If you want a quick overview of your performance data, there is a tool called GrinderAnalyzer that you can download from here. Usage is pretty straightfoward (check out the documentation). Extract it to your user home directory. Then set jython to run from command line.
To set jython:Go to my computer->properties->advanced system setting->Advanced tab->Environment variablesFrom system variables, select path and and then click edit.Add the end of the Variable value put ;c:\jython2.5.2Click ok
To run the analyzer tool from Windows, Linux or OS X:
1. Open a command-line shell and cd into the grinderAnalyzer directory. 2. Execute the script like this: jython ./analyzer.py c:/users/ijain/grinder/logs/localhost-0-data.log c:/users/ijain/grinder/logs/localhost-0.log
OUTPUT
All graphs and html are created in a directory named 'grinderReport'. The .png files may be viewed in the tool of your choice. The html report includes a summary table with sortable columns.
To store the analyzer results long-term, you will need to move or rename the 'grinderReport' directory, since it is reset every time you run the analyzer.
Conclusion
Grinder is a pretty flexible and powerful performance-testing tool. I’m hoping that after this tutorial you have enough knowledge to go out and start creating and running tests of your own. Like I mentioned before, this guide is by no means comprehensive. It’s simply something to get you started. If you want more information, I highly recommend looking at the excellent Grinder Documentation.
Parameterizing the test
There are a few tests where you would like to supply data (for example, when adding a contact). In the context of performance-testing, this is called parameterizing. Since Grinder uses Jython, it’s actually pretty easy to parameterize a test. First, let’s take a look at a TCPProxy-generated Jython-script. I’m only going to reproduce parts of the script since the entire script itself is rather large (however, if you want to look at the entire script, you can download it):
view source print ? 001.# The Grinder 3.2 002.# HTTP script recorded by TCPProxy at Jun 17, 2009 4:57:02 PM 003. 004.from net.grinder.script import Test 005.from net.grinder.script.Grinder import grinder 006.from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest 007.from HTTPClient import NVPair 008.connectionDefaults = HTTPPluginControl.getConnectionDefaults() 009.httpUtilities = HTTPPluginControl.getHTTPUtilities() 010. 011.# To use a proxy server, uncomment the next line and set the host and port. 012.# connectionDefaults.setProxyServer("localhost", 8001) 013. 014.# These definitions at the top level of the file are evaluated once, 015.# when the worker process is started.
016. 017.connectionDefaults.defaultHeaders = \ 018. ( NVPair('Accept-Language', 'en-us,en;q=0.5'), 019. NVPair('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.7'), 020. NVPair('Accept-Encoding', 'gzip,deflate'), 021. NVPair('User-Agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.11) Gecko/2009060308 Ubuntu/9.04 (jaunty) Firefox/3.0.11'), ) 022. 023.headers0= \ 024. ( NVPair('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'), 025. NVPair('Referer', 'https://local.infusiontest.com:8443/?msg=You%27ve+been+logged+out+-+thanks+for+playing%21&amp;amp;notification=You%27ve+been+logged+out+-+thanks+for+playing%21'), )
026. 027.headers1= \ 028. ( NVPair('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'), 029. NVPair('Referer', 'https://local.infusiontest.com:8443/Admin/home.jsp'), )
030. 031.... 032.... 033. 034.headers9= \ 035. ( NVPair('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'), 036. NVPair('Referer', 'https://local.infusiontest.com:8443/Admin/home.jsp?revertCalUser=true'), 037. NVPair('Cache-Control', 'no-cache'), )
038. 039.headers10= \ 040. ( NVPair('Accept', 'image/png,image/*;q=0.8,*/*;q=0.5'), 041. NVPair('Referer', 'https://local.infusiontest.com:8443/?msg=You%27ve+been+logged+out+-+thanks+for+playing%21&amp;amp;notification=You%27ve+been+logged+out+-+thanks+for+playing%21'), ) 042. 043.url0 = 'https://local.infusiontest.com:8443'044. 045.# Create an HTTPRequest for each request, then replace the 046.# reference to the HTTPRequest with an instrumented version. 047.# You can access the unadorned instance using request101.__target__. 048.request101 = HTTPRequest(url=url0, headers=headers0) 049.request101 = Test(101, 'POST processLogin.jsp').wrap(request101) 050. 051.request102 = HTTPRequest(url=url0, headers=headers0) 052.request102 = Test(102, 'GET home.jsp').wrap(request102) 053. 054.request201 = HTTPRequest(url=url0, headers=headers1) 055.request201 = Test(201, 'GET popUpTask.jsp').wrap(request201) 056. 057.... 058....
059. 060.request3004 = HTTPRequest(url=url0, headers=headers0) 061.request3004 = Test(3004, 'GET defaultLogin.jsp').wrap(request3004) 062. 063.request3005 = HTTPRequest(url=url0, headers=headers0) 064.request3005 = Test(3005, 'GET index.jsp').wrap(request3005) 065. 066.class TestRunner: 067. """A TestRunner instance is created for each worker thread."""
068. 069. # A method for each recorded page. 070. def page1(self): 071. """POST processLogin.jsp (requests 101-102)."""
072. 073. # Expecting 302 'Moved Temporarily' 074. result = request101.POST('/login/processLogin.jsp', 075. ( NVPair('username', 'myusername'), 076. NVPair('password', 'myp@55W0rd'), 077. NVPair('Login', 'Login'), ), 078. ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))
079. 080. grinder.sleep(49) 081. request102.GET('/Admin/home.jsp')
082. 083. return result 084. 085. def page2(self): 086. """GET popUpTask.jsp (request 201)."""087. result = request201.GET('/files/popUpTask.jsp' +088. '?0.6697580414936783') 089. 090. return result 091. 092. def page3(self): 093. """POST calendarBackend.jsp (request 301)."""094. result = request301.POST('/Calendar/calendarBackend.jsp', 095. ( NVPair('calType', 'Day'), 096. NVPair('userId', '1'), 097. NVPair('calDate', '17'), 098. NVPair('calMonth', '5'), 099. NVPair('calYear', '2009'), 100. NVPair('weekStartDate', '-1'), 101. NVPair('weekStartMonth', '-1'), 102. NVPair('weekStartYear', '-1'), 103. NVPair('weekEndDate', '-1'), 104. NVPair('weekEndMonth', '-1'), 105. NVPair('weekEndYear', '-1'), ), 106. ( NVPair('Content-Type', 'application/x-www-form-urlencoded; charset=UTF-8'), ))
107. 108. return result 109. 110. ... 111. ... 112. 113. def page18(self): 114. """POST processContact.jsp (requests 1801-1802)."""
115. 116. # Expecting 302 'Moved Temporarily'
117. result = request1801.POST('/Contact/processContact.jsp', 118. ( NVPair('addingFormId', '0'), 119. NVPair('tabs_sel', 'main'), 120. NVPair('Contact0Id', '0'), 121. NVPair('Contact0FirstName', "Mikey"), 122. NVPair('Contact0LastName', "Mike"), 123. ... 124. ... 125. NVPair('Contact0Website', ''), 126. NVPair('Contact0Email', "[email protected]"), 127. NVPair('email_Contact0Email', '1'), 128. ... 129. ... 130. NVPair('view', 'add'), 131. NVPair('Save', 'Save'), ), 132. ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), )) 133. self.token_view = \ 134. httpUtilities.valueFromLocationURI('view') # 'edit' 135. self.token_tabs_sel = \ 136. httpUtilities.valueFromLocationURI('tabs_sel') # 'main' 137. self.token_msg = \ 138. httpUtilities.valueFromLocationURI('msg') # 'Person Added'
139. 140. grinder.sleep(46) 141. request1802.GET('/Contact/manageContact.jsp' +142. '?view=' +143. self.token_view +144. '&amp;amp;tabs_sel=' +145. self.token_tabs_sel +146. '&amp;amp;msg=' +147. self.token_msg)
148. 149. return result 150. 151. ... 152. ... 153. 154. def __call__(self): 155. """This method is called for every run performed by the worker thread."""156. self.page1() # POST processLogin.jsp (requests 101-102) 157. 158. grinder.sleep(350) 159. self.page2() # GET popUpTask.jsp (request 201) 160. 161. grinder.sleep(385)
162. self.page3() # POST calendarBackend.jsp (request 301)
163. 164. ... 165. ... 166. 167. grinder.sleep(407) 168. self.page29() # POST calendarBackend.jsp (request 2901) 169. 170. grinder.sleep(3661) 171. self.page30() # GET logout.jsp (requests 3001-3005)
172. 173.def instrumentMethod(test, method_name, c=TestRunner): 174. """Instrument a method with the given Test."""175. unadorned = getattr(c, method_name) 176. import new 177. method = new.instancemethod(test.wrap(unadorned), None, c) 178. setattr(c, method_name, method) 179. 180.# Replace each method with an instrumented version. 181.# You can call the unadorned method using self.page1.__target__(). 182.instrumentMethod(Test(100, 'Page 1'), 'page1') 183.instrumentMethod(Test(200, 'Page 2'), 'page2') 184.instrumentMethod(Test(300, 'Page 3'), 'page3') 185.... 186.... 187.instrumentMethod(Test(2900, 'Page 29'), 'page29') 188.instrumentMethod(Test(3000, 'Page 30'), 'page30')
From the above listing, you can get a general idea of the anatomy of a test script. Of course, there’s a little bit more that goes into this, so if you want a detailed description, take a look at this page (in my next article, I’ll go into more detail about the anatomy of a recorded script). By itself, this script is 95% useful. However, to make it exactly what we need, we need to parameterize certain areas. The first section we need to parameterize is where we add a new contact (look at the page18 method). Right now, it inserts a contact called “Mikey Mike” with the email “[email protected]”. That’s not particularly useful if we’re running this test a bunch of times. We want to parameterize this value. What we’re going to do is create a random first name, last name, and email address, and use those values in the request. Here’s how we do it (once again, I’m only showing the pertinent sections of the code i.e., the sections that have changed):
view source print ? 01.# The Grinder 3.2 02.# HTTP script recorded by TCPProxy at Jun 17, 2009 4:57:02 PM
03. 04.# We need to import java.util.Random for random number generation
05.from java.util import Random 06. 07.... 08.... 09. 10. def page18(self): 11. """POST processContact.jsp (requests 1801-1802)."""12. 13. chars = [ "b", "c", "d", "f", "g", "h", "j", "l", "m", "n", "p", "q", "r", "s", "t", "v", "w", "x", "y", "z", "th", "dh", "gh", "ph", "ch", "st",
"sw", "sp", "kh" ]; 14. vowels = [ "a", "e", "i", "o", "u" ]; 15. r = Random(); 16. 17. fname = ""; 18. lname = ""; 19. 20. for i in range(0, r.nextInt(3) + 2): 21. fname += chars[r.nextInt(len(chars))] + vowels[r.nextInt(len(vowels))]; 22. 23. for i in range(0, r.nextInt(3) + 2): 24. lname += chars[r.nextInt(len(chars))] + vowels[r.nextInt(len(vowels))]; 25. 26. # Expecting 302 'Moved Temporarily' 27. result = request1801.POST('/Contact/processContact.jsp', 28. ( NVPair('addingFormId', '0'), 29. NVPair('tabs_sel', 'main'), 30. NVPair('Contact0Id', '0'), 31. NVPair('Contact0FirstName', fname.capitalize()), 32. NVPair('Contact0LastName', lname.capitalize()), 33. ... 34. ... 35. NVPair('Contact0Website', ''), 36. NVPair('Contact0Email', fname + "." + lname + "@" + lname + ".com"), 37. NVPair('email_Contact0Email', '1'), 38. ... 39. ... 40. NVPair('Save', 'Save'), ), 41. ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), )) 42. self.token_view = \ 43. httpUtilities.valueFromLocationURI('view') # 'edit' 44. self.token_tabs_sel = \ 45. httpUtilities.valueFromLocationURI('tabs_sel') # 'main'
46. self.token_msg = \ 47. httpUtilities.valueFromLocationURI('msg') # 'Person Added'
48. 49. grinder.sleep(46) 50. request1802.GET('/Contact/manageContact.jsp' +51. '?view=' +52. self.token_view +53. '&amp;amp;tabs_sel=' +54. self.token_tabs_sel +55. '&amp;amp;msg=' +56. self.token_msg)
57. 58. return result
As you can see, we’ve now parameterized the first name and last name values that go into processContact.jsp. There is still one more thing we need to parameterize. If you go back and take a look at grinder.properties, you can see that I set grinder.threads to 5. This means that there are going to be 5 threads running. We don’t want the same user logging in more than once (our app doesn’t allow it). So what we also need to do is parameterize user logins:
view source print ? 01.# The Grinder 3.2 02.# HTTP script recorded by TCPProxy at Jun 17, 2009 4:57:02 PM
03. 04.from java.util import Random 05.from net.grinder.script import Test 06.from net.grinder.script.Grinder import grinder 07.from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest 08.from HTTPClient import NVPair 09.connectionDefaults = HTTPPluginControl.getConnectionDefaults() 10.httpUtilities = HTTPPluginControl.getHTTPUtilities() 11. 12.# To use a proxy server, uncomment the next line and set the host and port. 13.# connectionDefaults.setProxyServer("localhost", 8001) 14. 15.# These definitions at the top level of the file are evaluated once, 16.# when the worker process is started. 17. 18.# Setting up an array of dictionaries (i.e., hash/map/associative-array) of username and password name-value pairs 19.loginCredentials = [{"username":"vivin", "password":"abAB12!@"}, 20. {"username":"jimbo", "password":"abAB12!@"}, 21. {"username":"hippy", "password":"abAB12!@"}, 22. {"username":"flippy", "password":"abAB12!@"}, 23. {"username":"batman", "password":"abAB12!@"}];
24. 25.... 26.... 27. 28.class TestRunner: 29. """A TestRunner instance is created for each worker thread."""
30. 31. # A method for each recorded page. 32. def page1(self): 33. """POST processLogin.jsp (requests 101-102)."""
34. 35. loginCredential = loginCredentials[grinder.threadNumber]; 36. username = loginCredential["username"]; 37. password = loginCredential["password"]; 38. # Expecting 302 'Moved Temporarily' 39. result = request101.POST('/login/processLogin.jsp', 40. ( NVPair('username', username), 41. NVPair('password', password), 42. NVPair('Login', 'Login'), ), 43. ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), )) 44. 45. grinder.sleep(49) 46. request102.GET('/Admin/home.jsp') 47. 48. return result
To parameterize logins, I created an array of dictionaries that store username and password combinations. Then, based on the thread number (which is zero-based and which I can access via grinder.threadNumber) I select the appropriate username and password combination, to log in.
