6/15/04

6/15/04

CORRELATION FOR WEB SCRIPT

INTRODUCTION..........................................................................................................................................2

WHAT IS CORRELATION.........................................................................................................................3

WHY DO I HAVE TO CORRELATE?................................................................................................................3WHAT ERRORS MEAN I HAVE TO CORRELATE?............................................................................................4

HOW TO CORRELATE A WEB SCRIPT.................................................................................................5

FUNCTIONS FOR CORRELATION....................................................................................................................5HOW TO IDENTIFY VALUES TO CORRELATE.................................................................................................5

A hypothetical example...........................................................................................................................6

CORRELATION WITH THE AUTO-CORRELATION ENGINE.........................................................6

RULES CORRELATION (RECORD).................................................................................................................6CORRELATION STUDIO (REPLAY)................................................................................................................7

STEPS TO PERFORM MANUAL CORRELATION................................................................................9

1. Record two copies of the script with the same business process and data:..................................92. Identify the values to correlate with the help of the WinDiff tool:..............................................103. Manually correlate the script with the web_reg_save_param function......................................124. Replace the hard-coded value in the script with the parameter.................................................155. Recap...........................................................................................................................................16How to print out the parameter value...................................................................................................17How to handle special characters (\r \n \t “) when correlating script.................................................17Cannot find the recording snapshot under the script’s data directory.................................................17Error: "File no longer available" when launching the WinDiff tool....................................................17Error: “No match found for the requested parameter 'ParameterName'. If the data you want to save exceeds xx bytes, use web_set_max_html_param_len to increase the parameter size."......................18There is a pop-up “Correlation warning” during recording...............................................................19How to manually start the “Scan action for correlation” feature........................................................19“Scan Action for Correlation” window does not come up after replay...............................................20Why does Correlation Studio fail to detect all the dynamic values......................................................20

ADDITIONAL CORRELATION HELP RESOURCES..........................................................................21

6/15/04

Introduction

VuGen, when recording a script, simply listens to the client (browser) talking to the server (Web server) and writes it all down. The complete transcript of everything that was said, the dates and time, content, requests, and replies can be found in the Recording Log (for Single Protocol) or Generation Log (for Multiple Protocol). The script is an easier to read version of this. The main difference is that the script only contains the client’s communication.

If you imagine that VuGen is an impersonator pretending to be the client (browser), the script tells VuGen what to say to the server to successfully fool it. You want the server to believe that VuGen is a real client, and thus send it the requested information.

This script has the hard-coded information of the original conversation (Browser session) that occurred between the client and server. This hard coded information may not be enough to fool the server during replay however; it may have to be correlated.

What is correlation

Correlation is where the script is modified so that some of the hard-coded values in the script are no longer hard-coded. Rather then sending the original value to the server, we may need to send different values.

For example, the original recorded script may have included the server sending the client a session identification number, something to identify the client during that particular session. This Session ID was hard-coded into the script during recording.

During replay, the server will send the client a new Session ID. You need to capture this value, and incorporate it into the script so you can send it back to the server to correctly identify yourself for this new session. If you leave the script unmodified, you will send the old hard-coded Session ID to the server. The server will look at it and think it invalid or unknown, and so will not send the pages that have been requested. Replaying a script with an old Session ID will not successfully fool the server into believing it is a client.

Correlation is the capturing of dynamic values passed from the server to the client and back. You save this captured value into a parameter, and then use this parameter in the script in place of the original value. During replay, the replay engine will now listen to what the server sends to it, and when it makes requests of the server, send this new, valid value back to the server; thus fooling the server into believing it is talking to a real client.

6/15/04

Why do I have to correlate?

If you try to replay a script without correlating first, the script will most likely fail. The server will not reply to the requests it sends. Either the Session ID is invalid so the server will not allow you into the site, or it will not allow you to create new records because they are the same as existing ones, or the server will not understand your request because it is not what it is expecting.

The following diagram illustrates a typical example where correlation is required. While recording, the client request for page A with the ID “123.” This information (123) will be intercepted by the Web recorder and placed as a hard-coded value in the script. During replay, even though the ID is supposed to be “456,” the Web Vuser sends “123” to the server causing a replay failure. You will need to correlate this value so that the correct data is sent to the server.

As a rule of thumb, any value that changes every time you connect to the server is a candidate for correlation. A correlated script will send the server the information it is looking for thus allowing the script to replay. This will allow many Vusers to replay the script many times, placing load on your server.

6/15/04

What errors mean I have to correlate?

There are no specific errors that are associated with correlation, but there are errors that could occur because a value has not been correlated. For example, if an invalid Session ID is sent to a Web server, how that server responds depends on the implementation of that server. It might send a page specifically stating the Session ID is invalid and ask you to log in again. It might send an HTTP 404 Page not found error because the requesting user did not have permissions for the specified page, and so the server could not find the page.

In general, any error message returned from the server after the script makes a request that complains about permissions can point to a hard-coded value that needs to be correlated.

How to Correlate a Web Script

Functions for correlationThere are three functions that you can use for correlation:

A list of all the three functions, along with documentation and examples, can be found in the on-line documentation. From VuGen, go to Help Function reference Contents Web and Wireless Vuser Functions Correlation Functions.

Function Description

web_reg_save_param This is the latest function with some extra features for a more powerful usage. The syntax is as following:

web_reg_save_param ( “Parameter Name” , <List of Attributes>, LAST );

Each of these parameters is a pointer to a string. That means that if they are entered as literal text, they need to be enclosed in quotes, with each parameter separated by a comma. The supported attributes include LB (Left boundary), RB (Right boundary), RelFrameID, ORD, Search, SaveOffset, SaveLen, and Convert. These attributes can appear in any order because they contain within them what they are. Detail information about each attribute can be found on the function reference online.

Web_create_html_param Both of these functions are for backward compatibility only. It is recommended that you use the web_reg_save_param function. If you need to use this function for some reason, please refer to the function help reference.

Web_create_html_param_ex

6/15/04

How to identify values to correlateWhat values in the script need to be correlated? The simplest answer is “Any value that changes between sessions required for the script to replay.”

A hypothetical exampleYou are logging onto a website. When you send the server your user name and password, it replies with a Session ID that is good for that session. The Session ID needs to be correlated for replay. You need to capture this value during replay to use in the script in place of the hard-coded value.

There are two easy ways to identify the values and to correlate:1. Correlation with the auto correlation engine , and2. Manually correlation .

Correlation with the auto-correlation engineVuGen’s auto-correlation engine allows you to automatically correlate dynamic data using the following methods:

Correlation mechanism

Description

Rules Correlation

Rules Correlation operates during recording. There are two types of Rules correlation:

Built-in CorrelationThe Built-in Correlation detects and correlates dynamic data for supported application servers. If you are recording a session with a supported application server, you can use one of the existing rules. An application server may have more than one rule. You can enable or disable a specific rule by selecting or clearing the checkbox adjacent to the rule. VuGen displays the rule definitions in the right pane.

User-defined Rules CorrelationUser-defined Rules Correlation allows you to define your own correlation rules before you record a session. The rules include information such as the boundaries of the dynamic data you want to correlate and other specifications about the match such as binary, case matching, and the instance number.

Correlation Studio

Correlation Studio operates after replay, which means that you need to run the script at least once. After replay, the studio automatically tries to find matching values from the server’s responses and the client’s request, and estimates the need for correlation. The studio is usually used to initially find places to correlate, in unfamiliar environments.

6/15/04

Rules Correlation (Record)To record with the Rules Correlation mechanism, follow the steps below.

1. Enable auto-correlation.a. Enable the auto-correlation option in VuGen’s Tools menu, under Recording

options Internet Protocol: Correlation. Click on “Enable correlation during recording” to turn on the feature.

b. ( Built-in Correlation ) Indicate the servers to which you want to apply the correlation rules. Select the checkboxes adjacent to the server names to enable the rules for that server. To enable specific rules within a server group, click the plus sign to expand the tree and select the desired rules.

c. (User-define rule correlation) To add a new rule to an existing server, select one of the existing entries and click “New Rule.” Set the properties for the rule in the right pane.

d. Set the option on how you want the recorder to behave when there is a dynamic value matching the rule being detected. The available options are Issue a pop-up message and let me decide online (Tips: Correlation warning

window on recording ) Perform correlation in script

2. Record a script.Start a recording. The Web recorder will automatically correlate the dynamic values that match the correlation rule defined on step 1. After the recording, assuming there are dynamic values being detected and correlated, you will see something similar to the following in your script:

3. This is all regarding correlation rules. Replay the script as desired.

6/15/04

Correlation Studio (Replay)

Very often, you will notice that not all the dynamic values are correlated with the use of Rules Correlation. This is likely to happen when you are recording against a session on an unsupported application server whose context is not known, and you cannot determine any correlation rules.

To handle this, the Web protocol comes with the Correlation Studio, where after running the script, it will automatically compare the server’s response for record and replay, to look for dynamic values for correlation. The list of differences will be reported to you, for you to choose the ones to correlate.

To invoke this feature, 1. Run the script once. 2. After the run, you will see the following pop-up alert prompting for “Scan Action for

correlation.” Click on “Yes” to start the scan.

Tips: Scan Action for Correlation window does not come up after replayTips: How to manually start the ‘Scan Action for Correlation’ feature

3. During the scan, the Correlation Studio will automatically compare the record and replay response for dynamic values and report the differences to you. You can see the dynamic values that are detected on the lower panel, under the “Correlation Results” tab.

6/15/04

4. Look through the differences reported by the Correlation Studio. Correlate the values on those necessary. To make the correlation process easy, you can choose the option to correlate them either one at a time (Correlate) or all at once (Correlate All). Note: Correlation Studio often reports all the dynamic values, including those where you do not need to correlate. Because of this, Mercury recommends you to use the Correlate option, and look at the value to correlate one at a time.

When you correlate a value using this mechanism, VuGen inserts a web_reg_save_param function and a comment into your script indicating that a correlation was done for the parameter. It also indicates the original value.

5. In general, you will need to repeat steps 1-4 a few times, before you can correlate all the dynamic values. The Correlation Studio is not likely to report all the differences with just one run because your script is prone to fail on a very early stage. This causes the subsequent dynamic values to go undetected.

Example:Your business process includes the following:1. Log onto the site.2. Retrieve employee information.3. Update employee information.4. Logout.

If the logon process (step 1) fails on the first run and halts the replay, step 2 and above of the example will not be executed. For such, the Correlation Studio will NOT be able to report the dynamic values of step 2 and above. To handle this case, what you will need to do is to repeat the correlation processes. The steps needed are similar to the following:

1. Run the script once.2. Identify and correlate the values needed to allow logon process.3. Rerun the script.4. Identify and correlate the values needed to retrieve employee information.5. Run the script again.6. Identify and correlate the values needed to update employee information.

There are cases where some dynamic values may not be detected by the Correlation Studio. You may need to perform manual correlation as guided on the following chapter.

Tips: Why does Correlation Studio fail to detect all the dynamic values.

6/15/04

Steps to Perform Manual Correlation

1. Record two copies of the script with the same business process and data:a. Record a script on the desired business process and save it.b. Record another copy of the script, repeating all the steps as in step 1a. As much as

possible, during recording, enter the same values in both scripts, for example, user ID, password, and fields and edit selections.

2. Identify the values to correlate with the help of the WinDiff tool:a. From the second script, go to Tools Compare with Vuser, and choose the first

recorded script. b. WinDiff will open and display the two scripts side by side. It will automatically

highlight the lines with differences in them, showing the differences in red. (If not, go to Options View Show inline differences.)

Tips: Error: "File no longer available." launching WinDiff

c. Locate the first difference, take note of it, and search the script open in VuGen for that difference. That is the original value hard-coded into the script that was different in the second script. Highlight it and copy it.Note: Differences like "lr_think_time" can be ignored. They are pacing functions, and do not represent data sent to the server.

6/15/04

d. Go to the Recording Log (for Single Protocol) or Generation Log (for Multiple Protocol) and place your cursor at the top. Press Control-F (CTRL+F) to bring up the Find window.

Paste the value copied from step 2c and search downwards. You are looking for the first occurrence of this value in the Recording Log or Generation Log. If you do not find the value, verify you are looking in the correct script’s

Recording Log or Generation Log. Remember you have two almost identical scripts here.

If you find the value, scroll up in the log, and make sure the value was sent as part of a response from the server. The first header you come across while looking up the script should be prefaced by a receiving response. This indicates that the server sent the value to the client. If the value first appears as part of a sending request, then the value originated on the client side and

6/15/04

does not need to be correlated, but rather parameterized. That is a different topic all together. The response will have a comment before it that looks like this:

*** [tid=b9 Action1 2] Receiving response from host astra.merc-int.com:80 ( 25/11/2002 12:04:00)

e. So, you have a value that is different between subsequent recordings, it was sent from the server to the client; this value most likely needs to be correlated. If the value is not different between recordings and not originating on the server and sent to the client then the value probably does not need to be correlated.

3. Manually correlate the script with the web_reg_save_param function.

After confirming that the first occurrence was part of a received response from the server, you need to figure out where to place the web_reg_save_param( ) function and what to put in it before creating the function

a. Where to place the web_reg_save_param function?

From the last step, you located the dynamic value inside the Execution Log. Pick up the text that is before the dynamic value. This text should remain constant no matter how many times you replay the script; highlight it and copy it. This is the text that will identify where to find the start of the value you are capturing.

6/15/04

Now, to identify the place to put the function, you need to replay the script once in extended log. To do this, i. Go to the Vuser menu and select “Run-Time Settings.”ii. Go to General:Logiii. Select “Enable Logging,” “Always sends messages,” “Extended log,” and all

the options under extended log.

After replay, go to the Execution Log and search for the text that you just copied from the Recording Log (for Single Protocol) or Generation Log (for Multiple Protocol). You should see a corresponding Action1.c() at the beginning of that line with a number in the brackets. That is the number of the line in the script where you need to put web_reg_save_param( ) function. The function should go right above that line in the script.

So, add a couple of blank lines to your script before the function at that line, and then type in web_reg_save_param(“UserSession”, but give it a name that means more to you than UserSession.

6/15/04

b. Identifying the boundaries for web_reg_save_param function.

i. Identifying the left boundary - The starting point to correlate.

Go back to the Recording Log (for Single Protocol) or Generation Log (for Multiple Protocol), highlight the text to the left of the dynamic value and copy it.

The amount of text you highlight should be sufficient so that it is unique in this reply from the server. It is suggested that you copy as much as possible without copying any special characters. This is shown in the Execution Log as black squares, and the actual character they represent is uncertain. After selecting a boundary, go to the top of the Servers reply, and press CTRL+F and do a search for that boundary. You want to make certain what you have selected is the first occurrence in the Servers reply. If it is not select, use more text to make it unique, or consider using the ORD parameter of the web_reg_save_param function. Once you have finalized the static text that represents the left boundary, copy it into the web_reg_save_param statement.

Tips: How to handle special characters ( \r \n \t " ) when correlating script

The function you will have so far would be web_reg_save_param (“UserSession”, “LB=input type=hidden name=userSession value=”,

ii. Identifying the right boundary - The ending point to correlate.

6/15/04

You are now going to tell the replay engine how to identify the end of the value you are trying to capture, that is, the right boundary of what you are looking for. Again, look in the Execution Log and copy the static text that appears to the right of the dynamic value you are looking for. For example, let’s say the Execution Log contained the following:

… userSession value=75893.0884568651DQADHfApHDHfcDtccpfAttcf>…

Then, the example so far to save the number into the parameter UserSession would be

web_reg_save_param(“UserSession”, “LB=input type=hidden name=userSession value=”, “RB=>”,LAST);

Note: In choosing a right boundary, make sure you choose enough static text to specify the end of the value. If the boundary you specify appears in the value that you are trying to capture, then you will not capture the whole value.

4. Replace the hard-coded value in the script with the parameter.

6/15/04

Once you have created the parameter, the next step is to replace the hard-coded occurrences with the parameter. Look through the script for the original value. Where you find it, delete the value and replace it with the parameter. Note, only the value you want replaced is deleted. The characters around it remain.

Example:Change:"Name=userSession", "Value=75893.0884568651DQADHfApHDHfcDtccpfAttcf", , To:"Name=userSession", "Value= {UserSession}", ENDITEM,…

At this point, you are ready to run the script to test if it works, or if it needs further correlation, more work on this correlation.

5. RecapThat was a lot of looking through the logs and checking of values. Let’s just recap what you have done. You have identified a value that you think needs to be correlated. You then identified in the script where to place the statement that would ultimately capture

6/15/04

and save the value into a parameter. You then placed the statement, and gave the text strings that appear on either side of the value that you are looking for so that it can be found.

The flow of logic for this is the correlation functions tell the replay engine, what to look for in the next set of replies from the server. The replay engine makes a request of the server. The server replies. The replay engine looks thorough the replies for the left and right boundaries. If it finds them, what is in-between is then saved to a parameter of the name specified.

Remember, the parameter cannot have a value until after the next statement is executed. The correlation statement only tells the replay what to look for; it does not assign a value to the parameter. Assignment of a value to the parameter does not happen until after generating request to the server and looks in the reply. If you have a case where a correlation statement is followed by a function that attempts to use the parameter, the statement is in the wrong place, and the script will fail.This is always incorrect:

web_reg_save_param(…);Web_submit_data (… {Parameter}…);

In-between the two, there needs to be a request of the server that causes it to reply with the value you are trying to capture.

When you are done with the above, the next step is to replace the hard-coded occurrences with the parameter. The script is then ready for replay.

Tips and Tricks

How to print out the parameter valueSometimes you want to print out the value that was assigned to a parameter. To do this,

6/15/04

use the lr_eval_string function and the lr_output_message function. For example, to print the value of the parameter to the Execution Log:

lr_output_message(“ Value Captured = %s", lr_eval_string("{ParameterName}"));

If you find that the value being substituted is too long, too short, or completely wrong, printing out the value will help identify the changes you need to make to the correlation function.

How to handle special characters (\r \n \t “) when correlating scriptThere are some special characters and cases when specifying the boundaries. Special characters (e.g., '\', '"\) in the boundaries must be "escaped" by a backslash.

Example:

Recording Log (for Single Protocol), or Generation Log (for Multiple Protocol)

Left boundary Right boundary

Value="57685" "Value=\"" "\""

Value_"\item\"value’7875’ "Value_\"\\item\\\"value’ " "’"

Value="7898756"

"Value=\r\n\"" "\""

Cannot find the recording snapshot under the script’s data directoryIf there is no recording snapshot displayed for the selected step, check the following possible reasons: The script was recorded with a VuGen version 6.02 or earlier. The imported Actions do not contain snapshots. The Vuser files are stored in a read-only directory, and VuGen could not save the

replay snapshots. Snapshots are not generated for certain types of steps. For example, a snapshot will

not be generated if the step represents navigation to a resource. The following option was turned off to disable snapshot generation:

Tools General options Correlation tab Save correlation information during replay.

Error: "File no longer available" when launching the WinDiff tool

The Compare tool in VuGen makes use of a standard utility called WinDiff. WinDiff has certain limitations, in that it cannot handle directories and files with spaces. To resolve the issue, move the script directory to a new drive without any spaces, and try to keep the names short. Alternatively, open WinDiff manually, and open up the scripts via the menu options. Simply compare the action1.c sections of the scripts in question.

Error: “No match found for the requested parameter 'ParameterName'. If the data you want to save exceeds xx bytes, use web_set_max_html_param_len to increase the parameter size."By default, the web_reg_save_param function issues the above error if the boundaries cannot be found. If you need to use this function for other purposes, and would like to

6/15/04

avoid errors, use the "Notfound=warning" attribute so that the replay will just issue a warning.

Now, the above error actually suggested two possible causes for the error,

1. The boundaries defined cannot be found, and2. The data you want to save exceeds 256 bytes.

The advice given is just a recommendation that you must take into consideration. Was the value you were trying to capture more than 256 characters long? In the above example, it was only 40 characters long. Have a look at the Recording Log (for Single Protocol) or Generation Log (for Multiple Protocol) and see how long the original value is. Have a look at the second recording made earlier and see how long it was in that script. Turn on the Extended log (Run-Time Settings Log Extended log data returned from server) and have a look at how long it is in the Execution Log.

If all occurrences of the script were much shorter that the maximum parameter length, then the problem is either with the web_reg_save_param in the wrong place, or that the boundaries are incorrect. Go back and look at the boundaries that you selected, look at the placement of the web_reg_save_param. Is it immediately before the statement that causes the server to reply with the data you are looking for?

If, at any time, any of these values was close to being around 200 characters, or exceeded 200 characters, then add a web_set_max_html_param_len statement to the start of the script to make the maximum longer than 256 characters. For example,

web_set_max_html_param_len ( “length” );

This statement tells the replay engine to look for larger matches for the left and right parameter. When it finds the left boundary, it will look ahead the max parameter length for the right boundary. This setting is script wide and takes effect from when it is executed; it only needs to appear in the script once. Looking for longer matches uses up more memory and CPU processing to search through the returned text from the server. For this reason, do not set it too high, or you will be making your script less scalable. That is, you will reduce the number of Vusers that can run it on a given machine. Try to have the maximum parameter length no more than 100 characters greater that what you are expecting.

There is a pop-up “Correlation warning” during recordingWhen you have auto-correlation with the option to “Issue a popup message and let me decide online” turned on, you will see a pop-up warning when dynamic values matching the correlation rule is detected.

Example:

6/15/04

You can set if you want to correlate it or ignore the value. You can also set the future behavior for correlating as needed.

How to manually start the “Scan action for correlation” featureTo manually run the “Scan Action for correlation” feature, run the script once. Then, select Vuser “Scan Action for Correlations” to start the scan.

“Scan Action for Correlation” window does not come up after replayThis window may not come up if you have set to disabled it before. To re-enable this feature:1. Go to Tools General options Correlation tab.2. Enable the option for “Show Scan for correlations popup after replay of Vuser.”

6/15/04

Why does Correlation Studio fail to detect all the dynamic valuesThe following are possible causes where Correlation Studio failed to detect all the dynamic values, and suggestion to overcome it.

1. The difference is only a few bytes in length.By default, the Web correlation only reports dynamic values that are greater than four characters. To change the limit:

a. Go to Tools General options Correlation Tab Advanced….b. Click on the “Advanced…” button.c. Modify the value for “Ignore Differences Shorter than x characters.”

2. The dynamic value is generated by the client and not by the server. For example, you

have a client side JavaScript that generates some important dynamic values. Since the Web replay does not execute client side scripting, you will need to add your custom code in the script to deal with this.

3. The replay response’s HTML has a different format than the recording’s. For this, you will need to perform manual correlation.

Additional Correlation Help ResourcesThe examples in the Function reference contain a lot of data and examples on how to use these functions. Looking them over is recommended.

The Customer Support website has a video for download that goes over correlation. You

6/15/04

can get it from http://support.mercury.com. After logging in, go to Downloads Browse. Select the Mercury Interactive downloads radio button, choose “LoadRunner” from the product selection drop-down box, and click on the “Retrieve” button. Under “Training,” select the “LoadRunner Web Script Correlation Training” link.