Fog Light Experience Viewer Guide

5.5.2Foglight®Foglight®Foglight®

Foglight Experience ViewerUser Guide

© 2009 Quest Software, Inc. ALL RIGHTS RESERVED.This guide contains proprietary information protected by copyright. The software described in this guide is furnished under a software license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose other than the purchaser's personal use without the written permission of Quest Software, Inc.

If you have any questions regarding your potential use of this material, contact:Quest Software World HeadquartersLEGAL Dept5 Polaris WayAliso Viejo, CA 92656www.quest.comemail: [email protected]

Refer to our Web site for regional and international office information.

TrademarksQuest, Quest Software, the Quest Software logo, AccessManager, ActiveRoles, Aelita, Akonix, AppAssure, Benchmark Factory, Big Brother, BusinessInsight, ChangeAuditor, ChangeManager, DeployDirector, DirectoryAnalyzer, DirectoryTroubleshooter, DS Analyzer, DS Expert, ERDisk, Foglight, GPOADmin, Imceda, IntelliProfile, InTrust, Invirtus, iToken, I/Watch, JClass, Jint, JProbe, LeccoTech, LiteSpeed, LiveReorg, LogADmin, MessageStats, Monosphere, NBSpool, NetBase, NetControl, Npulse, NetPro, PassGo, PerformaSure, Quest Central, Quest vToolkit, Quest vWorkSpace, ReportADmin, RestoreADmin, SelfServiceADmin, SharePlex, Sitraka, SmartAlarm, Spotlight, SQL LiteSpeed, SQL Navigator, SQL Watch, SQLab, Stat, StealthCollect, Storage Horizon, Tag and Follow, Toad, T.O.A.D., Toad World, vAutomator, vControl, vConverter, vFoglight, vOptimizer Pro, vPackager, vRanger, vRanger Pro, vSpotlight, vStream, vToad, Vintela, Virtual DBA, VizionCore, Vizioncore vAutomation Suite, Vizioncore vBackup, Vizioncore vEssentials, Vizioncore vMigrator, Vizioncore vReplicator, Vizioncore vTraffic, Vizioncore vWorkflow, WebDefender, Webthority, Xaffire, and XRT are trademarks and registered trademarks of Quest Software, Inc in the United States of America and other countries. Other trademarks and registered trademarks used in this guide are property of their respective owners.

DisclaimerThe information in this document is provided in connection with Quest products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Quest does not make any commitment to update the information contained in this document.

License Credits and Third Party InformationTo view license credit information, click the License Credits link on the Welcome to Foglight online help page.

User GuideAugust 2009Version 5.5.2

http://www.quest.com

mailto:[email protected]

Table of Contents

Introduction to this Guide ...................................................................................................................................9About Foglight ................................................................................................................................................................ 10About Foglight Experience Viewer ................................................................................................................................. 10About this Guide............................................................................................................................................................. 10Foglight Documentation Suite ........................................................................................................................................ 12

Core Documentation Set ....................................................................................................................................... 13Cartridge Documentation Sets .............................................................................................................................. 13Foglight Experience Viewer Documentation Set ................................................................................................... 13Feedback on the Documentation........................................................................................................................... 14

Text Conventions ........................................................................................................................................................... 14About Quest Software, Inc. ............................................................................................................................................ 15

Contacting Quest Software.................................................................................................................................... 15Contacting Quest Support ..................................................................................................................................... 15

Introducing the Foglight Experience Viewer (FxV) .........................................................................................17Architecture Overview .................................................................................................................................................... 18

How Are Hits Processed?...................................................................................................................................... 18What Are Hit Filters and Transaction Filters?........................................................................................................ 19How Is Hit Analysis Performed? ............................................................................................................................ 20How Do You Perform a Search? ........................................................................................................................... 20How Do You Examine a Session?......................................................................................................................... 21

Data Model Overview ..................................................................................................................................................... 22Custom Fields........................................................................................................................................................ 23Metrics ................................................................................................................................................................... 24Hits ........................................................................................................................................................................ 24Sessions ................................................................................................................................................................ 26Transactions .......................................................................................................................................................... 27

4 Foglight Experience ViewerUser Guide

FxV Browser Interface .................................................................................................................................................... 28Logging In .............................................................................................................................................................. 29Automatic Login via Foglight.................................................................................................................................. 29FxV Screen Elements ............................................................................................................................................ 30

Performing Searches ........................................................................................................................................ 35Time Constraints............................................................................................................................................................. 37Session Search............................................................................................................................................................... 38Transaction Search......................................................................................................................................................... 40Hit Search ....................................................................................................................................................................... 43Search Result Limits....................................................................................................................................................... 48Saved Searches ............................................................................................................................................................. 48Custom Search Screens................................................................................................................................................. 49

Analyzing User Sessions.................................................................................................................................. 51Session Explorer Toolbar ............................................................................................................................................... 52Details View .................................................................................................................................................................... 53Timeline View ................................................................................................................................................................. 54Hit Inspector View........................................................................................................................................................... 55Content Replay View ...................................................................................................................................................... 57Transactions View .......................................................................................................................................................... 61Combined Sessions View ............................................................................................................................................... 63

Defining Custom Search Screens.................................................................................................................... 65Creating and Editing Saved Searches............................................................................................................................ 66Creating and Editing Custom Search Screens ............................................................................................................... 67

Setting User Preferences.................................................................................................................................. 71Account Information........................................................................................................................................................ 72Session Explorer Settings............................................................................................................................................... 72Search Settings .............................................................................................................................................................. 75

Configuring the Hit Analysis Process ............................................................................................................. 79Hit Filters......................................................................................................................................................................... 80

Hit Filter Match Conditions..................................................................................................................................... 80Hit Filter Actions..................................................................................................................................................... 81

Table of Contents 5

Hit Filter Configuration............................................................................................................................................90Hit Filter Additional Information ..............................................................................................................................92

Transaction Filters...........................................................................................................................................................92Transaction Events.................................................................................................................................................93Transaction Event Match Conditions......................................................................................................................94Transaction Event Actions......................................................................................................................................97Transaction Filter Storage Configuration..............................................................................................................104Transaction Filter Configuration ...........................................................................................................................105Transaction Filter Additional Information..............................................................................................................106

Special Events ..............................................................................................................................................................106Special Events Configuration ...............................................................................................................................107Special Events Additional Information..................................................................................................................108

Custom Fields ...............................................................................................................................................................108Custom Fields Attributes ......................................................................................................................................109Custom Fields Configuration ................................................................................................................................113Custom Fields Additional Information...................................................................................................................115

Metrics...........................................................................................................................................................................115Metrics Configuration ...........................................................................................................................................115Metrics Additional Information ..............................................................................................................................117

Captured Metadata .......................................................................................................................................................117Scripts ...........................................................................................................................................................................118

Scripts Configuration ............................................................................................................................................119Scripts Additional Information...............................................................................................................................120

Sensitive Data Protection..............................................................................................................................................121Sensitive Hit Details .............................................................................................................................................121Sensitive Response Content Expressions ...........................................................................................................123

Hit Analysis Configuration Options................................................................................................................................125Session Configuration ..........................................................................................................................................125Archiver Configuration..........................................................................................................................................126Script Configuration..............................................................................................................................................127Other Configuration ..............................................................................................................................................127

Hit Analysis Configuration Change Log ........................................................................................................................128Hit Analysis Import/ Export Configuration .....................................................................................................................130

Hit Analysis Examples.....................................................................................................................................133


“Simple Data Extraction” Example................................................................................................................................ 134“Event Occurrence Counting” Example ........................................................................................................................ 134“Session-Level Event Occurrence Counting” Example................................................................................................. 135“Parsing HTML Content” Example................................................................................................................................ 136“Parameterized Metrics” Example ................................................................................................................................ 137“Buy Tunnel” Example .................................................................................................................................................. 138

“Buy Tunnel”: Hit and Transaction Filters Overview ............................................................................................ 139Transaction Event: Add to Cart ............................................................................................................................ 139Transaction Event: Checkout............................................................................................................................... 140Transaction Event: Shipping ................................................................................................................................ 141Transaction Event: Payment................................................................................................................................ 141Transaction Event: Confirmation.......................................................................................................................... 143Transaction Event: Cart Abandoned.................................................................................................................... 145Hiding Sensitive Data........................................................................................................................................... 147

Configuring the Analysis Repository ............................................................................................................ 149Overview....................................................................................................................................................................... 150Architecture................................................................................................................................................................... 151

The Session Table ............................................................................................................................................... 151The Transaction Tables ....................................................................................................................................... 153

Configuration ................................................................................................................................................................ 154Analysis Repository Initial Setup.......................................................................................................................... 154Analysis Repository Detailed Configuration......................................................................................................... 159

Data Transfer................................................................................................................................................................ 160Access .......................................................................................................................................................................... 161

Accessing the Analysis Repository from Foglight ................................................................................................ 161Accessing the Analysis Repository from Toad for MySQL .................................................................................. 161

Performing Advanced FxV Administration ................................................................................................... 165Archivers....................................................................................................................................................................... 166Collectors...................................................................................................................................................................... 167Superuser Tasks........................................................................................................................................................... 168

Appliance Maintenance........................................................................................................................................ 168Metrics ................................................................................................................................................................. 172

Managing User Account Settings .................................................................................................................. 175

Table of Contents 7

Users and User Groups ................................................................................................................................................176User Preference Settings .....................................................................................................................................176Pre-Defined User Accounts..................................................................................................................................176Managing User Accounts .....................................................................................................................................177User Group Settings.............................................................................................................................................178Pre-Defined User Groups.....................................................................................................................................181Managing User Groups ........................................................................................................................................182

Resource Groups ..........................................................................................................................................................183Preference Groups........................................................................................................................................................184

Appendix: Java Regular Expressions in FxV Hit Analysis ..........................................................................187Regular Expressions Overview .....................................................................................................................................188Regular Expressions in FxV..........................................................................................................................................192

Looking for “Matches” (Yes or No) .......................................................................................................................192Extracting a Single Value .....................................................................................................................................193Identifying Sensitive Response Content...............................................................................................................195

FxV Regular Expression Tester ....................................................................................................................................198

Appendix: FxV Metrics ....................................................................................................................................199Archiver Metrics ............................................................................................................................................................200

Hit Summary Metrics ............................................................................................................................................200Analysis Repository Metrics .................................................................................................................................208Archiver Capacity Metrics.....................................................................................................................................210Archiver Health Metrics ........................................................................................................................................215Archiver Appliance Metrics...................................................................................................................................224Archiver Search Usage Metrics............................................................................................................................228Script Metrics........................................................................................................................................................233

Collector Metrics ...........................................................................................................................................................234Collector Health....................................................................................................................................................234

Server Metrics ...............................................................................................................................................................236Server Health .......................................................................................................................................................237Server Sessions ...................................................................................................................................................239Server Replay Cache ...........................................................................................................................................239Server Requests to Archivers...............................................................................................................................240

Error/Warning Metrics ...................................................................................................................................................241


Errors ................................................................................................................................................................... 242Warnings.............................................................................................................................................................. 242

Appendix: Glossary ........................................................................................................................................ 243Component Architecture Terms.................................................................................................................................... 244Data Model Terms ........................................................................................................................................................ 250Product Feature Terms................................................................................................................................................. 253Networking and Database Terms ................................................................................................................................. 256

Index ................................................................................................................................................................. 259

Introduction to this Guide

This chapter provides information about what is contained in the FxV User Guide. It also provides information about the Foglight documentation suite and Quest Software.

This chapter contains the following sections:

About Foglight .............................................................................................................................10About Foglight Experience Viewer ..............................................................................................10About this Guide ..........................................................................................................................10Foglight Documentation Suite .....................................................................................................12Text Conventions .........................................................................................................................14About Quest Software, Inc...........................................................................................................15


About FoglightFoglight is an application management solution that reduces or eliminates service disruptions to unify IT and the business. Unlike other solutions, it provides a correlated, 360 degree view of your applications from end user to database and from service levels to infrastructure—to source the root cause of every incident impacting your business and fix them quickly. Foglight correlates data from multiple perspectives into a single version of the truth to provide deep insight into the service relationships that exist between end users, the business and infrastructure components. Its unique adaptive technology rapidly adjusts to change for improved application performance and service levels, reduced operational cost and risk, and enhanced visibility for all stakeholders.

About Foglight Experience ViewerFoglight Experience Viewer (FxV) is a network-based appliance that allows for capture, storage, analysis, and replay of actual end user sessions in their entirety. Data is provided to the Foglight Experience Viewer by the Foglight Experience Monitor (FxM), which gathers the raw Web traffic from a switch span port or other network tap.

Foglight Experience Viewer allows customers to:

• Understand the impact of infrastructure problems on end users and business.

• Identify the business impact associated with a poor user experience.

• Determine the affected end users in order to better prioritize incident and problem resolution.

About this GuideThe FxV User Guide provides in-depth coverage of the Foglight Experience Viewer features, focusing on strategies, tactics, and real-world examples.

The intended audience for this document are all users who need a deeper understanding of the Foglight Experience Viewer appliance.

The FxV User Guide is organized as follows:

Chapter 1, Introducing the Foglight Experience Viewer (FxV)—provides an overview of the FxV data model used for hit analyzing, hit searching, and session replay via the FxV browser interface.

Introduction to this Guide 11About this Guide

Chapter 2, Performing Searches—presents the FxV functionality available to users when performing data searches, and outlines the types of data searches and the results they provide.

Chapter 3, Analyzing User Sessions—describes how FxV users can analyze a session of interest, and outlines the analysis methods that are currently available.

Chapter 4, Defining Custom Search Screens—describes the settings that can be configured when configuring a Custom Search Screen.

Chapter 5, Setting User Preferences—outlines the preferences that users can view and edit (depending on their privileges); that is, account information, replay settings, and search settings.

Chapter 6, Configuring the Hit Analysis Process—presents the resources available to users for configuring the FxV Hit Analysis process (that is, resources defining the conditions and actions used to analyze hits and sessions, as they are captured).

Chapter 7, Hit Analysis Examples—provides several examples illustrating common uses of Hit Filters and Transactions Filters that FxV users can define via the FxV browser interface.

Chapter 8, Configuring the Analysis Repository—presents the operations that an administrator can perform to configure a secondary database for long-term storage of session and transaction data captured by FxV. This database is configured via the FxV user interface, and the data stored can be accessed through Foglight or via direct SQL queries.

Chapter 9, Performing Advanced FxV Administration—presents the operations that an administrator can perform to configure and administer the system via the FxV browser interface.

Chapter 10, Managing User Account Settings—presents the concepts an administrator must know in order to manage the FxV users and the capabilities they have when using the FxV resources.

Appendix A, Java Regular Expressions in FxV Hit Analysis—provides guidance on how to create regular expressions and how to use them with in FxV.

Appendix B, FxV Metrics—provides the complete list of FxV system metrics.

Appendix C, Glossary—provides complete Foglight Experience Viewer terminology.


Foglight Documentation SuiteThe Foglight documentation suite is made up of the core documentation set, plus the documentation set for each Foglight cartridge that you deploy. Documentation is provided in a combination of online help, PDF and HTML formats.

• Online Help: You can open the online help by selecting the Help tab from Foglight’s action panel.

• PDF: The complete Foglight documentation set is available in Adobe PDF from SupportLink. The PDF documentation can also be found in the Documentation folder on the Foglight DVD. The following subset is available from the computer Foglight is installed on: Administration and Configuration Guide, User Guide, Command-Line Reference Guide, Web Component Guide and the Web Component Tutorial. In addition, the cartridges ship with PDF guides. To view the installed PDF guides, in Windows go to Start > Programs > Quest Software > Foglight 5.5.2 > Documentation. The default location of the documentation after installation is <foglight_home>/docs. Adobe Reader is required.

• HTML: Release Notes are provided in HTML format.

• Videos: Tutorial videos are not provided with the product, but are available on SupportLink. They provide an easy and accessible way to learn about key features and help you get started with Foglight.

Introduction to this Guide 13Foglight Documentation Suite

Core Documentation SetThe core documentation set consists of the following documents:

• Release Notes (HTML)

• Middleware Release Notes (HTML)

• Getting Started Guide (PDF)

• What’s New Guide (PDF)

• System Requirements and Platform Support Guide (PDF)

• Upgrade Guides (PDF)

• Installation and Setup Guide set (all in PDF format)

• Administration and Configuration Guide (PDF and online help)

• Foglight User Guide (PDF and online help)

• Command-Line Reference Guide (PDF and online help)

• Transition Guide (PDF)

• Web Component Guide (PDF and online help)

• Web Component Tutorial (PDF and online help)

• Web Component Reference (online help)

Cartridge Documentation SetsWhen you deploy a cartridge, the documentation set for it is installed. The online help for the cartridge is integrated automatically with the core Foglight help. When you open the help, the name of the cartridge is displayed in a top level entry within the table of contents.

Most cartridges include additional PDF guides, which may be one or more of the following: a Getting Started Guide, an Installation Guide, a User Guide, and a Reference Guide. Cartridges also generally include their own Release Notes.

Foglight Experience Viewer Documentation SetThe Foglight Experience Viewer documentation set consists of the following documents:

• Release Notes (HTML)


• Quick Installation Guide for Dell PowerEdge Systems (PDF)

• Installation and Administration Guide (PDF)

• User Guide (PDF)

• Upgrade Field Guide (PDF)

• Security and Compliance Field Guide (PDF)

• Dell Remote Access Controller Guide (PDF)

Feedback on the DocumentationWe are interested in receiving feedback from you about our documentation. For example, did you notice any errors in the documentation? Were any features undocumented? Do you have any suggestions on how we can improve the documentation? All comments are welcome. Please submit your feedback to the following email address:

[email protected]

Please do not submit Technical Support requests to this email address.

Text ConventionsThe following table summarizes how text styles are used in this guide:

Convention Description

Code Monospace text represents code, code objects, and command-line input. This includes:• Java language source code and examples of file contents• Classes, objects, methods, properties, constants, and events• HTML documents, tags, and attributes

Variables Monospace-plus-italic text represents variable code or command-line objects that are replaced by an actual value or parameter.

Interface Bold text is used for interface options that you select (such as menu items) as well as keyboard commands.

Introduction to this Guide 15About Quest Software, Inc.

About Quest Software, Inc.Quest Software, Inc., a leading enterprise systems management vendor, delivers innovative products that help organizations get more performance and productivity from their applications, databases, Windows infrastructure and virtual environments. Through a deep expertise in IT operations and a continued focus on what works best, Quest helps more than 100,000 customers worldwide meet higher expectations for enterprise IT. Quest provides customers with client management as well as server and desktop virtualization solutions through its subsidiaries, ScriptLogic and Vizioncore. Quest Software can be found in offices around the globe and at www.quest.com.

Contacting Quest Software

Refer to our web site for regional and international office information.

Contacting Quest SupportQuest Support is available to customers who have a trial version of a Quest product or who have purchased a commercial version and have a valid maintenance contract. Quest

Files, components, and documents

Italic text is used to highlight the following items:• Pathnames, file names, and programs• The names of other documents referenced in this guide

Convention Description

Email [email protected]

Mail Quest Software, Inc.World Headquarters5 Polaris WayAliso Viejo, CA 92656USA

Web site www.quest.com

mailto:[email protected]




Support provides around the clock coverage with SupportLink, our web self-service. Visit SupportLink at: http://support.quest.com.

From SupportLink, you can do the following:

• Quickly find thousands of solutions (Knowledgebase articles/documents).

• Download patches and upgrades.

• Seek help from a Support engineer.

• Log and update your case, and check its status.

View the Global Support Guide for a detailed explanation of support programs, online services, contact information, and policy and procedures. The guide is available at: http://support.quest.com/pdfs/Global Support Guide.pdf.

http://support.quest.com

http://support.quest.com/pdfs/Global Support Guide.pdf

1

Introducing the Foglight Experience Viewer (FxV)

Foglight Experience Viewer allows for capture, storage, analysis, and replay of actual end user sessions in their entirety. Data is provided to the Foglight Experience Viewer by the Foglight Experience Monitor, which gathers the Web traffic from a switch span port or other network tap.

Foglight Experience Viewer allows customers to:

• Understand the impact of infrastructure problems on end users and business

• Identify the business impact associated with a poor user experience

• Determine affected end users in order to better prioritize incident and problem resolution

This chapter is an overview of the FxV data model used for hit analyzing, hit searching, and session replay via the FxV browser interface.


Architecture Overview .................................................................................................................18Data Model Overview ..................................................................................................................22FxV Browser Interface .................................................................................................................28


Architecture OverviewThis section is intended for FxV’s first-time users, and provides information about the following topics:

• “How Are Hits Processed?” on page 18

• “How Is Hit Analysis Performed?” on page 20

• “What Are Hit Filters and Transaction Filters?” on page 19

• “How Do You Perform a Search?” on page 20

• “How Do You Examine a Session?” on page 21

How Are Hits Processed?The Foglight Experience Viewer (FxV) and the Foglight Experience Monitor (FxM) components work together for data capture and storage, as illustrated in the following figure.

Logical Architecture Model Figure 1

Introducing the Foglight Experience Viewer (FxV) 19Architecture Overview

The FxM Agent consumes raw HTTP packets, determines the session ID for each hit, and passes raw hit data to its local Collector (through a local, shared memory interface). The FxM Collector accumulates these raw hits and then sends them across a network socket, to the FxV Tcpbatches listener. The FxV Tcpbatches process opens and listens on a TCP socket (FxV’s port 7623), and when hit data is received, it forwards it to its local Archiver, through a shared memory interface.

The FxV Archiver reads each hit, performs hit processing and analysis, then updates the searchable database (for the hit, the session that contains the hit, and any related transactions). The Archiver maintains a subset of the capture database on local disk, and manages the transfer of database segments to the storage tier (if present).

A secondary Analysis Repository can be implemented as part of the Storage service, to provide access to session and transaction data over extended periods of time.

What Are Hit Filters and Transaction Filters?A Hit Filter is a collection of conditions and actions used to analyze hits as they are captured. Hit Filters can be used to detect and alert on any per-hit condition, to mark interesting hits for later searching, and to manage hit storage. Hit Filters are powerful in their ability to populate custom fields and increment metrics. Hit Filters can also be used to define events within Transaction Filters.

A Transaction Filter is a collection of conditions and actions used to analyze sessions, as they are captured, and to identify the distinct transactions within those sessions. Transaction Filters can be used to detect and alert on any activity within a session, and to calculate custom metrics across aggregated sessions or transactions. A Transaction Filter can detect multiple events that provide information about site problems and client activity within a related set of pages. Like Hit Filters, Transaction Filters give relevant context to the captured data to make searching and reporting easier.

Important A key distinction between Hit Filters and Transaction Filters is that Hit Filters work on a single hit, while Transaction Filters can work across multiple hits in the same session.

Hit Filters and Transaction Filters can also invoke Groovy scripts to perform additional capture-time processing, such as evaluating complex conditional expressions or regular expressions, transforming hit data for use in searchable fields or metrics, or interacting with external systems through a generic HTTP interface.


How Is Hit Analysis Performed?The Archiver performs real-time analysis with Hit Filters and Transaction Filters that profile and alert upon end user behaviors and problems.

The Archiver takes the following actions for each hit:

• Validates and parses the hit into a hit model object.

• Determines the type of the hit (HTML versus non-HTML) by scanning response headers and content.

• For HTML hits, creates a string representation of the content for content analysis (which may require decompression).

• For HTML hits, extracts the page title from the response content.

• Finds and updates the session model for the hit, creating a new session model if necessary.

• Applies Hit Filters, which calculate the final hit status and optionally update the hit/session or metrics.

• Applies Transaction Filters, which identify specific transactions within the session, calculate transaction status, and optionally update the session/ transaction(s) or metrics.

• Optionally for HTML hits, breaks the response content down into keywords for full-text searching (prepares to update inverted index).

• Writes hit information to the capture database for later searching and replay.

How Do You Perform a Search?The user can perform a hit, session, or transaction search using the FxV browser interface, or as the result of a drilldown from Foglight.

When performing a search, the FxV Server accesses data stored on various Archiver and Storage components in the appliance group, assembles the results, and returns the merged view of the search results. If a storage tier is present, the Storage components are searched first, to avoid any possible work for the Archivers.

The data stored in the Analysis Repository is not accessible directly from the FxV user interface. Instead, this data can be accessed from within Foglight, via a Foglight data source (as part of the Foglight Experience Viewer cartridge). It can also be queried directly via SQL, using Quest’s Toad® for MySQL or any other third-party reporting tool.

Introducing the Foglight Experience Viewer (FxV) 21Architecture Overview

How Do You Examine a Session?After users have searched and found a session of interest, they can inspect trace-level details for the user session, or visually replay the session (presenting the Web applications as the end user originally saw them). Session exploring provides visibility into every aspect of the user experience:

• Complete hit details for the entire HTTP request and response for any hit in the session

• A visual presentation of each page as originally served by the Web server

• A visual presentation of each page with user input (forms/links)

• The original HTML source for each page generated by the Web server

• Any errors (HTML or other source) detected by Hit Filters and Transaction Filters

• Session and transaction durations, hit counts, and other aggregate details

The FxV Session Explorer offers several ways of analyzing the hit/session/transaction search results, via the following types of views: Details, Timeline, Hit Inspector, Content Replay, Transactions, and Combined Sessions.

The following figure illustrates an example of a Timeline view for a user session.

User Session Visual Replay Example Figure 2


Data Model OverviewAll data captured and stored by FxV starts with the individual hits processed by the FxM Collectors and passed to the FxV Archivers. As illustrated in the following figure, hits are grouped into sessions based on their session ID. Every hit is part of one session, at most. A session may contain any number of transactions, as defined by Transaction Filters.

Hit-Session-Transaction Diagram Figure 3

The Hit, Session, and Transaction data types represent the core of the FxV data model. Other elements include Custom Fields and Metrics. The interactions between the elements of the data model are complex, as illustrated in the following diagram.

Introducing the Foglight Experience Viewer (FxV) 23Data Model Overview

Data Model Diagram Figure 4

For additional details about the FxV data model, see these sections:

• “Custom Fields” on page 23

• “Metrics” on page 24

• “Hits” on page 24

• “Sessions” on page 26

• “Transactions” on page 27

Custom FieldsThe FxV Hit Analysis process is highly customizable by virtue of user-defined Hit Filters and Transaction Filters. These filters allow the captured data to be customized by adding custom fields to any hit, session, or transaction. A custom field can consist of


any piece of data that might be useful as a search term or a search result. The values stored in custom fields can come from a variety of sources. Typically, they are some variation on a value extracted from a hit. In many cases, it is useful to extract data from a specific hit and store it as a custom field on the enclosing session and/or transaction, in order to allow the session to be found based on that data (for example, a username).

Custom fields on individual hits are less common, since most of the data typically used to populate a custom field value is already part of the hit. However, it may be useful to extract a header, field, or cookie value and store it as a hit-scoped custom field, in order to allow it to appear directly in hit search results.

MetricsFxV metrics are distinct from any hit, session, or transaction. However, like custom fields, metrics are updated by Hit Filters and Transaction Filters. Essentially, FxV user-defined metrics are global numeric custom fields, whose values are not set explicitly (they are only incremented or decremented).

Although metrics are not directly associated with hits, sessions, or transactions, metric updates are. This means that for any hit, session, or transaction, it is possible to see any metrics that were modified during its processing, specifically:

• If a Hit Filter updates a metric, that update is associated with the hit being processed, as well as with the active session. However, that update is NOT associated with any transactions.

• If a Transaction Filter updates a metric, that update is associated with the active transaction, as well as with the active session. However, that update is NOT associated with any hits.

HitsA hit is a single request made by a client browser and the corresponding response from the Web server. FxV captures and stores the following data for each hit:

• Overall Hit Status (determined by Hit Filters)

• Timing Details

• End to End Time• Client Time• Processing Time


• Request Details

• Client IP address• HTTP Method (GET, POST, etc.)• Full URL• Server• Path

• Request Headers (for example, User-Agent and Referer)

• Request Fields (including form field values)

• Response Details

• HTML Title• HTML Base HREF• Timestamp• Server IP• HTTP Version• HTTP Code• Total Response Length

• Response Headers (for example, Content-Type)

• Cookies

• Custom Fields values (determined by Hit Filters)

• Metric Updates caused (determined by Hit Filters)

• Archiver Details

• (FxM) Collector ID• Collector Version• Hit Identifier• Batch ID• Session ID• FxM URL ID• FxM Referer ID• Exception(s)• Response Contains HTML Tag• Response Contains Frameset Tag


• Hit Filter Matches (determined by Hit Filters)

• Response Content (which include the Response Content Keywords)

Note The following values are available on each hit for hit analysis, but are only stored at the session level (therefore, you can’t search for hits based on them):

• City

• Region

• Country

• ISP

• Subnet

• Browser Type

• Username

SessionsA session is a list of hits (with the same session ID) that were submitted by the same browser client in the same visit to the site. The following data is calculated and stored for each session:

• Details

• Session ID• Start Time• Last Hit Time• Duration• Session Active (In Progress)• Stop Reason• Time of First Error• Time of Last Error

• Client Details

• Client IP• Browser Type• ISP• City• Region


• Country• Subnet• Username

• Hit Counts

• Total Hits• Stored Hits• HTML Hits• Total Errors• Total Warnings• HTML Errors• HTML Warnings

• Transaction Counts

• Total Completed Transactions• Completed Transaction Errors• Completed Transaction Warnings

• Timings

• Total Client Time• Total End to End Time• Total Processing Time

• Custom Field values (determined by Hit Filters and/or Transaction Filters)

• Metric Updates caused (determined by Hit Filters and/or Transaction Filters)

• Hit Filter Match Counts (determined by Hit Filters)

TransactionsA transaction is a portion of a session that matched a Transaction Filter. Transactions are considerably different than hits and sessions, in that they only exist as a result of specific Transaction Filter definitions. The following data is stored for each transaction:

• Details

• Transaction Filter Name• Session ID• Start Time


• Last Hit Time• Duration• Transaction Active (In Progress)• Stop Reason• Time of First Error• Time of Last Error• Transaction Status (determined by Transaction Filters)

• Hit Counts

• Total Hits• Stored Hits• HTML Hits• Total Errors• Total Warnings• HTML Errors• HTML Warnings

• Timings

• Total Client Time• Total End to End Time• Total Processing Time

• Event occurrences (determined by Transaction Filters)

• Custom Field values (determined by Transaction Filters)

• Metric Updates caused (determined by Transaction Filters)

FxV Browser InterfaceThe FxV browser interface is the main visual interface to the Foglight Experience Viewer appliance. It is a local Web application that runs as part of the FxV Server software component. The browser interface is used for configuring the appliance and for user session reporting and visualizations (including visual replay). The browser interface also supports workflows where customers drill down from the Foglight or Foglight Experience Monitor interfaces.

Introducing the Foglight Experience Viewer (FxV) 29FxV Browser Interface

This section provides details about the FxV browser interface and its elements, how to log into the GUI, and how to get quick tips about the GUI elements (via the tips). For details about these topics, see the following sections:

• “Logging In” on page 29

• “Automatic Login via Foglight” on page 29

• “FxV Screen Elements” on page 30

Logging InTo access the Foglight Experience Viewer browser interface:

1 Enter one of the following addresses in your Web browser:

• http://[myappliance]/console• https://[myappliance]/console (for SSL access)

Note To force all browser interface traffic to use https, log into the browser interface, navigate to Configure > Superuser Tasks > Server Configuration, and set SSL Redirection Enabled to Yes.

where [myappliance] is the IP address allocated to the eth0/Gb1 port of the appliance configured as Server (for information about how to allocate an IP address, see the FxV Installation and Administration Guide).Note If your system is configured to not use the default http/https ports (that is, 80/443),

you need to add port numbers to the URLs, as follows:http://[myappliance]:7630/consolehttps://[myappliance]:7643/console

2 Enter your username and password at the Foglight Experience Viewer login screen, then click Login. For information about default accounts, see the FxV Installation and Administration Guide.

Automatic Login via FoglightFoglight users that want to drill down in to an FxV appliance can automatically follow an URL to a remote FxV appliance, without the need to re-authenticate themselves. In order to avoid explicitly logging in to the FxV appliance, an FxV administrator must enable the Automatic Login via Foglight feature for the specific Foglight users. For details, refer to the FxV Installation and Administration Guide.


FxV Screen ElementsThe FxV browser interface is illustrated in the following figure.

FxV Screen Elements Figure 5

The FxV screen consists of the following elements:

• Menu bar—contains application-specific menus which provide access to the FxV functions and help information.Note Some menu options are displayed only for users with full administrative privileges.

• Action panel—contains the various actions that you can perform, depending on the menu option selected from the menu bar (for example, define search conditions for a hit/session/transaction search, display the results of a data search, edit the replay preferences, configure the FxV system, manage users and group accounts, etc.).

For a complete list of menu options, see the following table.


Menu Option Description

Search Transactions Performs a search for transactions in the capture database. Once an interesting transaction has been found, the transaction or the entire session can be analyzed from the search results screen.For more information, see “Transaction Search” on page 40.

Sessions Performs a search for sessions in the capture database. Once an interesting session has been found, it can be analyzed from the search results screen. For more information, see “Session Search” on page 38.

Hits Performs a search for hits in the capture database. Once an interesting hit has been found, the entire session can be analyzed from the search results screen. For more information, see “Hit Search” on page 43.

Custom Search Performs a hit search based on search conditions specified by a Custom Search Screen. For more information, see “Custom Search Screens” on page 49.Note This menu option is available only after a

Custom Search Screen has been defined.

Preferences Provides account information, a list of replay settings, and a list of search settings. These fields can be edited or just viewed, depending on the user’s privileges. For more information, see “Setting User Preferences” on page 71.


Configure1 Hit Analysis Creates, configures, or displays:• Hit Filters. See “Hit Filters” on page 80.• Transaction Filters. See “Transaction Filters”

on page 92.• Special Events. See “Special Events” on

page 106.• Custom Fields. See “Custom Fields” on

page 108.• Metrics. See “Metrics” on page 172.• Captured Metadata. See “Captured Metadata”

on page 117.• Scripts. See “Scripts” on page 118.• Sensitive Hit Details. See “Sensitive Hit

Details” on page 121.• Sensitive Response Content Expressions. See

“Sensitive Response Content Expressions” on page 123.

• Hit Analysis Configuration Options. See “Hit Analysis Configuration Options” on page 125.

• Analysis Repository. See “Configuring the Analysis Repository” on page 149.

• Hit Analysis Configuration Change Log. See “Hit Analysis Configuration Change Log” on page 128.

Allows importing and exporting the Hit Analysis configuration. See “Hit Analysis Import/ Export Configuration” on page 130.

Custom Search Screens

Creates and configures Custom Search Screens. For more information, see “Defining Custom Search Screens” on page 65.Note The first step towards creating a Custom Search

Screen is to create a Saved Search. Once a saved search has been created, it can be used to build a Custom Search Screen.



Users Creates and configures User accounts for persons who need to log into the FxV browser interface. For more information, see “Users and User Groups” on page 176.

User Groups Creates and configures User Groups.A User Group is a group of users sharing the same permissions. For more information, see “Users and User Groups” on page 176.

Preference Groups

Creates and configures Preference Groups.A Preference Group is a group of users with common default user preference settings. For more information, see “Preference Groups” on page 184.

Resource Groups

Creates and configures Resource Groups.A Resource Group is a collection of resources (Custom Search Screens and Saved Searches) that can be associated with one or more User Groups, in order to grant access to those resources. For more information, see “Resource Groups” on page 183.

Archivers Adds and configures new Archivers to the system. For more information, see “Archivers” on page 166.

Collectors Displays the list of Collectors that have made contact with the FxV Server, and manages the Collector Groups defined in the system. For more information, see “Collectors” on page 167.

Superuser Tasks

Provides access to the Appliance Maintenance settings and allows operations with metrics (viewing metrics, resetting metrics, and running metric-based diagnostics). For more information, see “Superuser Tasks” on page 168.



By default, the display area shows tips about each page element. These tips contain supplementary information regarding their corresponding element.

To view a tip:

• Click the button associated with it.

To turn tips on/off:

• Click Hide Tips/Show Tips, on the upper right side of the screen.

Help Contents Provides the list of documents available via the FxV browser interface, in PDF or HTML format:• FxV User Guide• FxV Installation and Administration Guide• Groovy Scripting API

About Provides information about the product version, technical support contact details, and Quest’s copyright information.

Logout Logs you out of the FxV browser interface.

1 This menu is available only for users with the necessary privileges.


2

Performing Searches

This chapter presents the FxV functionality available to users when performing data searches, and outlines the types of searches that can be performed and the results they provide.

Data captured by FxV can be retrieved via the search screens. Three types of searches are available in the FxV browser interface. They are distinguished by the types of data that they return: individual hits, user sessions, and transactions.

For each of these data types, two types of search interfaces are available: default search screen and custom search screens.

Each row in the search results screen represents a single session, transaction, or hit, depending on the type of search that was performed. The columns that appear in the search results depend on two factors: custom fields and user preferences. In addition to a default set of column headings, a column is added for each custom field (with the appropriate scope) that has been configured to appear in the search results.

To hide any of these columns, click the Edit Hidden Columns icon in the upper-left corner of the search results screen. Alternatively, configure the set of hidden columns via the Preferences screen, by editing the Hit/ Session/ Transaction Search Results Hidden Columns settings.

The column by which the results are sorted can be changed by clicking any column heading.

The full search results can be exported as a comma-separated text file (CSV) by clicking the Export Results (CSV) button in the search results screen.



Time Constraints..........................................................................................................................37Session Search............................................................................................................................38Transaction Search......................................................................................................................40Hit Search....................................................................................................................................43Search Result Limits....................................................................................................................48Saved Searches ..........................................................................................................................48Custom Search Screens..............................................................................................................49

Performing Searches 37Time Constraints

Time ConstraintsThis section presents the generic time constraints imposed on all types of data searches.

All searches are constrained to a user-specified time window. Depending on the type of search, several parameters define the search window, as follows:

• Time Range Mode—for session/transaction searches only, defines the type of sessions and transactions to be searched for, in terms of their status (“active” versus “completed”). The options available for this parameter include:

• Sessions/Transactions that started within (a specified time range)• Sessions/Transactions that ended within (a specified time range)• Sessions/Transactions that were in progress at (a specified time)• Sessions/Transactions that are currently in progress (no additional time

constraints are specified)• Time Range—restricts the search to the specified time window. It ranges from

“the last minute” to “the last 72 hours”. Alternatively, you can choose to specify start and end times (rather than defaulting the end time to “now”).Note The maximum (72 hours) is determined by the user’s Preference Group (and

possibly, though not typically, by the user preference settings).

• Timestamp—for session/transaction searches only, specifies a specific time for session/transaction searches when the Time Range Mode is set to “...were in progress at...”.

• Start Time—defines the time to start the search, when Time Range is set to “specified start and end times...”. It is specified as a date and a time (down to the minute).

• End Time—restricts how much captured data can be included in the search, when Time Range is set to “specified start and end times...”. It is specified by selecting a search window duration (such as five minutes, one hour, etc.). The UI displays the actual end time based on the start time and the selected window.

• Sessions/Transactions to Search—for session/transaction searches only, specifies whether to include active or completed sessions/transactions. The options available for this parameter are:

• Search completed and active sessions/transactions• Search completed sessions/transactions only • Search active sessions/transactions only


Note If any options are not available for these generic parameters defining the search window, it is due to the currently selected value of one of the other time-related fields.

Certain combinations are not allowed by the user interface. For example, if you select “Sessions that are currently in progress” for the Time Range Mode parameter, the option “Search completed sessions only” for the Sessions to Search parameter is disabled, since these two values are contradictory.

Session SearchThis section presents the parameters available to define a session search and the type of information displayed in session search result screens.

Session searches return information about completed and/or active user sessions, based on search conditions that are grouped into seven basic categories (in addition to the time constraints presented in section “Time Constraints” on page 37):

• General Conditions—for session search, the only field in this category is the Session ID. This provides direct access to a known session, based on its unique ID.

• Custom Field Conditions—any number of session custom fields can be defined by Hit Filters and Transaction Filters. These filters then populate custom field values for each session. The value of any session-level custom field can be used as a search condition. Note Custom fields defined as numeric can be searched using operators such as “>” and

“<=”.

• Metric Update Conditions—Hit Filters and Transaction Filters can be configured to update metrics when a particular condition is met. When searching for sessions, the search can be constrained to only include sessions during which a specified metric increased by a specified amount. Comparison operators allow for full control over the types of increases to be included (for example, increase > 0 or increase <= 10, etc.).

• Client Detail Conditions—restricts the session search by one or more captured session details relating to the end-user client for the session, including Browser Type and client location related details.

Performing Searches 39Session Search

• Client IP—restricts the session search based on the IP address of the client browser. An asterisk (“*”) can be used as a wildcard character: “66.35*” would search for any IP addresses that begin with “66.35”. Note An IP address cannot be used to reliably identify a single client on public Web

sites. It is not uncommon for a single client visit to use multiple IP addresses, or for traffic from simultaneous clients to be multiplexed onto a single originating IP address.

• Browser Type—restricts the session search based on the client’s “Browser Type”. An asterisk (“*”) can be used as a wildcard character. Click the Find Browser icon to choose from a list of browser options.

• ISP—restricts the session search based on the ISP (Internet Service Provider) of the client. An asterisk (“*”) can be used as a wildcard character. Click the Find ISP icon to choose from a list of ISP options.

• City—restricts the session search based on the City of the client. An asterisk (“*”) can be used as a wildcard character. Click the Find City icon to choose from a list of city options.

• Region—restricts the session search based on the Region of the client (for example, State). An asterisk (“*”) can be used as a wildcard character. Click the Find Region icon to choose from a list of region options.

• Country—restricts the session search based on the Country of the client. An asterisk (“*”) can be used as a wildcard character. Click the Find Country icon to choose from a list of country options.

• Hit Filter Conditions—the match conditions defined for each Hit Filter in the system are evaluated against every hit processed by FxV. This type of condition allows you to search for sessions that contain a particular number of hits that matched a specified Hit Filter.

For example, if you have defined a Hit Filter called “Failed Login” that matched hits to the page displayed to a user when they entered incorrect login information, to find sessions in which users have hit this page five times or more, you could specify the “Failed Login” Hit Filter with a value of “>= 5” and all such sessions would be returned by the session search.

• Hit Count Condition—these types of conditions allow the user to search for sessions that contain a specified number of a particular category of hit.

• Transaction Count Conditions—Transaction Filters allow FxV to delineate transactions as subsets of the hits that make up a session. These conditions allow the user to search for sessions that contain a specified number of transactions.


The session searches have the following limitation when searching on metric deltas or hit filter matches: users can only match on zero when using the search criteria “= 0”. All other searches, “>”, “>=”, “<“, “<=”, and “!=”, do not match on zero.

For example:

• If you use the criterion “< 5”, it returns sessions where the entry is 1, 2, 3, or 4, but not 0.

• Also, the criterion “!= 3” returns sessions where this value is 1, 2, or >3 (not 0 or 3).

Note This limitation does not apply to transaction searches.

The session search results screen renders the information in table format. Each row in the session search results table contains two icons that can be clicked to trigger activities related to that session:

• Explore Session—click this icon to open the Session Explorer view. This is a tabbed view that provides multiple ways to get details about the session: Details, Transactions, Timeline, Hit Inspector, and Content Replay.

• View Session Details—click this icon to open a new window containing all available details about the session.

Transaction SearchThis section presents the parameters available to define a transaction search and the type of information displayed in transaction search result screens.

Transaction searches return information about completed and/or active transactions of a specific type. The first step in any transaction search is to specify the type of transactions (that is, the Transaction Filter) to search for. Transactions can then be found based on search conditions that are grouped into five basic categories (in addition to the time constraints presented in section “Time Constraints” on page 37):

• General Conditions—for transaction search, the only field in this category is the Session ID. This provides direct access to any transactions that occurred within a known session based on its unique ID.

• Status Conditions—every transaction can have a status of OK, Warning, or Error, as defined by the Transaction Events that make up the Transaction Filter. By

Performing Searches 41Transaction Search

default, all transactions have a status of OK. The Transaction Filter may define conditions that change this status.

• Custom Field Conditions—if the type of transaction being searched defines transaction-scope custom fields, these fields can be used as search constraints.Note Custom fields defined as numeric can be searched using operators such as “>” and

“<=”.

• Hit Count Conditions—these types of conditions allow the user to search for transactions that contain a specified number of a particular category of hit.

• Event Conditions—every Transaction Filter is made up of one or more Transaction Events. Some events may not occur in every transaction, and some events may occur more than once, if allowed by the event definition. The occurrence (or lack thereof) of a specific event can be used as a search condition. For example: “Find transactions in which event A occurred, event B did not occur, and event C occurred three or more times.”

The transaction search results screen renders the information in table format. Transaction search results are similar to session search results, including the two icons (Explore Session and View Session Details) described in session “Session Search” on page 38.

Unique to the transaction search result screen is the display of the following Transaction Event information:

• Each event that makes up the Transaction Filter definition (displayed above the main result table).

• For each event, a count of the number of transactions in which the event occurred.

• The percentage of the total number of transactions returned by the search that included the event.Note For some types of transactions, this information is not meaningful. If all events are

required and therefore occur in every transaction, the number shown for each event is the same and the percentage is always 100%. However, for a transaction like a “buy tunnel”, the event counts and percentages can provide valuable information about conversion rates and common transaction exit points.

One column unique to the transaction search result screen is Events. This column shows a sequence of single-character labels for all Transaction Events that occurred within the transaction, in the order that they occurred. This provides a quick summary of the activities that took place (or did not take place) as part of the transaction.

Example:

Consider a Transaction Filter made up of the following events:


• Add To Cart (A)

• Shipping (B)

• Billing (C)

• Confirmation (D)

• Orders Placed (E)

• Error Occured (R)

• Abandoned (X)

The Events column might contain values such as:

• ABCDE—user added one item to the shopping cart, calculated shipping, calculated billing, got a confirmation for the transaction, and placed the order.

• ABCX—user added one item to the shopping cart, calculated shipping, calculated billing, and abandoned the shopping cart.

• AX—user added one item to the shopping cart, then abandoned it.

• ABCDRX—user added one item to the shopping cart, calculated shipping, calculated billing, got a confirmation for the transaction, then an error occured and user abandoned the shopping cart.

Performing Searches 43Hit Search

The following figure illustrates an example of results returned by the transaction search matching the conditions imposed by this Transaction Filter.

Hit SearchThis section presents the parameters available to define a hit search and the type of information displayed in hit search result screens.

Hit searches return information about individual captured hits (HTTP Requests/Responses) based on search conditions that fall into six basic categories (in addition to the time constraints presented in section “Time Constraints” on page 37):

• General Conditions—includes a wide range of hit details, such as the page title (for HTML hits), the Web server, the path, and response content keywords:

• Content Keywords—restricts the hit search based on keywords found within page content. This allows searching for visible text displayed to clients, or for strings hidden within HTML or JavaScript code.

Foglight Experience Viewer breaks each page down into a list of keywords, single words that are more than four characters long and less than 64 characters long. The simplest search is for one of several words: green blue


(find green or blue, any order). To find an exact phrase (several words in order), use quotes around the keywords: “Quest Software” (exact phrase).

Advanced operators include: asterisk (*) for multiple-character wildcard, plus sign (+) for required words/phrases, minus sign for words/phrases to omit: +re* -read (find region and repeat but not read).Note Keyword indexing must be enabled by one or more Hit Filters. If keyword search

is not working as expected, verify that such a Hit Filter exists and that its keyword indexing is currently enabled.

• Client IP—restricts the hit search based on the IP address of the client browser. An asterisk (“*”) can be used as a wildcard character: “66.35*” would search for any IP addresses that begin with “66.35”.Note IP address cannot be used to reliably identify a single client on public Web sites.

It is not uncommon for a single client visit to use multiple IP addresses, or for traffic from simultaneous clients to be “multiplexed” onto a single originating IP address.

• Server IP—restricts the hit search based on the IP address of the web server that served the hit. An asterisk (“*”) can be used as a wildcard character: “66.35*” would search for any IP addresses that begin with “66.35”.

• HTML Title—restricts the hit search based on the HTML page title. An asterisk (“*”) can be used as a wildcard character: “Foo*” would search for any HTML pages with titles that begin with “Foo”.

An HTML hit is defined as a hit with a “Content-Type” response header starting with “text/html”, containing a proper HTML tag in the response content, and having a response code of 200. Non-HTML hits do not have searchable titles. Click the Find HTML Title icon to choose from a list of captured titles.

• HTTP Method—restricts the hit search based on the HTTP request method. The most commonly used methods are “GET” and “POST”, though there are additional methods defined by the HTTP specification. An asterisk (“*”) can be used as a wildcard character.

• Collector ID—restricts the hit search based on the Collector used to capture the hit. This is especially useful if multiple Collectors are being used for different applications or in different datacenters.

• Collector Group—restricts the hit search based on the Collector Group used to capture the hit. This is especially useful if multiple Collectors are being used for different applications or in different datacenters.


• Web Server—restricts the hit search based on the Web server that handled the hit request. Search values should include the protocol scheme (that is, HTTP vs. HTTPS), port number, and either a DNS name or IP address.

http://mysite:8080.com

https://12.106.87.32

An asterisk (“*”) can be used as a wildcard character: “https*” would search for any hits served over SSL. Click the Find Server icon to choose from a list of captured web servers.

• Path—restricts the hit search to the specified path, often used to narrow the search to a specific page. The path is defined as the portion of the request URL after the hostname (and optional port), with any query parameters removed. Paths always begin with a leading slash. An asterisk (“*”) can be used as a wildcard character: “/index*” would search for any paths beginning with “/index”. Click the Find Path icon to choose from a list of captured paths.

• Type—restricts the hit search to hits of the specified type. An HTML hit is defined as a hit with a “Content-Type” response header starting with “text/html”, containing a proper HTML tag in the response content, and having a response code of 200.

• Session ID—restricts the hit search to hits with the specified Session ID.• Exception—restricts the hit search based on network/protocol/capture

Exceptions noted by the FxV system when the hit was captured. Exceptions are stored with the hit as a comma separated list (for example, CorruptedResponse,MissingProperty). To search for a single value, the search value should be surrounded with a wildcard characters (“*”); for example, “*CorruptedResponse*”. Click the Find Exceptions icon to choose from a list of possible exceptions.

• HTTP Header/Field Conditions—any cookie, request header, response header, or request field captured as part of a hit can be used as a search constraint. Request field searches are particularly useful for finding hits based on values entered into form fields.Note The most frequently captured header/field names are included in the list of captured

metadata. Less frequently captured data may be excluded form this list. This feature applies only to fields, cookies, request headers, and response headers.

• Custom Field Conditions—Hit Filters can be defined to set custom field values on any hit. These values can be used to constrain hit searches. Note Custom fields defined as numeric can be searched using operators such as “>” and

“<=”.


• Metric Update Conditions—Hit Filters can be configured to update metrics when a particular condition is met. When searching for hits, the search can be constrained to include only hits whose processing caused a specified metric to be increased by a specified amount. Comparison operators allow for full control over the types of increases to be included (for example, increase > 0 or increase <= 10, etc.).Note Hit searches have the same constraint that the session searches have. That is, hits

whose metric update value is “0” are not included in the results; hits whose metric value is less then three match on “1” or “2” (but not 0). Also, hit searches have the added constraint that they do not support searching for “0” (=0). The browser interface displays an error message to the user if they attempt this.

• Status Conditions—every hit can have a status of OK, Warning, or Error, as defined by any Hit Filter whose match conditions are met by the hit. By default, all hits have a status of OK. It is up to the Hit Filters to define conditions that change this status. Note Any Hit Filter can change the status of the hit. The “worst” status recorded by any Hit

Filter is the hit's final status (for example, if one Hit Filter leaves the status as OK, another sets it to Error, and another sets it to Warning, the final hit status is Error, regardless of the order in which the Hit Filters were processed).

In addition to the overall hit status as defined by Hit Filters, it is also possible to search for hits based on their HTTP Response Code. This status may not (necessarily) correspond to a value such as OK, Warning, or Error, but it does define the status of the hit from the perspective of the Web server.

• Hit Filter Conditions—hits can be found if they met the match conditions of a specific Hit Filter.

• Timing Constraints—restricts the hit search based on the measured performance of the individual hits. Hits can be found by performing hit searches on:

• End to End Time—total end-to-end time recorded for the hit.• Client Time—time spent by the client (browser) acknowledging the response

sent from server.• Processing Time—time spent processing the hit.

The hit search results screen renders the information in table format. Each row in the hit search results table represents an individual hit (an HTML page, an image, a CSS file, etc.). Each row in the hit search results table contains two icons that can be clicked to trigger activities related to the session containing that hit:

• Explore Session containing Hit—click this icon to open the Session Explorer view. This is a tabbed view that provides multiple ways to get details about the


session containing that hit: Details, Transactions, Timeline, Hit Inspector, and Content Replay.

• View Session Details—click this icon to open a new window containing all available details about the session.

Two additional icons are located beside each hit’s URL:

• View Hit Details—click this icon to open a window containing all available information about the selected hit.

• Hit Content Preview—click this icon to open, in the corner of the search results table, a small window that shows a visual representation of the hit:

• For image hits, this preview shows the actual image.• For text files, such as CSS or JavaScript files, the actual text is displayed.• For HTML hits, the page is rendered as it appeared to the user.

Hit search results can also be viewed in Aggregate. In this mode (selected from the Search Result View drop-down list, above the results table), the individual hits are not displayed. Instead, each unique URL is displayed along with a count of the number of times a hit with that URL appeared in the search results.

Note Since this view does not show individual hits, the details/replay/preview icons are not displayed.

Two variations of the aggregate hit search results table are available:

• full URL—the entire URL is used when identifying common hits.

• ignore URL params—any query parameters that appear in the URL are ignored when identifying common hits.

For example, the following two hits would be considered the same when using “ignore URL params” mode, but would be considered different in “full URL” mode:

http://www.myveryownwebsite.com/somepage?a=foo&b=bar

http://www.myveryownwebsite.com/somepage?a=dog&b=baz

In “ignore URL params” mode, these URLs would be listed as:

http://www.myveryownwebsite.com/somepage


Search Result LimitsThis section presents the concept of limiting the number of results brought back for visualization, and why these limits are necessary. It also presents how users (with certain privileges) can modify the pre-defined limits.

When a search is performed, the FxV Server sends the query to all available Archivers configured in the system. Each Archiver limits the number of results it returns to a configurable maximum number of results. By default, the limits are set to 200 hits, sessions, or transactions. These limits can be modified as needed, up to a maximum value of 1000 per Archiver. If any Archiver reaches its limit, a message is displayed together with the search results, indicating that some results that match the query conditions have not been returned.

Important If a search is performed in parallel by several Archivers, the maximum search result limit is not multiplied by the number of Archivers involved the search, but applies to each Archiver individually.

These search result limits are configurable at the Preference Group level. Each user is a member of a single Preference Group. By default, the search result limits are not configurable by individual users, but it is possible to unlock these limits and allow each user to configure their own limits via the Preferences menu on the menu bar. For more information, see “Setting User Preferences” on page 71.

Saved SearchesUsers that expect to perform repeatedly the same search have the option of saving these search conditions as Saved Searches. Saved searches make searching easier because there is no need to remember all the details for frequently performed searches. Saved searches can also be shared between users, as shared resources.

To create or edit Saved Searches:

• See “Creating and Editing Saved Searches” on page 66.

To perform searches based on a Saved Search:

• In the Hit/Session/Transaction Search page, restore the saved search conditions by selecting a saved search from the Load Saved Search list.Note The Load Saved Search list is empty if no saved searches have been created yet.

Performing Searches 49Custom Search Screens

Once loaded, these conditions can be modified as needed before executing the search operation.

The second purpose of saved searches is to serve as the basis for the creation of Custom Search Screens. The saved searches must specify search condition values that the custom screen can use or override. For more information, see “Defining Custom Search Screens” on page 65.

Custom Search ScreensA Custom Search Screen is a simplified search screen, usually defined for frequently used searches or for less-technical users. This section presents how to use an already defined Custom Search Screen, via the Custom Search menu on the FxV menu bar. This menu is available only to users who have access to at least one Custom Search Screen resource. Therefore, it is not displayed to any users until as least one custom screen has been created.

To create or edit Custom Search Screens:

• See “Creating and Editing Custom Search Screens” on page 67.

To perform a search based on a Custom Search Screen:

1 In the FxV browser interface, on the menu bar, click Custom Search.

A Custom Search Screen page appears.

2 Select a Custom Search Screen from the Search Screen list.

Each Custom Search Screen created in the system is unique. When you select a Custom Search Screen to use, you are presented with the standard time controls and the list of search fields that are defined for that Custom Search Screen.


3 Fill out the form as needed, then click Search.

The search screen results page appears, listing all hits that matched the custom search settings. Search results are consistent with the type of search being performed (hit/session/transaction).

3

Analyzing User Sessions

This chapter describes how FxV users can analyze a session of interest, and outlines the available analysis views. To access these views, click the Session Explorer icon on the search results screen.


Session Explorer Toolbar.............................................................................................................52Details View.................................................................................................................................53Timeline View ..............................................................................................................................54Hit Inspector View........................................................................................................................55Content Replay View ...................................................................................................................57Transactions View .......................................................................................................................61Combined Sessions View............................................................................................................63


Session Explorer ToolbarThe Session Explorer toolbar is located on the upper right corner of the screen. The following table presents the functionality of the toolbar’s buttons.

Button Name Button Icon

Button Description

Combine this Session with others for Session Explorer

Enables FxV to combine the hits from multiple sessions for the Session Explorer.When you click this button, the Combined Sessions view becomes visible and displays the results of the session combining.Note The button is shown only when session combining

is enabled and when the active session has a value for the session combining custom field. To configure the Session Combining settings, see “Session Explorer Settings” on page 72.

Return to un-combined Session

Disables the display of combined sessions in the Session Explorer. When you click this button, the Combined Sessions view is hidden.Note The button is shown only when viewing combined

sessions.

Return to Search Re-displays the page containing the results of your last search operation.Note The button is shown only when opening the

Session Explorer from an FxV search result screen.

Refresh Refreshes the information displayed in the Session Explorer views for sessions that are “in progress”.Note This button is shown only when the session is “in-

progress”.

Edit Session Explorer Preferences

Opens a dialog box which allows you to modify the default Session Explorer settings. For more information, see “Session Explorer Settings” on page 72.

Analyzing User Sessions 53Details View

Details ViewThe Details view presents all data elements (and their values) that define the session of interest.

To access this view, click the Session Explorer icon on any search result screen, then click the Details tab. The same information can also be accessed via the hit or session search result screen, by clicking the View Session Details icon.

Details View Figure 1

The session details are grouped in several categories and presented in a tabbed format. The following data is calculated and stored for each session (for a complete list of elements see “Sessions” on page 26):

• Client, Counts and Timings—contains Details, Client Details, Transaction Counts, Hit Counts, and Timings

• Custom Fields—contains custom fields set by Hit Filters and/or Transaction Filters

• Metric Updates—contains metrics updated by Hit Filters and/or Transaction Filters

• Hit Filter Matches—contains matches determined by Hit Filters


Timeline ViewThe Timeline view presents a time-accurate visual representation of how long hits within the session lasted, and their time-relation to other hits.

To access this view, click the Session Explorer icon on any search result screen, then click the Timeline tab.

Timeline View Figure 2

The Timeline view displays individual “bars” that represent the hits. Each “bar” has banded colors which further break down the duration of each hit into client time, processing time, and latency. Additionally, any transactions that occurred during the session are displayed in relation to the hits (and other transactions). This view gives great visibility into the parallel nature of how hits occur in the “real world”.

Each hit bar can be accompanied by icons and labels, which are made visible/hidden by clicking the Labels and Icons buttons, respectively. The icons are the same as in the hit search results screen and in the Content Replay view. The hit labels are a shortened version of the hit’s URL (which is usually the “filename” part); this shortening is due to limited space available for hits’ display. The number within brackets is part of the label (for example, [2.5]) and is intended to match up hits between the three views that deal with hits (that is, Timeline, Hit Inspector, and Content Replay). These numeric labels are only relevant to the session views and they may change within a particular session if you refresh a session that is “in progress”.

Analyzing User Sessions 55Hit Inspector View

To analyze a hit of interest:

1 In the Timeline view, click the hit icon or label.

The dialog box that appears presents a short list of hit details.

2 For additional information, do one of the following:

• To view the complete list of hit details, click View Full Hit Details.

A new window appears, displaying the data collected and stored for this hit (for a complete list of elements see “Hits” on page 24).

• To view the Web page corresponding to this hit, click Show Hit in Content Replay.

The Content Replay view opens, displaying the hit’s component Web page, as the end user originally viewed it.

To display/hide non-HTML hits:

• In the Timeline view, click Non-HTML.

The display’s background is split into three sections: the upper section corresponds to hours, the middle section to minutes, and the lower section to seconds. To zoom in/out an area of interest, click the plus/minus icons on the right side of the view. To move the focus of your interest on a different area of the time line, click anywhere on the view and drag left or right.

Hit Inspector ViewThe Hit Inspector view presents timing details and other information for every hit that occurred during the session.

To access this view, click the Session Explorer icon on any search result screen, then click the Hit Inspector tab.


Hit Inspector View Figure 3

The upper pane of the Hit Inspector view lists all hits that make up the session of interest and a brief summary of their details. The column by which the hits are sorted can be changed by clicking any column heading.

The lower pane of the Hit Inspector view provides complete details about the hit selected in the upper panel.

The hit details are grouped in several categories and presented in a tabbed format. The following data is collected and stored for each hit (for a complete list of elements see “Hits” on page 24):

• Request—contains Request Details, Request Headers, and Request Fields

• Response—contains Response Details and Response Headers

• Cookies—contains cookies

• Status Analysis—contains status information as determined by Hit Filters

• Timings—contains Timing Details

• Hit Custom Fields—contains custom fields set by Hit Filters

• Metric Updates—contains metrics updated by Hit Filters

• Other—contains Archiver Details

Analyzing User Sessions 57Content Replay View

Content Replay ViewThe Content Replay view allows FxV users to visualize a session of interest by displaying its component Web pages as the end user originally viewed them.

Note The Content Replay view is available only to FxV users who are part of a User Group for which the Hit Content Replay privilege has been enabled.

To access this view, click the Session Explorer icon on any search result screen, then click the Content Replay tab. The following figure illustrates the Content Replay GUI.

Content Replay GUI Figure 4

The screen consists of the following elements:

• Replay menu bar—provides access to the replay operations, replay settings, and results display modes.

• Hit Selection pane—contains two tabs:


• Captured Hits—displays the list of hits that were captured by FxV for the session being replayed. This is essentially the list of hits that were actually sent over the network to the end user’s browser.

• Page Visit Order {pv#}—displays the list of HTML Pages in the order in which FxV determines the end user visited the pages.Note Captured hits may be visited more than once during a session (for example, by

using the Back button) and thus they may appear in the Page Visit Order more than once.

• Hit Details pane—displays brief details about the currently selected hit, along with some end user navigation information.Note You can hide/ display this pane by clicking the collapse / expand icons.

• User Input pane—contains the User Input tab, which displays field values submitted with HTML forms from the currently selected hit, if applicable.Note You can hide/ display this pane by clicking the collapse / expand icons.

• Hit Replay pane—displays data captured by FxV for the currently selected hit. Depending on the display mode selected, this pane may “replay” pages as the end user has originally seen them, as source code, or as a list of complete hit details.

The Hit Selection pane and the Hit Details pane use the same color scheme in presenting the list of captured hits (that is, Referer = previous page visited, highlighted hit = current page, and Navigated To = next page visited).

Analyzing User Sessions 59Content Replay View

The following table presents the functionality of the Replay menu bar.


Previous Page Visited Navigates to the previous hit in the Page Visit Order, and displays it in the Hit Replay pane.

Next Page Visited Navigates to the next hit in the Page Visit Order, and displays it in the Hit Replay pane.

View Show Only HTML Hits In the Hit Selection pane, toggles between these options:• ON—displays only the HTML hits in

the selected session.• OFF—displays all hits in the selected

session, including non-HTML pages such as images and style sheets.

Display Page Titles for HTML Hits

In the Hit Selection pane, toggles between these options:• ON—displays the HTML page title

for hit labels (when available).• OFF—displays the URL for the

HTML hit labels.

Display Full URL for Hit Labels

In the Hit Selection pane, toggles between these options:• ON—displays the full URL for hit

labels, including the parameters (e.g., http://foo.com/bar?p1=v1).

• OFF—displays only a the URL path for the hit labels (e.g., /bar for the URL http://foo.com/bar).

Expand All Expands all nested elements in the Hit Selection pane > Captured Hits tab.


Expand Errors/Warnings Expands all nested elements marked as errors or warnings in the Hit Selection pane > Captured Hits tab.

Collapse All Collapses the tree of nested elements in the Hit Selection pane > Captured Hits tab.

Enable Tooltips Enables the display of tooltips in the Content Replay view.

Tools Export to ZIP Exports the session selected for replay (file is saved in ZIP format).

Display Mode

Hit Replay In the Hit Replay pane, displays a “Replay” of currently selected hit’s response content. For HTML Pages, this is an approximation of how the page was originally displayed to the end user.Note When replaying HTML Pages, any

page elements (for example, images, css, etc.) that were not captured with the session may be requested from the “live” Web server, depending on the user’s Replay preferences.

Hit Replay (Highlight User Input)

Same as “Hit Replay” mode, except that any form fields where the user entered data are populated with the data that was entered, and the links clicked are highlighted.

Hit Source In the Hit Replay pane, for HTML pages, displays the HTML source code captured for the selected hits. Note If a hit that is not an HTML page is

selected, the equivalent of the “Hit Replay” mode is displayed.


Analyzing User Sessions 61Transactions View

Important Visual replay is most reliable when applied to Web 1.0 applications that make light use of JavaScript. Heavy use of JavaScript and frames can negatively impact FxV’s ability to replay the user experience with complete accuracy.

Transactions ViewThe Transactions view presents information about any transactions that took place during the session being explored.

To access this view, click the Session Explorer icon on any search result screen, then click the Transactions tab. This tab is selected automatically when entering from a transaction search result page.

Hit Details In the Hit Replay pane, displays details about the selected hit. These details are stored by FxV during the data capture process.



Transactions View Figure 5

The upper pane of the Transactions view lists all transactions that occurred during the selected session, and a brief summary of their details. The column by which the transactions are sorted can be changed by clicking any column heading.

The lower pane of the Transactions view, named Transaction Details, provides complete details about the transaction selected in the upper panel.

The information presented in the Transaction Details pane can also be accessed via the transaction search’s result screen, by clicking the View Transaction Details icon.

The transaction details are grouped in several categories and presented in a tabbed format. The following data is calculated and stored for each transaction (for a complete list of elements see “Transactions” on page 27):

• Client, Counts and Timings—contains Details, Hit Counts, and Timings

• Events—contains information about the Transaction Events that occurred as defined by the Transaction Filter

• Custom Fields—contains custom fields set by Transaction Filters

• Metric Updates—contains metrics updated by Transaction Filters

Analyzing User Sessions 63Combined Sessions View

Combined Sessions ViewThe “combined sessions” feature allows FxV to combine hits from multiple sessions for the Session Explorer. This functionality should only be used when proper sessionizing (of hits into unified sessions) is not possible. By default, this feature is disabled.

The Combined Sessions view presents information about all the combined sessions, when displaying a “combined” session. This view is visible on the Session Explorer only when the “combined sessions” feature is enabled. To configure the Session Combining Settings, see “Session Explorer Settings” on page 72 and “Preference Groups” on page 184.

To access the Combined Sessions view:

1 Click the Session Explorer icon on any search result screen.

2 Click the Combine this Session with others for Session Explorer icon on the Session Explorer toolbar.

3 Click the Combined Sessions tab.

Combined Sessions View Figure 6

The information presented in this view is grouped into several categories:

• Primary Session details—includes Session ID, Started, Last Hit, Combined Start, and Combined Last Hit.Note In this case, the other Session Explorer views show values which are relevant only

to the Primary Combined Session.


• Combined Sessions criteria—includes Custom Filed Name, Custom Field Value, Time Range, and a list of links to all other sessions combined with the Primary Session.Note When you click one of the links to other sessions, the Session Explorer views show

values relevant to the selected session, and the Combined Sessions view disappears.

4

Defining Custom Search Screens

A Custom Search Screen is a simplified search screen, usually defined for frequently used searches or for less-technical users. They allow searches to be performed using simple screens that contain just a few familiar fields that apply specifically to the business (for example, Name, Address, Account Number, etc.). The first step towards defining a Custom Search Screen is to create a Saved Search.


Creating and Editing Saved Searches.........................................................................................66Creating and Editing Custom Search Screens ............................................................................67


Creating and Editing Saved SearchesUsers that expect to repeatedly perform the same search have the option of saving these search conditions as Saved Searches.

The purpose of the Saved Search is to serve as a foundation for the Custom Search Screen. Every search performed using a Custom Search Screen uses the values defined on its underlying Saved Search. For example, if you want to define a custom search that looks for sessions in which a particular metric was incremented by at least one, your Saved Search would include a condition such as “myMetric > 0”. If you then want to allow users to search for these sessions based on the user’s login (stored in a custom field), you would not specify the login field in the Saved Search (since you want that value to be entered by the person performing the search). Instead, you would add that field in the next step when you create the Custom Search Screen.

The Saved Search must specify search condition values that the custom screen can use or override.

To create a Saved Search:

Note This procedure illustrates the example of a session saved search. For hit and transaction saved searches, similar procedures apply.

1 In the FxV browser interface, on the menu bar, click Search > Sessions.

The Search Sessions page appears.

2 Set the search conditions as needed for your session search.

3 In the Save Search As text box, specify a unique name for the search, then click Save.

The Select Resource Groups page appears.

4 Select the check box(es) beside the resource groups to which this resource must belong, then click OK.

The Search Sessions page re-appears. The search conditions are now saved and the name of the Saved Search appears in the Load Saved Search list.Note The Select Resource Group screen appears if you have access to more that one

Resource Group configured to store saved searches. This screen may not appear depending on the configuration of your system. For more information about Resource Groups, see“Resource Groups” on page 183.

Defining Custom Search Screens 67Creating and Editing Custom Search Screens

To edit Saved Searches:

1 In the Hit/Session/Transaction Search page, click Saved Searches.

The Saved Searches page opens, listing the already defined saved searches.

2 Perform one of the following operations, as needed:

• To view the set of conditions defined for a Saved Search, using the main search screen, click the View icon.

• To change the Resource Groups a Saved Search is associated with, click the Share icon.

• To delete a Saved Search from the system, click the Delete icon.Note Depending on the state of the Saved Search and your user privileges, some of these

icons may not appear for certain Saved Searches.

3 Click OK.

Creating and Editing Custom Search ScreensA Custom Search Screen is a simplified search screen, usually defined for frequently used searches or for less-technical users.

Note The first step towards creating a Custom Search Screen is to create a Saved Search. For more information, see “Creating and Editing Saved Searches” on page 66.

To create Custom Search Screens:

1 In the FxV browser interface, on the menu bar, click Configure > Custom Search Screens.

The Custom Search Screens page appears.

2 Click Create Custom Search Screen.

The Custom Search Screen page appears.

3 Select the type of search to be performed (Hit, Session, or Transaction) from the Search Type list.Note These options are available if at least one Saved Search of the same type has been

created and is available to this user.


4 Fill in the Custom Search Screen Name. Upon creation, this name will be displayed in the list of available screens, in the Custom Search menu, on the menu bar.Note The Custom Search menu is only displayed to users who have access to at least

one custom search screen. Therefore, it is not displayed to any users until at least one custom screen has been created.

5 Add search fields and order them using the Move Up and Move Down arrows. The names of these fields will appear (in the same order) on the Custom Search Screen. These names must be meaningful to the screen’s users (for example, Login, First Name, Account Number, etc.).

6 Select one or multiple search phases from the Search Phases list and order them using the Move Up and Move Down arrows.

When a user submits a search via a Custom Search Screen, one or more Saved Searches are executed. While it is often sufficient to execute a single search, it is possible to specify multiple searches (phases) to be executed in sequence, until at least one result is found. This can be useful, for example, if there are multiple pages where the user can log in. Each page can be searched by its own Saved Search in order to cover all possible entry points.

For each Search Phase, mappings are defined from the fields entered by the user of the custom search screen to fields and/or cookies sent from the client.

7 Click OK.

The Custom Search Screen is now saved and appears in the list of screens available for this user.Note If this is the first Custom Search Screen created for this user, then the Custom

Search menu appears on the menu bar.

To edit Custom Search Screens:

1 In the FxV browser interface, on the menu bar, click Configure > Custom Search Screens.

The Custom Search Screens page appears, showing the Custom Search Screens available for this user.Note If no custom screen has been defined yet, or the user has no access to a resource of

this type, the list is empty.

2 Perform one of the following operations, as needed:

• To view the settings defined for a Custom Search Screen, click the View icon.

Defining Custom Search Screens 69Creating and Editing Custom Search Screens

• To modify a Custom Search Screen, click the Edit icon, and update the fields as needed.

• To delete a Saved Search from the system, click the Delete icon.

3 Click OK.


5

Setting User Preferences

This chapter outlines the preferences that users can view and edit (depending on their privileges): account information, replay settings, and search settings.


Account Information.....................................................................................................................72Session Explorer Settings ...........................................................................................................72Search Settings ...........................................................................................................................75


Account InformationThe fields in the Account Information section allow you to edit basic information about your account.

To edit the Account Information settings:

1 In the FxV browser interface, on the menu bar, click Preferences.

The Preferences page appears.

2 In the Account Information area, identify the fields that need editing. The following fields are provided: Login (not editable), Last Name, First Name, Email, and Password.Note The Password field is not directly editable on this page. To change your password,

click Change Password and follow the instructions.

To modify the settings for user accounts other than your own, see “Managing User Account Settings” on page 175.

Session Explorer SettingsThe Session Explorer settings are grouped in the following categories:

• Default Session Explorer Tab—allows you to specify the initial tab shown when a session is displayed in the Session Explorer.Note If the option Based on search context is selected, the initial tab is based on the

Search View from which the Session Explorer opens.

• Content Replay Settings—allow you to change the behavior of the Content Replay view:

• Define the default replay parameters and display mode.• Customize the way user input data is displayed in the Replay pane.• Improve the way Web pages are rendered during a replay.

• Session Combining Settings—allow you to enable/disable and configure the session combining feature.

To edit the Session Explorer settings, use one of the following methods:

• Via the FxV browser interface—on the menu bar, click Preferences.

The Preferences page appears; it provides a section dedicated to Session Explorer Settings.

Setting User Preferences 73Session Explorer Settings

• Via the Session Explorer screen—on the toolbar, click the Edit Session Explorer Preferences icon.

The Session Explorer Settings dialog box appears.

Note An individual user can edit the Replay Settings only if this operation is enabled via the Preference Group to which that user belongs to. Settings that are not locked in a Preference Group serve as defaults, and may be changed by individual users. Locking a setting immediately overrides any user-specified values. For more information about this topic, see “Preference Groups” on page 184.

The following Content Replay Settings may be configured:

• Rewrite JavaScript—specifies if JavaScript should be enabled while replaying captured HTML hits.Caution Because JavaScript may sometimes cause problems during replay, this setting is

set to “NO” by default.

• Less Strict Replay With User Input (Forms)—when set to “Yes”, improves the replay of user input data, in cases where JavaScript is used to manipulate form field values.

• Less Strict Replay With User Input (Links)—when set to “Yes”, may improve replay of user clicked links, in the cases where hit referrers are not present.

• Show Only HTML Hits—specifies if non-HTML hits (such as images and style sheets) should be listed in the Hit Selection pane, when first opening the Replay browser interface.Note This setting may be changed during replay by clicking the Show Only HTML Hits

icon.

• Hit List Display Mode—specifies how the URLs are displayed in the Hit Selection pane, when first opening the Replay browser interface. The following options are available for selection:

• Show Title (or Path)—displays the HTML page title, or the path (when the title is unavailable).

• Show Title (or Full URL)—displays the HTML page title, or the URL (when the title is unavailable).

• Show Path Only—displays only the path to the hit (for example, “/bar” for the URL “http://foo.com/bar”).


• Show Full URL—displays the full URL for the hit, including any parameters (for example, “http://foo.com/bar?p1=v1”).

Note This setting may be changed during replay by clicking the Display Page Titles for HTML Hit Labels and the Display Full URL for Hit Labels icons.

• Display Mode—defines the default display mode to be used on the Replay pane, when first opening the Replay browser interface. The following options are available for selection:

• Hit Replay—displays hits as the user has seen them, including any errors.Note If the content is unavailable (for example, if the browser used a cached page),

the replay system may attempt to obtain the content from other sessions or from the live Web server, depending on the values of the “Use Elements From Unrelated Sessions” and “When Element Content Not Available” settings.

• Hit Replay (with user form input)—displays hits similarly to “Hit Replay” mode, except that any form fields where the user has entered data are populated with the data that was entered, and links clicked are highlighted.

• Hit Details—displays network and HTTP details including response time, hit analysis (including Hit Filter matches), request and response fields and headers, cookies, and Archiver details.

• Hit Source—displays the source code that was captured for HTML hits, useful for debugging dynamic Web applications.

• User Input Highlighting - Primary Color—specifies the primary color to be used when highlighting user input. Any valid CSS color value may be used (for example, #FC0, RGB(255,204,0), lime).Note The color input field uses the specified color as its background color. If the field has a

blank background, this indicates an invalid color code (or white).

• User Input Highlighting - Secondary Color—specifies the secondary color to be used when highlighting user input. This color is used for highlighting specific items, such as the option a user selected from a pull-down menu. Any valid CSS color value may be used (for example, #FF0, RGB(255,255,0), yellow).Note The color input field uses your selected color as its background color. If the field has

a blank background, this indicates an invalid color code (or white).

• Use Elements From Unrelated Sessions—specifies if page elements (images, stylesheets, etc.) from other unrelated sessions should be used for Replay.

• When Element Content Not Available—specifies what FxV should do during replay if the content for a page element (images, stylesheets, etc.) is not available in the session being replayed or in other unrelated sessions (when the Use

Setting User Preferences 75Search Settings

Elements From Related Sessions option is enabled). The following options are available:

• Redirect to ‘live’ web server—page elements may be used from the live Web server. Note This option is a bad choice if the live server is not reachable from the Replay

browser interface or the live Web server/application requires a login to reach page elements.

• Send 404 Response—the replay system sends a “404” error code to the browser replaying the session. For missing images, a “broken image” icon is displayed; if style sheets or javascript are missing, then the HTML page may not visually render correctly.

The following Session Combining Settings may be configured:

• Session Combining Custom Field—specifies the name of the custom field for combining other sessions for the Session Explorer. For example, if “Client IP” is the selected Session Combining Custom Field and the current session being explored has the custom field name/value: “Client IP”/”10.0.0.10”, hits from other sessions with the same custom field name/value pair should be combined with the session currently being explored. By default, this field is disabled.Caution This functionality should be avoided, unless proper sessionizing hits into unified

sessions is not possible.

Note Combining sessions only combines the hits from multiple sessions for Replay. Session statistics/details and Transaction Filter matching are not affected by this setting.

• Session Combining Time Range—specifies the number of minutes (measured before and after the time range of the session currently being explored) to combine for the Session Explorer other sessions that match the custom field selected in Session Combining Custom Field.

Search SettingsThe Search settings allow you to control the search conditions and the display of results, as follows:

• Limit the amount of data to be included into a search and the maximum time range over which to perform the search.

• Specify the default type of view for the search result screen.

• Define the type of information to be included by default in the result screen.


To edit the complete list of Search settings:

• In the FxV browser interface, on the menu bar, click Preferences.

The Preferences page appears; it provides a section dedicated to Search Settings.

Note An individual user can edit the Search Settings only if this operation is enabled via the Preference Group to which that user belongs to. Settings that are not locked in a Preference Group serve as defaults, and may be changed by individual users. Locking a setting immediately overrides any user-specified values. For more information about this topic, see “Preference Groups” on page 184.

The following Search settings may be configured:

• Hit Search Result Limit—defines the maximum number of hits that can be returned from a single Archiver in response to a single hit search. The default value is 200; the maximum value allowed is 1000. Larger values may increase the time required to complete a search.Note This limit is applied per Archiver searched. For an installation with three Archivers

and a result limit of 200 hits per search, up to 600 hits may be returned in one search.

• Session Search Result Limit—defines the maximum number of sessions that can be returned from a single Archiver in response to a single session search. The default value is 200; the maximum value allowed is 1000. Larger values may increase the time required to complete a search.Note This limit is applied per Archiver searched. For an installation with three Archivers

and a result limit of 200 sessions per search, up to 600 sessions may be returned in one search.

• Transaction Search Result Limit—defines the maximum number of transactions that can be returned from a single Archiver in response to a single transaction search. The default value is 200; the maximum value allowed is 1000. Larger values may increase the time required to complete a search.Note This limit is applied per Archiver searched. For an installation with three Archivers

and a result limit of 200 transactions per search, up to 600 transactions may be returned in one search.

• Search Time Range Limit—defines the largest possible (and default) time range limit used when searching. All Search screens are constrained to use a specific time range (for example, the last 48 hours).Note The default value is 48 hours. The maximum value allowed is 72 hours. Larger

values may increase the time required to complete a search.

Setting User Preferences 77Search Settings

• Search View—specifies the search screen to be used when logging into the FxV browser interface, and when clicking Search on the menu bar.Note The default value is “Find Transactions”. If “Find Transactions” is selected and there

are no Transaction Filters defined, then “Find Sessions” is used by default.

• Hit Search Result View—specifies the view first used when displaying hit search results (default value is “Hits”). The options are the following:

• Hits—shows the individual hits found by the hit search. From this view, you can preview the content of a hit and other hit details.

• Aggregate (full URL)—groups hits together based on URL (for example, if a search returned several hits against the exact same URL, this would be displayed as a single result row).

• Aggregate (ignore URL params)—aggregates hits by URL, but ignores URL query parameters (the portion of the URL after the “?”).

Note The view type can also be changed via the search results screen, by selecting an option from the Search Result View drop-down menu.

• Show Only HTML Search Results—specifies if non-HTML hits should be hidden when first displaying hit search results (default value is “No”).Note This setting can be toggled by clicking the HTML icon in the Type column on the

search results screen.

• Hit Search Results Hidden Columns—specifies the columns to be hidden on the hit search results screen. All custom field columns and the pre-defined hit detail columns are included in the list of columns that may be hidden. To edit the list of hidden columns, click the Edit Hidden Columns icon, use the arrow icons to select the columns that should be Displayed or Hidden in the search result screen, and click OK.Note The hidden columns can also be edited via the hit search results screen (when the

Hits view is selected), by clicking the Edit Hidden Columns icon in the top left corner.

• Session Search Results Hidden Columns—specifies the columns to be hidden on the session search results screen. All custom field columns and the pre-defined session detail columns are included in the list of columns that may be hidden. To edit the list of hidden columns, click the correspondent Edit Hidden Columns icon, and use the arrow icons to select the columns that should be Displayed or Hidden in the search result screen.Note The hidden columns can also be edited via the hit search results screen (when the



• Transaction Search Results Hidden Columns—specifies the columns to be hidden on the transaction search results screen. All custom field columns and the pre-defined transaction detail columns are included in the list of columns that may be hidden. To edit the list of hidden columns, click the correspondent Edit Hidden Columns icon, and use the arrow icons to select the columns that should be Displayed or Hidden in the search result screen.Note The hidden columns can also be edited via the hit search results screen (when the


6

Configuring the Hit Analysis Process

This chapter presents the resources available to users for configuring the FxV Hit Analysis process, that is, resources defining the conditions and actions used to analyze hits or sessions as they are captured.


Hit Filters .....................................................................................................................................80Transaction Filters .......................................................................................................................92Special Events...........................................................................................................................106Custom Fields............................................................................................................................108Metrics ....................................................................................................................................... 115Captured Metadata.................................................................................................................... 117Scripts........................................................................................................................................ 118Sensitive Data Protection ..........................................................................................................121Hit Analysis Configuration Options ............................................................................................125Hit Analysis Configuration Change Log.....................................................................................128Hit Analysis Import/ Export Configuration..................................................................................130

For Hit Analysis examples see “Hit Analysis Examples” on page 133.

For details about how to configure a secondary database for long-term storage of session and transaction data captured by FxV, see “Configuring the Analysis Repository” on page 149.


Hit FiltersA Hit Filter is a collection of match conditions and actions to be performed when a hit matches those conditions. Every Hit Filter is evaluated against each hit as it enters an FxV Archiver.

Hit Filters can be used to detect and alert on any per-hit conditions, to mark interesting hits for later searching, and to manage hit storage. They can also be used to define events within Transaction Filters.

Hit Filters are helpful because they provide a human context to the captured hits. For example, it is easier to search and alert on Login Attempts using a Hit Filter than to search and alert by the server, path, and fields used by the Web application.

Hit Filters are very powerful in their ability to extract custom fields and update metrics. Using regular expressions, Hit Filters can pull data from fields, headers, cookies, or from page content.

For detailed information about Hit Filters, see the following sections:

• “Hit Filter Match Conditions” on page 80

• “Hit Filter Actions” on page 81

• “Hit Filter Configuration” on page 90

• “Hit Filter Additional Information” on page 92

Hit Filter Match ConditionsIn order for a Hit Filter to have an impact on a hit, the filter’s Match Conditions must apply to the hit. Match conditions may include elements such as the request URL, request field values, HTTP response code, etc. When necessary, the match conditions may even evaluate the output of a custom script (written in the Groovy programming language) or analyze the full content of the HTTP response.

A full range of comparison operators is available when defining match conditions. These include “equals”, “contains”, “less than”, “matches” (regular expression), etc.

Note The most frequently captured header/field names are included in the list of captured metadata. Less frequently captured data may be excluded from this list. This feature applies only to fields, cookies, request headers, and response headers.

Configuring the Hit Analysis Process 81Hit Filters

Defining Match ConditionsWhen defining a Hit Filter match condition, follow these guidelines:

• Make match conditions as specific as possible.

• Avoid the use of response content matches. Performing regular expression matching on every hit can negatively impact the performance of the capture system.

• Avoid the use of scripting. Executing a script on every hit can negatively impact the performance of the capture system.

• Avoid creating hit filters that match every hit, unless you truly want to perform an action on every hit.

Grouping Match ConditionsMultiple match conditions may be combined using arbitrary boolean logic operators (for example, AND, OR, and NOT), as needed.

Example:

Request path contains “shopping” AND the value of the “cartValue” cookie is >0 AND the HTTP response code is >=500.

Hit Filter Match Conditions can be combined in complex grouping of match conditions. When at least three match conditions are defined, you can specify an arbitrary expression that defines how the conditions are to be evaluated.

Example:

With conditions 1, 2, and 3, you could write:

AND(1 OR(2 3))

This would cause a match to occur when condition 1 occurs AND either condition 2 OR condition 3 occurs.

Hit Filter ActionsWhen a Hit Filter’s match conditions match a given hit, any number of actions may be performed. For additional details, see the following sections:

• “Execute Scripts” on page 82

• “Hit Filters: Set Custom Fields” on page 82


• “Hit Filters: Increment Metrics” on page 86

• “Configure Storage Settings” on page 87

Execute ScriptsFxV Hit Analysis supports the concept of script execution using the Groovy programming language. Scripts can be used to perform complex operations which are not supported directly by the Hit Analysis model, including the execution of complex algorithms or remotely accessing external systems (for example, to look up user account information based on data in the hit, such as a user ID).

Scripts can set output variables that can be referenced in other areas of the Hit Filter (for example, custom fields and metrics). If the only purpose of a script is to set a particular output variable, it doesn’t need to be explicitly executed as an action of the Hit Filter. Instead, the script output variable can be referenced when updating the custom field or metric, and the script is executed as needed.

For detailed information about scripting, see “Scripts” on page 118.

Hit Filters: Set Custom FieldsCustom Fields allow specialized information to be associated with individual hits, sessions, and transactions. A Hit Filter whose match conditions match a given hit can add (or update) custom field values for that hit or for the session that contains it (transaction custom fields are only accessible in Transaction Filters).

You can add a custom field to a Hit Filter by creating a new Custom Field or by selecting (and customizing, if necessary) an existing one. In the first case, the following Custom Field attributes can be configured:

• “Name” on page 83

• “Description” on page 83

• “Storage (Scope)” on page 83

• “Storage (Type)” on page 84

• “Value Source” on page 85

• “Value Population Policy” on page 85

• “Value Assignment Mode” on page 85

• “Update If Blank” on page 86


In the latter case, you must specify only the value source and the value population policy. Optionally, you may also override any of the following three attributes defined for the Custom Field: Value Assignment Mode, value Separator (for append modes), and Update If Blank. The other attributes get the default settings defined for that Custom Field.

Name

This field defines the name of the Custom Field.

Note This name may contain only letters, numbers, and spaces. Other characters are not allowed. Names must be less than 80 characters long, and cannot contain leading or trailing spaces.

Description

This field is optional. If populated, it is used to build tooltips and reports, when the custom field name appears as a column heading in a search results screen.

Storage (Scope)

A Hit Filter can set hit-scoped or session-scoped custom fields. Within these two categories, there are additional options to be considered. The main thing to decide is where the custom field needs to be seen.

The following table presents the values you can select for hit-scoped custom fields.

Value (Update Hit) Description

Show in Search Results For fields that should appear in search results.

Not in Search Results For fields that should not be in search results, but can be seen when examining hit details for a specific hit.

First Column in Search Results For identifying fields (for example, username) that should appear first in search results (far left side of the results screen).


The following table presents the values you can select for session-scoped custom fields.

Storage (Type)

A Custom Field must be declared as either a string (Store Values as Text) or numeric (Stores Values as Numbers) field. Explicitly defining a custom field as numeric enables searching based on numerical comparison operators (for example, is the value of custom field “shopping cart value” greater than 100?). Numeric sorting is also enabled.

Value (Update Session) Description1

1 The options that end with “(+ Audit Log)” can be used to flag custom fields for audit logging when sessions are viewed by FxV users. This allows you to track the data seen by individual FxV users.

Available only during Hit Analysis

In some cases, a piece of data extracted from a hit is needed by a Transaction Filter. This setting allows such data to be stored for transaction processing purposes without exposing the value at the session level.


Not in Search Results For fields that should not be in search results, but can be seen when examining session details for a specific session.


Show in Search Results (+ Audit Log)

For fields that should appear in search results, and be used for audit logging.

Not in Search Results (+ Audit Log)

For fields that should not be in search results, but can be seen when examining hit details for a specific hit, and be used for audit logging.

First Column in Search Results (+ Audit Log)

For identifying fields (for example, username) that should appear first in search results (far left side of the results screen). Also used for audit logging.


Value Source

A custom field’s value source defines where the value for the field comes from. The following basic types of value sources are available:

• Hit Details—any data already associated with the hit (request field values, cookie values, URL, etc.).

• Script Output—any data exposed as an “output variable” by a Groovy script.Note The script is executed as needed in order to populate the custom field.

• Fixed Values—any hard-coded value.

A regular expression may be applied to the value source in order to extract only a portion of the value. For example, if request field “price” contains values in the form “123.00USD”, a regular expression could be used to extract the value “123.00” and store it in a “numeric price” custom field.

To add a regular expression, click the two slashes (//) at the far right of the Value Source entry section.

Value Population Policy

Custom field values are typically updated any time a Hit Filter’s match conditions apply to a hit. However, it is possible to configure updates to occur conditionally based on the hit status (either the final hit status, or the hit status determined by the current Hit Filter).

Value Assignment Mode

This attribute controls how the Custom Field is updated, which is most useful for fields added to sessions. The field value can be “reset” for each match, or “set on the first match” to lock in just the initial value. String fields allow “appending” to an existing field using a custom separator; numeric fields allow “incrementing”. When appending strings, a maximum of 2 KB is stored in a Custom Field value.

When a custom field already has a value, several different options are available.

Value Description1

Reset value on each match Default mode in which the custom field value is always set to the value defined by the current value source [2].


Update If Blank

Blank values (as defined by any value source) are ignored by default (that is, a previous custom field value is not replaced by a blank value). This behavior can be changed by setting the Update If Blank flag to “Yes”.

Hit Filters: Increment MetricsMetrics are essentially numeric, global custom fields with a value assignment mode of “increment value on match”. You can add a metric to a Hit Filter by creating a new metric or by selecting (and customizing, if necessary) an existing one. Like custom fields, metric values can be incremented by:

• Fixed Values—any hard-coded value (typical “counter” metrics are set to increment by a hard-coded value of one).

• Script Outputs—any data exposed as an “output variable” by a Groovy script.Note The script is executed as needed in order to determine the metric increment amount.

Set value on first match if not already set

Once the custom field value is set, it is not changed [1].

Append to value on each match Concatenates values as they are encountered (an optional separator can be inserted between the values) [1232].

Increment value on each match Increments a numerical custom field value by the amount defined by the current the value source [8].

Append to value on each match (if unique)

Similar to Append to value on each match, but only unique values are added [123].

Append to value on each match (if unique, ignoring case)

Similar to Append to value on each match, but values like “foo” and “Foo” are considered identical [123].

1 The values in brackets represent the final custom field value in the case where three assignments to the same field occur in the following sequence: 1, 2, 3, 2.

Value Description1


• Hit Details—any numeric data already associated with the hit (for example, request field values, cookie values, extracted response content, etc.).

Like custom fields, metric values are typically updated any time a Hit Filter’s match conditions apply to a hit. However, it is possible to configure updates to occur conditionally, based on the hit status (either the final hit status, or the hit status determined by the current Hit Filter).

When entering the name for a metric, it is sometimes useful to parameterize the name based on some piece of data from the hit.

Example

You may want to define a series of metrics such as:

Errors from Spain

Errors from France

Errors from Italy

These can be defined via a single metric that contains a substitution token. The name of the metric would be defined as Errors from $0.

When $0 is included in the metric name, the Hit Filter UI displays a field called $0 Source used to identify the value source for the $0 token (that is, the data that should be used to replace $0 in the metric name).

In this example, you would select Country from the list of hit details, and the hit’s country would be used in the metric name.

Important All value sources available for populating custom field and metric values are available here as well. In order to avoid the creation of an unusable number of metrics, ensure that there is a reasonable number of unique values for the substitution token.

Configure Storage SettingsThis section allows for the configuration of storage-related policies for the hits that match this Hit Filter's match conditions.

Define Hit Storage Restrictions

The default storage policy is to store all hits, without filtering sensitive data and without indexing the response content. A Hit Filter may specify that hits that match the filter’s match conditions and (optionally) have a specific status value should be given special


consideration in terms of how they are stored. Various combinations of the following options may be selected:

• Do not store hit—hits matching the conditions are dropped completely (that is, no data is stored), and they are not available for search/replay. This setting saves on storage space, but results in complete loss of data for hits matching the Hit Filter’s match conditions. This setting requires that the Do not store response content setting is selected.

• Do not store response content—hit details are stored without response content.

• Index response content—response content is stored and indexed, to make the hit’s content keyword-searchable. This setting requires that the Do not store hit and Do not store response content settings are not selected.

• Do not store hit details marked “Always Sensitive”—hit is stored after applying any Sensitive Hit Detail rules defined as “Always Sensitive” (to avoid storing sensitive fields, headers, or cookie values). This setting requires that the Do not store hit setting is not selected.

• Do not store response content marked “Always Sensitive”—response content is stored after applying any Sensitive Response Content Expressions defined as “Always Sensitive” (to hide sensitive data contained in the response content before storing the hit). This setting requires that the Do not store hit and Do not store response content settings are not selected.

Note If a hit matches multiple Hit Filters, any overrides to the default storage policy are honored.

Storage restrictions can be customized based on the final status of the hit (for all matching filters), or just the status defined by this Hit Filter alone. You can choose to keep all matching hits, or just the errors, or tailor your storage to make best use of the available disk space.

Define Session Storage Restrictions

By default, all sessions are transferred to the Analysis Repository (if configured). This section allows sessions to be excluded from the Analysis Repository based on a Hit Filter match, by selecting the Do not store session in Analysis Repository check box.

Mark Hit

You can also determine (via Mark Hits menu) whether a Hit Filter match should be stored in the capture database for later searching. The following options are available:

• Always—to be able to search on the Hit Filter.


• Never—to only support Transaction Filters or to manage hit storage.

• Conditional—to mark hits according to the status of the matched hits. This option is useful when a large number of hit filter matches is expected and the disk space is limited.

“Mark Hit” is the only action performed by default when a Hit Filter matches a hit. Marking a hit means that the hit is annotated as having matched the Hit Filter. This allows users to easily search for hits that matched a given Hit Filter.

In some cases, it may be desirable to only mark hits as matching a Hit Filter when the hit has a particular status (for example, Error). In this case, hit marking can be defined as “conditional” based on the status of the hit.

In the case where there is no need to search for hits that match a given Hit Filter and the Hit Filter is not used by a Transaction Filter, hit marking may be set to Never. In this case, the other actions defined by the Hit Filter are performed, but there is no explicit way to see that the Hit Filter matched the hit.

Hit Filters that match every hit should usually be set to “do not mark the hits”, since no additional information is added by doing so (marking every hit just wastes storage space).

By default, Hit Filters configured to mark hits that meet their match conditions can be used as search criteria when searching for hits. For example, you can search for hits that occurred during a specified time range that matched Hit Filter X. If searching based on Hit Filter matches is not desired, it can be disabled (that is, the Hit Filter does not appear in the list of Hit Filter options on the search screen) by setting the “searchable” flag to NO.

Note Hit Filters whose match conditions are set to match every hit should always be configured as “non-searchable”, since every hit would be returned by a search looking for matches to that Hit Filter.

Update Hit Status

Every hit carries a status of OK, Warning, or Error. Unless explicitly changed, a hit status is always OK (the Hit Analysis process has no intrinsic definition of what constitutes a Warning or Error condition). When a hit matches a Hit Filter’s match conditions, any error or warning conditions are evaluated as well. These conditions are defined just like the match conditions (that is, you can evaluate any aspect of the hit in order to determine its status).


In some situations, it is desirable to say that any time a hit matches a Hit Filter’s match conditions, its status should be set to Error. In fact, the FxV user interface supports this concept as the default when setting a non-OK status. If a status value must be set only in certain situations, an additional step is necessary to define the error and warning conditions.

Multiple Hit Filters can potentially evaluate differently the status of a given hit. The basic rule prevailing in this case is “worst status wins”. That is, if any Hit Filter sets a hit’s status to Error, the hit’s final status is Error. If one Hit Filter sets the status to Warning and another sets it to Error, the hit’s final status is Error. There are situations where it may be important to distinguish between the status of the hit according to Hit Filter X and the final status of the hit (once all Hit Filters have evaluated the hit and set the hit status). The Hit Filter and Transaction Filter interfaces make this distinction and refer to the “final status of the hit” versus the “status recorded by Hit Filter X” where it is useful to differentiate between these two types.

Example:

Hit Filter A does not set the status.

Hit Filter B sets the status to Warning.

Hit Filter C sets the status to Error.

In this case, the final status of the hit is Error. The status according to Hit Filter A is OK, the status according to Hit Filter B is Warning, and the status according to Hit Filter C is Error.

Hit Filter ConfigurationYou can create, edit, view, copy, and delete Hit Filters defined in your system, via the FxV browser interface.

Several “default” Hit Filters are provided with the initial FxV configuration. These filters can be modified as needed to meet the requirements of your installation. They also provide examples of what can be accomplished by defining Hit Filters.

The following Hit Filters are installed by default:

• Default Code Exclusions—drops hits with an HTTP response code of 304. These hits are generally uninteresting responses that tell the browser that it can use the content (usually images) it already has cached from a previous hit. Dropping these hits helps reduce confusion and clutter during replay.


• Default HTTP Error Codes—marks hits with an Error status value due to HTTP Error Code; specifically, the status of hits with an HTTP response code of 400 or greater (with the exception of 401) is set to Error.

• Default Session Custom Fields—adds a custom field called Initial Referer to every session.

• Default Session Page Visit Custom Fields—adds an Entry Page and a Last Page Visited custom field to every session with HTML hits.

• Default Type Exclusions—excludes hits by response content type. This Hit Filter is disabled by default. It is provided as a template for customers wishing to reduce the volume of data captured by the system.

• Optional Keyword Indexing—turns ON content keyword indexing for all HTML hits. Hits can be indexed based on two factors:

• A hit filter has to “mark” the hit for indexing.• The response content type has to start with either “text/html”, “text/xml”, or

“text/plain”.To configure the Hit Filters in your system:

1 On the menu bar, click Configure > Hit Analysis.

The Hit Analysis page appears.

2 Expand the Hit Filters section.

The Hit Filters already defined in your system are displayed in table format.

3 Do one of the following:

Column Name Description

Name Hit Filter name.The Disabled icon following a filter name indicates that the Hit Filter is currently disabled.

Used By Lists any Transaction Filters (or Special Events) currently using this Hit Filter.The Disabled icon following a filter name indicates that the Transaction Filter is currently disabled.If the hit filter match counts for this filter are stored in the Analysis Repository, the Analysis Repository icon is also displayed.


• To view an existing Hit Filter, click the View icon.• To edit an existing Hit Filter, click the Edit icon.• To copy an existing Hit Filter, click the Copy icon.• To remove a Hit Filter from the system, click the Delete icon.

Note If a filter is “in use”, you can delete it from the system only after removing all references to it (including removing its mapping in the Analysis Repository, if applicable).

• To create a new Hit Filter, click Create Hit Filter, then fill in the information requested.

4 Click OK.

Hit Filter Additional InformationThis section is displayed every time you view or edit a hit filter defined in your system. It contains the following attributes:

• Created—the name of the user who created this object and the creation date.

• Last Update—the name of the user who last updated this object and the update date.

• Version—The version number for this object. Each time a change is made, the version number is incremented.

Transaction FiltersA Transaction Filter is a collection of conditions and actions used to analyze sessions as they are captured, and to identify the distinct transactions within those sessions. While they are most commonly used to identify a series of hits that make up some sort of logical transaction, Transaction Filters can be used to detect any activity within a session. A key distinction between Hit Filters and Transaction Filters is that Hit Filters can only examine the hit currently being processed, while Transaction Filters can act on a series of hits within the active session.

A session is a set of hits with the same Session ID that were submitted by the same browser client in the same visit to the site. Sessions are tracked based on simple timeout rules and configurable Global Stop Events. A transaction is a portion of a session that met the criteria specified by a Transaction Filter.

Configuring the Hit Analysis Process 93Transaction Filters

A Transaction Filter can detect multiple events that provide information about site problems and client activity within a related set of pages. Like Hit Filters, Transaction Filters give a helpful and relevant context to the captured data, making the searching and reporting easier.

For detailed information about Transaction Filters, see the following sections:

• “Transaction Events” on page 93

• “Transaction Event Match Conditions” on page 94

• “Transaction Event Actions” on page 97

• “Transaction Filter Storage Configuration” on page 104

• “Transaction Filter Configuration” on page 105

• “Transaction Filter Additional Information” on page 106

Transaction EventsTransaction Filters are made up of a series of Transaction Events (events). There are a number of different types of events, but the most common type is “Hit Filter match”—an event that occurs as the result of a Hit Filter’s match conditions matching an incoming hit. Like Hit Filters, Transaction Events consist of matching conditions (for example, the fact that Hit Filter X matched a hit) and a collection of actions to be performed when those conditions are matched. The different types of match conditions and events are described in the following sections.

Each Transaction Event has a name and an optional one-character label. The labels are used to summarize the events that occurred in a given transaction.

Example:

A Transaction Filter has a start event, three intermediate steps, and a stop event, and uses the labels S, 1, 2, 3, and F for these events. When viewing a list of transactions, the events may appear as S123F / S12F / S321F etc., indicating which events occurred for each individual transaction.

All transactions flow through the following series of states:

Not Started -> Active -> Stopped

Each event specifies the “entry state” required for the event to occur. Events that trigger the beginning of a transaction specify an entry state of Not Started. Most other events


specify an entry state of Active. Certain special events can be defined with an entry state of Stopped. These events use special match conditions that only apply to post-transaction processing.

When determining whether or not to evaluate an event’s match conditions, the Hit Analysis system checks the value of the “allow multiples” flag, in addition to checking the entry state. If a given event has already occurred for the current transaction and this flag is set to “No”, the event is not evaluated again. When this flag is set to “Yes”, multiple occurrences of a given event can be registered within a single transaction. Each time the match conditions are met, the event’s actions are executed and the event is recorded. Using the event labels described earlier, this may result in event label sequences such as S112233123321F, if the events with labels 1, 2, and 3 are all configured to allow multiple occurrences.

Transaction Event Match ConditionsA Transaction Event is fully processed only when its match conditions are met. Several types of match conditions are available, the most common match condition for a Transaction Event being the “Hit Filter match”.

For a description of each match condition type, see these sections:

• “Hit Filter Match” on page 94

• “Hit Count by Hit Filter” on page 95

• “Hit Filter Ordering” on page 95

• “Session Custom Field” on page 95

• “Transaction Custom Field” on page 95

• “Script Output” on page 95

• “Session/Transaction Hit Counts” on page 95

• “Session/Transaction Duration” on page 96

• “Always True” on page 96

• “Post Transaction: Stopped by Global Stop Condition” on page 96

• “Post Transaction: Stopped by Transaction Stop Condition” on page 97

Hit Filter MatchThis type of match condition specifies a specific Hit Filter that triggers the event. If an incoming hit matches the match conditions of the specified Hit Filter, the event is


considered to have occurred. Optionally, a specific hit status can be specified (for example, if the hit matches Hit Filter Foo and has a status of “Error”).

Hit Count by Hit FilterThis type of match condition specifies a specific Hit Filter as well as the number of times the Hit Filter must match within a given session or transaction in order to trigger the event. This type of condition can be used to identify recurring behavior, such as failed login attempts (for example, if Hit Filter Failed Login Page occurs five times in the same session). Optionally, a specific hit status can be specified (only Hit Filter matches with the specified status are counted).

Hit Filter OrderingThis type of match condition specifies two Hit Filters and allows you to define things such as “if (within a given session) a hit matching Hit Filter Bar occurs before a hit matching Hit Filter Foo occurs”. Specific status values may be specified.

Session Custom FieldThis type of match condition allows you to trigger an event based on the value of a Session Custom Field (which may be set by a Hit Filter or a Transaction Filter).

Transaction Custom FieldThis type of match condition allows you to trigger an event based on the value of a Transaction Custom Field defined in this Transaction Filter. This event condition is not typically used, since the condition that caused the custom field to be updated in the first place is usually sufficient.

Script OutputThis type of match condition allows you to trigger an event based on the value of an output variable set by a script.

Session/Transaction Hit CountsThis type of match condition triggers based on the total number of hits since the session started, or since the transaction started. Hit counts can include all hits or can be limited to HTML hits only. Hit status may be taken into account as well. This allows for conditions such as “if the total number of HTML hits in the session with status Error is greater than 100”.


Session/Transaction DurationThis type of match condition triggers based on the duration of the session or transaction. The duration of a session is based on the time passed since the first hit in the session. The duration of a transaction is based on the time passed since the transaction started, as defined by some other Transaction Event (one which changed the state of the transaction to In Progress).

The following timing values can be evaluated:

• Client Time—the amount of time spent on the client side during client-server communication with multiple bursts. It is the sum of all the Client Time delays that occur between each burst of server data.

• Total Time—total “wall clock” time for the session or transaction. In other words, the amount of time that has passed since the session or transaction started.

• Processing Time—the total time spent by the Web server(s) processing hits that occurred within the session or transaction.

• End-to-End Time—the total time spent downloading the contents of the URLs processed within the session or transaction.

Always TrueThis type of match condition triggers on any hit. The purpose of this type of condition is to allow for the creation of transactions that start as soon as the session starts. By using an Always True condition to trigger a transaction state change to “In Progress”, the Transaction Filter causes a new transaction to start with every new session.

Post Transaction: Stopped by Global Stop ConditionThis is a special type of match condition used only for post-transaction processing. It can only be triggered after a transaction has been stopped. It provides a final opportunity to annotate the transaction (or perform other actions) based on the manner in which the transaction was stopped. When this type of condition occurs, it indicates that a system-wide global stop condition forced the transaction to stop. Such conditions include:

• The global stop-transaction event occurred

• The global stop-session event occurred

• The maximum session hit count was reached

• The maximum session duration was reached

• The “long session” timeout was exceeded


• The “short session” timeout was exceeded

Post Transaction: Stopped by Transaction Stop ConditionThis is a special type of match condition used only for post-transaction processing. It can only be triggered after a transaction has been stopped. It provides a final opportunity to annotate the transaction (or perform other actions) based on the manner in which the transaction was stopped. This type of condition occurs only when a transaction is stopped by a Transaction Event explicitly setting the transaction state to “Stopped”. In most cases, the event that stopped the transaction can do any final processing itself. However, if there are multiple events that can potentially stop the transaction, this special condition allows you to define any post-transaction processing in a single location.

Grouping Match ConditionsLike Hit Filter Match Conditions, Transaction Filter Match Conditions can be grouped together as needed (with any combination of the AND, OR, and NOT logic operators). This is rarely necessary for Transaction Event Match Conditions.

Transaction Filter Match Conditions can be combined in complex grouping of match conditions. For more information, see “Grouping Match Conditions” on page 81.

Transaction Event ActionsWhen a Transaction Event’s match conditions are met, any number of actions may be performed. These actions are grouped in the following categories:

• “Set Transaction Status” on page 97

• “Change Transaction/Session State” on page 98

• “Execute Scripts” on page 98

• “Transaction Event: Set Custom Fields” on page 99

• “Transaction Event Increment Metrics” on page 103

Set Transaction StatusBy default, all transactions have a status of “OK”. Each Transaction Event has an opportunity to set the transaction status to “Error” or “Warning”.


Change Transaction/Session StateAs described in section “Transaction Events” on page 93, all transactions flow through a the following series of states:

Not Started -> Active -> Stopped

Certain event conditions identify the beginning or end of a transaction. These events are responsible for changing the transaction state from Not Started to Active (that is, start the transaction) or from Active to Stopped (that is, stop the transaction). In some cases, a single event may start and stop a transaction (that is, take it from Not Started directly to Stopped). This is useful for identifying activities such as repeated failed logins (for example, “When Hit Filter FailedLogin occurs five times, start/stop a HackerAttempt transaction and set a session custom field that flags the suspicious session”).

In addition to changing the state of a transaction, it is also possible to change the state of a session (to Stopped). This allows events such as a user clicking on a logout link to immediately end a session (rather than waiting for the session to eventually time out).

Note A session should only be marked as Stopped in cases where you are certain that the Web application has ended the session.

Some events do not impact the state of the transaction (or session). Once a transaction is active (that is, it has been started by some event), other interesting events may occur that leave the transaction in the Active state.

Execute ScriptsFxV Hit Analysis supports the concept of script execution using the Groovy programming language. Scripts can be used to perform complex operations not supported directly by the Hit Analysis model, including the execution of complex algorithms or the remote access of external systems (for example, to look up user account information based on captured data such as a user ID).

Scripts can set output variables that can be referenced in others areas of the Transaction Event (custom fields and metrics). If the only purpose of a script is to set a particular output variable, the script does not have to be explicitly executed as an action of the Transaction Event. Instead, the script output variable can be referenced when updating the custom field or metric, and the script is executed as needed.

For detailed information about how scripting works, see “Scripts” on page 118.


Transaction Event: Set Custom FieldsCustom Fields allow specialized information to be associated with individual hits, sessions, and transactions. A Transaction Event whose match conditions have been met can add (or update) custom field values for the current transaction or for the session that contains it (hit custom fields are only accessible in Hit Filters).

You can add a custom field to a Transaction Event by creating a new Custom Field or by selecting (and customizing, if necessary) an existing one. In the first case, the following Custom Field attributes can be configured:

• “Name” on page 99

• “Description” on page 99

• “Storage (Scope)” on page 100

• “Storage (Type)” on page 101

• “Value Source” on page 101

• “Value Assignment Mode” on page 102

• “Update If Blank” on page 102

In the latter case, you must specify only the value source and the value population policy. Optionally, you may also override any of the following three attributes defined for the Custom Field: Value Assignment Mode, value Separator (for append modes), and Update If Blank. The other attributes get the default settings defined for that Custom Field.

Name

This field defines the name of the Custom Field.


Description

This field is optional. If populated, it is used to build tooltips and reports, when the custom field name appears as a column heading in a search results screen.


Storage (Scope)

A Transaction Filter can set transaction-level or session-level custom fields. Within these two categories, there are additional options to be considered. The main thing to decide is where the custom field needs to be seen.

The following table presents the values you can select for hit-scoped custom fields.

The following table presents the values you can select for session-scoped custom fields.

Value (Update Transaction) Description

Available only within this Transaction Filter

For fields that are only needed within the definition of the Transaction Filter (perhaps as a value source for some other custom field or metric).



Not in Search Results For fields that should not be in search results, but can be seen when examining transaction details for a specific transaction.


Available only for Transaction Filters

Makes the custom field value available to any Transaction Filter defined in the system, but the value does not appear when viewing session search results or session details.




Storage (Type)

A Custom Field must be declared as either a string (Store Values as Text) or numeric (Stores Values as Numbers) field. Explicitly defining a custom field as numeric enables searching based on numerical comparison operators (for example, is the value of custom field “shopping cart value” greater than 100?). Numeric sorting is also enabled.

Value Source

A custom field’s value source defines where the value for the field comes from. Three basic types of value sources exist:

• Other Custom Fields—a custom field associated with the active session or transaction.

• Script Outputs—any data exposed as an “output variable” by a Groovy script.Note The script is executed as needed in order to populate the custom field.

• Fixed Values—any hard-coded value.











Value Assignment Mode

This attribute controls how the Custom Field is updated. The field value can be “reset” for each event, or “set on the first event” to lock in just the initial value. String fields allow “appending” to an existing field using a custom separator; numeric fields allow “incrementing”. When appending strings, a maximum of 2 K is stored in a Custom Field value.


Update If Blank

Blank values (as defined by any value source) are ignored by default (that is, a previous custom field value is not replaced by a blank value). This behavior can be changed by setting the Update If Blank flag to “Yes”.

Value Description1


Reset value each time this event occurs

Default mode in which the custom field value is always set to the value defined by the current value source [2].

Set value the first time this event occurs

Once the custom field value is set, it is not changed [1].

Append to value each time this event occurs

Concatenates values as they are encountered (an optional separator can be inserted between the values) [1232].

Increment value each time this event occurs

Increments a numerical custom field value by the amount defined by the current the value source [8].

Append value on each match (if unique)

Similar to Append to value each time this event occurs, but only unique values are added [123].

Append value on each match (if unique, ignoring case)

Similar to Append to value each time this event occurs, but values like “foo” and “Foo” are considered identical [123].


Transaction Event Increment MetricsMetrics are essentially numeric, global custom fields with a value assignment mode of “increment value on match”. You can add a metric to a Transaction Event by creating a new metric or by selecting (and customizing, if necessary) an existing one. Similarly to custom fields, metric values can be incremented by:

• Fixed Values—any hard-coded value (typical “counter” metrics are set to increment by a hard-coded value of one).

• Script Outputs—any data exposed as an “output variable” by a Groovy script.Note The script is executed as needed in order to determine the metric increment amount.

• Numeric Custom Fields—any numeric custom field associated with the active session or transaction.

Metric updates defined by Transaction Events do no take place until the transaction stops. Updates can be configured to take place unconditionally (as soon as the transaction ends) or conditionally based on the Commit Pending Metric Update policy defined for the Transaction Filter.

When entering the name for a metric, it is sometimes useful to parameterize the name based on some piece of data from the transaction.

Example

You may want to define a series of metrics such as:

Errors from Spain

Errors from France

Errors from Italy

These can be defined via a single metric that contains a substitution token. The name of the metric would be defined as Errors from $0.

When $0 is included in the metric name, the Transaction Filter UI displays a field called $0 Source used to identify the value source for the $0 token (that is, the data that should be used to replace $0 in the metric name).

In this example, you would select Country from the list of transaction details, and the transaction’s country would be used in the metric name.

Important All value sources available for populating custom field and metric values are available here as well. In order to avoid the creation of an unusable number of metrics, ensure that there is a reasonable number of unique values for the substitution token.


Transaction Filter Storage ConfigurationThe following settings default to “Always”, but can be configured to allow for conditional storage and updates of certain data:

• Store Transactions—can be set to “Always”, “Never”, or “Conditional”, based on the final status of the transaction. Some Transaction Filters may only truly define a transaction when certain events occur (beyond the “start” event). For example, if a session timeout occurs, the transaction may not be meaningful and should not be stored.

• Store Custom Fields—can be set to “Always” or “Conditional”, based on the final status of the transaction. This setting applies to transaction custom fields and allows custom field values to be conditionally stored based on the final transaction status.

• Store Events—can be set to “Always”, “Never”, or “Conditional”, based on the final status of the transaction. In some cases, the occurrence of the individual events provides no additional value. The most common such case would be a Transaction Filter whose events always occur in the same order without variation. Storing the fact that events A, B, C, and D occurred for every transaction is not helpful.

• Commit Pending Metric Updates—can be set to “Always” or “Conditional”, based on the final status of the transaction. Metric updates defined by the events that make up the Transaction Filter can be set to update unconditionally as soon as the transaction stops (by setting the metric’s increment policy to “Always increment when transaction stops”). These metric updates are not impacted by this higher-level setting. However, metric updates with an increment policy of “Conditionally increment when transaction stops” are updated according to this setting. This allows metrics that are only meaningful in certain situations (for example, when the transaction completes successfully) to be updated only when appropriate.

• Store Transaction in Analysis Repository—controls whether or not to store transactions in the Analysis Repository (if enabled). It can be set to “Always” (for transactions that should always be stored), “Conditional” (to base storage on the final status of the transaction), or “Never” (to temporarily disable storage of these transactions in the Analysis Repository).Note Transactions must be stored in the Archiver database in order to be eligible for

transfer to the Analysis Repository. You must also configure the Analysis Repository database schema to define a table for this Transaction Filter.


• Store Session in Analysis Repository—controls whether or not to store sessions in the Analysis Repository (if enabled).Note Transaction Filters and Hit Filters can be configured to flag sessions for exclusion

from the Analysis Repository. If a session is marked for exclusion by any filter, it is excluded regardless of the storage policy defined by any other filters.

Transaction Filter ConfigurationYou can create, edit, view, copy, and delete Transaction Filters defined in your system, via the FxV browser interface. No “default” Transaction Filters are provided with the initial FxV configuration.

To configure the Transaction Filters in your system:



2 Expand the Transaction Filters section.

If Transaction Filters are already defined in your system, they are displayed in table format.


• To view an existing Transaction Filter, click the View icon.• To edit an existing Transaction Filter, click the Edit icon.• To copy an existing Transaction Filter, click the Copy icon.

Note When you copy a Transaction Filter that has transaction-scoped custom fields, you are implicitly creating new custom fields (with transaction scope, but for the new Transaction Filter).

• To remove a Transaction Filter from the system, click the Delete icon.Note If a filter is stored in the Analysis Repository, you can delete it from system only

after you remove its mapping to the Analysis Repository.


Name Transaction Filter name.

Used By Displays the Analysis Repository icon if transactions identified by this filter are stored in the Analysis Repository.


• To create a new Transaction Filter (using the full-featured user interface), click Create Transaction Filter, then fill in the information requested.

• To create a new Transaction Filter and new Hit Filters in a single step, click Quick Create, then fill in the information requested. This interface supports only very simple Hit Filters that match on a single URL path or HTML page title.Note You can use the Quick Create interface to quickly create multiple Hit Filters and

the Transaction Filter that uses them and then go back and modify the individual filters as needed.

4 Click OK.

Transaction Filter Additional InformationThis section is displayed every time you view or edit a transaction filter defined in your system. It contains the following attributes:




Special EventsThe model used to describe the events that define each Transaction Filter is also used to define three global events. These events are not specific to any particular Transaction Filter, but they run as part of the Hit Analysis process that occurs for each hit captured by the system. These events defined outside Transaction Filters stand on their own.

Certain aspects of the Transaction Event model do not apply to the special events. Specifically, anything that depends on the enclosing Transaction Filter or the active transaction is not applicable, since these events are not evaluated in the context of a transaction.

The three events that can be defined are:

• Global Stop Session Event—this event is used to define conditions that must trigger the end of a session. Without a Global Stop Session Event, the only way to identify the end of a session is by a session timeout. This event allows activities

Configuring the Hit Analysis Process 107Special Events

that are known to stop sessions (user logout being the most common example) to immediately close a session, without having to wait for the session to time out.

• Global Stop Transaction Event—this event works much like the Global Stop Session Event, but rather than closing the entire user session, it stops any active transaction(s). Global Stop Transaction Events are rarely meaningful, as it is unlikely that a common activity applies to all Transaction Filters in the system, unless it is something such as a logout page (in which case the Global Stop Session Event is probably a better choice).

• Post Hit Analysis Event—this event is fired as the very last step for each hit processed by the system. This is the only place where certain data is guaranteed to be updated prior to processing.

For detailed information about Special Events, see the following sections:

• Special Events Configuration

• Special Events Additional Information

Special Events ConfigurationThe special events provided with the initial FxV configuration are undefined. Once you define them, you can further modify them as needed to meet the requirements of your installation.

To define a Special Event in your system:



2 Expand the Special Events section.

The list of events is displayed.Note No Special Events are defined in the initial FxV configuration.

3 Click the Edit icon beside the event you want to define.Note Editing is the only operation available for an undefined special event.

The Default <Event Name> page appears.

4 Fill in the information to customize this event, as necessary, then click OK.

The newly defined special event is now marked as IS Defined.


To configure a Special Event defined in your system:



2 Expand the Special Events section.

The special events that have been defined in your system are marked as IS Defined.


• To view an existing Special Event, click the View icon.• To edit an existing Special Event, click the Edit icon.• To cancel all customizations made to a Special Event, click the Delete icon.

4 Click OK.

Special Events Additional InformationThis section is displayed every time you view or edit a special event defined in your system. It contains the following attributes:




Custom FieldsCustom Fields allow specialized information to be associated with individual hits, sessions, and transactions.

For detailed information about Custom Fields, see the following sections:

• “Custom Fields Attributes” on page 109

• “Custom Fields Configuration” on page 113

• “Custom Fields Additional Information” on page 115

Configuring the Hit Analysis Process 109Custom Fields

Custom Fields AttributesCustom fields are highly configurable. The following sections describe the attributes that can be configured for each Custom Field:

• Name

• Description

• Storage (Scope)

• Storage (Type)

• Value Assignment Mode

• Update If Blank

NameThis field defines the name of the Custom Field. Custom Field names must be unique within a given scope, but the same name may be used in different scopes (that is, you can have a Hit Custom Field named foo and a Session Custom Field named foo). Each Transaction Filter defines its own unique scope (that is, two different Transaction Filters can define Custom Fields named foo).


DescriptionThis attribute is optional. If populated, it is used to build tooltips and reports.

Storage (Scope)This attribute controls how the Custom Field is stored in the capture database and how it should be used for search results (that is, not shown at all, included with search results, or included in the first column in the results). Once you define a custom field, you can not change its scope, but you can change the storage details within that scope.

Important The Storage (Scope) attribute cannot be overridden by filters that update this Custom Field.


Select one of the Update Hits options to store the field with all matching hits for easy searching later. The following table presents the values you can select for hit-scoped custom fields.

Select one of the Update Session options to store the field with the sessions of all matching hits, for use by Transaction Filters and/or when searching for sessions. The following table presents the values you can select for session-scoped custom fields.

Value (Update Hit) Description


For transient values that do not need to be stored in the database.


Not in Search Results For fields that should not be in search results, but can be seen when examining hit details for a specific hit.




For transient values that do not need to be stored in the database.







Custom Fields with transaction scope can only be created when creating or editing the Transaction Filter that defines their scope.

Storage (Type)A Custom Field must be declared as either a string (Store Values as Text) or numeric (Stores Values as Numbers) field. Explicitly defining a custom field as numeric enables searching based on numerical comparison operators (for example, is the value of custom field “shopping cart value” greater than 100?). Numeric sorting is also enabled.

Once you define a custom field, you can not change its storage type.

Important The Storage (Type) attribute cannot be overridden by filters that update this Custom Field.

Value Assignment ModeThis attribute controls how the Custom Field is updated, which is most useful for fields added to sessions. The field value can be “reset” for each match, or “set on the first match” to lock in just the initial value. String fields allow “appending” to an existing field using a custom separator; numeric fields allow “incrementing”. When appending strings, a maximum of 2 K is stored in a Custom Field value.

Important This setting can be overridden by filters that update this Custom Field.









Update If BlankThis attribute determines how blank values should be treated. By default, this attribute is set to No, and blank values are ignored (that is, a previous custom field value is not replaced by a blank value). Set to Yes to retain blank values when setting or appending strings.

Important This attribute can be overridden by filters that update this Custom Field.

Value Description1


Reset value Default mode in which the custom field value is always set to the value defined by the current value source [2].

Set value only if not already set Once the custom field value is set, it is not changed [1].

Append to value Concatenates values as they are encountered (an optional separator can be inserted between the values) [1232].

Increment value Increments a numerical custom field value by the amount defined by the current the value source [8].

Append value (if unique) Similar to Append to value, but only unique values are added [123].

Append value (if unique, ignoring case)

Similar to Append to value, but values like “foo” and “Foo” are considered identical [123].


Custom Fields ConfigurationYou can create, edit, view, copy, and delete all Custom Fields defined in your system, via the FxV browser interface.

Several “default” Custom Fields are provided with the initial FxV configuration. These Custom Fields can be modified as needed to meet the requirements of your installation. They also provide examples of what can be accomplished by defining custom fields.

The following Custom Fields are installed by default:

• Entry Page—identifies the first HTML page visited by the user during this session.

• Initial Referer—identifies the Web page the user was visiting prior to starting this session.

• Last Page Visited—identifies the last HTML page visited by the user during this session.

To configure the Custom Fields defined in your system:



2 Expand the Custom Fields section.

The Custom Fields already defined in your system are displayed in table format.


Name Name of the Custom Field, preceded by two icons indicating the scope and the type of the field:• Hit Scope • Session Scope • Transaction Scope • Values store as text • Values stored as numbers



• To view an existing Custom Field, click the View icon.• To edit an existing Custom Field, click the Edit icon.• To copy an existing Custom Field, click the Copy icon.• To remove a Custom Field from the system, click the Delete icon.

Note If a Custom Field is in use, you can delete it from the system only after you remove all references to it (including any Analysis Repository mappings, if applicable).

• To create a new Custom Field, click Create Custom Field, then fill in the information requested.Note Custom Fields can also be created from within the Hit Filter, Transaction Filter,

and Special Event editors.

4 Click OK.

The Custom Field page that appears when you view or edit a Custom Field contains a section named Updated By. This is a read-only table that lists all of the objects (Hit Filters, Transaction Filters, or Special Events) that update the value of that Custom Field. The primary purpose of this table is to show any overridden values. Since updating objects can override the “assignment mode”, “appended values separator”, and “update if blank policy”, this table has columns for each of these three settings. The values in these columns are blank unless the updating object overrides the value. If it does, the overridden value is shown in the table.

The only exception to this rule is when a Custom Field has an assignment mode of “append” and defines a separator character. If a Hit Filter, Transaction Filter, or Special

References A list of objects that either set the value of the Custom Field or access its value. Each reference is preceded by two icons:• The filter or event reads the value of the Custom Field • The filter or event sets the value of the Custom Field • Hit Filter • Transaction Filter • Special Event If this Custom Field is stored in the Analysis Repository, the Analysis Repository icon is also displayed.


Configuring the Hit Analysis Process 115Metrics

Event overrides and removes the separator, the Used By table displays a value of “{none}” in the Separator column.

Any changes made to a Custom Field are automatically propagated to its “updating objects”, unless they are overriding the value. Overrides can be cleared by editing either the Custom Field or the overriding object, and modifying the values so that they match.

For details on referencing and updating Custom Fields from Hit Filters, Transaction Filters, and Special Events, see the following sections: “Hit Filters: Set Custom Fields” on page 82, “Transaction Event: Set Custom Fields” on page 99, and “Special Events” on page 106.

Custom Fields Additional InformationThis section is displayed every time you view or edit a custom field defined in your system. It contains the following attributes:




MetricsA Metric is a global numeric value incremented by Hit Filters, Transaction Filters, and Special Events.

For detailed information about Metrics, see the following sections:

• “Metrics Configuration” on page 115

• “Metrics Additional Information” on page 117

Metrics ConfigurationYou can create, edit, view, copy, and delete all Metrics defined in your system, via the FxV browser interface.

A single “default” Metric is provided with the initial FxV configuration:


• HTTP Response Code $0—counts occurrences of individual HTTP Response Codes.

This metric may be modified as needed to meet the requirements of your installation. It provides an example of what can be accomplished by defining metric values.

To configure the Metrics defined in your system:



2 Expand the Metrics section.

The Metrics already defined in your system are displayed in table format.


• To view an existing Metric, click the View icon.


Name Name of the Metric.Metric names that contain the substitution characters $0 are considered “dynamic metrics”. These metric names result in the creation of multiple metrics. The $0 is replaced at Hit Analysis time based on the $0 Source defined by the Hit Filters, Transaction Filters, or Special Events that updated the metric value.Note The $0 Source is not part of the Metric definition, since it can vary

depending on where the metric is updated. The $0 Source fields are only displayed by the editors that define the actual metric updates.

References A list of objects that either update the value of the Metric or access it (in the case of the Analysis Repository). Each reference is preceded by an icon indicating the type of the updating object:• Hit Filter • Transaction Filter • Special Event • Analysis Repository Note If a Metric is not updated by any object in the Hit Analysis model, this

metric is not displayed in the Analysis Repository Configuration page.

Configuring the Hit Analysis Process 117Captured Metadata

• To edit an existing Metric, click the Edit icon.• To copy an existing Metric, click the Copy icon.• To remove a Metric from the system, click the Delete icon.

Note If a Metric is in use, you can delete it from the system only after you remove all references to it (including any Analysis Repository mappings, if applicable).

• To create a new Metric, click Create Metric, then fill in the information requested.

4 Click OK.

For details on referencing and updating Metrics from Hit Filters, Transaction Filters, and Special Events, see the following sections: “Hit Filters: Increment Metrics” on page 86, “Transaction Event Increment Metrics” on page 103, and “Special Events” on page 106.

Metrics Additional InformationThis section is displayed every time you view or edit a metric defined in your system. It contains the following attributes:




Captured MetadataMetadata captured by the system can be used when searching and when building Hit Filters and Transaction Filters.

To display metadata captured by your system:



2 Display the Captured Metadata menu options.

3 To display a specific metadata, click one of the following links:

• View Captured Web Servers


• View Captured Paths• View Captured Field Names• View Captured Request Header Names• View Captured Response Header Names• View Captured Cookie Names• View Captured HTML Titles• View Captured Browser Types• View Captured ISPs• View Captured Cities• View Captured Regions• View Captured CountriesNote Data displayed by clicking these links can be exported to CSV files.

FxV users can control the amount of metadata that is included in the Captured Field Names, Captured Request Header Names, Captured Response Header Names, and Captured Cookie Names lists by editing the Metadata Sensitivity property in the Server Configuration page (for details, see “Server Configuration” on page 170).

The Metadata Sensitivity property specifies the minimum reference count required for a name to be included in the list. The reference count is the number of hits that contain the field, header, or cookie name in any of the database segments. For example, setting the Metadata Sensitivity property to “1” results in including all names in the list; setting this property to “1000” results in including only those names that appear in at least 1000 hits in a database segment.

Note This property affects only the Captured Field Names, Captured Request Header Names, Captured Response Header Names, and Captured Cookie Names metadata lists.

ScriptsFxV uses scripts (snippets of custom logic written in the Groovy scripting language) to allow its users to extend the functionality of the Foglight Experience Viewer system. Scripts can be used to integrate FxV with other systems, as well as perform more complex analysis of the data than is allowed by the standard hit analysis filters (such as evaluating complex match expressions, transforming hit data, applying complex regular expressions, etc.). A script is executed by a Hit Filter or a Transaction Filter, which can use the output variables published by the script to update custom fields or metrics.

Configuring the Hit Analysis Process 119Scripts

For detailed information about Scripts, see the following sections:

• “Scripts Configuration” on page 119

• “Scripts Additional Information” on page 120

Scripts ConfigurationTo implement scripts into the FxV Hit Analysis model:

1 Determine the script requirements.

2 Select the appropriate Hit Analysis APIs.

3 Select the script’s execution frequency.

4 Determine the script’s output.

5 Write the script.

6 Test the script.

7 Add the script to the Hit Analysis model.

For detailed information about implementing scripts into the FxV Hit Analysis Model, see the Groovy Scripting API documentation.

For additional information about the Groovy scripting language, see http://groovy.codehaus.org/Documentation.

To manage Scripts via the FxV browser interface:



2 Expand the Scripts section.

The Scripts already defined in your system are displayed in table format.


Name Script name.



• To view an existing Script, click the View icon.• To edit an existing Script, click the Edit icon.• To copy an existing Script, click the Copy icon.• To remove a Script from the system, click the Delete icon.• To create a new Script, click Create Script. In the Script definition page, fill

in the information requested.• To access the Groovy Scripting API documentation, click Create Script. In

the Script definition page, click the Groovy Scripting API link in the Script Definition area.

4 Click OK.

Since scripts commonly make use of regular expressions, the Script create/edit page provides direct access to the FxV Regular Expression Tester. For additional information about this topic, see “Appendix: Java Regular Expressions in FxV Hit Analysis” on page 187.

Scripts Additional InformationThis section is displayed every time you view or edit a script defined in your system. It contains the following attributes:



Executed/Errors

The number of times this script has been executed and the number of times it has failed in the current day (since 12:00 AM).Note The number of failures is incremented when an uncaught exception

occurs while executing the script, keeping the script from running to completion.

Used By Lists any Hit Filters or Transaction Filters currently using this Script.The Disabled icon following the name indicates that the filter is currently disabled.


Configuring the Hit Analysis Process 121Sensitive Data Protection


Sensitive Data ProtectionThis section focuses on FxV features that allow sensitive data (for example, credit card numbers, passwords, etc.) captured by FxV to be hidden from some or all users of the FxV browser interface.

Note The features described in this section are just a subset of the options available for dealing with sensitive data.

The following considerations are important when planning to protect sensitive data:

• Full-disk encryption can be configured on the FxV appliance to ensure that sensitive data cannot be extracted directly from stolen hard drives.

• The FxV browser interface can be configured such that all traffic is sent via https (SSL).

• All configuration changes of the FxV capture system are logged to ensure that unauthorized changes can be identified.

• The Foglight Experience Monitor (FxM) can be configured such that certain data is never passed to FxV.

• Entire hits can be dropped (not stored), by configuring the Storage Restrictions for a Hit Filter.

Sensitive Hit DetailsIn many cases, sensitive data is passed into the appliance as a result of the user entering the data into a form. Such data appears in FxV as request fields. For example, when a user enters a credit card number into a payment form and submits the form, a hit containing a request field cc_num=4123456781234567 may be captured. Such data can be hidden from display be defining a Sensitive Hit Detail.

A Sensitive Hit Detail is defined by the following attributes:

• Name—is the name given to the Sensitive Detail, for identification purposes.

• Description—reserved for describing the sensitive data.


• Hit Detail Type—can be one of the following options: “Request Field” (the most common), “Request Header”, “Response Header”, and “Cookie”.

• Matching Logic—specifies how the Hit Detail Name must be interpreted. A value of “=” indicates that the name must match exactly in order to be identified as sensitive. Other values options include “Starts With”, “Ends With”, “Contains”, and “Matches” (which treats the Hit Detail Name as a regular expression). FxV also provides “ignore-case” versions of each of these options (except for “Matches”), which allow for case-insensitive comparisons.

• Hit Detail Name—is the name of the sensitive field/header/cookie (for example, cc_num). Depending on the value of the Matching Logic attribute, this value may be an exact name, a partial name, or a regular expression.

• Replacement Text—specifies a fixed string to be displayed in place of the sensitive data. This field is optional.

• Always Sensitive—indicates whether the sensitive data should be considered sensitive regardless of the privileges of the FxV user. This flag is set to “Yes” by default. When set to “No”, users with special privileges are able to see the actual value of the sensitive data, while other users see only the Replacement Text (or nothing, if no replacement text is specified). In order to see sensitive data flagged as “not always sensitive” a user must be a member of a User Group that has the Sensitive Data Display privilege turned on.

Example:

Hit Detail Type = Request Field

Hit Detail Name = cc

Matching Logic = Starts With (IC)

Replacement Text = ****

Always Sensitive = No

In this scenario, the value of any request field starting with the letters cc (or CC or cC or Cc) is replaced with the string “****” when displayed to a typical user. However, users who are members of a User Group with the Sensitive Data Display privilege can see the actual credit card numbers.

Note Regardless of the value of the Always Sensitive flag, the sensitive data is always captured and stored in the database. The replacement occurs when the data is queried by an FxV

Configuring the Hit Analysis Process 123Sensitive Data Protection

user. For information on disabling the storage of sensitive data completely, see “Define Hit Storage Restrictions” on page 87.

Data marked as sensitive is not available for use during Hit Analysis. For example, if the request field cc_num is identified as sensitive, this setting cannot be circumvented by creating a Hit Filter that populates a custom field with the value of the cc_num request field. If this were attempted, the replacement text would be stored in the custom field, not the original value. This is true regardless of the value of the Always Sensitive flag, since the Hit Analysis processing is performed outside the context of any specific user.

Sensitive Response Content ExpressionsSensitive Hit Details are ideal in situations where the user is entering the sensitive data to be hidden. However, some Web sites render sensitive data directly in the content of a page. For example, after a user creates a new account, the next page might contain text such as “Thank you for joining. Your username is superdog and your password is 5uP3rd0G”. In this case, the user’s password is part of the HTML response and cannot be hidden using a Sensitive Hit Detail. Sensitive Response Content Expressions can handle such situations.

A Sensitive Response Content Expression is a regular expression that gets applied to the response content and hides or replaces the specified portion of the page.

Note Sensitive Response Content Expressions are only applied to text/html hits.

A Sensitive Response Context Expression is defined by the following attributes:

• Name—is the name given to the Sensitive Response Content Specification, for identification purposes.

• Description—reserved for describing the sensitive data.

• Regular Expression—a regular expression that identifies the sensitive text on the page. This expression usually contains one or more sets of “grouping parenthesis” that identify the portion of the expression to be hidden.

• Sensitive Groups—one or more numbers that identify which of the “groups” defined by the parenthesis in the expression are to be hidden. If there are no grouping parenthesis, the value of this attribute is always “0”, which refers to the entire regular expression. Pairs of parenthesis are numbered “1..n”, starting from the left.


For example, the expression “abc(def)ghi(jkl)mno” has two groups. Group #1 has the value “def”. Group #2 has the value “jkl”. Group #0 always represents the entire expression, so the value of group #0 is “abcdefghijklmno” in this case.Note The most common scenario is to define a regular expression with a single pair of

grouping parenthesis and the Sensitive Groups attribute set to “1”.

• Replacement Text—optional field that allows for the specification of a fixed string to be displayed in place of the sensitive data.

• Always Sensitive—flag that indicates whether the sensitive data should be considered sensitive regardless of the privileges of the FxV user. It is set to “Yes” by default. When set to “No”, users with special privileges are able to see the actual value of the sensitive data, while other users see only the Replacement Text (or nothing, if no replacement text is specified). In order to see sensitive data flagged as “not always sensitive” a user must be a member of a User Group that has the Sensitive Data Display privilege turned on.

Example:

Regular Expression = “and your password is (.*)\.”

Sensitive Groups = 1

Replacement Text = “[HIDDEN BY FXV]”

Always Sensitive = No

In this scenario, consider that the Web site displays the following message (text shown as HTML code) after a user introduces sensitive information into a form:

Thank you for joining. Your username is superdog and your password is 5uP3rd0G.

The Regular Expression finds the password between the and tags. The parenthesis identify the specific text that is sensitive (anything between the tags, that is, the password itself). Since there is one group of parenthesis, the Sensitive Group is “1”.

When displaying this page to a typical user, the content displays as:

Thank you for joining. Your username is superdog and your password is [HIDDEN BY FXV].

When displaying this page to a user with Sensitive Data Display privileges, the original response is displayed:

Thank you for joining. Your username is superdog and your password is 5uP3rd0G.

Configuring the Hit Analysis Process 125Hit Analysis Configuration Options

Note Similarly to Sensitive Hit Detail data, sensitive content identified by Sensitive Response Content Expressions is stored in its original form. Replacement occurs at data access time rather than capture time.

Regardless of the value of the Always Sensitive flag, the sensitive data is always captured and stored in the database. The replacement occurs when the data is queried by an FxV user. For information on disabling the storage of sensitive data completely, see “Define Hit Storage Restrictions” on page 87.

Hit Analysis Configuration OptionsThis section contains various settings that relate to the Hit Analysis process:

• “Session Configuration” on page 125

• “Archiver Configuration” on page 126

• “Script Configuration” on page 127

• “Other Configuration” on page 127

Session ConfigurationThis section provides several configuration options for how the system should identify the end of a session. These settings apply to all Archivers in the system. The following options are available:

• Maximum Hits Per Session—defines the maximum number of hits that can be contained within a single session. When this limit is reached, the session is stopped and a new session (with the same Session ID) is started.

• Maximum Session Duration—defines the maximum time range that a single session can span before being stopped. When this limit is reached, the session is stopped and a new session (with the same Session ID) is started.

• Minimum Hits For “Long” Session—defines how many hits must be present within a session before it is considered “long” for timeout purposes.

• Count Only HTML Hits for “Long” Session—defines if all hits or only HTML hits should be counted when determining if a session is “long” for timeout purposes.


• Timeout for “Short” Sessions—defines the length of idle time before a “short” session is timed out.Note A “short” session is a session that contains one hit or a very small number of hits.

“Short” sessions usually “time-out” fairly quickly, which allows them to be identified as closed more quickly then “long” sessions. This also improves the performance of the Archiver, since fewer active sessions must be maintained in memory.

• Check Only HTML Hits for “Short” Session Timeout—for “short” sessions, determines if all hits or only HTML hits should be interpreted as active use by the client.

• Timeout for “Long” Sessions—defines the length of idle time before a “long” session is timed out. This value should always be set to match the session timeout defined by the Web server, to ensure that sessions timed out by the application are marked as timed-out by FxV at the same time.

• Check Only HTML Hits for “Long” Session Timeout—for “long” sessions, determines if all hits or only HTML hits should be interpreted as active use by the client.

To configure the session timeout settings in the system:



2 Expand the Hit Analysis Configuration Options menu.

3 Edit the settings in the Session Configuration area as required, then click Save Changes.

The settings are now saved and are applied to all Archivers in the system.

Note In addition to these settings, session completion can be further configured by defining a Global Stop Session Event.

Archiver ConfigurationThis sections provides several configuration options that allow you to control the way captured data is segmented by the capture system. These settings apply to all Archivers in the system:

• Maximum DB Segment Span—data captured by the FxV Archivers is stored in database segments. The size of each segment is limited to a fixed maximum size, as well as a maximum time span (whichever comes first). This setting allows that

Configuring the Hit Analysis Process 127Hit Analysis Configuration Options

time span to be configured. Typically, this value is set to 24 hours. It can be changed to a smaller number when it is desirable to purge data from the system after a small period of time (data is removed from the system one segment at a time, based on the value of the Maximum DB Segment Retention option).

• Maximum DB Segment Retention—data captured by the FxV Archivers is stored in database “segments”. This setting determines how long each segment is stored.Note In addition to removing segments based on their age, Archivers also remove old

segments (oldest first) as needed to free up space for new segments. A value of 0 (zero) is interpreted as “no maximum”, in which case segments are removed only as needed to make space for new data.

For detailed instructions on how to configure Archivers in a system via the FxV browser interface, see the FxV Installation and Administration Guide.

Script ConfigurationThe settings in this section are applied to all Scripts defined in the system:

• Maximum Number of Uncaught Failures Per Script—defines the maximum number of times that a given script can fail (by throwing an exception not handled by the script) before the script is disabled.

• HTTP Connection Timeout—defines the maximum amount of time (in milliseconds) that a script requesting an HTTP connection waits before timing out.

• HTTP Socket Timeout—defines the maximum amount of time (in milliseconds) that a script using an HTTP connection waits for a response before timing out.

Other ConfigurationThe settings in this section allow certain aspects of the Hit Analysis process to be customized:

• Additional Parsed Response Types—allows you to control the response content types that are parsed. By default, when processing hits, the response content is only evaluated for hits whose response content type is set to text/html. Additional response types can be included by specifying in this field a comma-separated list of content types (for example: text/xml,text/plain).

These content types are utilized at capture time only. The primary use of this setting is to make the content of certain kinds of hits available to hit filters, for use


in match conditions or as a source in a custom field or metric update. The following cases may occur:

• Case 1: If the content type on a hit is text/html, FxV applies sensitive content rules, makes it available to hit filters, and performs keyword indexing on it.

• Case 2: If the content type on a hit is not text/html, but is in the list of content types specified in the Additional Parsed Response Types field, FxV does not apply sensitive content rules, but makes it available to hit filters, and performs keyword indexing on it.

• Case 3: If neither one of the conditions specified in Case 1 and Case 2 applies, FxV does not apply sensitive content rules, does not make it available to hit filters, and does not perform keyword indexing on it.

Note The content types are case sensitive.

Hit Analysis Configuration Change LogThis section allows you to view the changes made to the Hit Analysis Configuration and roll back to any of the previous configuration versions. The number of versions shown is configurable (Show Last 5, 25, 100, or 1000).

To explore the Hit Analysis Configuration versions:



2 Expand the Hit Analysis Configuration Change Log section.

The log information is displayed in table format.


Version Every time a change is made to anything on the Hit Analysis page, a new version of the Hit Analysis Configuration model is created. Version numbers are incremented sequentially.

User The name of the FxV user who modified the model.

Update Type The type of the change made: Create, Delete, Import, Rollback, or Update.

Configuring the Hit Analysis Process 129Hit Analysis Configuration Change Log


• To view the changes made on a specific version, click the View icon in that row.

The Hit Analysis Configuration Summary page for that version opens, showing everything that was in a previously-saved version of the Hit Analysis configuration.

Review the changes, then click OK.

• To import selected changes made on a specific version, click the View icon in that row.

The Hit Analysis Configuration Summary page for that version opens, showing everything that was in a previously-saved version of the Hit Analysis configuration.

a Review the changes, then click Import.

The screen changes to a mode that allows you to select the items that you want to import.

b Select the check box for each of the items you want to import, or select the Import check box to import all items in the list.

c Click Import Selected Items.

The selected items are now imported and the Hit Analysis Configuration Change Log is updated to reflect this operation.

• To roll back to any of the previous configuration versions, click the Rollback icon.

All changes made in the versions you are rolling back are lost. You need to confirm that you want to roll back that version, before proceeding with the

Object Type The type of object that was created, updated, or deleted (for example, Hit Filter, Custom Field, etc.).Note This column is blank if the Update Type is Rollback or Import.

Name The name of the object that was created, updated, or deleted.

Date/Time The time when the change took place.



operation. The Hit Analysis Configuration Change Log is updated to reflect the rollback operation.Note When you use the Rollback option, the entire Hit Analysis configuration is imported

(that is, you cannot choose individual pieces of the Hit Analysis configuration to be imported).

• To configure the number of versions shown in the Hit Analysis Configuration Change Log, select one of the available options from the Show Last list (5, 25, 100, or 1000).

The Hit Analysis Configuration Change Log is updated accordingly.

Hit Analysis Import/ Export ConfigurationThe Export Configuration and Import Configuration buttons on the Hit Analysis page allow you to perform the following operations, respectively:

• Export the Hit Analysis Configuration to a portable XML format. The resulting file may be later imported, to recreate the saved Hit Analysis Configuration.

• Import the Hit Analysis Configuration from an external XML file. This feature is useful to quickly augment or replace existing Hit Analysis Configuration settings.

Important The only things imported/exported by this FxV feature are the settings found on the Hit Analysis page. Users, user groups, resource groups, custom search screens, etc. are not part of this export/import.

To export the Hit Analysis Configuration of an FxV system:

1 In the FxV browser interface, on the menu bar, click Configure > Hit Analysis.


2 Click Export Configuration.

The File Download dialog box appears.

3 To save the Hit Analysis Configuration file, click Save.

Configuring the Hit Analysis Process 131Hit Analysis Import/ Export Configuration

4 Select a name and a destination directory for the Hit Analysis Configuration file, then click Save.

The file is now downloaded in the specified directory.

To import an external Hit Analysis Configuration into your FxV system:



2 Click Import Configuration.

The Import Configuration page appears.

3 Specify the XML Hit Analysis Configuration file to be imported and the path to its source directory, then click Load File.


The configuration is validated, and the Hit Analysis Configuration Summary page opens, displaying all Hit Analysis objects available for import.

4 By selecting one of the options available in the Import Mode list, you can choose how you want to handle the objects existing in your system when importing an old configuration:

• Drop current configuration; replace with selected items—nothing from the currently-active Hit Analysis Configuration is retained. The only possible exception are the Configuration Options (if you choose not to import the old settings, the current ones continue to be used).

• Keep current configuration; add/replace selected items only—the current configuration remains in place. Selected objects from the old configuration are added to the current configuration. If a selected object exists in both models, the current version is replaced.

5 Select the check box for each of the items you want to import, or select the Import check box to import all items in the list.

6 Click Import Selected Items.

The selected items are now imported into your system, and the Hit Analysis Configuration Change Log is updated to reflect this operation.

7

Hit Analysis Examples

This chapter provides several examples illustrating common uses of Hit Filters and Transactions Filters that FxV users can define via the FxV browser interface.


“Simple Data Extraction” Example.............................................................................................134“Event Occurrence Counting” Example .....................................................................................134“Session-Level Event Occurrence Counting” Example .............................................................135“Parsing HTML Content” Example.............................................................................................136“Parameterized Metrics” Example .............................................................................................137“Buy Tunnel” Example ...............................................................................................................138


“Simple Data Extraction” ExampleA very simple use of Hit Filters is to extract data from a hit and store it as a Custom Field, either on the hit itself, or on the session. By copying data to Custom Fields, you make that data available in search results, therefore, you make that data easier to search for.

This example shows you how to transfer the credit card type from a request field to a session custom field:

Name: Credit Card Type Extraction

Match Condition: Request Path = /submitcc.do

Custom Field:

Name: Card Type

Storage: Update Session, Show in Search Results

Value Source: Request Field card_type

Populated: Always

Value Assignment Mode: Append value on each match (if unique); Separator: /

Update If Blank: No

In this example, the value of the card_type field is appended to the Card Type custom field. This would result in a value such as Visa/MC if a customer were to make two different payments in the same session (using different card types).

“Event Occurrence Counting” ExampleWeb site owners are often interested in knowing how frequently certain types of events are occurring on their Web site. For example, they may be interested in counting the number of times that a certain error page is displayed. Using a simple Hit Filter with a Metric update, the frequency of any hit-level occurrence can easily be tracked.

This example shows you how to track the occurrence of a specific hit-level event:

Name: Error Page Counter

Match Condition: Request Path contains error

Hit Analysis Examples 135“Session-Level Event Occurrence Counting” Example

Metric:

Name: Error Pages

Increment By: Fixed Amount (1)

Incremented: Always

“Session-Level Event Occurrence Counting” ExampleA different technique is required to count the number of sessions in which some specific event occurs. If the event can potentially occur more than once per session, the technique described in the “Event Occurrence Counting” Example section is not appropriate, since each matching hit would increment the metric.

This example shows you how to modify the Hit Filter defined in the “Event Occurrence Counting” Example section to count the number of sessions in which at least one error page occurred. Starting from that Hit Filter, you must remove the metric update (since it increments the metric each time an error page occurs):

Name: Error Page Counter

Match Condition: Request Path contains error

You also have to define a simple Transaction Filter that starts a transaction as soon as an error page occurs:

Name: Sessions With Errors

Event Name/Label: Error/E

Allow Multiples: No

Transaction Entry State: Not Started

Event Condition: Hit matches Hit Filter Error Page Counter

Set Transaction Status: No Change

Start/Stop Actions: Start Transaction

Metric:

Name: Error Sessions

Increment By: Fixed Value (1)

Increment Policy: Always increment when transaction stops


Storage:

Store Transactions: Never

Store Custom Fields: Never

Store Events: Never

Commit Pending Metric Updates: Always

This type of transaction demonstrates the power of Transaction Filters to do more than just track a series of steps that make up a logical transaction. Transaction Filters are useful for many types of analysis that involves more than one hit per session.

Notice that in this example you have configured the Transaction Filter to not store transaction information. The only purpose of the Transaction Filter is to increment the Error Sessions metric. There is no need to store the transaction as well.

Also notice that the no “stop” condition is necessary for the transaction, which simply ends when the session ends. This is the expected behavior, since your goal is to count only one error page per session. This is specified by setting the Allow Multiples option to No for the one event in this filter.

Note Setting the Allow Multiples option to a value other than No results in each Hit Filter match incrementing the metric.

“Parsing HTML Content” ExampleIn situations where the data necessary to evaluate a condition can only be found in the response content sent from the Web server, the FxV hit analysis process provides mechanisms to allow this to be done. There are a few key factors to keep in mind when defining Hit Filters that reference the response content:

• Parsing response content is expensive. While the hit analysis process is designed to handle this sort of processing, it should not be done excessively. A hit analysis configuration with a large number of Hit Filters parsing the response content of every page could reduce the capture capacity of the system.

• When parsing the response content is necessary, keep the pages being parsed to a minimum. In many cases, this can be accomplished by adding additional match conditions to reduce the number of pages that have to be parsed.

Suppose the system being monitored has been known to return an error page with the following response content:

Hit Analysis Examples 137“Parameterized Metrics” Example

SQL Exception

Database error 12345 has occurred. Please call support.

A simple approach to tracking these sorts of errors might be defined as follows:

Hit Filter Name: SQL Exceptions

Match Conditions: Response Content Contains “SQL Exception”

While this filter would work, it has the unfortunate effect of causing every hit to be parsed for the string “SQL Exception”. Whenever possible, an additional condition should be added to reduce the number of pages being scanned. In many cases, the URL or Request Path can be used to reduce the set of pages potentially containing the text being searched for. In this example, you can use the HTTP Response Code.

Hit Filter Name: SQL Exceptions

Match Conditions: Response Code = 500 AND Response Content Contains “SQL Exception”

Important Match conditions are evaluated in the order that they appear. Always make sure that any evaluation of the response content appears last in your list of match conditions.

“Parameterized Metrics” ExampleIn addition to updating simple metrics such as “Exception Count”, Hit Filters and Transaction Filters can also be used to update parameterized metrics. A parameterized metric is a metric whose name varies based on some piece of data from the captured data.

This example presents one of the default Hit Filters included with FxV:

Hit Filter Name: Default HTTP Error Codes

Match Conditions: Response Code = 400 OR Response Code > 401

Error Conditions: Every hit matching the Match Conditions will be marked with the ERROR status.

Metric:

Name: HTTP Response Code $0

$0 Source: Response Code

Increment By: Fixed Amount (1)


Incremented: Conditional: when the status recorded by this Hit Filter is Error

The key feature of this Hit Filter is the use of $0 in the name of the metric. This is a special token that the system replaces with the value of the “$0 Source” in order to create multiple unique metrics. For example, if a hit is captured with a response code of 404, a metric called Http Response Code 404 would be incremented. Note that the “$0 Source” can be just about anything. This example uses the Response Code here, but it could as well use the value of a request field, some other hit detail such as Country, or even some piece of data extracted from the page content.

Note When using parameterized metrics, ensure that “$0 Source” has a reasonably small number of possible values. Using something like the session ID or some other highly dynamic piece of data may produce hundreds or thousands of distinct values, resulting in an unmanageable number of distinct metric names.

“Buy Tunnel” ExampleThis section documents a typical Transaction Filter and its component Hit Filters used to track a common “buy tunnel” operation (that is, the process an end user goes through when purchasing a product online). The intent of this example is to show how Hit Filters and Transaction Filters work together to capture valuable information about a customer’s online experience. During this example, note how Custom Fields are used as a tool to transfer hit-level data from Hit Filters to Transaction Filters (and from the session to the transaction).

This example includes eight Hit Filters, one Transaction Filter, and the definition of a Global Stop Session Event. For details, see these topics:

• ““Buy Tunnel”: Hit and Transaction Filters Overview” on page 139

• “Transaction Event: Add to Cart” on page 139

• “Transaction Event: Checkout” on page 140

• “Transaction Event: Shipping” on page 141

• “Transaction Event: Payment” on page 141

• “Transaction Event: Confirmation” on page 143

• “Transaction Event: Cart Abandoned” on page 145

• “Hiding Sensitive Data” on page 147

Hit Analysis Examples 139“Buy Tunnel” Example

“Buy Tunnel”: Hit and Transaction Filters OverviewStart by setting up the following Hit Filters:

• Login—populates a session custom field containing the user’s email address when a user logs in to the system.

• Add to Cart—triggered when a product is added to the shopping cart.

• Checkout—triggered when a user goes to check out.

• Shipping Selected—triggered when a user chooses a shipping preference.

• Confirmation—triggered when a user is taken to the confirmation page. It updates a transient custom field (to be used by the Transaction Filter) containing the final shopping cart value.

• Transaction Complete—triggered when a user finalizes a transaction.

• View Cart—triggered when the user’s shopping cart is displayed. It updates a session custom field that contains the total value of the items in the cart.

• Logoff—triggered when a user logs out the system.

The “Buy Tunnel” Transaction Filter relies heavily on many of these Hit Filters. The remainder of this example walks you through the events that make up the Transaction Filter, providing more detail about each of the Hit Filters as they are encountered.

Transaction Event: Add to CartStart tracking a transaction as soon as a user adds something to the shopping cart, although you can start earlier (for example, as soon as the user starts browsing the product catalog).

The basic setup for this event is the following:

Event Name/Label: Add to Cart /A

Allow Multiples: No

Transaction Entry State: Not Started

Every time you define an event that starts a transaction, the entry state for the event must be Not Started (that is because you cannot start a transaction if it is already started).

The match condition for this event is the following:

Hit matches Hit Filter Add to Cart (with status Any)


In order to understand this match condition, you must look at the Add to Cart Hit Filter. This Hit Filter is very simple. Its sole purpose is to trigger the beginning of a transaction. Its only “action” is to flag a hit as having matched. The match conditions for this Hit Filter are:

Request Path Ends with /product_info.php AND Request Field action = add_product

These match conditions would successfully match a URL such as:

http://www.myonlinestore.com/catalog/product_info.php?action=add_product&products_id=26

Now that you know what it takes to trigger the beginning of this transaction, you can look at the actions you need to perform in the first event. Since the point of this event is to mark the beginning of the transaction, you must first configure the system accordingly. In the Transaction Filter definition page, set:

Start/Stop Actions: Start Transaction

In addition to starting the transaction, you can also keep count of the total number of transactions that have been started. You can do this by incrementing a metric:

Name: Entered Tunnel



Transaction Event: CheckoutThis event is a simple progress check. By adding steps throughout the transaction, you can gain insight into the process (for example, at what point in the process users who do not complete their transactions are leaving the “buy tunnel”). The purpose of this step is to track users who click the Check Out button.

The setup for this event is the following:

Event Name/Label: Checkout / B

Allow Multiples: No

Transaction Entry State: Active

The match condition for this event is the following:

Hit matches Hit Filter Checkout (with status Any)


The Checkout Hit Filter has the following match conditions:

Request Path Ends with /checkout_shipping.php AND Response Code = 200

In this case, some occurrences of this filter return a 302 (redirect) response code. These hits are not interesting for the purpose of this example, so you must specify a specific HTTP Response Code, to narrow the match conditions.

The Checkout Transaction Event has no impact on the status or state (Not Started/Active/ Stopped) of the transaction. The only action defined for this event is another metric increment, allowing you to track the number of users who continue on to this step:

Name: Checkout Clicked



Transaction Event: ShippingThis event is similar to the Checkout Transaction Event, but with a different path and a different metric name (Shipping Selected). For more information, see “Transaction Event: Checkout” on page 140.

Transaction Event: PaymentThis event occurs when the user reaches the Order Confirmation (payment) page. The match conditions for this event are similar to those in the previous steps (see “Transaction Event: Checkout” on page 140).

However, in this case, the Hit Filter used to trigger this event is going to extract the final value of the shopping cart (which is, the amount being paid) into a custom field. The definition of this Hit Filter custom field is the following:

Name: Final Cart Value

Storage: Update Session, Available only for Transaction Filters; Store Values as Numbers

Value Source: Response Content >Sub-Total:<.*?\$([0-9,]+\.\d\d)

Populated: Always

Value Assignment Mode: Reset value on each match

Update If Blank: No


The key part of this custom field definition is the value source. The page we are dealing with contains a table with the following text:

Sub-Total: $69.99

Flat Rate (Best Way): $5.00

Total: $74.99

Your goal is to extract now the Sub-Total (the cost of the items, excluding shipping). In order to identify how to extract this value from the response content, you need to look at the HTML source for the page. This particular table is rendered by the following HTML:

<table border=”0” cellspacing=”0” cellpadding=”2”>

<tr>

<td align=”right” class=”main”>Sub-Total:</td>

<td align=”right” class=”main”>$69.99</td>

</tr>

<tr>

<td align=”right” class=”main”>Flat Rate (Best Way):</td>

<td align=”right” class=”main”>$5.00</td>

</tr>

<tr>

<td align=”right” class=”main”>Total:</td>

<td align=”right” class=”main”>$74.99</td>

</tr>

</table>

Since there are multiple amounts on the page, you need to take care to extract the one correct one. The following expression successfully returns 69.99:

>Sub-Total:<.*?\$([0-9,]+\.\d\d)

This expression means:

“Search for the text ‘>Sub-Total:<’ followed by any number of characters, until you find a ‘$’ followed by one or more digits/commas (the comma allows for values such as $1,234.56), followed by a decimal point, followed by two more digits.”


Everything after the dollar sign is wrapped in parenthesis, indicating the portion of the match to be returned as the value of the custom field.

Important Notice the use of the “?” in this regular expression. Placing the “?” after the “.*” means “match any character, but do not be greedy”.

If you leave off the “?”, the regular expression becomes:

>Sub-Total:<.*\$([0-9,]+\.\d\d)

and it returns the value 74.99, since the “.*” would match all characters until the last dollar sign on the page.

These sorts of subtle details demonstrate the importance of using the embedded regular expression tester to validate your expressions against real data.

Transaction Event: ConfirmationThe final step in the “buy tunnel” (for successful transactions) occurs when the user submits the payment information and is redirected to the “Checkout Success” page. This event is triggered by a very simple Hit Filter, one that matches on a specific path, and otherwise does nothing. But in this case, the Transaction Event also performs several important actions when this Hit Filter matches.

The first action the event must perform is to mark the transaction complete:

Set Transaction Status: No Change

Start/Stop Actions: Stop Transaction

Next, you are going to set two transaction custom fields based on data collected during the processing of the transaction. One of these custom fields, Email, comes from a Hit Filter that was not discussed yet. At some point during the user’s session, users were required to log in with their email address. This step is not tracked as part of the transaction, but a Hit Filter (Login) is set up to extract the user’s email address and to add it to the session.

The match conditions for this event are the following:

Request Path Ends with /create_account.php OR Request Path Ends with /login.php

These conditions cover the case where an existing user logs in to the system, as well as the case where a new user creates an account. Since your Web application uses the same


request field for the email address on both pages, you are able to use a single Hit Filter to handle both cases.

This Hit Filter also defines a custom field:

Name: Email

Storage: Update Session, First Column in Search Results; Store Values as Text

Value Source: Request Field email_address

Populated: Always

Value Assignment Mode: Reset value on each match

Update If Blank: No

This custom field becomes part of the session data for this user and appears as the first column in any session search results screen.

Getting back to the final Transaction Event (Confirmation), you can use this custom field value to store the user’s email address with the transaction as well. This way, the value can also be displayed in transaction search results.

The custom field definition in the event is the following:

Name: Email (The name is the same, but it is a different custom field, since it is in transaction scope here.)

Storage: Update Transaction, First Column in Search Results

Value Source: Session Field—Email

Value Assignment Mode: Reset value each time this event occurs

Update If Blank: No

This technique of copying a value from a session custom field to a transaction custom field is very common. It allows hit-level data to be transferred to the transaction (which has no direct access to hit data). In fact, you can use the same technique for one more custom field:

Name: Total Sale Amount

Storage: Update Transaction, Show in Search Results

Value Source: Session Field—Final Cart Value

Value Assignment Mode: Reset value each time this event occurs

Update If Blank: No


Recall that in the payment step (see “Transaction Event: Payment” on page 141), you extracted the final cart value from the page content and stored it in a transient custom field (that is, the value was associated with the session for the specific purpose of moving it over to the transaction at this point; the value is not stored in the database at the session level).

Finally, you can now also use this event to update two more metrics. The first metric is similar to the other counters incremented in previous steps (see “Transaction Event: Add to Cart” on page 139 and “Transaction Event: Checkout” on page 140):

Name: Completed Transactions



The second metric uses a different “Increment By” strategy, to track the total value of all completed transactions:

Name: Total Value of Completed Transactions

Increment By: Session Custom Field Final Cart Value


Note You use here the same session custom field used to set the Total Sale Amount transaction custom field to increment a metric by the value of the item(s) purchased.

Transaction Event: Cart AbandonedThis event is very different from the other events that make up the “Buy Tunnel” Transaction Filter definition. The purpose of this event is to track transactions that are not successfully completed, in other words, transactions that match the start event (Add to Cart), but do not match the stop event (Confirmation).

The first important difference in the definition of this event is the entry state:

Transaction Entry State: Stopped

This means that this event can only take place after the transaction has stopped. In order for this condition to make sense, you need a corresponding match condition for the event: Transaction stopped due to global stop condition. This special match condition fires when one of the following occurs:

• The Global Stop Session Event occurs


• The Global Stop Transaction Event occurs

• The session times out (based on the session timeout specified in the Session Configuration section of the Hit Analysis screen)

In this example, you are interested in either the Global Stop Session Event occurrence or a session timeout. Either of these events would indicate that the user never completed the purchase initiated. To better understand what the Global Stop Session Event means, you need to look at another Hit Filter (Logoff). All that this Hit Filter does is to mark hits when they match its simple match condition:

Request Path Ends with logoff.php

The Default Stop Session Event is an event like any other Transaction Event, except that it is not contained within a Transaction Filter. In our case, it is defined with a very simple match condition:

Hit matches Hit Filter Logoff

The only purpose of this Default Stop Session Event is to stop the active session (and therefore any active transactions) when the Logoff Hit Filter matches. The entry state (N/A) and default action (Stop Session) are not editable on the Default Stop Session Event.

Getting back to the Cart Abandoned event, you can now see that its match condition of Transaction stopped due to global stop condition is triggered as a result of a session timeout or a user logout.

The only thing left now is to look at the Custom Fields and Metrics set by this event. Before you look at these values, look at another Hit Filter, called View Cart.

The View Cart Hit Filter is not explicitly referenced by the “Buy Tunnel” Transaction Filter. However, it does set a session Custom Field that you can use to update a Custom Field or a Metric in the Cart Abandoned event.

When shoppers add items to their cart, they are taken to the /shopping_cart.php page. This page shows the current contents of their cart, including the total cost. The View Cart Hit Filter is triggered by hits matching this request path. Much like the Hit Filter used by the Payment event, it extracts the current value of the shopping cart using a regular expression against the response content, and stores it in a session Custom Field. This field always contains the current value of the user’s shopping cart (whether they are still in the middle of the transaction, they completed it, or they abandoned it). This is the Custom Field you can use to update the transaction in the Cart Abandoned event.

The Custom Fields set by the Cart Abandoned event are the following:


• Lost Sale Amount—this field is set to the value of a session Custom Field called Cart Value. The intent is to capture the value of the abandoned shopping cart.

• Email—as you did on successful completion of a transaction, you can transfer the value of the session Custom Field called Email into a transaction Custom Field, so that both sessions and transactions can be identified by the user’s email address.

The Metrics set by the Cart Abandoned event are:

• Abandoned Shopping Carts—a simple counter, incremented by “1” each time the event occurs.

• Total Value of Abandoned Shopping Carts—incremented by the value of the Cart Value session custom field.

Hiding Sensitive DataThe final item of note in this example is a Sensitive Hit Detail definition used to prevent FxV users from viewing credit cart numbers entered by users of the Web site. Setting up a Sensitive Hit Detail is relatively simple once you know the name of the request field used to transfer the credit card number to the Web server.

In this example, you can navigate to the page on which the credit card number is entered and view the HTML source for the page. Within the HTML, you would find:

<td class=”main”>Credit Card Number:</td>

<td width=”10”><img src=”images/pixel_trans.gif” border=”0” alt=”” width=”10” height=”1”></td>

<td class=”main”><input type=”text” name=”cc_number_nh-dns”></td>

This HTML tells you that the name of the credit card request field is cc_number_nh-dns, so the Sensitive Hit Detail should be defined as follows:

Hit Detail Type: Request Field

Matching Logic: =

Hit Detail Name: cc_number_nh-dns

Replacement Text: *****HIDDEN*****

Always Sensitive: Yes


Once this Sensitive Hit Detail is in place, anyone viewing captured sessions (including sessions captured before the Sensitive Hit Detail was in place) would only see *****HIDDEN***** in place of the credit card number.

8

Configuring the Analysis Repository

This chapter presents the operations an FxV user can perform to configure a secondary database for long-term storage of session and transaction data captured by FxV. This database is configured via the FxV user interface. The stored data can be accessed via direct SQL queries or through the Foglight.


Overview....................................................................................................................................150Architecture ...............................................................................................................................151Configuration .............................................................................................................................154Data Transfer.............................................................................................................................160Access .......................................................................................................................................161


OverviewThe FxV Analysis Repository is a secondary database that offers you long-term storage of session and transaction data captured by FxV. This add-on feature eliminates the primary capture database’s restrictions, that is, session and transaction search screens returning a small number of sessions/transactions, limited FxV capture capacity, and inability to write report-style queries against the FxV data.

The Analysis Repository provides the following key features:

• Longer term storage of session and transaction data.

• Configurable database structures.

• Simple MySQL database format, accessible:

• From within Foglight, via a Foglight data source (as part of the Cartridge for Foglight Experience Viewer)

• Via standard SQL query tools, such as Quest’s Toad for MySQL• Via third-party reporting tools

The Analysis Repository is designed to run on any FxV appliance. For single-appliance installations with minimal traffic (or for POC installations), the Analysis Repository can co-exist with the FxV Server and Archiver components, sharing space with the primary capture database.

Sites with multiple Archivers require a Storage appliance in order to consolidate the session/transaction data into a single database. The Analysis Repository makes use of the FxV appliance’s local disk, so the Storage appliance does not necessarily need to have an attached SAN device. Users with a multi-Archiver installation may enable the Analysis Repository without adding a Storage appliance to their system. However, in this configuration each Archiver appliance has its own Analysis Repository containing only the sessions/transactions captured by that Archiver. Therefore, in order to obtain a complete view of the Analysis Repository, users would have to perform identical queries against multiple databases and merge the results.

Important The addition of a Storage appliance is strongly recommended for FxV systems with more than one Archiver.

The Analysis Repository is disabled by default. Leaving this feature disabled does not impact the performance of existing FxV systems. Users that plan to enable this feature must first ensure that their system meets the installation and configuration requirements

Configuring the Analysis Repository 151Architecture

(for details, see the FxV Installation and Administration Guide), then configure the database structure (for details, see “Configuration” on page 154).

ArchitectureThe structure of the Analysis Repository is designed to be very simple. It provides two types of tables:

• A single session table—contains one row for each captured session, indexed by the sessionID, startTime, and lastHitTime.

• One transaction table for each Transaction Filter—the tables contain one row for each captured transaction. The tables are indexed by the sessionID, startTime, and lastHitTime.

The columns in these tables can contain session/transaction details (location information, hit counts, etc.), custom field values, metric update values, and Transaction Event counts. The specific columns defined for each table are user-configurable.

The Session TableThe session table stores information about the sessions captured by FxV. Each row in this table corresponds to a single session. The columns in this database are user-configurable with the exception of the following standard columns:

• sessionID—the session identifier for each session.

• startTime—the time at which each session started.

• lastHitTime—the time at which each session ended (this is the timestamp of the last hit in the session, not the time at which the session timed out).

The user-configurable columns in the session table correspond to any of the following types of data:

• Session Details—pre-defined details related to the session:Note The timing values of type DOUBLE are expressed in seconds (and fractions of a

second).

• Browser Type (TEXT)• City (TEXT)• Client IP (TEXT)• Client Time (DOUBLE)


• Country (TEXT)• End to End Time (DOUBLE)• Errors (INTEGER)• HTML Errors (INTEGER)• HTML Hits (INTEGER)• HTML Warnings (INTEGER)• ISP (TEXT)• Processing Time (DOUBLE)• Region (TEXT)• Stop Reason (TEXT)• Stored Hits (INTEGER)• Subnet (TEXT)• Time of First Error (DATETIME)• Time of Last Error (DATETIME)• Total Hits (INTEGER)• Total Time (DOUBLE)• Transaction Count (INTEGER)• Transaction Errors (INTEGER)• Transaction Warnings (INTEGER)• Username (TEXT)• Warnings (INTEGER)

• Hit Filter Matches (INTEGER)—one column for each Hit Filter in the system. The values indicate the number of hits the Hit Filter matched during the session.

• Custom Fields (TEXT or DOUBLE, depending on the Custom Field type)—one column for each session-scoped Custom Field.

• Metric Deltas (INTEGER)—one column for each metric. The values indicate changes to the metric values made within the scope of the session.

Users can choose which of these columns are included in the database structure, on a column-by-column basis. The column names are user-configurable; each type of column has a fixed prefix (for example, SDT_ for Session Details, HFM_ for Hit Filter Matches, etc.), but the core of the column name is specified by the user or populated with the default names.

Configuring the Analysis Repository 153Architecture

The Transaction TablesIn addition to the session table, the Analysis Repository may optionally contain an additional table for each Transaction Filter defined in the system. The rows in these tables correspond to transactions as defined by the Transaction Filter.

Each transaction table contains the following standard columns:

• sessionID—the session identifier for each transaction.

• startTime—the time at which each transaction started.

• lastHitTime—the time at which each transaction ended (this is the timestamp of the last hit in the transaction, not the time at which the transaction timed out).

The user-configurable columns in the transaction tables correspond to any of the following types of data:

• Transaction Details—pre-defined details related to the transaction:Note The timing values of type DOUBLE are expressed in seconds (and fractions of a

second).

• Client Time (DOUBLE)• End to End Time (DOUBLE)• Errors (INTEGER)• HTML Errors (INTEGER)• HTML Hits (INTEGER)• HTML Warnings (INTEGER)• Processing Time (DOUBLE)• Status (TEXT)• Stop Reason (TEXT)• Stored Hits (INTEGER)• Time of First Error (DATETIME)• Time of Last Error (DATETIME)• Total Hits (INTEGER)• Total Time (DOUBLE)• Warnings (INTEGER)

• Transaction Events (INTEGER)—one column for each Transaction Event defined by the Transaction Filter. The values indicate the number of times the event occurred within the scope of the transaction.


• Custom Fields (INTEGER or TEXT, depending on the Custom Field type)—one column for each transaction-scoped Custom Field.

• Metric Deltas (INTEGER)—one column for each metric. The values indicate changes to the metric values made within the scope of the transaction (only changes made by the Transaction Filter are included).

Users have complete control over which Transaction Filters have corresponding transaction-specific tables in the Analysis Repository and over the types of columns each of these transaction-specific tables contains. However, the Transaction Details columns are defined only once, and are applied to all transaction-specific tables. For example, if you choose to include the Client Time transaction detail, it is added as a column to every transaction table you define.

Users can choose which of these columns are included in the database structure, on a column-by-column basis. The column names are user-configurable; each type of column has a fixed prefix (for example, TDT_ for Transaction Details, TEV_ for Transaction Events, etc.), but the core of the column name is specified by the user or populated with the default names.

The names of the transaction-specific tables are user-configurable; they contain a common prefix (txn_) and a core name specified by the user.

ConfigurationUsers can configure the Analysis Repository structure via the FxV browser interface. The configuration process contains two major phases:

1 “Analysis Repository Initial Setup” on page 154

2 “Analysis Repository Detailed Configuration” on page 159

Analysis Repository Initial SetupThe initial setup phase consists of allocating a fixed-size space for the Analysis Repository in the FxV appliance, that is, defining the maximum number of elements that the database can contain. This phase is crucial in configuring the Analysis Repository and must be performed correctly. Once you have allocated the space for this

Configuring the Analysis Repository 155Configuration

database, any changes to the column counts result in all Analysis Repository data being lost.

Note Changing the database size and/or the table retention limits does not result in losing the previously captured data.

To initialize the Analysis Repository:



2 Expand the Analysis Repository menu options.Note By default, the Analysis Repository is disabled.

3 Enable the Analysis Repository by setting the Maximum Analysis Repository Size to a value other than zero.Important The Maximum Analysis Repository Size defines the maximum size for the

Analysis Repository. Depending on the current size of the repository and the current growth rate (how quickly the size of the repository is increasing), decreasing the maximum size may result in the immediate termination of any queries actively running against the Analysis Repository.

4 Define the maximum number of hours of data that an individual table in the Analysis Repository can retain, by setting the Maximum Table Retention field. Distinct tables are created for each segment in the primary capture database, so this value must be greater than the maximum segment span, as defined in the Hit Analysis Configuration Options page (see “Archiver Configuration” on page 126).Note The default value of zero sets no time limit for the Analysis Repository; in this case,

data is deleted only when the size of the Analysis Repository nears its limit.

5 Define the maximum amount of time (in minutes) that an individual query against the Analysis Repository is allowed to run, by setting the Maximum User Query Duration field. Queries running longer than this limit are terminated.

6 Specify the time at which the Analysis Repository maintenance is performed each day, by setting the Maintenance Time field. At this time, any queries running against the Analysis Repository are terminated in order to allow routine database maintenance activities to be performed.

7 Define the maximum number of columns of each data type that can be configured to store data in the session table, as well as the maximum number of these columns that can be indexed for improved search performance, by typing in the


values in the provided fields. For details about this data type, see “Session Table Data Types” on page 156.Tip Optimize the table size while also allowing for future growth.

8 Define the maximum number of columns of each data type that can be configured to store data in each transaction table, as well as the maximum number of these columns that can be indexed for improved search performance, by typing in the values in the provided fields. For details about this data type, see “Transaction Table Data Type” on page 158.Tip Optimize the table size while also allowing for future growth. All transaction tables will

have the same limits.

9 Click Save Changes.

The Analysis Repository is now enabled and its initial setup phase completed.

Session Table Data TypesA key point when defining the maximum number of columns of each data type is that the numbers should be set to allow for future database growth. The session table has a limit of 1,000 total columns. Each transaction table can also have up to 1,000 columns. Unused columns take up space, so while it is recommended to allocate more columns than are currently needed, it is also recommended to maintain a low count of unused columns.

Example

Consider the number of TEXT columns in the session table. A maximum of nine pre-defined session details can be stored in TEXT columns, if they are turned on. In addition, one column is needed for each non-numeric session custom field.

Consider that users want to capture all nine session details and 10 non-numeric custom fields. At a minimum, they need to configure 19 TEXT columns for the session table. However, users are strongly encouraged to set the value much higher than 19, to allow for future database growth. Each additional non-numeric custom field (beyond the original 10) that they want to include will require a new TEXT column. A value such as 30 might be appropriate in this case.


The following table illustrates the type of columns that can be defined in the session table.

Column Type Column Description

TEXT Defines the maximum number of database columns that can be configured to store textual data in the session table, as well as the maximum number of these columns that can be indexed for improved search performance.Data stored in TEXT columns includes nine session details (Browser Type, City, Client IP, Country, ISP, Region, Stop Reason, Subnet, and Username) and session custom fields configured to store values as text.

INTEGER Defines the maximum number of database columns that can be configured to store numeric (integer) data in the session table, as well as the maximum number of these columns that can be indexed for improved search performance.Data stored in INTEGER columns includes nine session details (Errors, HTML Errors, HTML Hits, HTML Warnings, Stored Hits, Total Hits, Transaction Count, Transaction Errors, Transaction Warnings, and Warnings), and metric deltas.

DOUBLE Defines the maximum number of database columns that can be configured to store decimal (double) data in the session table, as well as the maximum number of these columns that can be indexed for improved search performance.Data stored in DOUBLE columns includes four session details (Client Time, End to End Time, Processing Time, and Total Time) and session custom fields configured to store values as numbers.

DATETIME Defines the maximum number of database columns that can be configured to store timestamp data in the session table, as well as the maximum number of these columns that can be indexed for improved search performance.Data stored in DATETIME columns includes two session details (Time of First Error and Time of Last Error).Note No user-configurable data is stored in DATETIME columns. The only

reason to allocate more than two columns of this type would be to prepare for potential product changes in the future (for example, new session details).


In addition to defining the number of columns of each data type, users can also define the maximum number of these columns that can be indexed. Indexed columns should be used for data that is expected to commonly appear in search criteria. The session table has a limit of 60 indices.

Example

If users intend to do a lot of queries based on location, the SDT_CITY, SDT_REGION, and SDT_COUNTRY columns would be good candidates for indices.

Note Configuring the number of indexed columns of each type also implicitly sets the maximum number of non-indexed columns of each type. For example, if you allocate 50 INTEGER columns and 10 of them indexed, the maximum number of non-indexed INTEGER columns allowed is 40.

Transaction Table Data TypeA key point when defining the column/index counts for transaction tables is that the counts apply to each table, not all tables in aggregate.

Example

If you define 10 TEXT columns, you can have 10 TEXT columns in each Transaction-specific table, and they can be different columns.

The following table illustrates the type of columns that can be defined in each transaction table.


TEXT Defines the maximum number of database columns that can be configured to store textual data in each transaction-specific table, as well as the maximum number of these columns that can be indexed for improved search performance.Data stored in TEXT columns includes two transaction details (Status and Stop Reason) and transaction custom fields configured to store values as text.


Analysis Repository Detailed ConfigurationThe repository’s detailed configuration phase consists of specifying the elements to be captured in this database. It assumes that the initial configuration phase is already completed (for details, see “Analysis Repository Initial Setup” on page 154).

To define the Analysis Repository structure:

1 In the Analysis Repository section of the Hit Analysis page, click Configure Database Schema.

INTEGER Defines the maximum number of database columns that can be configured to store numeric (integer) data in each transaction-specific table, as well as the maximum number of these columns that can be indexed for improved search performance.Data stored in INTEGER columns includes seven transaction details (Errors, HTML Errors, HTML Hits, HTML Warnings, Stored Hits, Total Hits, and Warnings), transaction event counts, and metric deltas.

DOUBLE Defines the maximum number of database columns that can be configured to store decimal (double) data in each transaction-specific table, as well as the maximum number of these columns that can be indexed for improved search performance.Data stored in DOUBLE columns includes four transaction details (Client Time, End to End Time, Processing Time, and Total Time) and transaction custom fields configured to store values as numbers.

DATETIME Defines the maximum number of database columns that can be configured to store timestamp data in each transaction-specific table, as well as the maximum number of these columns that can be indexed for improved search performance.Data stored in DATETIME columns includes two transaction details (Time of First Error and Time of Last Error).Note No user-configurable data is stored in DATETIME columns. The only

reason to allocate more than two columns of this type would be to prepare for potential product changes in the future (for example, new transaction details).



The Analysis Repository Configuration page appears, listing all the possible columns and tables (for Transaction Filters) that can be added to the database. Initially, all of the column names are blank, which means that no data is stored, other than the three pre-defined standard columns (see “Architecture” on page 151). Also, all transaction table names are initially blank, which means that no transaction filter-specific tables are created.

2 In the provided Column Name fields, type in names for all columns you want to include in the session and transaction-specific tables.

Alternatively, click the Set to Default icons beside any of these fields to populate them with their default names.Note Click the Clear icon beside each of these fields to clear the column name.

Tip The Set to Default and Clear icons at the top of each section allow you to auto-populate or clear an entire section at once.

3 For improved search performance, configure certain columns to be indexed, by selecting Indexed from the list associated with them.

4 Click OK.

The Analysis Repository structure is now defined.Note Columns can be added, removed, or renamed at any time, but the changes only

apply to data loaded into the Analysis Repository after the changes have been made. For example, if you have three days of data in your Analysis Repository and realize that you want to add two columns, you can do that. The three days of data continues to be in the database, but their rows are not updated to populate the new columns. These columns contain null values for all previously-loaded data.

The configuration of the Analysis Repository is part of the Hit Analysis Configuration, which can be exported as XML and imported back into the system. This contains all settings related to the Analysis Repository, including the individual column mappings. For more information, see “Hit Analysis Import/ Export Configuration” on page 130.

Data TransferThe Analysis Repository is not populated in real time like the primary capture database. Instead, database segments are transferred to the Analysis Repository when they close. The duration of each segment is defined on the Archivers Configuration section in the Hit Analysis Configuration Options page (see “Archiver Configuration” on page 126). It can be set to as little as one hour, in which case completed sessions would appear in the Analysis Repository no later than one hour after completion.

Configuring the Analysis Repository 161Access

AccessThe Analysis Repository is a MySQL database, accessible:

• From within Foglight, via a Foglight data source (as part of the Cartridge for Foglight Experience Viewer). For more information, see “Accessing the Analysis Repository from Foglight” on page 161.

• Via standard SQL query tools, such as Quest’s Toad for MySQL. For more information, see “Accessing the Analysis Repository from Toad for MySQL” on page 161.

• Via third-party reporting tools.

Access to the Analysis Repository is read-only. You cannot delete rows from the Analysis Repository tables. Data in the Analysis Repository is removed, as needed, based on the configured Maximum Analysis Repository Size and Maximum Table Retention. When data is removed from the Analysis Repository, it is removed according to database segment boundaries.

Accessing the Analysis Repository from FoglightData in the Analysis Repository is accessible from within Foglight, via a Foglight data source (as part of the Cartridge for Foglight Experience Viewer). For details, see the Cartridge for Foglight Experience Viewer documentation.

Accessing the Analysis Repository from Toad for MySQLData in the Analysis Repository is also accessible via standard SQL query tools (such as Quest’s Toad for MySQL), by using the settings specified in the following table.

Setting Value/ Description

Host The FxV appliance’s host/IP address.For multiple-appliance installations, the Analysis Repository is stored on the Storage appliance.

Port 3306

Account Name fxv


Quest’s Toad for MySQL is freeware. You can download it from Quest’s Web site (http://www.quest.com/toad-for-mysql) and install it on your local machine. For more information about accessing the FxV Analysis Repository via Toad for MySQL, see the Quest’s Toad for MySQL documentation set.

To access an Analysis Repository using Toad for MySQL:

1 Open Toad for MySQL.Note This procedure assumes that you have already downloaded and installed Toad for

MySQL on your local machine.

2 Create a connection to the database by clicking File > New > Connection.

3 In the Create New Connection dialog box define the fields as follows:

• Connection type—select TCP from the list.• Host—use the IP address of the FxV appliance.• User—type in fxv.• Password—type in fxv.• Database—type in fxv.• Port—select 3306 from the list.• Name—type in a name for the database to display (for example, FxV

Analysis Repository).

Password fxvThis is the default value. To change the password, see “Appliance Maintenance” on page 168.

Database Name fxv

Table Names session and txn_* (as defined when configuring the database)Note The session table can be joined with the txn_ tables, but this

process is quite complex. Sessions are identified by their sessionID, startTime, and lastHitTime (this is because you can have multiple sessions with the same session ID). To get the transactions for a specific session, you have to use a WHERE clause. For example:WHERE session.sessionID = txn_xxx.sessionID AND session.startTime <= txn_xxx.startTime AND session.lastHitTime >= txn_xxx.lastHitTime

Setting Value/ Description

http://www.quest.com/toad-for-mysql

http://www.quest.com/toad-for-mysql

Configuring the Analysis Repository 163Access

When the connection is established, the name of the database appears in the Connection Manager pane.

4 Display the information stored in the FxV Analysis Repository by selecting its name in the Connection Manager pane, and then clicking Tools > Database Explorer.

The FxV Analysis Repository tab appears in the Database Explorer pane.

5 Select the Views tab.

The list of session and transaction tables available in this Analysis Repository appears.

6 Explore the tables by clicking their names in the left pane.

7 View the data stored in each of these tables by clicking the Data tab in the right pane.


9

Performing Advanced FxV Administration

This chapter presents the operations that a user with full administrative privileges can perform in order to configure and administer the system via the FxV browser interface.


Archivers....................................................................................................................................166Collectors...................................................................................................................................167Superuser Tasks........................................................................................................................168


ArchiversArchivers capture hits collected by Collectors, then apply Hit Filters and Transaction Filters, and update the capture database for later search and replay. All FxV appliances have a pre-installed default Archiver.

To view the Archivers already configured in your system:

• In the FxV browser interface, on the menu bar, click Configure > Archivers.

The Archivers page appears, showing all Archivers in the system. You can view, edit, or remove the existing components, or add new ones. In order to add an Archiver, the Archiver must be running and must be accessible from the Server.

Depending on your system’s configuration, an administrator may be required to do the following operations with Archivers:

• Remove the default Archiver from FxV appliances configured as Server, Server/Storage, or Storage.

• Register a remote Archiver/Storage component with the FxV Server (which informs the Server where to perform searches).Note Remote Archiver/Storage refers to those components configured on appliances

other than the Server appliance. One or multiple Archiver and Storage components can be registered remotely with a Server.

• Register a local Archiver/Storage component with the FxV Server.Note Each Storage component must be registered with the local Server before it can

begin processing data.

• Test the ability of the FxV Server to communicate with all currently configured Archivers (by clicking Test Connections).

For detailed instructions on how to configure Archivers in a system via the FxV browser interface, see the FxV Installation and Administration Guide.

Performing Advanced FxV Administration 167Collectors

CollectorsThe Collector components run on Foglight Experience Monitor appliances, but the Collector Group configurations (that govern those Collector components) are centrally defined in the Foglight Experience Viewer Server.

Collector Groups allow load-balancing configuration to be centrally controlled across all Foglight Experience Monitor and Foglight Experience Viewer appliances. Each Collector polls the Foglight Experience Viewer Server for load-balancing and security configuration needed for data capture (including the correct private route for monitored traffic). Collector Groups are also used to roll-up health metrics, to alert on capture errors or other data quality problems.

To view the Collector Groups configured in your system:

• In the FxV browser interface, on the menu bar, click Configure > Collectors.

The Collectors page appears, showing the following sections:

• Attached Collectors—lists all the Collectors that have made contact with the Server.

• Collector Groups—lists all the Collector Groups defined in the system.

For detailed instructions on how to configure Collector Groups in a system, see the FxV Installation and Administration Guide.


Superuser TasksThis section presents the list of “superuser tasks” that an administrator can perform via the FxV browser interface. For details about these topics, see the following sections:

• “Appliance Maintenance” on page 168

• “Metrics” on page 172

Appliance MaintenanceAn administrator can use the buttons available in the Appliance Maintenance section to perform the following operations, as part of the FxV appliance maintenance:

• Manage SSH—turns on/off the SSH (secure shell) service for all FxV hosts.Note For detailed instructions on how to enable/disable the SSH access via the FxV

browser interface, see the FxV Installation and Administration Guide.

• Manage Foglight Integration—configures the transmission of metric data to a remote Foglight 5 Server. For more information, see “Manage Foglight Integration” on page 169.

• Server Configuration—sets various configuration parameters for your system. For more information, see “Server Configuration” on page 170.

• Generate Support Bundle—creates a ZIP file that can be sent to the FxV support team for analysis.Note For detailed instructions on how to generate a support bundle, see the FxV

Installation and Administration Guide.

• Upgrade Appliance(s)—applies an upgrade to all FxV hosts. Before using this feature, you must obtain a .runx file necessary for the upgrade. The upgrade menu gives you the option of applying the upgrade immediately or at a future date/time.Note For detailed instructions on how to upgrade an older FxV installation to the latest

FxV version, see the FxV Upgrade Field Guide.

• Kill All Analysis Repository User Queries—kills all currently active Analysis Repository queries.Note Due to the open nature of the FxV Analysis Repository, it is possible to perform

queries that may take an extremely long time to complete.

Performing Advanced FxV Administration 169Superuser Tasks

• Reset Analysis Repository Password—resets the password for the fxv MySQL user account for the Analysis Repository.

Note The two Analysis Repository tasks described in this section are only available if the Analysis Repository has been enabled.

Manage Foglight IntegrationIn order to view user-defined metric data (as defined by Hit Filters and Transaction Filters), the Foglight Experience Viewer Server must be configured to send metrics to a remote Foglight Management Server. Data collected is pushed to the Foglight Management Server, where monitored services can be viewed in various formats and level of details.

The Manage Foglight Integration feature allows end-user metrics to be collected and analyzed alongside metrics from the backing application, database, and network services.

To configure the communication with a remote Foglight 5 Server for metric integration:

1 In the FxV browser interface, on the menu bar, click Configure > Superuser Tasks.

The Superuser Tasks page appears.

2 Click Manage Foglight Integration.

The Foglight Integration page appears.

3 Fill in the following fields:

• Foglight Server IP Address—specifies the IP address of the remote Foglight 5 Server where metrics should be collected. The FxV Server contacts the


Foglight Server using this IP address, so this address must resolve from the FxV Server appliance.

If you want to send metrics from FxV to Foglight via SSL, you must specify the HTTPS/SSL port number for Foglight in the Foglight Server IP Address field (for example, 10.10.11.1:8443). By default, the HTTPS/SSL port number is 8443.

To disable Foglight 5 integration, leave the Foglight Server IP Address field blank.

• Appliance Group Identifier—specifies a unique identifier for the appliance group managed by this FxV Server.Note This field needs to be modified only when multiple FxV Servers feed data into a

single Foglight 5 Server.

• FxV Hostname/IP Override—specifies the Hostname or IP address of the Foglight Experience Viewer (FxV) Server that should be sent to the Foglight 5 Server. This is the Hostname/IP that Foglight 5 must use to create links back to FxV. Leave this field blank to determine the address automatically.

4 Click OK.

The Foglight 5 integration settings are now configured.

Server ConfigurationThis feature allows you to set advanced configuration parameters that impact the entire FxV system.

To set the advanced configuration parameters:



2 Click Server Configuration.


The Server Configuration page appears.

3 Fill in the available fields as needed, then click OK.Note If you want all browser traffic to be secure (over HTTPS), change the SSL

Redirection Enabled setting to Yes. The SSL Redirection Hostname may be left blank.

The advanced configuration parameters are now set.

Important These settings rarely need to be changed. Exercise extreme caution when making any changes, as the impact to the system may be significant.


MetricsA metric is an instance of time-series data used to indicate and track health or performance factors, and to trigger actions when thresholds are breached. Examples of metrics include the value of abandoned shopping carts per day in dollars, or the total size of the database in TB.

During hit processing and analysis, the FxV Archiver performs several actions, including updating the metrics for each hit. The metric updates are recorded in the database for later searching.

The FxV Server maintains all metrics related to user sessions. It also pushes metric data up to a remote Foglight Management Server, if configured to do so. For more information, see “Manage Foglight Integration” on page 169.

The FxV browser interface allows the administrator to perform the following operations with metrics:

• View Metrics—view system health metric data collected for the Foglight Experience Viewer.

• Run Metric Based Diagnostics—analyze, capture, and replay health metrics over the past 24 hours (user-defined metrics can only be viewed in the Foglight Management Server). The resulting report lists any problems found and recommends solutions for each of these problems.

• Reset Metric History—delete all metric history.Important Metric history cannot be recovered once deleted.

To view the FxV metrics:



2 Click View Metrics.

The Metrics page appears, showing the collected system health metrics organized in the following four groups:

• Errors/Warnings• Archivers• Collectors


• ServerNote For a description of metrics that can be viewed in the FxV browser interface, see

“Appendix: FxV Metrics” on page 199.

3 Review the metrics collected for the FxV appliance by navigating through the available tabs. Use the Hide or Show icons to hide or show (respectively) an entire category.

Up to seven icons may appear to the right of any given metric value:

• Move your mouse pointer over this icon to view the timestamp that the Metric’s value was last updated.

• Move your mouse pointer over this icon to quickly view a graph of the metric’s history over the last 60 minutes.

• Click this icon to view the Metric History. This screen displays a large graph of the metric’s history and provides options for displaying alternate time ranges.

• Click this icon to open the Alarm Log for the metric. This screen shows all alarms (that is, any time the metric’s status changed) for the metric. Note This icon is only displayed for metrics that have alarm conditions defined.

• Click this icon to view the alarm conditions defined for the metric.Note This icon is only displayed for metrics that have alarm conditions defined.

• Click this icon to delete the metric.Note This option is only displayed when the system determines the metric is no longer

being updated.


To run a metric-based diagnostic on the FxV metrics collected during the last 24 hours:



2 Click Run Metric Based Diagnostics.

The Diagnostics page appears. The resulting report lists any problems found and recommends solutions for each of these problems.Note Only certain predefined health metrics are checked by these diagnostics. Custom

metrics defined by hit or transaction filters are not included.

10

Managing User Account Settings

This chapter presents the settings that the FxV browser interface provides to FxV administrators for managing the FxV users and the permissions they have when using the FxV resources.

The following diagram illustrates the relationship between the concepts presented in this chapter.

Figure 1


Users and User Groups.............................................................................................................176Resource Groups ......................................................................................................................183Preference Groups ....................................................................................................................184


Users and User GroupsEach person who logs into the FxV system requires a distinct User account. Each User belongs to one or more User Groups. For detailed information about Users, see the following sections:

• “User Preference Settings” on page 176

• “Pre-Defined User Accounts” on page 176

• “Managing User Accounts” on page 177

User Groups define permissions for FxV users logging into the FxV browser interface and accessing system resources. For detailed information about User Groups, see the following sections:

• “User Group Settings” on page 178

• “Pre-Defined User Groups” on page 181

• “Managing User Groups” on page 182

User Preference SettingsThe default user preference settings for a User account are determined by the Preference Group to which the user belongs. The Preference Group also defines which settings can be overridden by the user.

Each User can belong to only one Preference Group. The user preference settings consist of Session Explorer Settings (which allow you to change the behavior of the Replay browser interface) and Search Settings (which allow you to change the behavior of the Search pages).

For more information about preference settings, see “Setting User Preferences” on page 71 and “Preference Groups” on page 184.

Pre-Defined User AccountsTwo Users are pre-defined and delivered by default with new FxV systems:

• admin—defined as part of the administrators user group, for which all administrative privileges are enabled (except for “Foglight Auto-Login” and “Sensitive Data Display”). When logged in as this user, you can manage user accounts, user groups, preference groups, and resource groups.

Managing User Account Settings 177Users and User Groups

• guest—defined as part of the guests user group, for which all administrative privileges are disabled (except for “Hit Content Replay”). When logged in as this user, you can only use the resources offered by the FxV browser interface for hit, session, transaction, and custom searches.

Note From the system’s perspective, these default users are not special in any way. They can be modified or deleted from the system like any other user.

Managing User AccountsTo manage a User account:

1 Log into the FxV browser interface.Note To manage User accounts, you must log into FxV as a user who is in a User Group

configured with the “Users” capability.

2 On the menu bar, click Configure > Users.

The Users page appears, displaying the list of existing users.


• To view an existing user account, click the View icon.• To edit an existing user account, click the Edit icon.

Editing includes defining the user’s account settings (Last Name, First Name, Email, Preference Group, and User Groups), changing user’s password (click Change Password, then fill in the information requested), and managing resources owned by the user (see “To manage the resources owned by a User:” on page 178).

• To remove a user account from the system, click the Delete icon.• To disable/enable a user account, click the Disable / Enable icon.

Note The Disabled icon appears beside a User name when it is disabled.

• To create a new user account, click Create User, then fill in the information requested.Note A “personal” User Group and a “personal” Resource Group are automatically

created for each new User in the system. Each personal group corresponds to a single user and cannot be deleted.


• To disable all user accounts defined in the system, click Disable All Users, then confirm that you want to disable all users in the system, except yours.Note The disabled users are automatically logged out of the system and are treated as

though they do not exist. The Disabled icon papers beside the User names when they are disabled.

• To enable all user accounts defined in the system, click Enable All Users, then confirm that you want to enable all users currently disabled in your system.

Note Depending on the state of the user and your privileges, some of these icons may not appear for certain Users. Specifically, users who own resources cannot be deleted from the system. To remove such a user, first edit it, click View Resources, and either delete or re-assign ownership of any resources owned by this user.

To manage the resources owned by a User:

1 Log into the FxV browser interface.Note To manage User accounts, you must log into FxV as a user who is in a User Group

configured with the “Users” capability.

2 On the menu bar, click Configure > Users.

The Users page appears, displaying the list of existing users.

3 Click the Edit icon beside the User for which you want to manage resources.

The User page appears, displaying the fields available for editing.

4 Click View Resources.

The User Resources page appears, displaying the list of resources owned by the selected User, or a warning message in the case the User does not own a particular type of resource.

5 Do one of the followings:

• To re-assign the ownership of a resource, click the Change Owner icon.• To change the Resource Groups to which a resource belongs to, click the

Share icon.• To delete a resource from the list of owned resources, click the Delete icon.Note Depending on the state of the resource and your privileges, some of these icons

may not appear for certain resources.

User Group SettingsFor each User Group, the following settings must be configured:


• Administrative Privileges—determines the administrative permissions available to the users of the system. Most of these privileges correspond to the options in the Configure menu on the FxV menu bar. They are grouped in the following categories:

• User and Resource Group Management—allows users in this User Group to manage User accounts, User Groups, and Resource Groups, as specified in the following table.

• System Configuration—allows users in this User Group to administer the FxV system (that is, running system diagnostics, resetting the capture database, upgrading the appliance, etc.), as specified in the following table.

Setting Name Setting Description

Resources Manages individual resources (that is, deletes resources or re-assigns ownership of resources).

User & Resource Groups

Manages User Groups and Resource Groups.

Users Creates, edits, and deletes User accounts.


Archivers Configures the data capture appliances by managing the Archivers in your system.

Collector Groups Configures the data capture appliances by defining the Collector Groups in your system.

Edit Hit Analysis Allows the users in this User Group to edit the Hit Analysis settings.

Sensitive Data Configures the sensitive data settings (Sensitive Hit Details and Sensitive response Content Expressions).


• Other Privileges—allows users in this User Group to manage other miscellaneous privileges, as specified in the following table.

• Members—defines the list of users who are part of a User Group.

Superuser Performs FxV superuser tasks, which include managing SSH, configuring the Server, generating support bundle, upgrading the appliance, importing/exporting configurations, viewing metrics, etc.

View Hit Analysis Allows the users in this User Group to view the Hit Analysis settings.


Content Replay Zip Export

Exports user sessions as ZIP files.

Foglight Auto-Login Enables users in this User Group to automatically log in to the FxV appliance when drilling down from Foglight, without the need for re-authentication.

Hit Content Replay Displays the Content Replay view when exploring a session of interest.

Sensitive Data Display Makes visible to users sensitive data captured by the system, flagged as “not always sensitive”. For more information, see “Sensitive Data Protection” on page 121.Note Enabling the “Sensitive Data Display” option

negates the protection provided by any Sensitive Data definitions in the system not marked as “Always Sensitive”. This feature must be used only for very specific users, who are trusted to view such data. These users must be added to a User Group created specifically for this purpose.



• Resource Groups—defines the Resource Groups to which a User Group is linked. For detailed information about resources and Resource Groups, see “Resource Groups” on page 183.

Each User Group can be linked to any number of Resource Groups. These links determine which resources the users who belong to the User Group can access. For each Resource Group linked to this User Group, you can specify the privileges that members of this User Group have with respect to the resources in that Resource Group, as follows:

• Create—users allowed to create new resources in the corresponding Resource Group.

• Edit Owned—users allowed to edit resources in the corresponding Resource Group if they are the owner of the resource.

• Remove Owned—users allowed to delete resources in the corresponding Resource Group if they are the owner of the resource.

• Edit Not Owned—users allowed to edit resources in the corresponding Resource Group even if they are not the owner of the resource.

• Remove Not Owned—users allowed to delete resources in the corresponding Resource Group even if they are not the owner of the resource.

• View—users allowed to view resources in the corresponding Resource Group. Granting “view access” allows users to view resource definitions without being able to edit the resources.

• Use—users allowed to use resources in the corresponding Resource Group. Granting use access allows users to use resources that they cannot otherwise access.

Pre-Defined User GroupsFour User Groups are pre-defined and delivered by default with new FxV systems:

• administrators—group for which all administrative privileges are enabled (except for “Foglight Auto-Login” and “Sensitive Data Display”). The admin user is pre-defined as part of this group.

• guests—group for which all administrative privileges are disabled (except for “Hit Content Replay”). The guest user is pre-defined as part of this group.

• Personal UG:admin—”personal” group automatically created when the admin user is defined. This group can be deleted only by deleting the admin user.


• Personal UG:guest—”personal” group automatically created when the guest user is defined. This group can be deleted only by deleting the guest user.

Note From the system’s perspective, these default user groups are not special in any way. They can be modified or deleted from the system like any other user group.

Managing User GroupsTo view, edit, delete, or create a new User Group:

1 Log into the FxV browser interface.Note To manage User Groups, you must log into FxV as a user who is in a User Group

configured with the “User & Resource Groups” capability.

2 On the menu bar, click Configure > User Groups.

The User Groups page appears, displaying the list of existing User Groups.


• To view additional information about a User Group, click the View icon.• To edit a User Group, click the Edit icon.

The editing page allows you to add or remove users from the User Group, but also to add, edit, and remove links from the User Group to the Resource Group.

• To remove a User Group from the system, click the Delete icon.• To create a new User Group, click Create User Group, then fill in the

information required.

a Fill in the Group Name (must be unique).

b Enable the privileges necessary for this User Group by selecting the appropriate check boxes in the Administrative Privileges. For details about administrative privileges, see “User Group Settings” on page 178.

c From the Non-members list, using the >> and << buttons, move into the Members list the Users who should be part of this User Group.Note The Non-members list originally contains all User accounts already defined for

this system.

Managing User Account Settings 183Resource Groups

d Add Resource Groups that should be linked to this User Group, and specify the privileges users should have with respect to the resources in these groups. For details about resource privileges, see “User Group Settings” on page 178.

Note Depending on the state of the User Group and your privileges, some of these icons may not appear for certain User Groups.

Resource GroupsA Resource Group is a collection of resources that can be associated with one or more User Groups, in order to customize the users’ access to those resources.

The following types of resources can be created in the system: Custom Search Screens and Saved Searches (for more information about these topics, see “Creating and Editing Custom Search Screens” on page 67 and “Creating and Editing Saved Searches” on page 66).

Each resource is owned by a single User, but it can be associated with one or more Resource Groups. For different User Groups, you can grant/restrict the access privileges to these resources. For more information about these topics, see “Users and User Groups” on page 176.

Three Resource Groups are pre-defined and delivered by default with new FxV systems:

• Default—group that associates all resources with the two pre-defined User Groups (administrators and guest).

• Personal RG:admin—“personal” group automatically created when the admin user is defined. This group is deleted only when the admin user is deleted.

• Personal RG:guest—“personal” group automatically created when the guest user is defined. This group is deleted only when the guest user is deleted.

Note From the system’s perspective, these default resource groups are not special in any way. They can be modified or deleted from the system like any other resource group.

To view, edit, delete, or create a new Resource Group:

1 Log into the FxV browser interface.Note To manage Resource Groups, you must log into FxV as a user who is in a User

Group configured with the “User & Resource Groups” capability.

2 On the menu bar, click Configure > Resource Groups.


The Resource Groups page appears, displaying the list of existing resource groups.


• To view additional information about a Resource Group, click the View icon.

• To edit the information of a Resource Group, click the Edit icon.

The editing page allows you to define the type of resources that can be placed in this Resource Group (by selecting the corresponding Allowed Resource Types check boxes), to view the list of resources associated to this Resource Group (by clicking View Resources), and to control the mapping between the User Group and Resource Groups (that is, specify the privileges users should have with respect to the resources in the Resource Groups linked to any number of User Groups).

• To remove a Resource Group from the system, click the Delete icon.Note Each “personal” Resource Group is linked with the user’s personal User Group

and cannot be deleted.

• To create a new Resource Group, click Create Resource Group, then fill in the information requested.

Note Depending on the state of the Resource Group and your privileges, some of these icons may not appear for certain Resource Groups.

Preference GroupsA Preference Group is a group of FxV Users with common default user preference settings. Each User belongs to exactly one Preference Group.

Within a group, you can:

• Manage the list of Users assigned to the Preference Group.

• Define which values can be changed by the Users in the group, by selecting/clearing the lock check box beside a particular setting in the Session Explorer Settings or the Search Settings lists. Settings that are locked cannot be overridden by individual Users. Settings that are unlocked serve as defaults, and may be changed by individual Users. Locking a setting overrides immediately any user-specified values.

Managing User Account Settings 185Preference Groups

For a detailed description of Session Explorer Settings and Search Settings options, see “Session Explorer Settings” on page 72 and “Search Settings” on page 75.

Only one Preference Group (Default Preferences) is pre-defined and delivered by default with new FxV systems.

To view, edit, delete, or create a new Preference Group:

1 Log into the FxV browser interface.Note To manage Preference Groups, you must log into FxV as a user who is in a User

Group configured with the “User & Resource Groups” capability.

2 On the menu bar, click Configure > Preference Groups.

The Preference Groups page appears, displaying the list of existing Preference Groups.


• To view additional information about a Preference Group, click theView icon.

• To edit a Preference Group, click the Edit icon. Editing includes moving users into this group.

• To remove a Preference Group from the system, click the Delete icon.Note Preference Groups that have users associated with them cannot be deleted.

• To create a new Preference Group, click Create Preference Group, then fill in the information required.

Important Users cannot be explicitly removed from a Preference Group.

To remove users from a Preference Group:

• Add the users to a different Preference Group.

The users are then automatically removed from the Preference Group in which they were previously members.


A

Appendix: Java Regular Expressions in FxV Hit Analysis

Regular expressions allow you to identify and extract data during several steps of the FxV Hit Analysis process. The details of how they are used varies slightly depending on the circumstance.

This appendix examines these differences thoroughly. For details, see these topics:

Regular Expressions Overview .................................................................................................188Regular Expressions in FxV ......................................................................................................192FxV Regular Expression Tester .................................................................................................198


Regular Expressions OverviewBefore looking at the specifics, it is important to understand the basic concepts that apply to all regular expression usage in FxV. The purpose of a regular expression is to define a pattern that can match any number of similar strings. For example, consider a Web page that contains the following text, which may be slightly different, depending on the active user:

Username: joe

Username: john

Username: sarah

A regular expression can be used to extract the unique portion of the text from each page. In this case, the expression might look like this:

Username: (\w+)

Here, the next “Username: ” is matched exactly, followed by one or more letters or digits (the “\w” stands for “letter or digit” and the “+” represents “one or more”). The parenthesis around “\w+” indicate that the text matched by that part of the expression is the text to be returned.

The following table lists many of the most common characters used when writing regular expressions. Note that this table presents two types of examples:

• Example 1—the intent is to show that for the boolean question “does this expression match this input”, the answer would be “yes”:

expression matches sample-text

• Example 2—the intent is to show that for the request “return the portion of the input text that matches this expression”, the answer would be the result:

expression returns result for input sample-text

Characters Description Examples

Anything other than [\^$.|?*+()

Non-special characters that match the exact text specified.

Dog matches Dog

\ (backslash) followed by any one of [\^$.|?*+()

Matches the special character, ignoring its special meaning.

5\+3=8 matches 5+3=8

Appendix: Java Regular Expressions in FxV Hit Analysis 189Regular Expressions Overview

\d Matches any digit (0-9). \d\d matches 12abc\d matches abc5

\w Matches any letter or digit. \w\w\w matches 123\w\w\w matches abc\w\w\w matches a2c

\s Matches any whitespace character (spaces, tabs, new lines, etc.).

a\sZ matches a Z

\D, \W, \S Negated versions of \d, \w, and \s. That is, they match non-digits, non-word-characters, and non-whitespace.

\D\D matches AB\W\W matches @#\S\S\S matches a1#

. Matches any single character. a.c matches abca.c matches a2ca.c matches a c

^ Matches at the start of the string being evaluated (start of line).

^foo matches foobar^foo does not match barfoo

$ Matches at the end of the string being evaluated (end of line).

bar$ matches foobarbar$ does not match barfoo

| Matches what is on the left or right of the “pipe” character.

a|b|c matches aa|b|c matches ba|b|c does not match d

? Makes the preceding item optional. This optional item is included in the match if it exists.

ab? matches aab? matches ab



* Matches zero or more of the preceding item. As much data as possible is included in the match. For more information, see “Greedy vs. Reluctant Expressions” on page 191.

a.*a returns abaca for input bbabacabba.*a matches aa

*? Matches zero or more of the preceding item. As little data as possible is included in the match. For more information, see “Greedy vs. Reluctant Expressions” on page 191.

a.*?a returns aba for input bbabacabba.*?a matches aa

+ Matches one or more of the preceding item. As much data as possible is included in the match. For more information, see “Greedy vs. Reluctant Expressions” on page 191.

a.+a returns abaca for input bbabacabba.+a does not match aa

+? Matches one or more of the preceding item. As little data as possible is included in the match. For more information, see “Greedy vs. Reluctant Expressions” on page 191.

a.+?a returns aba for input bbabacabba.+?a does not match aa

{n} Matches exactly n of the preceding item.

z{3} matches zzz

{n,m} Matches between n and m of the preceding item. As much data as possible is included in the match.For more information, see “Greedy vs. Reluctant Expressions” on page 191.

z{2,4} matches zzz{2,4} matches zzzz{2,4} matches zzzzz{2,4} returns zzz for input zzz


Appendix: Java Regular Expressions in FxV Hit Analysis 191Regular Expressions Overview

Greedy vs. Reluctant Expressions

The distinction between “*?” and “*” (and “+?” and “+”) is subtle, but important. The following example illustrates the main difference between these concepts. Consider the following HTML page:

<HTML><BODY>Your username is wilma and your password is bambam</BODY></HTML>

An initial attempt to extract the username from this page might look like this:

username is (.*)

This would seem to say “Look for the text ‘username is’, and then extract the value between the and tags.”. This is exactly what the expression does. The problem is that “.*” is a “greedy” operator, meaning that it matches as much as it can. In this case, the result is:

wilma and your password is bambam

What happened is that the expression found “username is ”, and then looked for the last occurrence of “”. Everything in between is returned as the result.

In order to extract just the username, you need to use a “reluctant” expression:

username is (.*?)

The result you get now is the one you were actually looking for:

wilma

{n,m}? Matches between n and m of the preceding item. As little data as possible is included in the match. For more information, see “Greedy vs. Reluctant Expressions” on page 191.

z{2,4}? returns zz for input zzz



Regular Expressions in FxVRegular expressions can be used in FxV in several places. Depending on the purpose, the type of expression needed varies slightly. The following sections outline these differences:

• “Looking for “Matches” (Yes or No)” on page 192

• “Extracting a Single Value” on page 193

• “Identifying Sensitive Response Content” on page 195

Looking for “Matches” (Yes or No)Regular expressions can be used to answer questions of the form:

Does some piece of data match some expression.

In this case, the job of the regular expression is to return a boolean result: yes (the data matched the expression), or no (it did not).

There are two places where this type of expressions are used within Hit Analysis: Hit Filter match conditions and Sensitive Hit Details.

Using “Matches” in Hit Filter Match ConditionsWhen defining a Hit Filter match condition, several operators are available for selection. One of these operators is “Matches”.

When you select “Matches”, the text specified after the operator is expected to be a regular expression. For example:

Request Field phoneNum Matches \d{3}-\d{3}-\d{4}

This expression considers a hit to have matched if it contains a request field called phoneNum with a value such as 303-555-1212 (three digits, a dash, three digits, a dash, four digits). It does not match values such as 3035551212 or (303)555-1212.

Using “Matches” in Sensitive Hit Detail SpecificationsWhen defining a Sensitive Hit Detail, several “Matching Logic” operators are available for selection. One of these operators is “Matches”.

When you select “Matches”, the text specified as the Hit Detail Name is expected to be a regular expression. For example:

Appendix: Java Regular Expressions in FxV Hit Analysis 193Regular Expressions in FxV

Request Field matches ^cc\d+$

This expression considers a request field to be sensitive if the name of the field starts with the letters “cc”, and ends with one or more digits (with nothing else between the “cc” and the digits). Fields such as cc1 and cc500 are considered sensitive while fields such as xcc1, cc500z, and ccx500 are not.

Extracting a Single ValueIt is often useful to extract some unique piece of data from a data source that contains additional, unwanted text. For example, you might want to extract the numeric value 123.45 from the text “Total Cost: $123.45”.

This type of operation can be performed anywhere in the Hit Analysis UI where the characters “/ /” appear. This includes Custom Field value assignment, Metric value assignment, and Hit Filter match conditions.

Note This operation is different than using the “Matches” operator in a Hit Filter match condition.

When extracting a value in this manner, the regular expression must contain a set of parenthesis that indicates the portion of the expression to be returned. Expressions defined without any parenthesis do not work.

Note Only one set of parenthesis may be used in this context. Specifying an expression such as (\d+)xyz(\d+) does not return data from both sets of parenthesis. Only the data matching the first set of parenthesis (“Group 1”) is returned.

Assigning Custom Field or Metric ValuesWhen defining the value source for a Custom Field or Metric, click the icon “/ /” to specify a regular expression that should extract some portion of the value source rather than the entire value.

For example, suppose a Request Field called “total” contains values such as 123.45USD and 5.00USD, and you want to extract only the numeric portion of these values into a numeric Custom Field. After selecting the “total” field as the value source, the following expression would extract the desired data:

(\d*\.\d\d)


This expression means “match zero or more digits, a period, and two more digits.” The entire expression is wrapped in parenthesis, indicating that the entire value must be returned.

The following expression can also be used for extracting the same desired data:

(\d*\.\d\d)USD

As long as all values are followed by the letters “USD”, this expression would produce the same results as the expression in the first example.

Similarly, the expression:

\d*\.\d\d(\S{3})

could be used to extract the three-letter currency code into a “currency” custom field. This expression can be read as “match zero or more digits followed by a period, two more digits, and three more non-whitespace characters. Return only the three non-whitespace characters.”.

The expression:

(\S{3})

would not produce the desired results, as it would return only the 123 for the input data 123.45USD.

Extracting a Portion of a Hit Detail in a Hit Filter Match ConditionThe same technique can be used to extract a portion of a hit detail when defining a Hit Filter match condition. Only the extracted portion of the data is used when evaluating the selected operator. For example, a match condition could be defined like this:

Client IP / ^(\d+)\.\d+\.\d+\.\d+$ / = 192

The text between the slashes is entered by clicking the icon “/ /”, and filling in the text box that appears. In this case, the expression is looking for an IP address (one or more digits, a period, one or more digits, a period, one or more digits, a period, one or more digits). Since only the first series of digits is surrounded by parenthesis, that portion of the address is used to evaluate the rest of the match condition. In this example, hits would match if the first number in the Client IP address is 192.


Identifying Sensitive Response ContentAnother way to use regular expressions in the Hit Analysis process is as part of a Sensitive Response Content Expression. In this case, the regular expression is used to identify the portion(s) of the response content considered “sensitive”.

There are a few aspects of this type of expression that are unique to this particular use.

Multiple MatchesA Sensitive Content regular expression is matched repeatedly until the end of the page being processed is reached. For example, consider the following HTML page:

<HTML><BODY>Your credit card number is 4123123412341234 and my credit card number is 4111222233334444</BODY></HTML>

The following regular expression would identify both credit card numbers as sensitive:

\d{16}

This expression means “Look for 16 consecutive digits. Once you find a match, start looking again.”. An expression like this does not require any wrapping parenthesis since everything that matches the expression is considered sensitive.

However, it is not always possible to define an expression like this that matches everything that is sensitive, without also matching non-sensitive data. For example, consider the following HTML page:

<HTML><BODY>Account:123-4567 Phone:303-555-1212</BODY></HTML>

and the expression:

\d{3}-\d{4}

This would match both 123-4567 and 555-1212. If the goal is to mark the account number as sensitive, but not the phone number, this expression is not sufficient. Instead, additional context should be provided and parenthesis should be used to identify the sensitive portion of the expression:

Account:(\d{3}-\d{4})

This expression is explicitly looking for the text “Account:” followed by an account number matching the expected format. Since the phone number is not preceded by “Account:”, it is not a match. The parenthesis indicate that only the account number itself (not the text “Account:”) should be considered sensitive.


Group NumbersData matched by Sensitive Response Content Expressions is returned in one or more “groups”. Expressions that do not contain parenthesis have only one group: Group 0. This group contains everything returned each time the expression matches. For example:

Expression: \d\d

Content: 12 34 foobar 56

Group 0 (match 1): 12



When one or more set of parenthesis are included in the expression, each set of parenthesis identifies a new “group”. Group 0 represents the entire expression. For example:

Expression: (\d)(\w)

Content: 1a 3b 5c

Group 0 (match 1): 1a


Group 2 (match 1): a

Group 0 (match 2): 3b


Group 2 (match 2): b

Group 0 (match 3): 5c


Group 2 (match 3): c

Any time you use parenthesis in a Sensitive Response Content expression, it is important to ensure that the appropriate group(s) are marked as sensitive. The most common case is to define a single group (Group 1) and mark it as sensitive. For example:

Regular Expression: Password=(\w+)

Content: Password=trinity

Sensitive Groups: 1


In this case, only the password itself (the text between the and tags) would be considered sensitive. However, if Group 0 were specified rather than Group 1, everything matching the expression (Password=trinity) would be considered sensitive.

Multiline MatchingThere are subtle details related to evaluating regular expressions against text that spans multiple lines. Consider the following HTML page:

<HTML>

<BODY>



Hello1

World2

Goodbye3



</BODY>

</HTML>

The character “$” matches the end of a line, not the end of the document. The expression:

World(\d)$

returns 2 for Group 1.

The wildcard operator (*) does not stop at the end of a line. The expression:

(.*)

returns for Group 1:

Hello1

World2

Goodbye3

Note The behavior described in this section corresponds to the Java regular expression flags DOTALL and MULTILINE. This note is addressed to readers familiar with Java regular expressions (or those using a Java regular expression tester other than the one provided in the FxV browser interface).


FxV Regular Expression TesterThe FxV browser interface has a built-in expression tester, which can be accessed by clicking the icon (.?) besides the Regular Expression text boxes, wherever they appear in the Hit Analysis interface. The regular expression tester runs in three slightly different modes, depending on the context:

• When opened from a “matches” context, it returns either “true” or “false”, to indicate whether or not the given expression matches the specified test input.

• When opened from an “extract value” context, it requires the presence of matching parenthesis, and returns the data that would be extracted by these parentheses.

• When opened from the Sensitive Response Content Expression screen, it returns a summary of all matches and all groups (including Group 0). In this context, to ensure that the desired data is being properly extracted, it is recommended to copy the entire HTML source from a sample page, and use it as the sample search text.

B

Appendix: FxV Metrics

Foglight Experience Viewer monitors two types of metrics:

• System metrics—can be viewed in FxV browser interface and in Foglight.

• User-defined metrics—can only be viewed in Foglight.

This appendix provides the complete list of FxV system metrics and contains the following topics:

Archiver Metrics.........................................................................................................................200Collector Metrics........................................................................................................................234Server Metrics ...........................................................................................................................236Error/Warning Metrics................................................................................................................241


Archiver MetricsYou can select an Archiver to display its individual metrics, or “Totals” to display total metrics across all Archivers in the system.

Note Certain metrics are displayed only for individual Archivers.

The Archiver metrics include the following categories:

• “Hit Summary Metrics” on page 200

• “Analysis Repository Metrics” on page 208

• “Archiver Capacity Metrics” on page 210

• “Archiver Health Metrics” on page 215

• “Archiver Appliance Metrics” on page 224

• “Archiver Search Usage Metrics” on page 228

• “Script Metrics” on page 233

Hit Summary MetricsThis Metric Category provides statistics about all hits captured and analyzed by the system. These Metrics are used primarily to troubleshoot capture and data-quality problems.

Note These metrics do not reflect the health of your Web applications, only the capture/replay system itself. Collector metrics show health of the capture system from the Foglight Experience Monitor viewpoint.

The following metrics are provided in this category:

Metric Name Displayed in Totals

% Of Hits Discarded No

% Of Hits Dropped By Hit Filter No

% of Hits With Errors Yes

Appendix: FxV Metrics 201Archiver Metrics

% of Hits With Warnings Yes

Exceptions - Capture Error No

Exceptions - Chunked Transfer Error No

Exceptions - Connection Reset No

Exceptions - Content Dropped By Archiver No

Exceptions - Content Dropped By Collector No

Exceptions - Content Dropped By Filter No

Exceptions - Corrupted Response No

Exceptions - Incomplete Content No

Exceptions - Missing Property No

Exceptions - No Response No

Exceptions - No Session ID No

Exceptions - Timeout No

Exceptions - Truncated Response No

Exceptions - Unknown Exception No

Hits Discarded Yes

Hits Dropped By Hit Filter Yes

Hits Failed Yes

Hits Processed Yes

Hits With Corrupt Cookie Names No



% Of Hits DiscardedReports percentage of hits sent to a Capture Appliance but discarded before processing could complete. Each Archiver has a limited amount of memory to buffer hits before processing and, when this buffer fills up, incoming hits are discarded.

% Of Hits Discarded = Hits Discarded / Hits Processed

% Of Hits Dropped By Hit FilterReports percentage of hits intentionally dropped by Hit Filters in order to conserve disk space. If close to 100%, it may be that a Hit Filter has inadvertently been created to drop all incoming hits.

% Of Hits Dropped By Hit Filter = Hits Dropped By Hit Filter / Hits Processed

% of Hits With ErrorsReports percentage of hits marked as errors by a Hit Filter. All hits are considered OK until an error is detected.

% of Hits With Errors = Hits With Errors / Hits Processed

% of Hits With WarningsReports percentage of hits marked as warnings by a Hit Filter. All hits are considered OK until an error is detected.

% of Hits With Warnings = Hits With Warnings / Hits Processed

Hits With Corrupt Request Field Names No

Hits With Corrupt Request Header Names No

Hits With Corrupt Response Header Names No

Hits With Errors Yes

Hits With Warnings Yes



Exceptions - Capture ErrorReports number of hits that were not captured properly. This indicates hits that were incomplete or corrupted, but no other error condition was present, such as a connection reset or timeout. This metric is automatically updated prior to hit analysis.

Note To find these metrics, do a hit search on the Exception property for this exception type.

Exceptions - Chunked Transfer ErrorReports number of hits using chunked encoding that could not be de-chunked due to invalid formatting. This metric is automatically updated prior to hit analysis. These hits were processed normally, however the content may not display properly in session playback.


Exceptions - Connection ResetReports number of hits that were terminated by a connection reset by either the client or the server. Due to the abnormal termination of the hit, not all hit details, including content, may have been captured. This metric is automatically updated prior to hit analysis.

Note To find these, do a hit search on the Exception property for this exception type.

Exceptions - Content Dropped By ArchiverReports number of non-text hits whose response content was dropped due to the fact that the captured content did not match the Content-Length header. This metric is automatically updated prior to hit analysis.



Exceptions - Content Dropped By CollectorReports number of non-text hits whose response content was dropped due to the fact that the captured content exceeded the maximum response size defined for this Collector. This metric is automatically updated prior to hit analysis.


Exceptions - Content Dropped By FilterReports number of hits whose content was dropped by a hit filter rule. This metric is automatically updated upon completion of hit analysis.


Exceptions - Corrupted ResponseReports number of hits that had a corrupt response. A corrupt response is usually due to either response headers that could not be parsed or compressed content that could not be inflated. This metric is automatically updated prior to hit analysis.


Exceptions - Incomplete ContentReports number of text hits whose captured content did not match the Content-Length header. This metric is automatically updated prior to hit analysis.


Exceptions - Missing PropertyReports number of hits that were missing one or more required properties. The required properties are: URL, Method, and ClientIP. This metric is automatically updated prior to hit analysis.



Exceptions - No ResponseReports number of hits for which no response was captured. This is often caused by a connection reset or timeout. Due to the abnormal termination of the hit, not all hit details, including content, may have been captured. This metric is automatically updated prior to hit analysis.


Exceptions - No Session IDReports number of hits that were not sessionized. This metric is automatically updated prior to hit analysis.


Exceptions - TimeoutReports number of hits that timed out. A timeout is determined by the Monitor when the server does not respond within the TCP timeout period. This metric is automatically updated prior to hit analysis.


Exceptions - Truncated ResponseReports number of text hits whose response content was truncated due to the fact that the captured content exceeded the maximum response size defined for this Collector. This metric is automatically updated prior to hit analysis.



Exceptions - Unknown ExceptionReports number of hits that had an unknown exception. This metric is automatically updated prior to hit analysis.


Hits DiscardedReports number of hits sent to a Capture Appliance but discarded before processing could complete. Each Archiver has a limited amount of memory to buffer hits before processing, and when this buffer fills up incoming hits are thrown away.

Note Discarded hits are not counted as Hits Processed. Discards can also occur at the Collector before hits are sent.

Hits Dropped By Hit FilterReports number of hits intentionally dropped by Hit Filters in order to conserve disk space. It is not uncommon to drop hits in static site areas, or hits for non-HTML resources (like images or style sheets).

Hits FailedReports the number of hits that could not be processed due to an Archiver runtime failure. The logs on the Capture Appliance should have further detail on these failures.

Note Discarded hits are not counted as Hits Processed.

Hits ProcessedReports number of hits captured by a Collector, sent to a Capture Appliance, analyzed, and processed successfully.

Not all of these hits were necessarily written to the capture database for later search and replay: some hits may have been dropped by a Hit Filter, or discarded while under load, or in rare cases may have failed to process successfully.

Note A steady increase in this value indicates steady incoming traffic from attached Collectors.


Hits With Corrupt Cookie NamesReports number of hits that had corrupt cookie names. This metric is automatically updated prior to hit analysis. These hits were processed normally, however the invalid details were not written to the capture database.

Note To find these metrics, do a hit search on the Corrupt Hit Details custom field using a value of “*” and the field value will indicate the detail type affected.

Hits With Corrupt Request Field NamesReports number of hits that had corrupt request field names. This metric is automatically updated prior to hit analysis. These hits were processed normally, however the invalid details were not written to the capture database.


Hits With Corrupt Request Header NamesReports number of hits that had corrupt request header names. This metric is automatically updated prior to hit analysis. These hits were processed normally, however the invalid details were not written to the capture database.


Hits With Corrupt Response Header NamesReports number of hits that had corrupt response header names. This metric is automatically updated prior to hit analysis. These hits were processed normally, however the invalid details were not written to the capture database.



Hits With ErrorsReports number of hits marked as errors by a Hit Filter. All hits are considered OK until an error is detected.

Hits With WarningsReports number of hits marked as warnings by a Hit Filter. All hits are considered OK until an error or warning is detected.

Analysis Repository MetricsThis Metric Category contains all Metrics for the Analysis Repository.



Age of Latest Data No

Age of Oldest Data No

Analysis Repository Current Size Yes

Analysis Repository Maximum Size No

Maintenance Cycles Yes

Queries Canceled Yes

Segments - Last Load Time No

Segments - Loaded Yes

Segments - Waiting To Load Yes

Total Sessions Yes

Total Transaction Tables Yes

Total Transactions Yes


Age of Latest DataReports the age (in hours) of the most recent data in the Analysis Repository. Capture segments are loaded after they close. This metric is normally between 0 and the number of hours that a capture segment stays open.

Age of Oldest DataReports the age (in hours) of the oldest data in the Analysis Repository.

Analysis Repository Current SizeReports the current size of the Analysis Repository. This metric increases throughout the day as each capture segment is loaded. Once the Analysis Repository fills up, data is removed during the daily Maintenance Cycle.

Analysis Repository Maximum SizeReports the maximum size of the Analysis Repository as specified in the Analysis Repository Configuration.

Maintenance CyclesReports the number of Maintenance Cycles that have been performed. Maintenance is performed each day at the time specified in the Analysis Repository Configuration. This metric is reset to 0 after each restart, and is primarily used to indicate when the Maintenance Cycle was performed.

Queries CanceledReports the number of user queries that were canceled. A query is canceled if it exceeds the maximum query time as specified in the Analysis Repository Configuration, or if it is active at the start of a Maintenance Cycle.

Segments - Last Load TimeReports the number of seconds used to load the last capture segment into the Analysis Repository.

Segments - LoadedReports the number of capture segments loaded into the Analysis Repository. This is a daily total and is reset to 0 at midnight each day.


Segments - Waiting To LoadReports the number of capture segments waiting to be loaded into the Analysis Repository. It is normal to see a small number of capture segments waiting to load from time to time. However, if this metric stays above 0 for a long time, this indicates that the capture segments are closing faster than they can be loaded. This may result in some capture segments not being loaded into the Analysis Repository.

Total SessionsReports the total number of sessions in the Analysis Repository.

Total Transaction TablesReports the total number of transaction tables in the Analysis Repository. There is one transaction table for each Transaction Filter selected in the Analysis Repository Configuration.

Total TransactionsReports the total number of all transactions in the Analysis Repository. This total includes transactions for all Transaction Filters that have been selected in the Analysis Repository Configuration.

Archiver Capacity MetricsThis Metrics Category provides sizing information based on hits written to the capture database. These Metrics are primarily used for capacity planning, some have alarm conditions defined by default.

Note These metrics do not reflect the health of your Web applications, only the capture/replay system itself.



% Of Hits With Content Keyword Indexing No

% Of HTML Hits No


% Of Hits With Content Keyword IndexingReports percentage of hits in the open database segment that were written to the content keyword index, as directed by a Hit Filter.

Note Indexing a large percentage of hits can negatively impact system performance and increase disk use.

Available Database Aspects No

Available Database Hits Yes

Available Database Segments Yes

Available Database Sessions Yes

Average Content Size No

Average HTML Content Size No

Capture Database Size Yes

Capture Rate Yes

Content Uniqueness No

Database Growth Rate Yes

HTML Content Uniqueness No

Open Segment Aspects No

Raw Capture Rate Yes

Storage Database Size No



% Of HTML HitsReports percentage of hits in the open database segment that were recognized as HTML hits.

An HTML hit is defined as a hit with a Content-Type response header starting with text/html, containing a proper HTML tag in the response content, and having a response code of 200.

Available Database AspectsReports number of aspects across all database segments.

Aspects are created dynamically as needed to track fields, cookies, headers, and other searchable database objects.

Note A large number of aspects can degrade search performance and may indicate a data-quality problem.

Available Database HitsReports number of hits in all segments of the capture database. This is the total count of hits available for search and replay.

Note Hits are removed from the database as segments expire.

Available Database SegmentsReports number of database segments currently used in the capture database. Each segment contains a time span of captured data.

Note More segments are required for more replay history, but a large number of segments can degrade search performance. Segments are deleted as they expire.

Available Database SessionsReports number of sessions in all segments of the capture database. This is the total count of sessions available for search and replay.

Note Sessions are removed from the database as segments expire.


Average Content SizeReports average size of unique response content for non-HTML hits, calculated for the open database segment.

A Non-HTML hit is defined as a hit with a Content-Type response header other than text/html* or without a proper HTML tag in the response content. These include images and style sheets.

Average HTML Content SizeReports average size of unique response content for HTML hits, calculated for the open database segment.

An HTML hit is defined as a hit with a Content-Type response header starting with text/html, containing a proper HTML tag in the response content, and having a response code of 200. These include HTML page hits from static sites and dynamic Web applications.

Capture Database SizeReports total size of all database segments currently stored in the Capture Appliance databases. If there are no Storage Appliances, this is the total volume of data available for searching and replay. If a storage tier is present, this measures just the short-term capture portion of the total database.

Note The capture database size grows as data is collected, and shrinks as old segments expire and are dropped over time.

Capture RateReports hits per second received from all attached Collectors over the last two minutes. This measures the number of hits written to the capture database for later searching.

Content UniquenessReports percentage of non-HTML hits in the open database segment that have unique values. The higher the uniqueness value, the more disk space is used.

A Non-HTML hit is defined as a hit with a Content-Type response header other than text/html* or without a proper HTML tag in the response content.


Database Growth RateReports change in the size of the open database segment database over the last two minutes. This measures the volume of data being written to the Capture Appliance disks.

HTML Content UniquenessReports percentage of HTML hits in the open database segment that have unique values. The higher the uniqueness value, the more disk space is used.

An HTML hit is defined as a hit with a Content-Type response header starting with text/html, containing a proper HTML tag in the response content, and having a response code of 200.

Open Segment AspectsReports number of aspects in the open database segment.

Aspects are created dynamically as needed to track fields, cookies, headers, and other searchable database objects.

Note A large number of aspects can degrade search performance and may indicate a data-quality problem.

Raw Capture RateReports raw hit traffic received from all attached Collectors over the last two minutes. This measures the total network bandwidth consumed between the Collectors and the Archiver.

Storage Database SizeReports total size of all database segments currently maintained by Storage Appliances. If there are no Storage Appliances, this value will be zero. Otherwise, this measures the total size of the long-term storage database.

Note The storage database grows over time as data is collected and eventually saturates at its maximum capacity.


Archiver Health MetricsThis Metric Category provides information about Archiver internal health. These Metrics are primarily used to troubleshoot capture problems, some have alarm conditions defined by default.




Batch Queue Count No

Batch Queue Limit No

Cached Items No

Clock Skew from Server No

Config Available No

Hit Analysis - Custom Fields Active No

Hit Analysis - Failures Yes

Hit Analysis - Rules Fired No

Hit Analysis - Session Hit Filters Active No

Hit Analysis - Session Timeouts No

Hit Analysis - Sessions Active Yes

Hit Analysis - Sessions Completed No

Hit Analysis - Transaction Hit Filters Active No

Hit Analysis - Transactions Active Yes


Hit Analysis - Transactions Completed No

JVM Free Memory No

JVM Total Memory No

JVM Used Memory No

Refresh Metric Time No

Segment Transfer Attempts No

Segment Transfer Completions No

Segments Closing No

Segments Copying No

Segments Dropped Before Transfer Yes

Segments Waiting to Copy No

Segments Waiting to Transfer No

Shared Memory Size No

Shared Memory Used No

TCPBatches JVM Free Memory No

TCPBatches JVM Total Memory No

TCPBatches JVM Used Memory No

Write Buffers Failed Yes



Batch Queue CountReports the number of raw hits waiting to be processed. Each Archiver has a limited number of hits that it can buffer before processing.

Note The Archiver queue may often be at the limit. The TCPBatches process buffers incoming hits in shared memory and feeds these raw hits to the Archiver for processing.

Batch Queue LimitReports total number of raw hits that can be stored prior to processing. Each Archiver has this limited number of hits that it can buffer.

Note This metric value does not change while the Archiver is running. It is needed for legacy reasons.

Cached ItemsReports number of unique data items currently held in the Archiver memory cache. A very large value may indicate a data quality problem or extreme load.

Clock Skew from ServerReports an estimated number of milliseconds that the clock of the Capture Appliance running this Archiver and the Foglight Experience Viewer Server are skewed.

Config AvailableReports whether configuration polling back to the Server was successful on the last attempt. When an Archiver is added to a Server, it begins to contact the Server periodically for configuration settings.

Important When an Archiver is restarted, it must be able to contact the Server successfully once before it is able to capture incoming hits.


Hit Analysis - Custom Fields ActiveReports number of custom field facts defined by all active Sessions and Transactions. One fact is created for each custom field value.

Note Larger values indicate increased memory usage.

Hit Analysis - FailuresReports number of times an internal failure has occurred in the rules engine. Any failures are most likely triggered by a recent change in the Hit Analysis Configuration. If failures have been reported, export your current configuration and send it to the Foglight Experience Viewer support team.

Hit Analysis - Rules FiredReports number of times a Hit Filter match condition or Transaction Filter event condition was true.

Note Larger values indicate increased CPU time spent on hit analysis. Creating hit filters or transaction filters tends to increase this value.

Hit Analysis - Session Hit Filters ActiveReports number of hit filter facts defined by all active Sessions. One fact is created for every session for every hit filter defined.

Note Larger values indicate greater memory usage. Creating hit filters tends to increase this value.

Hit Analysis - Session TimeoutsReports number of completed sessions that stopped due to a timeout rule rather than the more specific Default Session Stop Condition.


Hit Analysis - Sessions ActiveReports number of session facts currently active in the Archiver rules engine used for hit analysis. This value increases as new session IDs are captured, and decreases as sessions stop. This indirectly measures the number of concurrent client browsers.

Note Larger values indicate increased memory usage and higher concurrency. A large number of sessions may also indicate timeouts that are set too short.

Hit Analysis - Sessions CompletedReports number of sessions that stopped and were written to the capture database.

Hit Analysis - Transaction Hit Filters ActiveReports number of hit filter facts defined by all active Transactions. One fact is created for every transaction for every dependent hit filter type matched.

Note Larger values indicate increased memory usage.

Hit Analysis - Transactions ActiveReports number of transaction facts currently active in the Archiver rules engine used for hit analysis. This value increases as start events are met for defined Transaction Filters, and decreases as transactions stop.

Note Larger values indicate increased memory usage and CPU time spent on hit analysis. Creating transaction filters tends to increase this value.

Hit Analysis - Transactions CompletedReports number of transactions (across all Transaction Filters) that stopped and were written to the capture database.


JVM Free MemoryReports unused memory within the Java Virtual Machine (JVM) running the Archiver. This value should oscillate as normal “garbage collection” occurs. A consistently low amount of free memory may indicate a problem.

Note The amount of free memory can change very rapidly as the JVM automatically resizes the heap.

JVM Total MemoryReports total heap memory allocated to the Java Virtual Machine (JVM) running the Archiver. The JVM automatically resizes the size of the heap in order to keep “garbage collection” times to a minimum.

Note This value can increase suddenly when a spike in capture traffic occurs. It decreases gradually over periods of relative inactivity.

JVM Used MemoryReports amount of memory currently in use within the Java Virtual Machine (JVM) running the Archiver. This value should oscillate as normal “garbage collection” occurs. A consistently high amount of used memory may indicate a problem.

Note The amount of used memory can change very rapidly as the JVM automatically resizes the heap.

Refresh Metric TimeReports time required to recalculate dynamic metrics, including: Available Database Aspects/Hits/Segments/Sessions, Capture Rate, Database Size, and all metrics in the Archiver Appliance category.

Note These dynamic metrics values are updated in the background every two minutes. A large sustained value indicates the appliance is under heavy load.


Segment Transfer AttemptsReports number of times this Archiver tried to transfer a database segment to/from a remote Storage Appliance. Since transferring segments involves a bidirectional handshake, attempts are tracked on both the Capture Appliance and Storage Appliance.

Note This value is zero if segment transfer is not enabled. Otherwise, the value should increase linearly as attempts are made.

Segment Transfer CompletionsReports number of times this Archiver successfully transferred a segment to/from a remote Storage Appliance. Since transferring segments involves a bidirectional handshake, attempts are tracked on both the Capture Appliance and Storage Appliance.

This value is zero if segment transfer is not enabled. Otherwise the value should equal Segment Transfer Attempts, if no retries or errors occurred.

Segments ClosingReports number of segments currently being closed by the Archiver.

Note This value should be between zero and one when the Archiver is healthy.

Segments CopyingReports number of segments currently being copied to a remote Storage Appliance. Copying the segment is the most expensive and longest phase of the segment transfer process.

Note This value should be between zero and two when the Archiver is healthy.

Segments Dropped Before TransferReports number of times a database segment was deleted from the local disk before it could be transferred to a remote Storage Appliance.

Note If segment transfer is enabled, this value should be zero, unless errors have occurred. Otherwise, this value increases linearly.


Segments Waiting to CopyReports number of database segments that have been closed and flushed, but where the copy to the remote Storage Appliance has not yet started.

Note If segment transfer is enabled, this value should be small, unless errors have occurred. Otherwise, this value increases linearly.

Segments Waiting to TransferReports number of database segments that have been copied to the remote Storage Appliance, but where the initialization and transfer has not yet completed.

Note If segment transfer is enabled, this value should be small, unless errors have occurred. Otherwise, this value increases linearly.

Shared Memory SizeReports total size of shared memory which is used to store raw hits prior to being sent to the Archiver for processing. Each TCPBatches has a limited number of hits that it can buffer.

Note This metric value does not change while the Archiver is running.

Shared Memory UsedReports the amount of shared memory consumed by raw hits waiting to be processed. Each TCPBatches has a limited amount of memory to buffer hits before being sent to the Archiver for processing, and when this buffer fills up TCPBatches stops reading batches from the network which may cause hits to be discarded.

Note TCPBatches should normally use very little shared memory, but short periodic bursts are normal. A sustained value above 25% of the shared memory size indicates the Capture Appliance is having difficulty keeping up with incoming traffic.


TCPBatches JVM Free MemoryReports unused memory within the Java Virtual Machine (JVM) running TCPBatches. This value should oscillate as normal “garbage collection” occurs. A consistently low amount of free memory may indicate a problem.


TCPBatches JVM Total MemoryReports total heap memory allocated to the Java Virtual Machine (JVM) running TCPBatches. The JVM automatically resizes the size of the heap in order to keep “garbage collection” times to a minimum.


TCPBatches JVM Used MemoryReports amount of memory currently in use within the Java Virtual Machine (JVM) running TCPBatches. This value should oscillate as normal “garbage collection” occurs. A consistently high amount of used memory may indicate a problem.


Write Buffers FailedReports the number of times a write buffer was lost due to an internal error. The support bundle includes all the error details.


Archiver Appliance MetricsThis Metrics Category provides information about the health of the Foglight Experience Viewer hardware running the Archiver (and the Server if one is configured). These Metrics are used to troubleshoot hardware and software problems.




Buffer Memory Used No

Cache Memory Used No

CPU I/O Wait Percent No

CPU Idle Percent No

CPU System Percent No

CPU User Percent No

Database Cache Blocks Available No

Database Cache Blocks Used No

Database Dirty Cache Blocks No

Database Read Cache Hit Rate No

Database Write Cache Hit Rate No

Filesystem Quest Used Percent No

Filesystem Root Used Percent No

Filesystem Tmp Used Percent No


Buffer Memory UsedReports amount of memory used by the appliance operating system to buffer disk writes.

Note This value is typically low even in high-volume installations. Archivers rely upon write buffering at the RAID controller, so there should be almost no write buffering in the operating system.

Cache Memory UsedReports amount of memory used by the appliance operating system for disk read caching.

Note The operating system uses free memory as a disk cache to improve I/O performance. A significant amount of memory (50% or more) may be allocated to disk caching, or this value may be as low as one-eighth of total memory.

Filesystem Var Used Percent No

Free Memory No

Processes Blocked No

Processes Running No

RAID Array Degraded No

Swap Memory Used No



CPU I/O Wait PercentReports percentage of CPU time spent waiting for disk operations, averaged over a five-second period when the Metric was last updated.

Note This value can spike up to 70%+ for very short periods of time (one observation), but should usually be 10% or less. A consistent high value indicates a drive failure or serious software problem.

CPU Idle PercentReports percentage of CPU time spent idle, averaged over a five-second period when the Metric was last updated.

Note A consistent low value indicates the appliance is heavily loaded.

CPU System PercentReports percentage of CPU time spent running system processes, averaged over a five-second period when the Metric was last updated.

Note This value should normally be 10% or less.

CPU User PercentReports percentage of CPU time spent running user processes, averaged over a five-second period when the Metric was last updated. These user processes include the Archiver and Server JVM processes.

Note A consistent high value indicates the appliance is heavily loaded.

Database Cache Blocks AvailableReports the number of block currently available in the database key buffer cache.

Database Cache Blocks UsedReports the maximum number of blocks that were allocated at one time from the database key buffer cache.


Database Dirty Cache BlocksReports the number of modified blocks in the database key buffer cache that have not been flushed to disk.

Database Read Cache Hit RateReports the read hit rate in the database key buffer cache.

Database Write Cache Hit RateReports the write hit rate in the database key buffer cache.

Filesystem Quest Used PercentReports percentage of space used on the “/quest” filesystem.

Filesystem Root Used PercentReports percentage of space used on the “/” filesystem.

Filesystem Tmp Used PercentReports percentage of space used on the “/tmp” filesystem.

Filesystem Var Used PercentReports percentage of space used on the “/var” filesystem.

Free MemoryReports amount of appliance operating system memory that is currently not used for any purpose.

Note It is normal to have almost no free memory, since the operating system tends to use any available memory either for disk read caching or write buffering.


Processes BlockedReports total number of processes that are waiting to run, averaged over a five-second period when the metric was last updated.

Note This value may be consistently zero for lightly loaded systems. Under heavy load, this value oscillates normally as I/O operations are synchronized. A consistent high value may indicate a threading problem.

Processes RunningReports total number of processes currently running, averaged over a five-second period when the Metric was last updated.

Note This value may be small for lightly loaded systems. Under heavy load, this value oscillates normally as I/O operations are synchronized.

RAID Array DegradedReports true if the performance of the appliance RAID array is currently degraded. This could be due to a disk failure, or because the RAID array is currently initializing or repairing in the background.

Swap Memory UsedReports amount of temporary swap memory used by the appliance operating system.

Note This value should always be zero, Archivers are configured to never swap.

Archiver Search Usage MetricsThis Metrics Category provides information about searches and other remote calls between the Server and the Archivers. These metrics indicate how heavily the FxV browser interface is being used.





Browser Types Calls No

Cities Calls No

Collector IDs Calls No

Content Calls No

Cookies Calls No

Countries Calls No

Fields Calls No

Hit Detail Calls No

Hit Detail Calls (empty) No

Hit Detail Calls (slow) No

HTML Titles Calls No

ISPs Calls No

Local Configuration Calls No

Metrics Calls No

Oldest Hits Calls No

Paths Calls No

Regions Calls No

Request Headers Calls No

Response Headers Calls No


Browser Types CallsReports number of requests for all Browser Types. This metadata is used in the FxV browser interface for searching and building filters.

Cities CallsReports number of requests for all Cities. This metadata is used in the FxV browser interface for searching and building filters.

Collector IDs CallsReports number of requests for all Collector IDs. This metadata is used in the FxV browser interface for searching and building filters.

Content CallsReports number of searches for hit content based on specific hit or session IDs, used for replay or export. Raw content returned by the Archiver is rewritten by the Server as needed for display purposes.

Search Hits Calls No

Search Hits Calls (empty) No

Search Hits Calls (slow) No

Search Sessions Calls No

Search Sessions Calls (empty) No

Search Sessions Calls (slow) No

Search Transactions Calls No

Search Transactions Calls (empty) No

Search Transactions Calls (slow) No

Web Servers Calls No



Cookies CallsReports number of requests for all Cookie names. This metadata is used in the FxV browser interface for searching and building filters.

Countries CallsReports number of requests for all Countries. This metadata is used in the FxV browser interface for searching and building filters.

Fields CallsReports number of requests for all Field names. This metadata is used in the FxV browser interface for searching and building filters.

Hit Detail CallsReports number of searches for hit details based on specific hit or session IDs, used for replay or export.

Hit Detail Calls (empty)Reports number of searches for hit details that returned zero results.

Hit Detail Calls (slow)Reports number of searches for hit details that took longer than 15 seconds to complete.

HTML Titles CallsReports number of requests for all HTML titles available. This metadata is used in the FxV browser interface for searching and building filters.

ISPs CallsReports number of requests for all ISPs. This metadata is used in the FxV browser interface for searching and building filters.

Local Configuration CallsReports number of requests for local Archiver configuration (hardware-specific settings).


Metrics CallsReports number of requests for raw metric values. The Server updates Metric History every two minutes, so a steady number of calls is normal.

Oldest Hits CallsReports number of requests for the “oldest hit”. This metadata used in the FxV browser interface when searching.

Paths CallsReports number of requests for all Paths. This metadata is used in the FxV browser interface for searching and building filters.

Regions CallsReports number of requests for all Regions. This metadata is used in the FxV browser interface for searching and building filters.

Request Headers CallsReports number of requests for all Request Headers. This metadata is used in the FxV browser interface for searching and building filters.

Response Headers CallsReports number of requests for all Response Headers. This metadata is used in the FxV browser interface for searching and building filters.

Search Hits CallsReports number of searches for captured hits.

Search Hits Calls (empty)Reports number of hit searches that returned zero results.

Search Hits Calls (slow)Reports number of hit searches that took longer than 10 seconds to complete.


Search Sessions CallsReports number of searches for captured sessions.

Search Sessions Calls (empty)Reports number of session searches that returned zero results.

Search Sessions Calls (slow)Reports number of session searches that took longer than 10 seconds to complete.

Search Transactions CallsReports number of searches for captured transactions.

Search Transactions Calls (empty)Reports number of transactions searches that returned zero results.

Search Transactions Calls (slow)Reports number of transactions searches that took longer than 10 seconds to complete.

Web Servers CallsReports number of requests for all Web Servers. This metadata is used in the FxV browser interface for searching and building filters.

Script MetricsThis Metric Category contains all Metrics concerning the Hit Analysis Scripts. These are dynamic metrics that are collected for each script defined.

Note They are displayed for individual Archivers, as well as for “Totals”.

The only metrics in this category are:

• <ScriptName> Errors—this metric reports how many times the script has failed (that is, exited with an error).

• <ScriptName> Execution Count—this metric reports how many times the script has executed.


Both metrics are reset to 0 at midnight and represent daily totals.

Collector MetricsYou can select a Collector to display its individual metrics, or “Totals” to display total metrics across all Collectors in the system.

Note Certain metrics are displayed only for individual Collectors.

The Collector metrics are currently grouped into one category, Collector Health.

Collector HealthThis Metric Category provides information about the health of any attached Collectors. These Metrics are primarily used to troubleshoot problems in the Foglight Experience Monitor, some have alarm conditions defined by default.




Clock Skew from Server No

Hits Captured Total Yes

Hits Discarded Total Yes

Hits Discarded - Dropped Packet Yes

Hits Discarded - Malformed URL Yes

Hits Discarded - Shared Memory Full Yes

Hits Discarded - Upload Queue Full Yes

Appendix: FxV Metrics 235Collector Metrics

Clock Skew from ServerReports the estimated number of milliseconds that the clock of the appliance running this Collector and the Foglight Experience Viewer Server are skewed.

Hits Captured TotalReports the total number of hits the Collector has sent to the Archiver, since the Collector was last restarted.

Hits Discarded TotalReports the total number of hits that were discarded at the Collector before being sent to an Archiver. There are a number of reasons why a hit may be discarded.

Hits Discarded - Dropped PacketReports the number of hits that were discarded at the Collector because a packet was dropped within the hit or SSL connection. This is typically because the Collector is overloaded.

Hits Discarded - Malformed URLReports the number of hits discarded at the Collector because of an unparseable or nonstandard URL.

Hits Discarded - Shared Memory FullReports the number of hits discarded at the Collector because the shared memory segment was full. Each Collector has a shared memory buffer used to transfer data, when shared memory fills up then incoming hits are discarded.

Hits In Progress No

Upload Queue - % Utilization No

Upload Queue - Hits Waiting to Upload No



Hits Discarded - Upload Queue FullReports the number of hits discarded at the Collector because the upload queue was full. Each Collector has a queue to buffer hits prior to being sent. When this queue fills up then hits are discarded.

Hits In ProgressReports the number of hits currently being captured by the Collector but not yet sent to an Archiver.

Upload Queue - % UtilizationReports the percentage of the upload queue (between the Collector and the Archivers) that is currently in use. Each Collector has a queue to buffer hits prior to being sent, when this queue fills up then hits are discarded.

Upload Queue - Hits Waiting to UploadReports the number of hits currently in the upload queue between the Collector and the Archivers. The Hits Captured metric is not incremented until the hit is removed from the upload queue.

Server MetricsThe Server metrics include the following categories:

• “Server Health” on page 237

• “Server Sessions” on page 239

• “Server Replay Cache” on page 239

• “Server Requests to Archivers” on page 240

Appendix: FxV Metrics 237Server Metrics

Server HealthThis Metric Category provides information about the internal health of the Server. These Metrics are primarily used to troubleshoot Server problems, some have alarm conditions defined by default.

Note These metrics do not reflect the health of the Web servers running your Web applications, only the Foglight Experience Viewer Server itself.

Database SizeReports size of Server database, used to store Metric History and system configuration.

Note Captured hits, sessions, and transactions are not stored in the Server database, which is relatively small.

JVM Free MemoryReports unused memory within the Java Virtual Machine (JVM) running the Server. This value should oscillate as normal “garbage collection” occurs. A consistently low amount of free memory may indicate a problem.


JVM Total MemoryReports total heap memory allocated to the Java Virtual Machine (JVM) running the Server. The JVM automatically resizes the size of the heap in order to keep “garbage collection” times to a minimum.



JVM Used MemoryReports amount of memory currently in use within the Java Virtual Machine (JVM) running the Server. This value should oscillate as normal “garbage collection” occurs. A consistently high amount of used memory may indicate a problem.


Last Event History Trimming Elapsed TimeReports time taken to trim old Alarm History data from the Server database. This action is performed in the background every 15 minutes.

Last Metric Update Elapsed TimeReports time taken to update the current values of all Metrics tracked by the Server. This action is performed in the background every two minutes.

Last Metric Update Value CountReports total number of Metrics currently being tracked by the Server. These metrics are updated in the background every two minutes.

Last Value History Trimming Elapsed TimeReports time taken to trim old Metric History data from the Server database. This action is performed in the background every 15 minutes.

Metadata Cache SizeReports number of items currently stored in the Server metadata cache. This metadata from the capture database is used to build lists of field, cookie, and header names for use in the FxV browser interface.

Open Database ConnectionsReports number of database connections currently open by the Server. A consistently high number of connections (especially when few users are logged in) may indicate a problem.

Appendix: FxV Metrics 239Server Metrics

Server SessionsThis Metric Category provides information about the Users currently logged into the Foglight Experience Viewer Server. These Metrics are primarily used to troubleshoot Foglight Experience Viewer Server problems.

Note These metrics do not reflect the activity on the Web servers running your Web applications, only the Foglight Experience Viewer Server itself.

Active Admin UsersReports number of Users with administrative privileges currently logged into the Server.

Active UsersReports number of Users currently logged into the Server, including administrators.

Server Replay CacheThis Metric Category provides information about the sequences (sessions or transactions) cached in memory by the Server. This cache improves performance by avoiding calls to the Archivers for captured data. These Metrics are primarily used to troubleshoot Server problems.

Hits in Replay CacheReports number of hits currently stored in the Replay Cache. This value normally increases and decreases as requests are made.

Missed Requests to the Replay CacheReports number of times the Server did not find a requested sequence (session or transaction) in the Replay Cache. The Server then had to call the Archivers for captured data.

Reloaded Requests to the Replay CacheReports number of times a requested sequence (session or transaction) was found in the Replay Cache, but the sequence had been cleared to reduce memory usage. The Server then had to call the Archivers for captured data.


Requests for SequencesReports total number of sequences (sessions or transactions) that have been requested from the Server.

Requests for Sequences Requiring Content ParsingReports number of sequences (sessions or transactions) requested that required the content to be parsed. This is typically because sequence contained at least one frameset. These sequences take longer to rewrite than sequences without frames.

Requests to the Replay CacheReports number of times the Server tried to find a sequence (session or transaction) in the Replay Cache.

Sequences in Replay CacheReports number of sequences (sessions or transactions) currently stored in the Replay Cache. This value normally increases and decreases as requests are made.

Server Requests to ArchiversThis Metrics Category provides information about the calls made by the Server to the Archivers for captured data and raw metric values. These Metrics are primarily used to troubleshoot Server or Archiver problems (especially search problems), some have alarm conditions defined by default.

Metrics Available From ArchiverReports if the Server was able to collect raw metric values from the specified Archiver. This is done in the background every two minutes as the Server updates Metric History.

If No (that is, if the Server is not able to collect raw metric values), this indicates a serious problem with the Archiver.

Searches for Hit DetailsReports the number of searches to the Archivers for hit details used for replay.

Appendix: FxV Metrics 241Error/Warning Metrics

Searches for Hit Details (Slow)Reports the number of searches to the Archivers for hit details (used during replay) that took more than 15 seconds to complete.

Searches for HitsReports the number of searches to the Archivers for captured hits.

Searches for Hits (Slow)Reports the number of searches to the Archivers for captured hits that took more than 10 seconds to complete.

Searches for SessionsReports the number of searches sent to the Archivers for captured sessions.

Searches for Sessions (Slow)Reports the number of searches to the Archivers for captured sessions that took more than 10 seconds to complete.

Searches for TransactionsReports the number of searches to the Archivers for captured transactions.

Searches for Transactions (Slow)Reports the number of searches to the Archivers for captured transactions that took more than 10 seconds to complete.

Error/Warning MetricsThe Error/Warning Metrics list contains all metrics that are currently in an “error” or “warning” state, as defined by their thresholds. There is no default “Error/Warning” set of metrics.

Only metrics which have thresholds defined (identified by the little View icon at the end of the metric row) are subject to appearing on this page.


ErrorsThis Metric Category contains all Metrics that currently have an Error status. These metrics are used to indicate serious problems that require immediate attention.

Note This summarizes all other metric categories.

WarningsThis Metric Category contains all Metrics that currently have a Warning status. These metrics are used to indicate potential problems that bear further investigation.

Note This summarizes all other metric categories.

C

Appendix: Glossary

This appendix provides the complete Foglight Experience Viewer terminology.

This appendix contains the following sections:

Component Architecture Terms .................................................................................................244Data Model Terms .....................................................................................................................250Product Feature Terms..............................................................................................................253Networking and Database Terms ..............................................................................................256


Component Architecture TermsThese terms are used when describing the following:

• Major hardware/software components and connections shown in the Foglight Named End User Configurations

• Component Architecture of the Foglight Experience Viewer

• Visual interface of the Foglight Experience Viewer

Term Definition

Agent A software component of the Foglight Experience Monitor that consumes raw HTTP packets, determines the session ID for each hit, and passes raw hit data to its local Collector to be transmitted to a Foglight Experience Viewer.

Analysis Repository

Secondary database that offers long-term storage of session and transaction data captured by FxV.

appliance group

Generic term for a collection of appliances (Foglight Experience Monitors and Foglight Experience Viewers), installed side-by-side or federated across different datacenters, where the intent is to unify appliances together for searching and reporting purposes.Every appliance group has one (and only one) Server component running. Data from multiple appliance groups can be combined in Foglight, but the appliance groups do not directly know about each other, and represent completely independent systems.

Appendix: Glossary 245Component Architecture Terms

Archiver A software component of the Foglight Experience Viewer that accepts hits from a local Tcpbatches component (through a shared memory interface), and writes user sessions into the database for later searching and reporting.The Archiver performs real-time analysis with Hit Filters and Transaction Filters that profile and alert upon end user behaviors and problems. Multiple Archivers can be configured in parallel (on separate Foglight Experience Viewer hosts) to scale up for large capture loads.The Archiver maintains its subset of the capture database on a local disk, and manages the transfer of database segments to the storage tier (if present). The Archiver responds to requests by the Server for search results or detailed session data. Each Archiver has a complete view of each individual user session it receives. If one Archiver of several fails, then its user sessions are simply unavailable until the Archiver is repaired, but the rest of the capture database can be used.

browser interface

The main visual interface to the Foglight Experience Viewer, a local Web application that runs as part of the Server software component. The browser interface is used to configure the appliance and to perform searches, and for user session reporting and visualizations (including visual replay). The browser interface also supports workflows where customers drill down from the Foglight or Foglight Experience Monitor interfaces.

capture database

General term for the searchable repository formed by all available Archiver and Storage components (across all available Foglight Experience Viewers). The capture database is a “distributed database” that is intelligently partitioned and managed across multiple appliance hosts.

capture tier Generic term for one or more Foglight Experience Viewers used to capture hits and store user sessions (a collection of Archiver components). The capture tier is wholly dedicated to the task of capturing data when a storage tier is present.

Collector A software component of the Foglight Experience Monitor that accumulates hits from its local Agent component (where raw HTTP packets are consumed), and then sends hits across a network socket to the Tcpbatches listener on the Foglight Experience Viewer. Within the Foglight Experience Monitor, data is transferred from the Agent to the Collector through a local shared memory interface.

Term Definition


Collector Group

Generic term for one or more Collector components (running on multiple Foglight Experience Monitors). Collector Groups, which are defined in the Foglight Experience Viewer browser interface, are primarily used to define configuration parameters—each Collector polls the Foglight Experience Viewer Server for load-balancing and security configuration needed for data capture (including the correct private route for monitored traffic). Collector Groups are also used to roll-up health metrics and to alert on capture errors or other data quality problems.

console See “browser interface” on page 245.

crossover cable

A physical cable that is often used to transfer hits from the FxM to FxV, or database segments between FxV appliances as part of a storage tier configuration. A crossover cable is generally considered more secure than sending sensitive data through a private switch.Note A standard (straight) network cable can be used in place of a crossover cable,

since the network interfaces on both FxV and FxM appliances support auto polarity correction.

database See “capture database” on page 245.

DB See “capture database” on page 245.

end user The visitor who makes use of the Web server, whose hits and pages are monitored by the Foglight Experience Monitor, and whose user session is captured and stored by the Foglight Experience Viewer.

Foglight Experience Monitor

Related appliance-based product that provides the Foglight Experience Viewer with data, also licensed and sold with Foglight for user performance and user session analysis. The Foglight Experience Viewer cannot be sold without the Foglight Experience Monitor.

Foglight Experience Monitor Agent

See “Agent” on page 244.

Term Definition


Foglight Experience Monitor browser interface

This is the main visual interface to the Foglight Experience Monitor, a Web application that runs as part of its Apache software component.

Foglight Experience Viewer

An appliance-based product (hardware and software) licensed and sold with Foglight as part of its user session analysis capability. This term refers somewhat interchangeably to the total package, the specific Dell or MBX hardware, and/or the general collection of software components running on the hardware. The appliance is imaged at the factory, and includes a Linux operating system (Suse Novell Enterprise Linux 10) with all other supporting components pre-loaded.

Foglight Experience Viewer Archiver

See “Archiver” on page 245.

Foglight Experience Viewer browser interface

See “browser interface” on page 245.

Foglight Experience Viewer Server

See “Server” on page 248.

Foglight Experience Viewer setup menu

See “setup menu” on page 248.

Foglight Experience Viewer Storage

See “Storage” on page 249.

Foglight user A customer who has an account for Foglight and the Foglight Experience Viewer, who wants to analyze and report on user sessions.

Term Definition


private switch A switch or hub that is used for interconnections between the end user appliances and the Foglight host. This does not necessarily mean that there is a single switch device dedicated to Foglight. It is common to create a private subnet on one or more shared switches to implement this concept in the field.

Rsync A software component that runs on the Foglight Experience Viewer that is used to transfer database segments from the capture tier to the storage tier. Rsync is not proprietary to Quest but is a standard preconfigured Linux service. Rsync operations are managed by the Archiver and Storage components, as part of the segment transfer process (configured from the setup menu).

SAN Acronym for storage area network. See “storage area network” on page 249.

Server A software component of the Foglight Experience Viewer that hosts the browser interface and provides the central point of control and configuration. The Server is used to define users, User Groups, Hit Filters, Transaction Filters, and all other Foglight Experience Viewer configuration data. The Archiver, Storage, and Collector components poll the Server for configuration periodically. When a user performs a search, it is the Server that contacts all available Archiver and Storage components, and then returns a merged view of the search results. The Server maintains all metrics related to user sessions and pushes metric data up to Foglight.

setup menu A text-menu program on the Foglight Experience Viewer that is used to manage all appliance-local settings, including network cards, time synchronization, firewall, disk encryption, and component services. This is accessed by logging in as the setup user locally, or by using the su setup command (switch user) from the support shell.

SSH A software component that optionally runs on the Foglight Experience Viewer to allow command-line remote access for administrative or management tasks. SSH is not proprietary to Quest but is a standard Linux service. An SSH client (such as Putty) is needed to connect to the SSH service. For security reasons, SSH access is not allowed by default, but can be turned on through the setup menu.

Term Definition


Storage A software component of the Foglight Experience Viewer that accepts database segments from another appliance for searching and long-term maintenance. The Storage component has a search API that is identical to the Archiver, but Storage cannot be used to capture hits like the Archiver. Ordinarily the portion of the capture database managed by the Storage component is physically located on a SAN, rather than local disk.

storage area network

A collection of storage disks that is made available to the Foglight Experience Viewer by physically installing a HBA card (QLogic preferred) and attaching a fiber connection to the storage array. Dell/EMC is Quest’s preferred vendor for SAN equipment. SAN installations are not limited by the number of drives installed on the Foglight Experience Viewer (450 GB/host) but can scale out to 24 TB/host or more.

storage tier Generic term for one or more Foglight Experience Viewers used for long-term storage and other advanced features (a collection of Storage components). The storage tier provides dedicated resources for searching and reporting that do not compete with real-time activities on the capture tier. Without a storage tier, searches and other use of the browser interface is always prioritized below the real-time capture of user sessions (which can be very demanding). A POC configuration (such as Antero) does not include a storage tier. Customers with a large number of concurrent users performing searches, requiring very fast searching in general (such as in call centers), or needing SAN integration should definitely have a storage tier. The Server should run on the storage tier whenever possible to free up resources for capture.

Tcpbatches A software component of the Foglight Experience Viewer that listens for hits sent by the Collector component of the Foglight Experience Monitor. Tcpbatches opens and listens on a TCP socket (port 7623 on the Foglight Experience Viewer) and when hit data is received, it forwards the hit data to its local Archiver through a shared memory interface.

Web server An installation of Apache, IIS, iPlanet, or other popular software used to host a customer Web site.

Term Definition


Data Model TermsThe data model terms are used to describe the followings:

• Data captured by the Foglight Experience Viewer

• Data models materialized by the Foglight Experience Viewer for searching and visualization features

• Visual interface of the Foglight Experience Viewer

Term Definition

aspect Aspects are the most basic data type in the database, used to encode hit, session, and transaction data elements. Aspects can represent numbers, de-duplicated strings, and other data types such as event or filter matches. Each aspect (such as hit request field X, or session custom field Y) consumes 2-6 tables within the database. This term is used only in a monitoring and health context, metric names in particular. A large number of aspects is one indication of data pollution, meaning that the collection of aspects grows without bounds.

database See “capture database” on page 245.

database segment

The database is divided into segments by time period. Each database segment can contain up to 24 hours or 1.5M hits of information. Each Archiver has only one of these segments open for writing at a time; this segment is called open segment. The remainder of the closed segments are accessed only when searching. This term is used primarily in monitoring and health context for attribute names, and when describing the transfer of data between the capture tier and storage tier.

captured hit A hit that was successfully captured into the database, and is online for searching and reporting.

discarded hit A hit that was lost prior to transmission to an Archiver, or lost by an Archiver that was overloaded.

event See “transaction event” on page 253.

failed hit A hit that could not be captured because an internal error occurred on the appliance.

Appendix: Glossary 251Data Model Terms

hit A hit is an HTTP request from an end user to a Web server, and the resulting HTTP response. A hit has a URL, response code, headers, and other data defined in the HTTP protocol. No two hits are identical. Every hit includes a Session ID, assigned by the Agent that gathered the hit data on the Foglight Experience Monitor. The Session ID is typically based on cookies set by the Web application. A “Web page” typically consists of several hits: an HTML hit that defines the page, and a series of non-HTML hits for page resources (like images and style sheets).

hit custom fields

Numeric and string fields generated by Hit Filters and stored with the hits for easy searching later. Clearly distinguished from HTTP request fields.

hit details The collection of all data elements (and their values) that make up a hit: Captured HTTP length, Client IP, Cookies, End-To-End Time, Client Time, Processing Time, Server IP, FxM RefererID, FxM UrlID, Exception, Monitor ID, Request Browser, Request Field, Request Header, Request Method, Request Path, Request Referer, Request Server, Request URL, Request Type, Response Code, Response Contains Frameset, Response Contains HTML tag, Response Content, Response Content Type, Response Header, Response HTML Title, Hit Identifier, Batch ID, Session ID, Custom Fields.

hit dropped by Hit Filter

A hit that was intentionally dropped by a Hit Filter that indicated the hit should not be stored. Metrics will still be updated for a dropped hit.

hit status The final status (OK, Error, Warning) of a hit after all Hit Filters were applied. A hit has a status of OK unless any problems are noted.

hit type Whether the hit contains HTML or non-HTML content.

marked hit A hit found to match a particular Hit Filter.

metric A metric is an instance of time-series data used to indicate and track health or performance factors, and to trigger actions when thresholds are breached. Examples of metrics include the value of abandoned shopping carts per day in dollars, or the total size of the database in TB.

metric update An individual change to a metric that is recorded in the database for later searching. Searching metric updates is an easy way to find user sessions that impacted a particular business indicator.

Term Definition


session A session is a set of hits (all having the same Session ID at the Agent that gathered the hit data) that represents the interaction of the end user with the Web server. A session is uniquely identified by its Session ID, Start Time (the earliest hit timestamp in the session), and Last Hit Time (the latest hit timestamp). A session begins when the first hit is seen for a given Session ID, and ends after a time-out period or other stop conditions. An active session is one that has started, but not yet stopped or timed out.

session custom fields

Numeric and string fields generated by Hit Filters or Transaction Filters and stored with the session for easy searching later.

session details The collection of all data elements (and their values) that make up a session: Session ID, Start Time, Last Hit Time, Time of First Error, Time of Last Error, Total Hits, HTML Hits, Total Errors, Total Warnings, HTML Errors, HTML Warnings, Total Transactions, Transaction Errors, Transaction Warnings, End-To-End Time, Client Time, Processing Time, City, Region, Country, ISP, Subnet, BrowserType, Username, Session Custom Fields.

sessionless hit A hit where a Session ID was not specified by the Agent, usually indicating that there was no sessionizing rule defined by the Foglight Experience Monitor that led to a meaningful result. The details of these hits can be inspected, but there is no user session for reporting or visual replay.

transaction A transaction is a portion of a session that matched a particular set of events defined by a Transaction Filter. Like a session, a transaction is uniquely identified by its Session ID, Start Time, Last Hit Time, and its transaction name. An example of a transaction might be “Buy a Book”, where the end user is tracked through an entire e-commerce interaction. Each transaction has a status based on its filter definition and a list of matched events.

transaction custom fields

Numeric and string fields generated by Transaction Filters and stored with the transactions for easy searching later.

transaction details

Similar to session details (see “session details” on page 252), but for transactions. Includes transaction events and transaction custom fields.

Term Definition

Appendix: Glossary 253Product Feature Terms

Product Feature TermsThese terms are used for describing the following:

• Specific marketable features provided by the Foglight Experience Viewer

• Configurable elements of the Foglight Experience Viewer

transaction event

Each transaction has a list of events that describe activity in that particular instance. An event is a checkpoint that occurred, did not occur, or occurred multiple times within the transaction. Event definitions are maintained as part of a Transaction Filter.

user session See “session” on page 252.

Term Definition

Term Definition

Custom Search Screen

A Custom Search Screen is a simplified search screen intended for less technical users or for frequently used searches. Each Custom Search Screen is based on a Saved Hit Search, a Saved Session Search, or a Saved Transaction Search (see “Saved Search” on page 254).

Hit Analysis Refers to the process by which an Archiver analyzes each hit according to rules configured on the Hit Analysis screen. This includes the processing and evaluation of Hit Filters, Transaction Filters, Special Transaction Events, Scripts, Sensitive Hit Details, Sensitive Response Content Expressions, Session Configuration settings, and Archiver Configuration settings. It results in updates to hit, session, and transaction data including the setting of Custom Fields and Metrics as well as updates to other hit, session, and transaction details.


Hit Filter A collection of conditions and actions used to analyze hits as they are captured. Hit Filters can be used to detect and alert on any per-hit conditions, to mark interesting hits for later searching, or to manage hit/session storage.Hit Filters are used to define events within Transaction Filters. Hit Filters can use powerful regular expressions or Scripts to extract data from any hit details. Hit Filters give a helpful and relevant context to the captured data to make searching and reporting easier.

Hit Search Find hits based on a collection of search criteria including custom fields, metric updates, and Hit Filter matches.

Metadata Information maintained in the database to make searching or filter creation easier, such as lists of the field names or Web servers that have been captured.

Preference Group

A group of users with common default user preference settings. Within a Preference Group, you can define which values can and cannot be changed by the users in the group. A user belongs to only one Preference Group.

Resource A general term for an object configured in the system and owned by a single user of the system. Resources are assigned to Resource Groups which allow access to them to be restricted based on User Groups. Resources include: Custom Search Screens and Saved Searches.

Resource Group

A collection of Resources that can be associated with one or more User Groups in order to grant users access to those resources.

Saved Search Used to save and quickly retrieve frequently used search criteria. Saved Searches can also be used to define search phases for Custom Search Screens.

Script A snippet of custom logic written in the Groovy scripting language. A Script is executed by a Hit Filter or Transaction Filter, which can use the output variables published by the Script to update custom fields or metrics. Scripts can perform advanced tasks, including: evaluating complex match expressions, transforming hit data, applying complex regular expressions, or integrating with external systems.

Term Definition

Appendix: Glossary 255Product Feature Terms

Sensitive Hit Detail

A specification of one or more request fields, request headers, response headers or cookies identified as “sensitive” (for example, fields used to submit credit cards or passwords). A Sensitive Hit Detail is not displayed to users in search results or replay. Instead, the data is replaced with alternate configurable text or left blank.

Sensitive Response Content Expression

A specification of a regular expression used to parse response content and remove “sensitive” data (for example, credit card numbers rendered within the page body by the Web server). These expressions are evaluated prior to pages being replayed, in order to avoid displaying sensitive data to the user.

Session Combining

Merging multiple sessions together during a visual replay (not at capture time). Used as a last resort when there are sessionizing problems.

Session Search

Find sessions based on a collection of search criteria including custom fields, metric updates, Hit Filter matches, errors found, or transactions completed.

Transaction Filter

A collection of conditions and actions used to analyze sessions as they are captured, and to identify the distinct transactions within those sessions. Transaction Filters can be used to detect and alert on any activity within a session, and to calculate custom metrics across aggregated sessions or transactions. A Transaction Filter can detect multiple events that provide information about site problems and client activity within a related set of pages. Transaction Filters give a helpful and relevant context to the captured data to make searching and reporting easier.

Transaction Search

Find transactions based on a collection of search criteria built from the filter definition, including custom fields, events, and metric updates if they are defined.

User Group Defines permissions for users logging into the browser interface. Every user belongs to at least one User Group.

Term Definition


Networking and Database TermsThese terms are used to define various networking and database concepts.

Term Definition

DNS Domain Name System. Is the way that Internet domain names are located and translated into Internet Protocol addresses. A domain name is a meaningful and easy-to-remember “handle” for an Internet address.Because maintaining a central list of domain name/IP address correspondences would be impractical, the lists of domain names and IP addresses are distributed throughout the Internet in a hierarchy of authority. There is probably a DNS server within close geographic proximity to your access provider that maps the domain names in your Internet requests or forwards them to other servers in the Internet.

Ethernet The most widely-installed local area network (LAN) technology. Specified in a standard, IEEE 802.3, Ethernet was originally developed by Xerox and then developed further by Xerox, DEC, and Intel.

HBA Host Bus Adapter. Is a circuit board and/or integrated circuit adapter that provides input/output (I/O) processing and physical connectivity between a server and a storage device. Because the HBA relieves the host microprocessor of both data storage and retrieval tasks, it can improve the server's performance time. An HBA and its associated disk subsystems are sometimes referred to as a disk channel.

HTTP Hypertext Transfer Protocol. Is the set of rules for transferring files (text, graphic images, sound, video, and other multimedia files) on the World Wide Web. As soon as a Web user opens their Web browser, the user is indirectly making use of HTTP. HTTP is an application protocol that runs on top of the TCP/IP suite of protocols (the foundation protocols for the Internet).

NIC Network Interface Card. Computer circuit board or card that is installed in a computer so that it can be connected to a network.

NTP Network Time Protocol. Is a protocol that is used to synchronize computer clock times in a network of computers. In common with similar protocols, NTP uses Coordinated Universal Time (UTC) to synchronize computer clock times to a millisecond, and sometimes to a fraction of a millisecond.

Appendix: Glossary 257Networking and Database Terms

SCP Secure Copy Protocol. Secure copy is a means of securely transferring computer files between a local and a remote host or between two remote hosts, using the Secure Shell (SSH) protocol.

SSH Secure Shell, sometimes known as Secure Socket Shell, is a Unix-based command interface and protocol for securely getting access to a remote computer. It is widely used by network administrators to control Web and other kinds of servers remotely. SSH is actually a suite of three utilities - slogin, ssh, and scp - that are secure versions of the earlier UNIX utilities, rlogin, rsh, and rcp. SSH commands are encrypted and secured in several ways. Both ends of the client/server connection are authenticated using a digital certificate, and passwords are protected by being encrypted.

TCP/IP Transmission Control Protocol/Internet Protocol. Is the basic communication language or protocol of the Internet. It can also be used as a communications protocol in a private network (either an intranet or an extranet). When you are set up with direct access to the Internet, your computer is provided with a copy of the TCP/IP program just as every other computer that you may send messages to or get information from also has a copy of TCP/IP.

URL Uniform Resource Locator (previously Universal Resource Locator). Is the unique address for a file that is accessible on the Internet. A common way to get to a Web site is to enter the URL of its home page file in your Web browser's address line. However, any file within that Web site can also be specified with a URL. Such a file might be any Web (HTML) page other than the home page, an image file, or a program such as a common gateway interface application or Java applet. The URL contains the name of the protocol to be used to access the file resource, a domain name that identifies a specific computer on the Internet, and a pathname, a hierarchical description that specifies the location of a file in that computer.

WAN Wide Area Network. Is a geographically dispersed telecommunications network. The term distinguishes a broader telecommunication structure from a local area network (LAN). A wide area network may be privately owned or rented, but the term usually connotes the inclusion of public (shared user) networks.

Term Definition


XML Extensible Markup Language. Is a flexible way to create common information formats and share both the format and the data on the World Wide Web, intranets, and elsewhere. XML can be used by any individual or group of individuals or companies that wants to share information in a consistent way.

Term Definition

Index

Aabout Foglight 10about Foglight Experience Viewer 10about Quest Software 15account information, setting preferences 72advanced administration

appliance maintenance 168Archivers 166Collectors 167Foglight integration 169metrics 172overview 165Server configuration 170superuser tasks 168

Analysis Repositoryaccessing

from Foglight 161from Toad for MySQL 161

architecture 151configuration 154data access 161data transfer 160detailed configuration 159initial setup 154overview 150session table 151transaction tables 153

analyzing, session 21appliance

advanced administration 165

architecture 18browser interface 28data model 22metrics 199overview 17terminology 243

Archiver configuration 125Archivers, administering 166

Bbrowser interface

automatic login via Foglight 29logging in 29screen elements 30tips 34

Ccaptured metadata, overview 117Collectors, administering 167Combined Sessions view 63component architecture terms 244configuration change log 128configuration import/ export 130configuring

Analysis Repository 149custom fields 113hit analysis process 79Hit Filters 90metrics 115


Sever 170special events 107storage settings 87Transaction Filters 105

contacting Quest 15Content Replay view 57creating

Custom Search Screens 67Saved Searches 66

custom fieldsadditional information 115attributes 109configuration 113overview 23, 108setting values 82

Custom Search Screenscreating 67defining 65editing 67performing a custom search 49

Ddata model

custom fields 23hits 24metrics 24overview 22sessions 26terms 250transactions 27

definingCustom Search Screens 65hit storage restrictions 87session storage restrictions 88

detailed configuration, Analysis Repository 159Details view 53documentation

cartridge 13core 13feedback 14

FxV 13suite 12

Eediting

Custom Search Screens 67Saved Searches 66search settings 75Session Explorer settings 72

export configuration 130

FFoglight integration 169

Hhit analysis

captured metadata 117configuration change log 128configuration options 125configuring process 79custom fields 108examples 133

Buy Tunnel 138Event Occurrence Counting 134Parametrized Metrics 137Parsing HTML Content 136Session-Level Event Occurrence Counting 135Simple Data Extraction 134

export configuration 130Hit Filters 80import configuration 130metrics 115performing 20scripts 118sensitive data protection 121special events 106Transaction Filters 92

Hit Filtersactions 81

Index 261

additional information 92configuration 90configure storage settings 87execute scripts 82increment metrics 86match conditions 80match conditions, defining 81match conditions, grouping 81overview 19, 80set custom field values 82

Hit Inspector view 55hit search, overview 43hit status, updating 89hit storage

defining restrictions 87hits

overview 24processing 18

https 29

Iimport configuration 130initial setup, Analysis Repository 154

JJava regular expressions

in FxV Hit Analysis 187overview 188testing in FxV 198using in FxV 192

Mmanaging

preference groups 184resource groups 183user account settings 175user and user groups 176

match conditionsdefining 81

grouping 81menu bar

Content Replay 59metrics

additional information 117Archiver 200Collector 234complete list 199configuration 115Error/Warnings 241incrementing 86overview 24, 115running a metric-based diagnostic 172Server 236viewing 172

Nnetworking terms 256

Pperforming

hit analysis 20searches 20

processing, hits 18product feature terms 253

Rregular expressions

assigning custom field or metric values 193extracting a portion of a hit detail 194extracting a single value 193greedy vs. reluctant 191group numbers 196identifying Sensitive Response Content 195looking for “matches” 192multiple matches 195mutiline matching 197using “matches” in Hit Filter match conditions 192


using “matches” in Sensitive Hit Detail specifications 192

SSaved Searches

creating 66editing 66overview 48

scriptsadditional information 120executing 82overview 118

search settings, editing 75searches

hit search 43performing 20, 35result limits 48saved searches 48session search 38time constraints 37transaction search 40

sensitive data protectionoverview 121sensitive hit details 121sensitive response content expressions 123

sensitive hit details 121sensitive response content expressions 123session configuration 125Session Explorer

Combined Sessions view 63Content Replay view 57Details view 53Hit Inspector view 55Timeline view 54toolbar 52Transactions view 61

Session Explorer settings, editing 72session search, overview 38session storage

defining restrictions 88

session tabledata types 156overview 151

session, analyzing 21sessions, overview 26setting

account information preferences 72custom fields values 82user preferences 71

Special Eventsadditional information 108configuration 107

special events, overview 106SSL 29storage

configuring settings 87mark hit 88update hit status 89

superuser tasksappliance maintenance 168Foglight integration 169overview 168Server configuration 170

support 16

Ttechnical support 16terms

component architecture 244data model 250networking 256product feature 253

text conventions 14time constraints, searches 37Timeline view 54toolbar

Session Explorer 52transaction event actions 97

changing transaction/session state 98executing scripts 98

Index 263

incrementing metrics 103setting custom field values 99setting transaction status 97

transaction event match conditions 94always true 96grouping 97hit count by Hit Filter 95Hit Filter match 94Hit Filter ordering 95post transaction stopped by global stop condition 96post transaction stopped by transaction stop

condition 97script output 95session custom field 95session/transaction duration 96session/transaction hit counts 95transaction custom field 95

Transaction Filtersadditional information 106configuration 105event match conditions 94overview 19, 92storage configuration 104transaction event actions 97transaction events 93

transaction search, overview 40transaction tables

data types 158overview 153

Transactions view 61transactions, overview 27

Uupdating, hit status 89user account

managing resource groups 183managing settings 175managing user and user groups 176preference groups 184

user groups

managing 182pre-defined 181

user preferencesaccount information 72search settings 75Session Explorer settings 72setting 71

user sessions, analyzing 51users

group settings 178managing user accounts 177pre-defined user accounts 176preference settings 176

Vviews

Combined Sessions 63Content Replay 57Details 53Hit Inspector 55Timeline 54Transactions 61

Documents

Fog Light Experience Viewer Guide