Upload
jermaine-mason
View
224
Download
0
Embed Size (px)
Citation preview
7/21/2019 Reporter Administrators Guide 9.x.b
1/134
Blue CoatSystemsReporter 9.x
Administrators Guide
Reporter Versions 9.4.x
7/21/2019 Reporter Administrators Guide 9.x.b
2/134
Blue Coat Reporter 9.x Administrators Guide
ii
Contact Information
Americas:Blue Coat Systems Inc.420 North Mary AveSunnyvale, CA 94085-4121
Rest of the World:Blue Coat Systems International SARL3a Route des Arsenaux1700 Fribourg, Switzerland
http://www.bluecoat.com/contact/customer-support
http://www.bluecoat.com
For concerns or feedback about the documentation:[email protected]
http://www.bluecoat.com/mailto:[email protected]:[email protected]://www.bluecoat.com/7/21/2019 Reporter Administrators Guide 9.x.b
3/134
iii
Copyright 1999-2013 Blue Coat Systems, Inc. All rights reserved worldwide. No part of this document may be reproduced by any means
nor modified, decompiled, disassembled, published or distributed, in whole or in part, or translated to any electronic medium or othermeans without thewritten consent of Blue Coat Systems, Inc. All right, title and interest in andto theSoftware anddocumentation areandshall remain the exclusive property of Blue Coat Systems, Inc. and its licensors. ProxyAV, ProxyOne, CacheOS, SGOS, SG,Spyware Interceptor, Scope, ProxyRA Connector, ProxyRA Manager, Remote Access and MACH5 are trademarks of BlueCoat Systems, Inc. and CacheFlow, Blue Coat, Accelerating The Internet, ProxySG, WinProxy, PacketShaper, PacketShaperXpress, PolicyCenter, PacketWise, AccessNow, Ositis, Powering Internet Management, The Ultimate Internet SharingSolution, Cerberian, Permeo, Permeo Technologies, Inc., and the Cerberian and Permeo logos are registered trademarks of BlueCoat Systems, Inc. All other trademarks contained in this document and in the Software are the property of their respective owners.
BLUE COAT SYSTEMS, INC. AND BLUE COAT SYSTEMS INTERNATIONAL SARL (COLLECTIVELY BLUE COAT) DISCLAIM ALLWARRANTIES, CONDITIONS OR OTHER TERMS, EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, ON SOFTWARE ANDDOCUMENTATION FURNISHED HEREUNDER INCLUDING WITHOUT LIMITATION THE WARRANTIES OF DESIGN,MERCHANTABILITYOR FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL BLUECOAT,ITS SUPPLIERS OR ITS LICENSORS BE LIABLE FOR ANY DAMAGES, WHETHER ARISING IN TORT, CONTRACT OR ANY OTHERLEGAL THEORY EVEN IF BLUE COAT SYSTEMS, INC. HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Americas: Rest of the World:Blue Coat Systems, Inc. Blue Coat Systems International SARL
420 N. Mary Ave. 3a Route des Arsenaux
Sunnyvale, CA 94085 1700 Fribourg, Switzerland
Document Number:Document Revision: Reporter 9.4.1: 11/2012
7/21/2019 Reporter Administrators Guide 9.x.b
4/134
Blue Coat Reporter 9.x Administrators Guide
iv
7/21/2019 Reporter Administrators Guide 9.x.b
5/134
iii
Contents
Chapter 1: Preface
About This Document ........................................................................................................................7Document Conventions .....................................................................................................................7Notes and Warnings...........................................................................................................................7Navigating This Document ............................................................................................................... 8
Chapter 2: Reporter Concepts
Chapter Contents ................................................................................................................................ 9Reporter Overview ............................................................................................................................. 9About the Reporter Architecture......................................................................................................9
About the Page View Combiner.....................................................................................................10Deployment Overview.....................................................................................................................12
Standard FTP DeploymentOne Server................................................................................12Standard FTP DeploymentTwo Servers..............................................................................14Direct ProxySG Streaming Deployment ................................................................................. 15Download Access Log Data from the Blue Coat Cloud........................................................16
About Optimizing Log Processing Configurations .....................................................................17About Access Log Naming Conventions................................................................................17About Chronological Ordering ................................................................................................19About Known Conditions for Efficiency/In-efficiency........................................................19About Database Purging ........................................................................................................... 20
About the Default Browse Time Calculations .............................................................................. 20The Page View Criteria Used for Browse Time .....................................................................20Examples......................................................................................................................................21
Report Field/Log Field Names.......................................................................................................21Log Field Best Practices ............................................................................................................. 22Main Logs .................................................................................................................................... 23
Reports/Log Field Matrix................................................................................................................24Main Log Field Matrix...............................................................................................................26Web Application Reports..........................................................................................................28Video Usage Reports..................................................................................................................28
Chapter 3: Administrative Tasks
How Do I...? .......................................................................................................................................31
Section A: Reporter Administration Tasks
Linux Root User Installation Procedure ........................................................................................ 32Uninstalling Reporter.......................................................................................................................33
http://-/?-http://-/?-7/21/2019 Reporter Administrators Guide 9.x.b
6/134
Blue Coat Reporter 9.x Administrator Guide
iv
About the Reporter Improvement Program.................................................................................33Securing the Reporter Web Server Transport Protocol...............................................................34
Default Certificate ...................................................................................................................... 34
Selected Certificate.....................................................................................................................35Connecting Reporter to E-mail Servers.........................................................................................38Creating ProxySG Policy That Backs Up Access Log Files.........................................................38Processing Log Files With Encoded Spaces in User Names.......................................................43Process Access Logs From the Blue Coat Cloud Service ............................................................ 44
Prerequisites................................................................................................................................44ThreatPulse Configuration .......................................................................................................44Reporter Configuration.............................................................................................................45Create a Database or Assign Log Source................................................................................47
Section B: Reporter Performance Best Practices
Basic Best Practices ........................................................................................................................... 48Reporter Server...........................................................................................................................48Log Processing............................................................................................................................48Managing Data ........................................................................................................................... 49Reporter System Maintenance..................................................................................................49
About Log File Names ..................................................................................................................... 49About UNC Paths.............................................................................................................................50
Section C: Advanced Filtering Tasks
Filtering Based on Custom Text File Contents ............................................................................. 51Filtering Based on IP Addresses with CIDR Notations .............................................................. 51
Section D: TroubleshootingHow do I...?........................................................................................................................................53About Compression Modes in a Direct ProxySG Connection Deployment............................ 53Configuring Reporter to Send Alerts.............................................................................................54Uploading System Diagnostics to Blue Coat................................................................................55Reviewing Reporter Event Logs.....................................................................................................55Troubleshooting HTTPS Configuration on Linux ....................................................................... 57
Chapter 4: Managing User Access to Reporter
About Users.......................................................................................................................................59
Section A: Planning the Role-Based Access
Recommended Database Fields for Roles.....................................................................................60Determining Role Access.................................................................................................................62
Section B: Configuring Reporter Role-Based Access
Section C: Authenticating Users With LDAP
About LDAP and Reporter..............................................................................................................64About LDAP Nested Group Support ............................................................................................ 65
7/21/2019 Reporter Administrators Guide 9.x.b
7/134
Contents
v
LDAP Procedure...............................................................................................................................66Use Case: Role-based Access for Managers Viewing Direct Report Data................................67
Section D: Auditing Reporter Users
Chapter 5: Web API
About the Web API ..........................................................................................................................71Additional Support...........................................................................................................................71About Web API Endpoints..............................................................................................................71
Security Requirements...............................................................................................................72Downloading Reports ...............................................................................................................72Parameter Syntax .......................................................................................................................72
Common Parameters........................................................................................................................72Parameter: username.................................................................................................................72Parameter: password ................................................................................................................. 73Parameter: reportId....................................................................................................................73Parameter: responseFormat......................................................................................................73
End Point: /api/create.....................................................................................................................74Parameter: database...................................................................................................................75Parameter: role............................................................................................................................75Parameter: label..........................................................................................................................75Parameter: format.......................................................................................................................75Parameter: summarizeBy..........................................................................................................76Parameter: columns ................................................................................................................... 76Parameter: rows..........................................................................................................................76
Parameter: sort............................................................................................................................76Parameter: action........................................................................................................................77Parameter: filterN.......................................................................................................................77Parameter: graphType...............................................................................................................77Parameter: graphColumns........................................................................................................77Parameter: dateRelativeUnit ....................................................................................................78Parameter: dateStart ..................................................................................................................78Parameter: dateEnd....................................................................................................................78Parameter: showLast .................................................................................................................78
End Point: /api/status.....................................................................................................................79
End Point: /api/cancel .................................................................................................................... 79End Point: /api/download ............................................................................................................. 79End Point: /api/listDatabases........................................................................................................79End Point: /api/listFields ............................................................................................................... 79Debugging .........................................................................................................................................80Relative Dates....................................................................................................................................81Trend Reports....................................................................................................................................81
http://-/?-http://-/?-7/21/2019 Reporter Administrators Guide 9.x.b
8/134
Blue Coat Reporter 9.x Administrator Guide
vi
7/21/2019 Reporter Administrators Guide 9.x.b
9/134
7
Chapter 1: Preface
About This DocumentAudience: Network Administrators (Security)
Abstract: This document describes various Reporter components, operationalconcepts, and how to view and manage reports, and provides proceduresassociated with tuning and troubleshooting Reporter performance.
This document assumes you have read and performed the tasks in the Blue CoatReporter 9.4 Initial Configuration Guide; that is, Reporter is installed on adedicated server and one or more ProxySG appliances are forwarding accesslogs by way of FTPthe exceptions are alternate deployment scenarios andupload methods, which are described in this document.
Document ConventionsThis document adheres to the following typographical and document designprinciples.
Notes and Warnings
The following is provided for your information and to caution you againstactions that can result in data loss or personal injury:
Table 11 Document Conventions
Conventions Definition
Italics The first use of a new or Blue Coat-proprietary term.
Courier font Screen output. For example, command line text, file names, andBlue Coat Content Policy Language (CPL).
Courier Italics A command line variable that is to be substituted with a literal
name or value pertaining to the appropriate facet of yournetwork system.
Courier Boldface A Blue Coat literal to be entered as shown.
Arial Boldface Screen elements in the Management Console.
{ } One of the parameters enclosed within the braces must besupplied
[ ] An optional parameter or parameters.
| Either the parameter before or after the pipe character can ormust be selected, but not both.
Note: Information to which you should pay attention.
7/21/2019 Reporter Administrators Guide 9.x.b
10/134
Blue Coat Reporter 9.x. Administrators Guide
8
Related Documentation
The following documents are available on the Blue Coat download site:
Blue Coat Reporter 9.4 Release NotesLinked to from the Reporter softwaredownload page.
Blue Coat Reporter 9.4 Initial Configuration Guide
Blue Coat Reporter Online Help System (available through the Reporteruser interface)
These document PDFs are available at:https://bto.bluecoat.com/documentation/pubs/Reporter
The intended reference path is:
Release Notes > Initial Configuration Guide > Help System > AdministratorsGuide
For any Reporter documentation issues, send e-mail to:[email protected].
Navigating This Document
List of Chapters and task descriptions. Chapter 2: "Reporter Concepts"on page 9Describes various Reporter
operations.
Chapter 3: "Administrative Tasks"on page 31Provides various commonadministrative procedures and lists some basic Reporter best practices.
Chapter 4: "Managing User Access to Reporter"on page 59Describeshow to plan and configure Reporter role-based access and externalauthentication (LDAP).
Chapter 5: "Web API"on page 71Describes scripting report creation andgeneration and supported HTTP endpoints.
Important: Critical information that is not related to equipment damage orpersonal injury (for example, data loss).
WARNING! Usedonlyto inform you of danger of personal injury or physical
damage to equipment. An example is a warning against electrostatic discharge(ESD) when installing equipment.
7/21/2019 Reporter Administrators Guide 9.x.b
11/134
9
Chapter 2: Reporter Concepts
This chapter discusses various components of Blue Coat Reporter.
Chapter Contents
This chapter contains the following sections:
"Reporter Overview"on page 9
"About the Reporter Architecture"on page 9
"About the Page View Combiner"on page 10
"Deployment Overview"on page 12
"About Optimizing Log Processing Configurations"on page 17
"About the Default Browse Time Calculations"on page 20 "Report Field/Log Field Names"on page 22
"Reports/Log Field Matrix"on page 24
Reporter Overview
Blue Coat Reporter is a key component in the Blue Coat Secure Web Gatewaysolution. Reporter generates and displays reports based on Web traffic accesslog data that is sent from one or more gateway ProxySG appliances. Analyzingreports gives insight regarding the integrity of the network and user Web
browsing habits and policy compliance. This allows you to:
Identify possible security threats (such as malware/spyware)
View user activity by user, group, URLs, or other aspect
View blocked Web traffic (such as categories and URLs)
Identify which users consume how much network bandwidth from Webuse
About the Reporter Architecture
The Reporter application accomplishes major tasks:
Processes raw log data received from ProxySG appliances and populatesdatabases.
Manages the databases and generates reports.
Manages the Reporter server functions.
Log processing itself involves the following components:
Log Reader: Reads access log data into memory on the Reporter server.
7/21/2019 Reporter Administrators Guide 9.x.b
12/134
Blue Coat Reporter 9.x Administrators Guide
10
Page View Combiner (PVC): This sub-component of the log reader attemptsto provide more realistic user browsing statistics by combining the initialrequest and its secondary referral requests as one page count. For detailedinformation about the PVC, see"About the Page View Combiner"on page10.
Log Processor: Populates the databases with the log data.
Figure 21 Access log to database process.
About the Page View Combiner
The Page View Combiner (PVC) is called during Blue Coat Reporter log
processing. The PVC combines multiple HTTP requests that are associated witha single Web page into a single log line. When a user browses to a Web page,most often that page triggers requests for more content, either from the sameWeb server or another server (for example, a media server that stores video orimage content). Rather than regard each of these as separate requests, the PVCcombines all of the bytes into the original request.
7/21/2019 Reporter Administrators Guide 9.x.b
13/134
Chapter 2: Reporter Concepts
11
Figure 22 PVC concept diagram.
The goals of the PVC are to:
Reduce the number of database entries from the original log file, whichimproves report generation performance.
More closely represent user browsing activity, as each object (requested bythe first page from content servers) is not counted as a separate entry.
It is possible that a Web request that would normally be combined to representone page view might be split into two page views. This occurs when, as a resultof internal processing, the log sources are halted or restarted or the request isrecorded across two log files.
LEGEND
A: Enterprise users
B: Gateway ProxySG appliance
C: Example.com Server Farm: C-1: main server; C-2: ad farm; C-3: media serverD: Reporter server (and log source staging)
E: Reporter Administrator User
PROCESS FLOW
1: An enterprise user initiates a Web request for a news story at: www.example.com.
2: example.comsends further requests to internal servers for advertisement and video content and
returns four data objects:
example.com/main.html
i.example.com/ads/sponsor1.gif
example.com/news/story1.html
example.com/news/video1.asf
3: The ProxySG appliance adds access log entries for requested content pages.4: The ProxySG appliance forwards the requested content to the user (successful policy check).
5: At a scheduled time, the ProxySG appliance sends (FTP) the log files to the Reporter server.
6: The PVC combines the log lines into one page view and saves that in the database. The Reporter user
generates and views a report that contains one page view entry for the original request to
www.example.com.
Internet
7/21/2019 Reporter Administrators Guide 9.x.b
14/134
Blue Coat Reporter 9.x Administrators Guide
12
If this occurs, no data is lost, but the database contains two page views.Continuing with the example in the previous illustration:
8:40:20 cnn.com/html
8:40:20 i.cnn.com/ads/sponsor1.gif
[------end of log file------------]
[----beginning of new log file----]
8:40:21 cnn.com/news/story1.html8:40:21 cnn.com/news/video1.asf
The first two entries are shown as one page view; the second two as anotherwithin the database. However, they represent a single page view requested by auser.
Requirements
The PVC requires the following fields in the logs:
cs-referer
sc-status
rs(Content-Type)
The Blue Coat-recommended log formats contain these fields (see"ReportField/Log Field Names"on page 22).
If these log fields are not present, no page-view combining occurs, and reportdata represents each and every Web request.
Deployment Overview
This section describes the Blue Coat-recommended deployment of Reporter.
Standard FTP DeploymentOne Server
In this deployment, gateway ProxySG appliances use FTP to send access logfiles to the same server on which Reporter is installed. For this deployment,Blue Coat strongly recommends staging the logs and installing the Reporterapplication on different physical hard disk drives.
Important: HTTPS logs donotcontain the sc-statusfield; therefore, PVCcalculations cannot occur. The field is not included because it would exposepersonal user data (such as bank account information).
7/21/2019 Reporter Administrators Guide 9.x.b
15/134
Chapter 2: Reporter Concepts
13
Figure 23 Same-server Reporter deployment.
You can install Reporter on the same server to which the ProxySG appliancesends log files if the system resources allow for efficient processing. Refer to theBlue Coat Reporter Sizing Guide.
LEGEND
A: Gateway ProxySG appliance
B: Reporter Server (with network card)
C: Reporter Administrator User
D: Reporter Role User
PROCESS FLOW
1: Enterprise users initiate Web requests (HTTP/HTTPS); they receive content or policy deny notices.
2: Using the mainaccess log format (a group of log field codes), the gateway ProxySG appliance stores
all Web activity in access log files. The ProxySG appliance FTP upload client periodically sends the
raw log files to the Reporter server (a dedicated server that stages the unprocessed log files and
contains the Reporter application).
3: Log files, or sources, are stored in file directories, named according to the source ProxySG appliance.
4: The Reporter Administrator creates a database, which listens for and detects unprocessed log files in
the log source.
5: The processed log data populates the fields in the database.
6: Users access the Reporter application by logging into the Management Console (using the server
network IP address). When a user clicks a report link, Reporter generates the data from the
associated database and displays the report. Administrator users have access to all reports and
configuration options. Role users only have access to log field data that is assigned to them.
Internet
FTP (LAN/WAN)
7/21/2019 Reporter Administrators Guide 9.x.b
16/134
Blue Coat Reporter 9.x Administrators Guide
14
Standard FTP DeploymentTwo Servers
Enterprises with numerous, larger access log sources require that Reporter isinstalled on a dedicated server, which then uses FTP to retrieve log files fromone or more log file staging servers.
Figure 24 One log server and Reporter server deployment.
Internet
FTP (LAN/WAN)
LEGEND
A: Gateway ProxySG appliance
B: Access Log Staging Server
C: Reporter Server (with network card)
D: Reporter Administrator User
E: Reporter Role User
PROCESS FLOW
1: Enterprise users initiate Web requests (HTTP/HTTPS); they receive content or policy deny notices.
2: Using the mainaccess log format (a group of log field codes), the gateway ProxySG appliances store all Web
activity in access log files. The ProxySG appliance FTP upload clients periodically send the raw log files to the
dedicated log file staging server.
3: Log files, or sources, are stored in file directories, named according to the source ProxySG.
4: The Reporter Administrator creates a database on the Reporter server; the database listens for and detects
unprocessed log files in the log source directories on the log file server.
5: When Reporter detects unprocessed log data in the log source, it retrieves (FTP operation) the log files and
populates the fields in the database.
6: Users access the Reporter application by logging into the Management Console by using a Web browser and
entering the server network IP address and the Reporter port number. When a user clicks a report link, Reporter
generates the data from the associated database and displays the report. Administrator users have access to all
reports and configuration options. Role users only have access to log field data that is assigned to them.
7/21/2019 Reporter Administrators Guide 9.x.b
17/134
Chapter 2: Reporter Concepts
15
Direct ProxySG Streaming Deployment
You can configure a ProxySG appliance to stream real-timelog data to aReporter system. Unlike the standard FTP deployment, you cannot archiveprocessed log files for back up purpose or future reprocessing. Furthermore, ifReporter encounters an issue and cannot continue to process log data, the
ProxySG stream begins caching the data to one of its local disks. If the ProxySGappliance is able to reconnect to the Reporter server before the cache fills tocapacity, the ProxySG appliance sends the backlog and processing continues. Ifthe ProxySG appliance is not able to reconnect before the ProxySG appliancelocal disk cacheoverflows, the ProxySG appliance begins overwriting the oldestdata file and the data is not recoverable for processing.
7/21/2019 Reporter Administrators Guide 9.x.b
18/134
Blue Coat Reporter 9.x Administrators Guide
16
Download Access Log Data from the Blue Coat Cloud Security
Service
If your enterprise has a Blue Coat Cloud Security Service (ThreatPulse) accountand is sending Web (HTTP/HTTPS) traffic for policy checks and reporting, youcan configure Reporter to download the cloud access logs for local processing.
This reporting provides flexibility across your enterprise.Communication between Reporter and the cloud service requires an API key,which is created in the cloud service interface. The key secures the link betweenReporter and the your cloud service account.
Reporter downloads cloud log data over a secure (HTTPS) connection to theDestination Directory that you specify (the procedure linked below describes this).
Upon the first successful communication with the cloud service, Reporterdownloads all available log data. After that, Reporter only downloads new logdata.
LEGEND
A: Users at the corporate office: Proxy Forwarding Access Method.
B: Branch office employees: Firewall/VPN Access Method (IPsec).
C: Remote Users: Client Connector.
D: Reporter Administrator/Reporting User
PROCESS FLOW
1: Corporate location users send their Web requests through the gateway ProxySG appliance, which connects to
the cloud service.
2: Branch office employees connect to the cloud service through an on-premise firewall device (IPsec); a remote
user connects through the Client Connector application, which is installed on their system.
3: Reporter receives logs from the ProxySG appliance (FTP) and from the cloud service (API key/HTTPS).
NOTE:The Reporter server must have Web access.
7/21/2019 Reporter Administrators Guide 9.x.b
19/134
Chapter 2: Reporter Concepts
17
Each access log contains a one hour segment of data. Reporter saves log files inthe Destination Directorywith date-formatted file names similar to:
cloud_###_##############.log.gz.
The second numerical portion of the sequence represents the following date/time information:
Year/Month/Day/Hour/Minute/SecondFor example, a sequence of201211221200means: This log file was collected bythe cloud service at 2012November22nd12pm.
To configure this method, see"Process Access Logs From the Blue Coat CloudSecurity Service"on page 46.
About Optimizing Log Processing ConfigurationsThis section describes some conditions that affect log processing efficiency.
About Access Log Naming Conventions
This section provides suggestions for ProxySG appliance access log namingconventions, especially for deployments that require processing a large numberof log files over a longer duration of time.
For optimal Reporter performance, configure your access logs to use thefollowing filename format:
xxxxxxxxxxxxxxxNddddddddddd.log.gz
where: xrepresents any valid character that can be used in naming a log file
(letters, digits, underscore, dash).
Nrepresents a non-decimal-digit character.
drepresents a decimal digit. This number, preceding the log fileextension, determines the order in which the log files are processed. Thelog file ordering is performed identically for FTP, cloud download, andlocal disk log sources. A date string representing the log line dateswithin the file is preferred. If you mix cloud files with on-premise files,use the 12 digit cloud date syntax described above.
.log.gzis the extension of the (compressed) log file.
DECIMAL DIGIT NOTES
The decimal digit number is the key part of the format.
If this number does not provide a complete ordering on the set of log files,then the log processing speed suffers because of internal log tablethrashing.
Note: The cloud service prevents the downloading of access logs that are lessthan two hours old.
7/21/2019 Reporter Administrators Guide 9.x.b
20/134
Blue Coat Reporter 9.x Administrators Guide
18
A filename format ofMMDDhhmmssis inadequate because the files processchronologically, except at year-end when they temporarily process out-of-order because of the December (MM= 12) rollover into January (MM= 01)where January files sort before December.
A filename format ofhhmmssis more problematic because log files are
processed out-of-order whenever one day rolls into the next. Given these constraints, to ensure the most efficient log file ordering, format
this eleven-digit number as: YYJJJhhmmss, where:
YY= two-digit year (00 99)
JJJ= three-digit Julian day of the year (001 366)
hh= two-digit hour of the day (00 23)
mm= two-digit minute of the hour (00 59)
ss= two-digit second of the minute (00 59)
Using this format allows Reporter to properly order log files through the
year 2021. The default filename format used for log files on the ProxySG has the
following text and specifiers: SG_%f_%c_%l%m%d%H%M%S.log.gz.
%f= log name (facility)
%c= name of the external certificate used for encryption, if any
%l= the fourth parameter of the ProxySG IP address (101.102.103.104)
%m= two-digit month (01 12)
%d= two-digit day (01 31)
%H= two-digit hour (00 23)
%M= two-digit minute (00 59)
%S= two-digit second (00 59)
.log.gz= extension
The suggested filename format for log files on the ProxySG applianceslightly alters the default and has the following text and specifiers:SG_%f_%c_%l%m%d_%y%j%H%M%S.log.gz.
%y= two-digit year, without century (00 99)
%j= three-digit Julian day within year (001 366)
The value of this naming convention for log files is very evident whenprocessing large numbers of log files (spanning multiple days and months)occurs. The value is less evident when log file generation and processingoccurs regularly (daily or more frequently) so that out-of-order files occurinfrequently. However, when re-processing large sets of log files, thenaming convention is essential.
7/21/2019 Reporter Administrators Guide 9.x.b
21/134
Chapter 2: Reporter Concepts
19
About Chronological Ordering
Each database creates and manages its own memory resident LogTable. EachLogTable is comprised of hour-tables containing data for each hour thedatabase LogProcessors spend reading log files. These tables constitute some ofthe most active memory in Reporter, and therefore have a significant impact on
overall log processing performance. If all log files were processed inchronological order, there would never be more than one hour-table necessaryin memory. It is common for the log processing process to encounter batches oflog files spanning multiple hours between them. If they are processed out ofchronological order, performance significantly improves by allowing thenumber of hour-tables to grow, provided there is sufficient process memory.Conversely, during low memory conditions, reducing the number of hour-tables prevents unnecessary memory starvation and subsequent diskoperations (swapping files in and out of memory).
Reporter orders log files based on a numeric field in the filename, when it ispresent. The field is part of the filename format described in the AccessLogging chapter (see Configuring the Upload Client) of theBlue Coat SGOSAdministration Guide(in pre-SGOS 5.5.x documention, Access Logging is itsown volume PDF). The default filenames created by the ProxySG contain aMonth/Day/Hour/Minute/Secondtimestamp immediately preceding the .logor.log.gzsuffix; for example: SG_Main_HQ-1_1102081500.log.gz. If the filenameends with .logor .log.gz, the LogProcessor parses it for any purely numericsequence immediately preceding the required suffix. If one is found, it is thenused to sequentially order that batch of log files. You can significantly improveLogProcessor performance by naming the log files with any ordered numericvalues that comply with this format. For example:anyfilenameprefix123.logorsome-other-prefix-84757.log.gz.
About Known Conditions for Efficiency/In-efficiency The many variables involved in processing log files prevents the ability to
present a comprehensive and complete set of recommended configurationsettings. Some of these variables include:
64 bit versus 32 bit operation systems and hardware.
Variant log file sizes, small to extremely large (dozens of gigabytes).
Available memory for Reporter resources.
Disk speed: setting up a stripped array or SAN might improveperformance.
In addition to knowledge of your systems, understanding the followingconditions that both aid and hinder Reporter log processing functionalityenables you to modify configuration options to optimize efficiency.
Known Condition for Efficient Processing
Do not run other applications on the Reporter server.
Adhere to the system guidelines set in theBlue Coat Reporter Sizing Guide.
7/21/2019 Reporter Administrators Guide 9.x.b
22/134
Blue Coat Reporter 9.x Administrators Guide
20
Known Conditions for In-efficient Processing
Having insufficient memory to retain all of the active data.
Consuming extra time to write processed data and inactive data frommemory to disk.
Reporter runs well, but other errors occur: Data is not available for report generation because it has not been
written to disk yet.
Other applications on user systems suffer from Reporters resource use.
About Database Purging
Most of the database is kept in memory. If the entire database is notoccasionally purged, it would continue to consume more of the processmemory as new log files are processed. As the database grows, configurationsettings that were previously beneficial might become detrimental.
As a general guideline, Blue Coat recommends that databases contain amaximum of 30 days of log data. However, the amount of log dataismorerelevant than the number of days in the data sets.
Reporter also allows the administrator to purge the database based on thenumber of log lines. Log lines can be purged by expiration, automatically(scheduled) and manually. This task can also be performed on demand asthe administrator does not need to schedule the task but can set a custompurge limit.
About the Default Browse Time Calculations
Some reports provide a datapoint called Browse Time. The intention of thisstatistic is toestimatehow long a user spends browsing a particular website orcategory of a website.
The Page View Criteria Used for Browse Time
Reporter calculates this by matching each source IP address and each user inthe logs with a website. After a match occurs, Reporter tracks the activity ofeach user as seen in the access logs.
As Reporter procesess each log line in each log file, it finds and adds up browsetime for each client IP address. If Reporter determines a request is a page view,the transaction is assigned 30 seconds of browse time. However, if another page
view is discovered within 30 seconds in the Page View Combiner (PVC) cachetime window (10 seconds by default), Reporter subtracts the time of theprevious page view from the next and counts the result. If the next page viewoccurs more than 30 seconds after the previous page view, the previous pageview remains 30 seconds. These page view calculations provides a moreaccurate algorithm than the one used in Reporter 8.x versions, especialy forcases where a user is using two browsers at the same time or has backgroundapplications making requests.
7/21/2019 Reporter Administrators Guide 9.x.b
23/134
Chapter 2: Reporter Concepts
21
Reporter calculates browse timein real timeduring log processing.Furthermore, Reporer can only subtract the time difference from the last pageview if it still exists in the PVC cache. For example, if a Reporter administratorsets the default browse time to 60 seconds per page and leaves the PVC cachetime windows to 10, the 60 second value applies by default unless another pageview is found for the same client IP address and user agent within the 30
second PVC window. Therefore, you could have pages with anywhere betweenzero and 30 seconds or 60 seconds of browse time. Typically, the default browsetime is set to 30 seconds by default, which means all pages have a browse timefrom zero to 30 seconds, but never more.
For related information, see"About the Page View Combiner"on page 10.
Examples
This sub-section provides browse time examples.
Example Notes
Scenario 1: Employee A visits cnn.com for 40 seconds, visits yahoo.com for 20seconds, and then leaves his browser on youtube.com for 2 minutes but doesnot watch a video or click links in the site.
Reporter calculates 30 seconds for cnn.com, 20 seconds for yahoo.com,and 30 seconds for youtube.comfor a total browse time of 80 seconds.
If, however, the same user browses videos on youtube.comevery 29seconds, the resulting browse time is 30 seconds for each video,resulting in a total browse time of 120 seconds.
Scenario 2: Employee A opens two concurrent browsersInternet explorerand Firefoxat the same time and performs the above scenario. The resultis double.
Scenario 3: Employee A uses the same browser. By default all page viewsare given the default browse time, which is 30 seconds (this value isconfigurable). If Reporter processes another page view from the same clientIP address on the same user agent while the first page view is still in thePVC cache (which is also a 30 second window by default), Reporter lowers
the browse time for the first page view to the time difference between thepage views.
Scenario 4: Employee B vists images.google.comfor 5 seconds and thenclicks a picture, views it for 15 seconds, clicks the back button, clicks adifferent picture, and views it for 45 seconds. Reporter records 5 seconds forimages.google.com, then 15 seconds for the the first picture (plus whatevertime it takes to click back and click on the second pic), and then 30 secondsfor the last picture.
Note: These examples assumes the default values of 30 seconds for browsetime with a default PVC cache of 30 seconds or fewer. For example, if a uservisits cnn.com and never loads another page (does not click through the variousarticles links) for three hours, the resulting browse time is 30 seconds.
7/21/2019 Reporter Administrators Guide 9.x.b
24/134
Blue Coat Reporter 9.x Administrators Guide
22
Report Field/Log Field Names
This section provides a reference table that lists the report field to log fieldassociation. Report fields are what comprise various reports, based on theinformation contained in the access log. The contents of an access log aredetermined by the log field names (which determine what data types are
captured during the ProxySG logging process). Some log field names correlateto absolute data (such as URLs), others derive information from access logvariables (such as browsing duration).
Log Field Best Practices
Certain access log fields are critical to proper Reporter operation.
To prevent Reporter from disregarding some log lines, the Reporter maindatabases require these fields: cs-host,cs-uri-host or cs-uri-hostname
sc-status
cs-uri-scheme
c-ip, x-client-ip, x-client-address, c-dns or x-cs-username-or-ip
rs(Content-Type)
sc-filter-result or x-exception-id
x-virus-id
For the PVC to operate correctly, Reporter requires these additional fields:
cs(Referer) or x-cs(Referer)-uri
x-exception-id or sc-filter-result (x-exception-id preferred)
sc-filter-category, cs-category, or cs-categories
For the PVC to operate correctly for video reports, Reporter requires theseadditional fields: cs-host cs-uri-host or cs-uri-hostname
cs-uri-scheme
c-ip, x-client-ip, x-client-address or c-dns, x-cs-username-or-ip
sc-status
sc-filter-result, or x-exception-id
x-virus-id
cs-method
time-taken
cs-uri-scheme s-session-id
To properly populate all default Dashboard reports, Reporter requires thesefields in addition to those above:
Note: For more PVC information, see"About the Page View Combiner"onpage 10.
7/21/2019 Reporter Administrators Guide 9.x.b
25/134
Chapter 2: Reporter Concepts
23
cs-username, x-cache-user, cs-userdn, x-radius-splash-username, x-
cs-session-username or x-ldap-attribute(displayName)
cs-category, sc-filter-category or cs-categories
sc-filter-result or x-exception-id
cs-host cs-uri-host or cs-uri-hostname
x-bluecoat-application-name
x-bluecoat-application-operation
To populate all default video reports, Reporter requires these fields: cs-host cs-uri-host or cs-uri-hostname
c-ip, x-client-ip, x-client-address, c-dns or x-cs-username-or-ip
x-cache-info
cs-auth-group or cs-auth-groups
x-rs-streaming-content
Main Logs
In the following table, italicized report field name text indicates thederiveddata.
Report Field Name Log Field Name
cs(Referer) cs(Referer)
browse_time Calculated at run-time from user session and stored as database
field.
c_ip c-ip
cs_auth_group cs-auth-group
cs_bytes cs-bytes
cs_host cs-host
cs_method cs-method
cs_uri_extension cs-uri-extension
cs_uri_path cs-uri-path
cs_url_query cs-url-query
cs_url_scheme cs-url-scheme
cs_user_agent cs(User-Agent)
cs_username cs-username
7/21/2019 Reporter Administrators Guide 9.x.b
26/134
Blue Coat Reporter 9.x Administrators Guide
24
Reports/Log Field Matrix
This section provides a table that lists which main-format access log fields arerequired to populate eachpre-definedreport in the User Behavior, Security, andBandwidth Usage groups on the Reports tab. Use this reference to understand howlog fields relate to report data and aid in your customization of reports.
date date
date_time date + time
day_of_week Derived from date.
hits Calculated from page_views+ all related log entries.
hour_of_day Derived from time.
month Derived from date.
requests (same as page
views or hits)
Calculated during database generation and stored as database
field.
rs_content_type rs(Content-Type)
s_action s-action
sc_bytes sc-bytes
sc_filter_category cs-categories (or cs-category or sc-filter-
category)
sc_status sc-status
time time
total_bytes cs-bytes + sc-bytes
url Combined from (uri-scheme://cs-host/cs-url-path
[cs-url-query]).
verdict x-exception-id(sc-filter-resultif x-exception-idis
not present).
week Derived from date.
x_virus_id x-virus-id
year Derived from date.
Report Field Name Log Field Name
7/21/2019 Reporter Administrators Guide 9.x.b
27/134
Chapter 2: Reporter Concepts
25
Log field Output
date + time YYYY-MM-DD + HH:MM:SS(GMT/UTC)
gmttime DD/MM/YYYY:hh:mm:ssGMT
localtime DD/MMM/YYYY:hh:mm:ss +nnnn
timestamp seconds since epoch in utc/gmt
x_timestamp_unix_utc seconds since epoch in utc/gmt
x_timestamp_unix seconds since epoch in local time
7/21/2019 Reporter Administrators Guide 9.x.b
28/134
Blue Coat Reporter 9.x Administrators Guide
26
Main Log Field Matrix
These reports are URL-centric; they display reports that reflect browsing activity.
Group Report Required Fields
UserBehavior Blocked Web Browsing perUser sc-filter-result, cs-username, cs-bytes, sc-bytes
Web Browsing per Category {cs-categories -or- sc-filter-
category}, cs-bytes, sc-bytes
Web Browsing per Day date, sc-bytes, cs-bytes
Web Browsing per Day of
Week
date, cs-bytes, sc-bytes, time,
time-taken
Web Browsing per Group cs-auth-group, cs-bytes, sc-bytes
Web Browsing per Hour of
Day
time, cs-bytes, sc-bytes, time-
taken
Web Browsing per Month date, cs-bytes, sc-bytes, time,time-taken
Web Browsing per Site cs-host, {cs-categories -or- sc-
filter-category}, cs-bytes, sc-
bytes, time_taken
Web Browsing per User cs-username, cs-bytes, sc-bytes
Web Browsing per User and
Category
cs-username, sc-filter-category or
cs-categories, sc-bytes, cs-bytes
Web Searches cs-uri-query
(Also requires Blue Coat Web Filter
(BCWF) enabled.)
7/21/2019 Reporter Administrators Guide 9.x.b
29/134
Chapter 2: Reporter Concepts
27
Security Blocked Web Browsing by
User Agent
sc-filter-result, cs(User-Agent),
cs-bytes, sc-bytes
Blocked Web Sites sc-filter-result, cs-host, {sc-
filter-category -or- cs-
categories}, cs-bytes, sc-bytes
Filtering Verdict Trend by Day date, sc-filter-result
Malware Requests Blocked by
Site
cs-bytes, cs-host, sc-bytes, sc-
filter-category, time-taken
Potential Malware Infected
Clients
c-ip, cs-bytes, cs-host, sc-bytes,
sc-filter-category, time-taken
Potential Threats x-virus-id, sc-filter-category
ProxyAV Malware Detected:
Client IP
c-ip, cs-bytes, sc-bytes, time-
taken, x-virus-id
ProxyAV Malware Detected:
Names
cs-bytes, sc-bytes, time-taken, x-
virus-id
ProxyAV Malware Detected:
Sites
cs-bytes, cs-uri-path, cs-uri-
query, cs-uri-scheme, sc_bytes,
time-taken, x-virus-id
Risk Groups sc-filter-category
SSL Certificate Categories {cs-username -or- c-ip}, s-action,
x-rs-certificate-hostname, sc-
bytes, cs-uri-port
SSL Certificate Errors x-rs-certificate-observed-errors,
x-rs-certificate-hostname, sc-
bytes, cs-uri-port
Trend of Potential Threats x-virus-id, sc-filter-category
Bandwidth
Usage
Bandwidth Cost per User date, cs-username, sc-bytes, cs-
bytes
Bandwidth Cost per User and
Site
cs-username, cs-host, sc-filter-
category or cs-categories, cs-
bytes, sc-bytes
Bandwidth Used per Day date, sc-bytes, cs-bytes
Bandwidth Used per Day of
Week
date, sc-bytes, cs-bytes
Bandwidth Used per Hour of
Day
date, sc-bytes, cs-bytes
Bandwidth Used per Month date, sc-bytes, cs-bytes
Requests per Content Type rs(Content-Type), cs-bytes, sc-
bytes
Requests per Protocol cs-uri-scheme, cs-bytes, sc-bytes
Web Requests per Client IP c-ip, cs-bytes, sc-bytes
7/21/2019 Reporter Administrators Guide 9.x.b
30/134
Blue Coat Reporter 9.x Administrators Guide
28
Web Application Reports
In the following table, italicized report field name text indicates thederiveddata.
Video Usage Reports
In the following table, italicized report field name text indicates thederiveddata.
Report Field Name Required Fields
Web Application Name x-bluecoat-application-name, hits, page-views,
browse-time, cost-time, total-bytes, cost-bytes,
sc-bytes, cs-bytes, cache-bytes, rs-bytes
Web Application
Operation
x-bluecoat-application-operation, hits, page-
views, browse-time, cost-time, total-bytes, cost-
bytes, sc-bytes, cs-bytes, cache-bytes, rs-bytes
Web Application
Detailed Report
x-bluecoat-application-name, x-bluecoat-
application-operation, c-ip, total-bytes, cost-
bytes, hits, sc-bytes, cs-bytes, page-views,
browse-time, cost-time, cache-bytes
Web Browsing per Web
Application Name and
Client IP
x-bluecoat-application-name, c-ip, total-bytes,
cost-bytes, sc-bytes, cs-bytes, hits, page-views,
browse-time, cost-time, cache-bytes
Web Browsing per Web
Application Name and
User
x-bluecoat-application-name, cs-username, total-
bytes, cost-bytes, sc-bytes, cs-bytes, hits, page-
views, browse-time, cost-time, cache-bytes
Report Field Name Field Name
Client IP Video c-ip, total-bytes, cost-bytes, sc-bytes, cs-
bytes, hits, page-views, browse-time, cost-time,
cache-bytes
Flash Streaming
Bandwidth Cost per Day
date, page-views, browse-time, sc-bytes, rs-
bytes, total-bytes, cs-bytes, cache-bytes
Group Video cs-auth-group, total-bytes, cost-bytes, sc-bytes,
cs-bytes, hits, page-views, browse-time, cost-
time, cache-bytes
Video Application
Delivery Method
x-rs-streaming-content, total-bytes, cost-bytes,
sc-bytes, cs-bytes, hits, page-views, browse-
time, cost-time, cache-bytes
Video Application Type x-cache-info, total-bytes, cost-bytes, sc-bytes,
cs-bytes, hits, page-views, browse-time, cost-
time, cache-bytes
7/21/2019 Reporter Administrators Guide 9.x.b
31/134
Chapter 2: Reporter Concepts
29
Video Applications x-rs-streaming-content, cs-host, total-bytes, sc-
bytes, cs-bytes, hits, page-views, browse-time,
cost-time, cache-bytes, cost-bytes
Video Page Detail cs-host, filename, c-ip, sc-bytes, cs-bytes, hits,page-views, browse-time, cost-time, cache-bytes,
total-bytes,
Video Site cs-host, total-bytes, sc-bytes, cs-bytes, hits,
page-views, browse-time, cost-time, cache-bytes
Report Field Name Field Name
7/21/2019 Reporter Administrators Guide 9.x.b
32/134
Blue Coat Reporter 9.x Administrators Guide
30
7/21/2019 Reporter Administrators Guide 9.x.b
33/134
31
Chapter 3: Administrative Tasks
This chapter describes various maintenance and performance tasks available tothe Reporter administrator, some of which are beyond the scope of the ReporterOnline Help System that is accessible directly in the Reporter ManagementConsole.
This chapter contains the following sections:
Section A: "Reporter Administration Tasks"on page 32
Section B: "Reporter Performance Best Practices"on page 50
Section C: "Advanced Filtering Tasks"on page 53
Section D: "Troubleshooting"on page 55
How Do I...?
How do I...? Tasks
I want to install Reporter as a root user. Procedure:"Linux Root UserInstallation Procedure"on page 32
I want to secure the connection thatReporter uses to communicate with theWeb server.
Procedure: "Securing the Reporter WebServer Transport Protocol"on page 35
What type of information does Blue Coatcollect from the Reporter Improvement
Program? Can I opt out?
"About the Reporter ImprovementProgram"on page 35
I want to configure Reporter to send e-mail to myself and/or others whenspecified events occur.
Procedure:"Connecting Reporter to E-mail Servers"on page 40.
Procedure:"Configuring Reporter toSend Alerts"on page 56.
I have configured the Direct ProxySGUpload Client and want to create back uplog files.
Procedure:"Creating ProxySG PolicyThat Backs Up Access Log Files"onpage 40.
My log files have spaces hardcodedbetween user names (first last) and in
reports the space displays as a %20symbol. Can I fix this?
Procedure:"Processing Log Files WithEncoded Spaces in User Names"on
page 45
I have a Blue Coat Secure Web Service(cloud) account and want to downloadaccess logs from there to use in Reporter.
Procedure:"Process Access Logs Fromthe Blue Coat Cloud Security Service"on page 46
I want to apply customized filteringoptions.
Section C: "Advanced Filtering Tasks"on page 53
7/21/2019 Reporter Administrators Guide 9.x.b
34/134
Blue Coat Reporter 9.x Administrators Guide
32
Section A: Reporter Administration Tasks
Section A: Reporter Administration Tasks
This section provides common tasks that Reporter administrators perform tofurther configure Reporter.
Linux Root User Installation Procedure
The Blue Coat Reporter Initial Configuration Guide provides a procedure forinstalling Reporter as a non-root user on a Linux server. This procedure is forinstalling Reporter as a root user.
Install the Reporter Application
Step 1: In a browser, enter:https://bto.bluecoat.com
Step 2: Access the software download page.
Step 3: Download the application files:
a. Click the Downloadtab. The Blue Coat Download page displays.
b. From the Downloadmenu, click Reporter.
c. Before installing any version of Reporter, Blue Coat strongly recommends reading the ReleaseNotes (Please Read link). System compatibility lists, new feature briefs, and any known issues arelisted in this document.
d. Click the download link that matches the system on which you are installing.Note:The Linux##RPMlink is the uncompressed installation file. If you select Linux##, this is thecompressed (gzip) file. Gunzip the file in the /opt/bcdirectory.
Step 4: Open a terminal and navigate to the
directory to which you downloaded theapplication. Invoke the installation script:
rpm -Uhv Reporter*.rpm
Alternate location:rpm -Uhv --
prefix=
Reporter*.rpm
Text similar to the following displays:Preparing #################### [100%]
1:bcreporter #################### [100%]
########################################
# The Blue Coat Reporter installation
completed successfully.
# Please change your current working
directory to ...
########################################
7/21/2019 Reporter Administrators Guide 9.x.b
35/134
Chapter 3: Administrative Tasks
33
Section A: Reporter Administration Tasks
Update Reporter Installation Procedure
The Blue Coat Reporter Initial Configuration Guide provides a procedure forinstalling Reporter as a non-root user on a Linux server. This procedure is forinstalling Reporter as a root user.You update by downloading and running anISO, which is 64-bit.
Step 5: Change the working directory to the Reporter installation directory:cd /opt/bc/reporter
Alternate, if you installed to a different location:cd
Step 6: Run the startup configuration. ./do-startup.sh
Step 7: Supply responses for the prompts: Username [admin]:
Password:
License key:
a. For the Username [admin]prompt, enter thedefault administrator access username.
b. For the Passwordprompt, create a passwordfor the default administrator user.
c. (Optional) If you purchased an Enterprise orPremium license andretrievedthe key, enter it
for the License keyprompt.Note:You can also enter the license key afterinstalling Reporter. Press .
Step 8: Text similar to the following displays: Blue Coat Reporter is already stopped
Starting Blue Coat Reporter: [OK]
Reporter is now up-and-running.
The RPM installation process also adds a start andstop script into the init.dfile, which automaticallystarts Reporter.
Install the Reporter Application
Install the Reporter Application
Step 1: In a browser, enter:https://bto.bluecoat.com
7/21/2019 Reporter Administrators Guide 9.x.b
36/134
Blue Coat Reporter 9.x Administrators Guide
34
Section A: Reporter Administration Tasks
Step 2: Access the software download page.
Step 3: Download the application files:
a. Click theDownload
tab. The Blue Coat Download page displays.b. From the Downloadmenu, click Reporter.
c. Before installing any version of Reporter, Blue Coat strongly recommends reading the ReleaseNotes (Please Read link). System compatibility lists, new feature briefs, and any known issues arelisted in this document.
d. Click the download link that matches the system on which you are installing.Note:The Linux##RPMlink is the uncompressed installation file. If you select Linux##, this is thecompressed (gzip) file. Gunzip the file in the /opt/bcdirectory.
Step 4: Open a terminal and navigate to thedirectory to which you downloaded theapplication. Invoke the installation script:
rpm -Uhv Reporter*.rpm
Alternate location:rpm -Uhv --
prefix=
Reporter*.rpm
Text similar to the following displays:Preparing #################### [100%]
1:bcreporter #################### [100%]########################################
# The Blue Coat Reporter installation
completed successfully.
# Please change your current working
directory to ...
########################################
Step 5: Change the working directory to the Reporter installation directory:cd /opt/bc/reporter
Alternate, if you installed to a different location:cd
Step 6: Run the startup configuration. ./do-startup.sh
Install the Reporter Application
7/21/2019 Reporter Administrators Guide 9.x.b
37/134
Chapter 3: Administrative Tasks
35
Section A: Reporter Administration Tasks
Uninstalling Reporter
The Readme file, which is located in the Reporter root folder/directory, containsprocedures that describe how to uninstall the Reporter application from Windowsand Linux servers.
About the Reporter Improvement Program
After it completes the installation process, Reporter displays a dialog thatdiscusses the Blue Coat Reporter Improvement Program. As described in thedialog, Reporter sends anonymous information to Blue Coat to assist Blue Coat
personnel in analyzing how Reporter is used. This basic information includesyour license, server specifications, system resource use, and Reporterconfiguration and use. No private, enterprise-sensitive data is transmitted.Although this feature is enabled by default, you have the option to declinesending this information. The General Settings > Systems Settings > Server Settingspage provides the Send anonymous system data to help improve Reporteroption; clearthis option and click Save.
Securing the Reporter Web Server Transport Protocol
By default, Reporter communicates with the Web server through the HTTPSprotocol. For increased security, you have the option to configure Reporter to use
HTTPS as the transport protocol. You must either accept the default certificate orspecify the location of a signed server certificate and unencrypted private key(2048-byte or larger key). Consult with your security administrator concerning thecreation of these.
Step 7: Supply responses for the prompts: Username [admin]:
Password:
License key:
a. For the Username [admin]prompt, enter thedefault administrator access username.
b. For the Passwordprompt, create a passwordfor the default administrator user.
c. (Optional) If you purchased an Enterprise orPremium license andretrievedthe key, enter itfor the License keyprompt.Note:You can also enter the license key afterinstalling Reporter. Press .
Step 8: Text similar to the following displays: Blue Coat Reporter is already stopped
Starting Blue Coat Reporter: [OK]
Reporter is now up-and-running.
The RPM installation process also adds a start andstop script into the init.dfile, which automaticallystarts Reporter.
Install the Reporter Application
Important: Reporter 9.x doesnotsupport SSL private keys that are passwordprotected.
7/21/2019 Reporter Administrators Guide 9.x.b
38/134
Blue Coat Reporter 9.x Administrators Guide
36
Section A: Reporter Administration Tasks
Proceed to the appropriate section:
"Default Certificate"on page 36
"Selected Certificate"on page 37
Default Certificate
Consider the following if you elect to employ the default certificate. The defaultcertificate is a Reporter generated self-signed test certificate; however, most
browsers correctly warn you to avoid using self-signed certificates because theyare not signed by a reputable certificate authority. To prevent browsers fromrejecting the improper default certificate after you change your host IP address,you also must generate a new default certificate. This is accomplished bychanging the host IP address (using the host's configuration tools), thenconfiguring Reporter to HTTP and restarting Reporter (to remove the old defaultcertificate), configuring back to HTTPS and restarting once again (to create thenew default certificate).
Configure HTTPS with the Default Certificate
Step 1: With administrator credentials, in the Reporter Management Console select General Settings >System Settings > Server Settings.
Step 2: Select the Default Certificate. a. In the Protocolarea, select HTTPS. The areaexpands to displays certificate options.The default Reporter secure port is 8082.Ifyourusers access Reporter with the 8081port,Reporter redirects the connection to the secureport.
b. In the Certificatearea, select Use defaultcertificate.
c. Click Save.
Step 3: Restart Reporter. a. Select General Settings > Shut Down/Restart.b. Click Restart Reporter.
7/21/2019 Reporter Administrators Guide 9.x.b
39/134
Chapter 3: Administrative Tasks
37
Section A: Reporter Administration Tasks
Selected Certificate
Reporter cannot use private keys that are password encrypted. If you already havea custom keypair that uses a password encrypted prviate key for your Reporterhost, then you must create an unencrypted version of the key. The public
certificate will continue to work with either key, but Reporter will only work withthe unencrypted key.
If you need to create a new unencrypted version of the existing keypair,proceed to"Creating an Unencrypted Private Key From an Existing Key".
If you have an existing keypair that isnotencrypted with passwords, proceedto"Configuring Reporter to Use HTTPS with a Selected Certificate"on page38.
Creating an Unencrypted Private Key From an Existing Key
The following procedures describe how to use the OpenSSL application that shipswith Reporter; the procedure contains steps for Windows and Linux operatingsystems.
Note: Blue Coat recommends creating certificate and key pair files in a privatefolder and not in the Reporter installation folders. If Reporter is uninstalled, youlose the files.
Create an Unencrypted Private Key From an Existing Encrypted Key
Step 1: Access the Open SSL application: In Windows, navigate to C:\Program Files\Blue Coat Reporter 9\utilities\ssl; double-click the openssl
application file. In Linux, run the opensslutility (to verify that the opensslpackage is installed on your system,
enter which openssl).
Step 2: Enter the following command: OpenSSL> rsa -inexisting_encryped_private.key -outnew_unencrypted_private.key
Step 3: The utility prompts you for the encrypted key password; enter that.
Step 4: Name the new key with a different name than the existing key, which prevents overwriting thestill valid encrypted key.
Step 5: The files are created and stored on the system. Proceed to the next procedure:"ConfigureReporter to use HTTPS with a Selected Certificate"on page 38.
7/21/2019 Reporter Administrators Guide 9.x.b
40/134
Blue Coat Reporter 9.x Administrators Guide
38
Section A: Reporter Administration Tasks
Configuring Reporter to Use HTTPS with a Selected Certificate
Forcing Renegotiation of SSL sessions
The longer connections exist, the more susceptible are they to man-in-the-middleattacks. You can configure Reporter to renegotiate SSL sessions, which preventsthese attacks by selecting the Force secure renegotiationoption (General Settings >System Settings > Server Settings Web Server Settings > SSL Settingsarea).
However, after this option is enabled, Reporter only supports clients that supportsecure renegotiation.
To support older clients that do not support the secure renegotiation option, clearthis option.
Anonymize DataTo comply with security requirements, sensitive data can be encrypted, or,anonymized. Anonymized data is defined on a per role basis. By default, data isviewed in clear text. Any or all fields can be anonymized. Reports are displayedafter applying the algorithm so that users with that role are able to view theanonymized data instead of viewing the original data.
Configure Reporter to use HTTPS with a Selected Certificate
Step 1: With administrator credentials, in the Reporter Management Console select General Settings >System Settings > Server Settings.
Step 2: Select the generated CSR and keypair files. a. In the Protocolarea, select HTTPS. The areaexpands to display certificate options.Note:If this option is not available, see"Troubleshooting HTTPS Configuration onLinux"on page 59.
b. Select Enter Certificate; the Server Certificateand Private Keyfields become active.
c. Enter the path to the generated certificateand keypair files (or click the folder icons to
navigate to their stored locations and selectthem).
d. (Optional, recommended) Click TestCertificate and Key to test their validity. If thetest fails, work with your securityadministrator to create valid files.
e. Click Save.
Step 3: Restart Reporter. a. Select General Settings > Shut Down/Restart.
b. Click Restart Reporter.
7/21/2019 Reporter Administrators Guide 9.x.b
41/134
Chapter 3: Administrative Tasks
39
Section A: Reporter Administration Tasks
Reporter Management Console Location:
Step 1: Administrator credentials: General Settings > Access Control > Roles.
Step 2: Databases: Click Newto define specific access rights for a role.
Move DatabaseUse this option to move the database to other physical locations. This is usefulwhen the drive on which database is located is running out of space. Beforemoving the database, Reporter calculates the size of the selected database and thespace available at the destination drive location.
Reporter Management Console Location
Step 1: Administrator credentials: General Settings > Data Settings > Databases.
Step 2: Select Move Databasesfrom the Actionsmenu for an existing database.Cancel move database operation restores the database back to the originallocation.
Step 3: Select General Settings > Data Settings> Databases.
Step 4: Select Cancel Databases from the Actions menu.
Internationalized Domain Name (IDN)
Enabling the Internationalized Domain Name (IDN) option converts domainnames back to the original Unicode domain name that were originally entered atthe client browser.Reporter Management Console Location:
Administrator credentials: General Settings > System Settings > Server Settings >ReportGeneration Settings.
Note: The move database operation fails if the destination drive size is lessthan 125% of the database size. While running the Move Database feature,Reporter unloads database and log sources.
7/21/2019 Reporter Administrators Guide 9.x.b
42/134
Blue Coat Reporter 9.x Administrators Guide
40
Section A: Reporter Administration Tasks
Connecting Reporter to E-mail Servers
To enable users to e-mail generated reports to recipients, Reporter must beconfigured to communicate with an SMTP mail server. This is also required toenable Reporter to send administrators alerts when system resources reach
specified use levels (see"Configuring Reporter to Send Alerts"on page 56).
Creating ProxySG Policy That Backs Up Access Log Files
If you configured the ProxySG appliance to continuously stream access log data tothe Reporter server, the data is not stored anywhere after it is processed. If thedatabase becomes corrupt or if you have another scenario that requiresreprocessing of legacy data, you cannot do so unless you configure the ProxySGappliance to send back up files (raw data) to another location.
The following procedure describes how to configure the ProxySG appliance toupload back up files and create a policy that implements the back up operation.
Connect Reporter to an SMTP (E-mail) Server
Step 1: With administrator credentials, in the Reporter Management Console selectGeneral Settings > Reporter Settings > System Settings > External Servers > Email.
Step 2: Enter the primary SMTP server IP address or hostname.
Step 3: Specify the Fromaddress used in e-mails; for example:[email protected](must be a valid e-mail address).
Step 4: (Optional) Enter the SMTP server access credentials if they are required by theserver.
Step 5: (Optional, recommended) Enter information for a backup SMTP server shouldthe primary server become unavailable.
Step 6: Click Save.
Note: This procedure is valid only in SGOS 5.x; in SGOS 4.x, you cannotsimultaneously stream and forward logs.
Configure the ProxySG to Store Back Up Raw Access Log Data
Step 1: In the ProxySG appliance Management Console, select the Configuration > Access Logging >Logs > Logtab.
Step 2: Click New. The Create Log dialog displays.
7/21/2019 Reporter Administrators Guide 9.x.b
43/134
Chapter 3: Administrative Tasks
41
Section A: Reporter Administration Tasks
Step 3: Configure the new log settings. a. In the Log Namefield, enter a name for thenew log file. For example, SG_HTTP_Backup.
b. From the Log Formatdrop-down list, selectbcreportermain_v1 (or whichever format youuse for Reporter logs).
c. (Optional) In the Descriptionfield, describethe new log.
d. Click OKto close the dialog
e. Click Applyto commit the new log file.
Step 4: Select the Configuration > Access Logging >Logs > Upload Clienttab.
Select the upload client:
a. From the Logdrop-down list, select thenewly-created log.
b. From the Upload Clientdrop-down list,select FTP Client.
c. Click Settings. The FTP Client Settingsdialog displays.
Configure the ProxySG to Store Back Up Raw Access Log Data
7/21/2019 Reporter Administrators Guide 9.x.b
44/134
Blue Coat Reporter 9.x Administrators Guide
42
Section A: Reporter Administration Tasks
Step 5: Configure the ProxySG to communicatewith the FTP server to be used for log
archiving.
Server connection options:
a. From the Settings for drop-down list, selectPrimary FTP Server.
b. In the Hostfield, enter the IP address orhostname of the FTP server; change thedefault port only if a different port is used.
c. In the Pathfield, enter the destinationfolder to be used for this log archive (toprevent log data duplication, do not pointto the same directory that is used forproduction data).
d. In the Usernamefield, enter the name
required to access this FTP server.e. If a server access password is required,
click Change Primary Passwordand enterthe information.
f. Click OKto close the dialog.
Step 6: Select the Configuration > Access Logging >Logs > Upload Scheduletab.
Upload schedule options:
a. From the Logdrop-down list, select thenew-created log.
b. In the Upload Typearea, select Periodically.
c. In the Upload the Log Filearea, set theinterval at which the ProxySG uploads thelog files. Blue Coat recommends once perhour.
d. Click Apply.
Step 7: Launch the Visual Policy Manager (VPM): Configuration > Policy > Visual Policy Managertab;click Launch.
Step 8: In a Web Access Layer, click Add Rule; or, select Policy > Add Web Access Layer. As policy bestpractice, do not create a new Web Access Layerif one already exists.
Configure the ProxySG to Store Back Up Raw Access Log Data
7/21/2019 Reporter Administrators Guide 9.x.b
45/134
Chapter 3: Administrative Tasks
43
Section A: Reporter Administration Tasks
Step 9: Add the FTP, HTTP, and HTTPS serviceobjects.
Steps:
a. Right-click a Service column cell; select Set.
The Set Service Object dialog displays.
b. Click Newand select Service Name. TheAdd Service Name Object dialog displays.
c. From the Service Namedrop-down list,select FTPand click OK.
d. Repeat Step band add the HTTP services.
Step 10: Create a combined object for the services. Steps:
a. Still in the Set Service Objects, click Newand select Combined Service Object.
b. Name the object.
c. Select each service and click Add.
d. Click OKto create the combined object.
e. With the combined object selected, click OKto add it to the rule.
Configure the ProxySG to Store Back Up Raw Access Log Data
7/21/2019 Reporter Administrators Guide 9.x.b
46/134
Blue Coat Reporter 9.x Administrators Guide
44
Section A: Reporter Administration Tasks
Step 11: Create an Action object that enables accesslogging.
a. Right-click the Actioncolumn cell; select Set.The Set Action Object dialog displays.
b. Name the object.
c. Click Newand select Modify Access Logging.The Add Access Logging Object dialogdisplays.
d. Select Enable logging to; from the drop-down list, select thelog file that you createdin Step 3.
e. Click OKto create the object.
f. With the object selected, click OKto add it tothe rule.
The rule is complete.
Step 12: To implement the policy, click Install Policy.
Configure the ProxySG to Store Back Up Raw Access Log Data
7/21/2019 Reporter Administrators Guide 9.x.b
47/134
Chapter 3: Administrative Tasks
45
Section A: Reporter Administration Tasks
Processing Log Files With Encoded Spaces in User Names
If Reporter generates reports that display %20in between user names (first andlast), it means that your access logs have encoded spaces. For example:
first lastversus a user name first.last
Although Reporter currently supports the Mainaccess log format. A hard-codedspace is a valid character in the Extended Log File Format (ELFF). The followingprocedure describes how to manually configure the databases to display hard-coded spaces as actual spaces rather than %20symbols.
Manually Editing the Database File to Process Encoded Spaces
Step 1: Create a new database, but donotassign alog source.
General Settings > Reporter Settings > Data
Settings > Databases
Step 2: Stop the Reporter service. General Settings > Reporter Settings > Shut Down/Restart
Step 3: Using a text editor, open the newly-created database configuration file.
By default, this file is located in the Blue CoatReporter 9 > Settings > Databasesdirectory.
Database configuration files are named withnumber and letter strings, not intuitive names.Look for the date and time stamp of the databaseyou created for this process.
Step 4: Inside of the database configuration file, search for the log field named cs_username, which islocated near the bottom of the file. The construct is similar to the following:
cs_username = {type = "flat"
index = "0"
name = "cs-username"
db_field = "cs_username"
} # cs_username
Step 5: Add encoded_spaces = "true"to the cs_usernameconstruct:cs_username = {
type = "flat"
index = "0"
name = "cs-username"
db_field = "cs_username"
encoded_spaces = "true"
} # cs_username
Step 6: Save the database configuration file.
Step 7: Re-start Reporter.
Step 8: Add a log source to the database to begin processing log data.
7/21/2019 Reporter Administrators Guide 9.x.b
48/134
Blue Coat Reporter 9.x Administrators Guide
46
Section A: Reporter Administration Tasks
Process Access Logs From the Blue Coat Cloud Security Service
Reporter can download access logs from the Blue Coat Cloud Security Service(ThreatPulse) and process the data, which enables unified reporting. To learnmore, see "Download Access Log Data from the Blue Coat Cloud Security Service"
on page 16.This procedure describes how to create the required cloud service API and how toconfigure Reporter to receive the log downloads.
Prerequisites
This procedure assumes that you have an account for the Web Securitymodule
in ThreatPulse. To receive log files from the cloud service, the Reporter server must have Web
access. If you already have Reporter deployed and that deployment inhibitsallowing Web access to the Reporter server, consider installing anotherinstance of Reporter on a different server at the external edge of the network.Then automate or otherwise move the log files to the existing server (toachieve unified reporting).
ThreatPulse Configuration
This section describes how to create the required API key.
Note: ThreatPulse refers to the Blue Coat cloud service product name. TheBlue Coat Cloud Security Service is a solution that includes all productsproduced by Blue Coat in the cloud.
Create an API Key in the ThreatPulse User Interface
Step 1: Access the ThreatPulse portal. a. In a browser, enter:https://portal.threatpulse.com
b. Log in with your account credentials.
Step 2: In Service mode, select Account Maintenance > Account Provisioning > API Keys.
7/21/2019 Reporter Administrators Guide 9.x.b
49/134
Chapter 3: Administrative Tasks
47
Section A: Reporter Administration Tasks
Reporter Configuration
This section describes how to configure Reporter to use the cloud API key tovalidate and begin access log downloads to specified location.
Step 3: Create an API key for Reporter.
a. Click Add API Key. The service displays the Create API Keys dialog.
b. Define a Usernameand Password. You will enter these during the Reporter configuration.
c. Click Add.
Step 4: Enable the key: select the key and click Enable.
Proceed to the next section to configure Reporter.
Create an API Key in the ThreatPulse User Interface
Configure Reporter to Download Access Logs from the Cloud Service
Step 1: With administrator credentials, in the Reporter Management Console (on the server where thelogs are to be staged) select General Settings > Data Settings > Cloud Download.
7/21/2019 Reporter Administrators Guide 9.x.b
50/134
Blue Coat Reporter 9.x Administrators Guide
48
Section A: Reporter Administration Tasks
Step 2: Select Enable Cloud Download; the other options become available to edit.
a. Select the Destination Directory, which is the folder that stages the cloud service access logs.
b. (Optional) If you have not previously created a folder for this purpose, click the Create New Foldericon.
Step 3: Set the Scheduleof how often Reporter
checks for new logs in the cloud service.The shortest increment is one hour, as all cloud logfiles contain one hour of data.
By design, the Blue Coat cloud service prevents thedownloading of logs that are less than two hours old.Furthermore, given that the minimum time chunk isone hour, allow some time for the data to accumulatein the Destination Directory.
Step 4: Enter the API key. a. Enter the Blue Coat cloud service APIUsernameand Passwordthat you created inthe ThreatPulse user interface.
b. Click Test Username and Password. If the test
fails, check the API key in the ThreatPulseuser interface (Service mode> AccountMaintenance > API Keys). Also check theexternal connection.
Step 5: Click Save.
Configure Reporter to Download Access Logs from the Cloud Service
7/21/2019 Reporter Administrators Guide 9.x.b
51/134
Chapter 3: Administrative Tasks
49
Section A: Reporter Administration Tasks
Create a Database or Assign Log Source
Now that you have a new source of log data, you can create a separate databaseand generate reports based on the cloud source only, add the newlog source to anexisting database to provide true unified reporting, or both.
Also, see"About Optimizing Log Processing Configurations"on page 17.
7/21/2019 Reporter Administrators Guide 9.x.b
52/134
Blue Coat Reporter 9.x Administrators Guide
50
Section B: Reporter Performance Best Practices
Section B: Reporter Per