Upload
others
View
50
Download
0
Embed Size (px)
Citation preview
ArcGIS Server Performance and Scalability - Testing
MethodologiesAndrew Sakowicz
Frank Pizzi
Introductions
•• Who are we?Who are we?-- Professional ServicesProfessional Services
-- Enterprise Implementation Services TeamEnterprise Implementation Services Team
•• Who are you?Who are you?
Audience
• Testers• Developers• Managers• System Architects
Presentation outline
• Test Objectives• Preparing test environment • Test data• Test scripts• Load test• Analysis• Reporting• Capacity planning
Test Objectives
Test Objectives
- Contractual Service Level Agreement- Bottlenecks- Capacity- Benchmark
Preparing Test Environment
Preparing Test Environment
Tuning• Critical step in successful GIS deployments
- Profile individual operations- Tuned systems optimize the use of critical resources- Map service performance directly linked to underlying
MXD document
• Tools- Mxdperfstat- Map Service Publishing Toolbar- Fiddler- NeXpert
Browser
Web Server
SOM
SOC
Total Response Time (t1-
t2)
Wait Time
Search & Retrieval
Time
Usage Time
SDE/DBMS
A test is executed at the web browser.
It measures web browser call’s elapsed time
(roundtrip between browser and data source)
t1 t2
Preparing Test Environment
Tuning Measure at web browser, e.g. stop watch or programmatically using API events
Preparing Test Environment
Tuning - Fiddler
HTTP RequestHTTP Request
HTTP ResponseHTTP Response
Inspector TabInspector Tab
Fiddler HTTP Debugging
Proxy
Preparing Test Environment
Tuning - Fiddler FiddlerFiddler’’s Timeline measuress Timeline measuresTransaction timeTransaction time
Preparing Test Environment
Tuning - NeXpert
Transaction Transaction MarkerMarker““StepStep””
Fiddler PlugFiddler Plug--ininNeXpertNeXpert provides provides
Transaction marker and Transaction marker and performance reportperformance report
Preparing Test Environment
Tuning – NeXpert ReportNeXpertNeXpert ReportReport
Preparing the test environment
Fiddler Add-on
Profile Web Requests with Fiddler
Preparing Test Environment
Tuning• Optimize using map service publishing toolbar
- Analyze Map- Preview Map- Save Map Service Definition- Publish to ArcGIS Server
Preparing Test Environment
• Tuning• Optimize mxd with mxdperfstat
http://resources.arcgis.com/gallery/file/enterprise- gis/details?entryID=6391E988-1422-2418-88DE- 3E052E78213C
Item At Scale Layer NameRefresh
Time
(sec)Recommendations
Feature
sVertices
Labelin
g
Geogra
phy
Phase
(sec)
Graphic
s Phase
(sec)
Cursor
Phase
(sec)
DBMS
CPUDBM
S LIO
41 1,000TaxlotDenseLabel 1.93
Simplify labeling, symbology:
GraphicsPhase=1.42; simplify
geometry and/or set label
scale; convert polygon to
polyline: vertices
fetched=200001; simplify
geometry and/or set label
scale: vertices fetched=200001; 1 200,001 TRUE 0.45 1.42 1.04 0.02 266
42 1,000TaxlotDenseNoLa
bel 0.53simplify geometry: vertices
fetched=200001; 1 200,001 FALSE 0.45 0.02 0.9 0.02 140
Preparing Test Environment
Concurrent UsersConcurrent Users
TunedTuned
Not TunedNot Tuned
Preparing Test Environment
Configuration• Critical step in successful GIS deployments
- Incorrect settings can limit access to hardware resources
• Configure ArcGIS Server Instances
Preparing Test Environment
• Optimize ArcMap document and configure correct number of ArcGIS Server instances
Corrected(Unloaded TT: 0.34 sec)
(2.1 Instances/core)
Uncorrected(Unloaded TT: 11.97 sec)
(1.6 Instances/core)
Test Data
Test Data
Bbox (Using Fiddler) Area of InterestArea of Interest
Selected ExtentSelected ExtentFrom HTTPFrom HTTPDebuggingDebugging
ProxyProxy
Test Data
Attribute Data
Test Data
Generate Bboxes
One simple example of Python One simple example of Python script to generate script to generate BboxesBboxes
Test Data
Heat Map based on Heat Map based on response times response times
from from ArcGISArcGIS ServerServer
Test Data Observe correlation between featureObserve correlation between feature
density and performancedensity and performance
Test Data - GIS Test Tool
• How to generate appropriate test data?
GIS Test Tool
• Test Data Generation by Area of Interest and Selected Scales
- Bounding Boxes- Point- Lines- Routes
- Start, Stop, Distance constraints- Attribute Data
Generating good test data
Demo GIS Test Tool
Test Scripts
Test Scripts
• Record user workflow based on application user requirements
• Create single user web test - Define transactions- Set think time and pacing based on application user
requirements- Parameterize transaction inputs- Verify test script with single user
Visual Studio Quick Introduction - WebTest
TransactionTransaction
HTTP RequestHTTP Request
Data sourceData source
Query String parameter referencinQuery String parameter referencindata sourcedata source
Testing with JMeter Build HTTP RequesBuild HTTP Reques
Testing with JMeter– Test Data from CSV
Set Test DataSet Test Data
Testing with JMeter
JMeter Plug-in
• http://code.google.com/p/jmeter-plugins/- Many new plug-in enhancements for JMeter
Testing with GIS Test Tool
Load Test
Load Test
• Create load test- Define user load- Create machine counters to gather raw data for analysis
• Execute
Visual Studio Quick Introduction – Load TestScenarios:Scenarios:
Test Mix (Test Mix (WebTestWebTest or Unit Test), or Unit Test), Browser Mix,Browser Mix,Network Mix, Network Mix, Step LoadsStep Loads
PerfmonPerfmon Counter Sets:Counter Sets:Available categories that may be Available categories that may be
mapped to a machine in mapped to a machine in the deploymentthe deployment
Run Settings:Run Settings:Counter Set Mappings Counter Set Mappings –– Machine metricsMachine metrics
Test durationTest duration
Demo Visual Studio - Load Test Run
Threshold Threshold rules rules
violatedviolated
Testing with JMeter – Create Step Load
Testing with JMeter – Set Thread Properties
Testing with JMeter - Results
GIS Test Tool
Testing with GIS Test Tool
GIS Test Tool– Performance Test Capabilities
• Define web tests including QA step for verification• Define transactions including think time• Define load tests• Execute load tests• Capture system metrics for multiple machines• View results and export to Excel
Analysis
Tips and Tricks
Analyze Results - Validation• Compare and correlate key measurements
- Response Time (increasing, higher than initially profiled for single user)
- Throughput- CPU on all tiers- Network on all tiers- Disk on all tiers- Passed tests- Failed test
Tips and Tricks
Analyze Results - Validation• Lack of errors does not validate a test
- Requests may succeed but return zero size image- Spot check request response content size
Tips and Tricks
Analyze Results – Reporting and Analysis• Exclude failure range, e.g. failure rate > 5% from the
analysis• Exclude excessive resource utilization range
Tips and Tricks
Analyze Results Validation Example – Unexpected response time decrease under heavy load
Unexpected curve shape:Unexpected curve shape:Response time should be increasing.Response time should be increasing.
Likely root cause: Likely root cause: failed or 0 size image requests.failed or 0 size image requests.
Tips and Tricks
Validation Example – Expected CPU and Response Time Correlation
Max CPUMax CPUutilizationutilization
Response Time increase Response Time increase correlates with:correlates with:
••User loadUser load••CPU utilizationCPU utilization
Tips and Tricks Validation Example – Test failure due to w3wp memory bottleneck
Root cause:Root cause:System memory bottleneck at System memory bottleneck at
Web Server processWeb Server processSymptom:Symptom:
System available memory System available memory is decreasingis decreasing
Tips and Tricks Validation Example – Unexpected CPU utilization while accessing cache services
Unexpected bottleneck in Unexpected bottleneck in cache application :cache application :
100% CPU utilization .100% CPU utilization .How to determined root cause?How to determined root cause?
Tips and Tricks
Determining System Capacity• Maximum number of concurrent users
corresponding to, e.g.:- Maximum acceptable response time- First failure or 5%- Resource utilization greater than 85%, for example CPU
• Different ways of defining acceptance criteria (performance level of service), e.g.
- 95% of requests under 3 sec- Max request under 10 sec
Tips and Tricks - Execute
• Ensure - Only target applications are running- Application data is in the same state for every test- Good configuration management is critical to getting
consistent load test results
5 6
Tips and Tricks - Common pitfalls
• Application stack is not properly configured and tuned
• Single error in a test can lead to a cascade of failing requests
• Test client machine is bottleneck- CPU- Memory- Networks (cache test in particular)
5
Tips and Tricks - Common pitfalls
Poor Configuration Example – Unexpected high CPU utilization due to lsass authentication not configured correctly
High CPU utilization (187%)High CPU utilization (187%)of Web server due toof Web server due to
authentication process authentication process ((lsass.exelsass.exe))
Report
Report
- Executive Summary- Test Plan
- Workflows- Work load
- Deployment documentation- Results and Charts
- Key Indicators, e.g. Response Time, Throughput- System Metrics, e.g. CPU %- Errors
- Summary and Conclusions- Provide management recommendations for improvements
- Appendix
Test Tools
Testing - Selecting Load Test Tool
Test Tools Open Source
Pros Cons
LoadRunner No •Industry Leader•Automatic negative correlations identified with service level agreements•Http Web Testing•Click and Script•Very good tools for testing SOA• Test results stored in database•Thick Client Testing•Can be used for bottleneck analysis
•High Cost•Test Development in in C programming language•Test metrics difficult to manage and correlate•Poor user community with few available examples
Silk PerformerNo •Good solution for testing Citrix
•Wizard driven interface guides the user•Can be used for bottleneck analysis
•Moderate to High Cost•Test metrics are poor•Test Development uses proprietary language•Test metrics difficult to manage and correlate•Poor user community with few available examples
Visual Studio Test Team No •Low to moderate cost•Excellent Test Metric reporting•Test Scripting in C# or VB .NET•Unit and Web Testing available•Blog support with good examples•Very good for bottleneck analysis
•No built in support for AMF•No Thick Client options•Moderate user community
JMeter Yes •Free•Tool
•Provides only response times•Poor User community with few available examples
6 2
Testing - Selecting Load Test Tool
• Tool selection depends on objective- Commercial tools all have system metrics and
correlation tools- Free tools typically provide response times and
throughput, but leave system metrics to the tester to gather and report on
Capacity Planning
Input Capacity Planning
• Find Input for Capacity Planning- Test Report
- Includes Throughput (Transactions per hour) - Includes System Metrics – %CPU Utilization, #Cores - Spec Rate from the machines tested
- Use this information to calculate Service Time for Transactions
Input Capacity Planning
• Capacity model expressed as Service Time
100%3600#
TH
CPUCPUST
Capacity Planning
• Estimate capacity for a different hardware platforms- Find your target (t) server on
http://www.spec.org/cpu2006/results/rint2006.html
t
b
t
tbt rCPUSpecRatePe
rCPUSpecRatePeCPU
THSTCPU
%3600100#
Capacity Planning
• SPEC.org- http://www.spec.org/cpu2006/results/rint2006.html- Find your server from the results
Capacity Planning
Capacity Planning
• Additional examples on capacity planning can be found here:
- http://resources.arcgis.com/gallery/file/enterprise- gis/details?entryID=6367F821-1422-2418-886F- FCC43C8C8E22
System Designer
Questions?
Contact us
• Professional Services- Andrew Sakowicz, [email protected] Frank Pizzi, [email protected]