Upload
kristin-banks
View
215
Download
1
Tags:
Embed Size (px)
Citation preview
Automated Capacity Planning and Support for Enterprise Applications
Dharmesh ThakkarSchool of Computing, Queen’s University, Canada
Performance Activitiesin The Waterfall SDLC
Requirements
Design
Implementation
Verification
MaintenanceVerification: Performance test need to be conducted to verify that the required performance objectives are met
2
Performance Activitiesin SDLC
Requirements
Design
Implementation
Verification
Maintenance
Maintenance: Continuous monitoring, capacity planning and tuning need to be done to ensure that the system continues to meet the performance objectives
3
Requirements
Design
Implementation
Verification
Maintenance
Requirements
Design
Implementation
Verification Phase Performance Activities
Maintenance Phase Performance Activities
PerformanceAnalyst
Execution Logs
Customer Engagement Repository
Customer
CustomerSupportAnalyst
PerformanceModel
PerformanceTests
LoadTests
StressTests
Deployment
Performance Issues!
1
2
3
3
4
4
5
6
6 7
8
9
9
Overview of Thesis
Customers engage the vendor’s support analyst to resolve the issues
with system
Performance issues arise in customers’ production
environment, which need troubleshooting
The capacity calculator helps customer determine required hardware resources to
support the anticipated workload
Build a capacity calculator based on the performance model
Challenge: The goal of performance testing is not to find bugs, but to establish benchmarks
4
Challenge: The support analyst needs to troubleshoot the
customer deployment
Challenge: Load test and stress test workloads need to
be compared against the customer workloadsModify the workload or the deployment
scenario to put excessive stress on one or more system resourcesRun the application for long periods under
high workloads
Overview of Thesis:Our ContributionsRequirements
Design
Implementation
Verification Phase Performance Activities
Maintenance Phase Performance Activities
PerformanceAnalyst
Execution Logs
Customer Engagement Repository
Customer
CustomerSupportAnalyst
PerformanceModel
PerformanceTests
LoadTests
StressTests
Deployment
Performance Issues!
1
2
3
3
4
4
5
6
6 7
8
9
9
Process Carrying Our Contribution
Regular Process
Requirements
Design
Implementation
Verification Phase Performance Activities
Maintenance Phase Performance Activities
PerformanceAnalyst
Execution Logs
Customer Engagement Repository
Customer
CustomerSupportAnalyst
PerformanceModel
PerformanceTests
LoadTests
StressTests
Deployment
Performance Issues!
1
2
3
3
4
4
5
6
6 7
8
9
9
Process Carrying Our Contribution
Regular Process
A technique for helping troubleshoot performance issues by retrieving relevant report from a customer engagement repository
Presented at the International Conference on Software Maintenance (ICSM 2008) Beijing, China.Acceptance rate: 26%
5
A framework for building performance model based capacity calculator
Presented at the Workshop on Software and Performance (WoSP 2008) Princeton, NJ.Acceptance rate: 38%
Frequency (transactions per hour)
20 40 60 80
Concurrency 250
Creating Customer Profile T111 T112 T113 T114Customer Login T121 T122 T123 T124Search Title T131 T132 T133 T134Purchase Title T141 T142 T143 T144
Concurrency 500
Creating Customer Profile T211 T212 T213 T214Customer Login T221 T222 T223 T224Search Title T231 T232 T233 T234Purchase Title T241 T242 T243 T244
Concurrency 1000
Creating Customer Profile T311 T312 T313 T314Customer Login T321 T322 T323 T324Search Title T331 T332 T333 T334Purchase Title T341 T342 T343 T344
Concurrency 1500
Creating Customer Profile T411 T412 T413 T414Customer Login T421 T422 T423 T424Search Title T431 T432 T433 T434Purchase Title T441 T442 T443 T444
Building a Measurement based Capacity Calculator
Frequency (transactions per hour)
20 40 60 80
Concurrency 250
Creating Customer Profile T111 T112 T113 T114
Customer Login T121 T122 T123 T124
Search Title T131 T132 T133 T134
Purchase Title T141 T142 T143 T144
7
Performance Modeling Challenges
■ Large number of tests to be executed■ Limited time for performance
modeling■ Manual process of setup, execution
and analysis■ Risk of error and repeating the full
modeling process
8
Addressing the Challenges of Performance Modeling
■We created the framework to:–Automate the process–Speed up the process–Focus research on specific
areas
9
Framework for Building Model based Capacity Calculator
Test Enumeration
Test Reduction
EnvironmentSetup
TestAnalysis
Test Execution
Test Transition
Control Flow Data Flow
PerformanceData
PerformanceModel
Model Building
■ Case Studies:– Dell DVD Store– A large enterprise application
10
Customization Cost
Test Enumeration
Test Reduction
EnvironmentSetup
TestAnalysis
Test Execution
Test Transition
PerformanceModel
Model Building
Minimal Minimal Minimal Minimal Minimal Minimal Minimal AnotherBuild
Extensive Reasonable Extensive Extensive Reasonable Reasonable Minimal Another Application
11
Maintenance Phase:Customer Support
ApplicationSupportAnalyst
SymptomsIdentified Problems
SolutionsExecution
Logs
Customer Engagement
Report
Customer Engagement Repository
Create Store
Contains
Attempted Workarounds
12
Retrieving Relevant Reports
■ State of Practice:– No systematic techniques to retrieve and use
information for future engagements– Keyword searching is limited:
• depends on the search skills and experience of the analyst and peculiarity of the problem
13
Customer SupportProblem Statement
■ We want to find customers with similar operational and problem profiles
■ We can reuse prior solutions and knowledge
Heavy Email, Light Web, Light calendar Light Email, Light Web, Light calendar Light Email, Heavy Web, Light calendar Heavy Email, Heavy Web, No calendar Light Email, Light Web, Heavy calendar
Other Customers
CompareNew
Customer Engagement
14
Using Logs forCustomer Support
■ Execution logs are readily available and contain – Operational Profile: usage patterns (heavy
users of email from device, or to device, or light users of calendar, etc.)
– Signature Profile: specific error line patterns (connection timeout, database limits, messages queued up, etc.)
■ Find the most similar profile
15
Execution Logs
■ Contain time-stamped sequence ofevents at runtime
■ Readily available representatives of both feature executions and problems
<time> Queuing new mail msgid=ABC threadid=XYZ<time> Instant message. Sending packet to client msgid=ABC threadid=XYZ<time> New meeting request msgid=ABC threadid=XYZ<time> Client established IMAP session emailid=ABC threadid=XYZ<time> Client disconnected. Cannot deliver msgid=ABC threadid=XYZ<time> New contact in address book emailid=ABC threadid=XYZ<time> User initiated appointment deletion emailid=ABC threadid=XYZ
16
Example
010002000
C2
0
2000
4000
C1
0
2000
4000
C3
MTH
MTH m
ore
requ
ests
MFH
View C
alend
ar
Mee
ting
Reque
st
Appoin
tmen
t syn
c
Mes
sage
mov
e
Conta
cts
HTTPM
DS
0500
1000150020002500
NewSupport Request
Other Customers
Compare
17
Technique to Retrieve Relevant Engagement Reports
CustomerExecution
LogsConvert Log
Lines to Events
Compare Event
Distributions
Customer Engagement Repository
Compare Event
Distributions
Identify Signature
Events
Signature Event Distribution
OUTPUT RESULT SET
Closest Customer Engagement Reports
wrtSignature Profile
Closest Customer Engagement Reports
wrtOperational Profile
Event Distribution
18
Case Studies
■ Case Study I– Dell DVD Store open source application– Code instrumentation done for event logging– Built the execution log repository by applying
synthetic workloads, changing the workload parameters each time
■ Case Study II– Globally deployed commercial application– More than 500 unique execution events
19
Case Study Results■ Dell DVD Store
– 100% precision and recall on both operational profile based and signature profile based retrieval
■ Commercial Application– 100% precision and recall for signature profile based
retrieval– Results for operational profile based retrieval:
ExperimentCount of Log Files
K-L Distance Cosine Distance
Precision Recall Precision Recall
Single Feature Group 28 67.71% 90.28% 67.71% 90.28%Multiple Feature Groups 28 60.71% 80.95% 75.00% 100.00%All Feature Groups 12 72.92% 97.22% 62.50% 83.33%Real World Log Files 12 54.17% 72.22% 68.75% 91.67%All the Log Files 80 59.93% 79.90% 56.72% 75.62%
20
Sources of Errors
■ Events that do not correspond directly to a particular operational feature, such as idle time events, server health check events, startup and shutdown events
■ Imbalance in the event logging
21
Summary:Our ContributionsRequirements
Design
Implementation
Verification Phase Performance Activities
Maintenance Phase Performance Activities
PerformanceAnalyst
Execution Logs
Customer Engagement Repository
Customer
CustomerSupportAnalyst
PerformanceModel
PerformanceTests
LoadTests
StressTests
Deployment
Performance Issues!
1
2
3
3
4
4
5
6
6 7
8
9
9
Process Carrying Our Contribution
Regular Process
Requirements
Design
Implementation
Verification Phase Performance Activities
Maintenance Phase Performance Activities
PerformanceAnalyst
Execution Logs
Customer Engagement Repository
Customer
CustomerSupportAnalyst
PerformanceModel
PerformanceTests
LoadTests
StressTests
Deployment
Performance Issues!
1
2
3
3
4
4
5
6
6 7
8
9
9
Process Carrying Our Contributions
Regular Process
A framework for building performance model based capacity calculator
Presented at the Workshop on Software and Performance (WoSP 2008) Princeton, NJ.
A technique for helping troubleshoot performance issues by retrieving relevant report from a customer engagement repository
Presented at the International Conference on Software Maintenance (ICSM 2008) Beijing, China.
23