Embed Size (px)
Citation preview
Continuous Validation of Load Test Suites
Mark D. Syer, Zhen Ming Jiang, Meiyappan Nagappan, Ahmed E. Hassan, Mohamed Nasser and Parminder Flora
1
2
Failures in ULS systems are typically due to performance issues
3
Load testing may detect
failures before they occur in the field
4
0
200
400
600
800
1000
1200
2005 2006 2007 2008 2009 2010 2011 2012
User Growth Over the Years (in millions)
Facebook Twitter LinkedIn WordPress Tumblr Google+ Pinterest
Field workloads change continuously
13.5M Mobile Users56M Mobile Users
New HTML5 App
Performance analysts can compare field and load test
workloads using execution logs
6
Comparing workloadsis difficult
7
Huge amount of data
Rapidly evolving systems
Workloads are made of hundreds or thousands of individual workers
GenerateSignatures
DetectOutliers
InspectOutliers
Our approach identifies events that differ between load testing
and field workloads
9
Test logs
Field logs
10
We generate a signature for each unique worker ID
Identify unique user IDs
00:01, Alice starts a conversation with Bob00:01, Alice says `hi' to Bob00:02, Alice says `are you busy?' to Bob00:11, Bob says `yes' to Alice00:12, Alice says `ok' to Bob00:18, Alice ends a conversation with Bob
11
Identify the logs attributable to each unique user ID
00:01, Alice starts a conversation with Bob00:01, Alice says `hi' to Bob00:02, Alice says `are you busy?' to Bob00:11, Bob says `yes' to Alice00:12, Alice says `ok' to Bob00:18, Alice ends a conversation with Bob
12
Abstract log lines to events
00:01, Alice starts a conversation with Bob00:01, Alice says `hi' to Bob00:02, Alice says `are you busy?' to Bob00:12, Alice says `ok' to Bob00:18, Alice ends a conversation with Bob00:11, Bob says `yes' to Alice
13
Count the events for each use ID
Alice BobUSER starts a conversation with USER 1 0USER says MSG to USER 3 1USER ends a conversation with USER 1 0
14
DetectOutliers
InspectOutliers
We identify and inspect outlying signatures
15
16
Can we detect...
17
Issue difference?
Intensity and feature differences?
Intensity difference?
Our Approach
18
State‐of‐the‐Practice
19
State‐of‐the‐practice is to compare event occurrence frequencies
Test logs
Field logsEvent Test FieldA 10/s 11/sB 5/s 7/sC 1/s 100/s
20
A datanodefails in the field
Hadoop distributes a workload across the nodes of a computing cluster
INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_id
INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStreamjava.io.IOException: Bad connect ackwith firstBadLink ip_address
WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_id bad datanode_id ip_address
INFO org.apache.hadoop.mapred.TaskTracker: attempt_id progress
INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_id
INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStreamjava.io.IOException: Bad connect ackwith firstBadLink ip_address
WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_id bad datanode_id ip_address
INFO org.apache.hadoop.mapred.TaskTracker: attempt_id progress
21
INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_id
INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStreamjava.io.IOException: Bad connect ackwith firstBadLink ip_address
WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_id bad datanode_id ip_address
INFO org.apache.hadoop.mapred.TaskTracker: attempt_id progress
Our approach flags eventswith high precision
0
20
40
60
80
100
Issue Feature and Intensity Intensity
Precision
Our Approach State‐of‐the‐Practice22
Our Approach
23
State‐of‐the‐Practice
88% 44%
24
