Embed Size (px)
Citation preview
Performance Regression AnalysisPerformance Regression Analysis
Cor-Paul Bezemer
Accomplishments and ChallengesAccomplishments and Challenges
““A burnt child dreads fire”A burnt child dreads fire”
Regression testing makes sure we do Regression testing makes sure we do not make the same mistake twicenot make the same mistake twice
Write codeExecute regressiontest
Deploy or fix changes
Regression testing also helps us to make Regression testing also helps us to make sure we are not re-introducing more bugs sure we are not re-introducing more bugs
Performance regression testing helps ensure Performance regression testing helps ensure we are not breaking performancewe are not breaking performance
Write codeExecute performance test
Deploy if performance is OK or fix changes
Performance regression testing Performance regression testing can/should be done on several metricscan/should be done on several metrics
Performance regression testing Performance regression testing can/should be done on several metricscan/should be done on several metrics
Performance regression testing Performance regression testing can/should be done on several metricscan/should be done on several metrics
Performance regression testing Performance regression testing can/should be done on several metricscan/should be done on several metrics
Directions in research on Directions in research on performance regression analysisperformance regression analysis
Directions in performance Directions in performance regression analysis researchregression analysis research
1. Execution profile comparison
2. Source code analysis
3. Visualization
Execution profile comparisonExecution profile comparison
Execution profile comparisonExecution profile comparison
Disadvantages of execution Disadvantages of execution profile comparisonprofile comparison
- Need to run performance tests several times because of variation
- Small changes may be difficult to interpret
Source code analysisSource code analysis
Identify risky commits and recommend those for testing (Huang et al. @ ICSE 2014)
Disadvantages of source Disadvantages of source code analysiscode analysis
- Can not replace 'real' performance measurements (yet)
Visualization of performance Visualization of performance regression analysis resultsregression analysis results
Control charts Control charts (Nguyen et al. @ MSR 2014)(Nguyen et al. @ MSR 2014)
Differential flame graphsDifferential flame graphs(Bezemer et al. @ SANER 2015)(Bezemer et al. @ SANER 2015)
Deciding on the baseline Deciding on the baseline performanceperformance
Deciding on the acceptable Deciding on the acceptable performance deviationperformance deviation
Reporting results to the Reporting results to the developerdeveloper
Availability of performance Availability of performance teststests
Execution time of testsExecution time of tests
Performance regression testing helps ensure Performance regression testing helps ensure we are not breaking performancewe are not breaking performance
Write codeExecute performance test
Deploy if performance is OK or fix changes
Directions in performance Directions in performance regression analysis researchregression analysis research
1. Execution profile comparison
2. Source code analysis
3. Visualization
Availability of performance Availability of performance teststests
Execution time of testsExecution time of tests
