B2conference performance 2004

©2004 BLACKBOARD, INC. ALL RIGHTS RESERVED.

Introduction to Software Performance Engineering

Blackboard Performance Center of Excellence

Steve Feldman, Tim Moore and Ryan O’[email protected]

Introduction to Software Performance Engineering

Blackboard Performance Center of Excellence

Steve Feldman, Tim Moore and Ryan O’[email protected]

15-Nov-07


Any statements in this presentation about future expectations, plans and prospects for the Company, including statements about the Company, the Building Blocks Programand other statements containing the words “believes,”“anticipates,” “plans,” “expects,” “will,” and similar expressions, constitute forward-looking statements within the meaning of The Private Securities Litigation Reform Act of 1995. Actual results may differ materially from those indicated by such forward-looking statements as a result of various important factors, including: product development, and other factors discussed in our Registration Statement filed on Form S-1 with the SEC. In addition, the forward-looking statements included in this press release represent the Company’s views as of July 26, 2004. The Company anticipates that subsequent events and developments will cause the Company’s views to change. However, while the Company may elect to update these forward-looking statements at some point in the future, the Company specifically disclaims any obligation to do so. These forward-looking statements should not be relied upon as representing the Company’s views as of any date subsequent to July 26, 2004.

Any statements in this presentation about future expectations, plans and prospects for the Company, including statements about the Company, the Building Blocks Programand other statements containing the words “believes,”“anticipates,” “plans,” “expects,” “will,” and similar expressions, constitute forward-looking statements within the meaning of The Private Securities Litigation Reform Act of 1995. Actual results may differ materially from those indicated by such forward-looking statements as a result of various important factors, including: product development, and other factors discussed in our Registration Statement filed on Form S-1 with the SEC. In addition, the forward-looking statements included in this press release represent the Company’s views as of July 26, 2004. The Company anticipates that subsequent events and developments will cause the Company’s views to change. However, while the Company may elect to update these forward-looking statements at some point in the future, the Company specifically disclaims any obligation to do so. These forward-looking statements should not be relied upon as representing the Company’s views as of any date subsequent to July 26, 2004.


Read the Fine PrintRead the Fine PrintAny statements in this presentation about future expectations, plans and prospects for the Company, including statements about the Company, the Building Blocks Program and other statements containing the words “believes,” “anticipates,” “plans,” “expects,” “will,” and similar expressions, constitute forward-looking statements within the meaning of The Private Securities Litigation Reform Act of 1995. Actual results may differ materially from those indicated by such forward-looking statements as a result of various important factors, including: product development, and other factors discussed in our Registration Statement filed on Form S-1 with the SEC. In addition, the forward-looking statements included in this press release represent the Company’s views as of July 26, 2004. The Company anticipates that subsequent events and developments will cause the Company’s views to change. However, while the Company may elect to update these forward-looking statements at some point in the future, the Company specifically disclaims any obligation to do so. These forward-looking statements should not be relied upon as representing the Company’s views as of any date subsequent to July 26, 2004.

Any statements in this presentation about future expectations, plans and prospects for the Company, including statements about the Company, the Building Blocks Program and other statements containing the words “believes,” “anticipates,” “plans,” “expects,” “will,” and similar expressions, constitute forward-looking statements within the meaning of The Private Securities Litigation Reform Act of 1995. Actual results may differ materially from those indicated by such forward-looking statements as a result of various important factors, including: product development, and other factors discussed in our Registration Statement filed on Form S-1 with the SEC. In addition, the forward-looking statements included in this press release represent the Company’s views as of July 26, 2004. The Company anticipates that subsequent events and developments will cause the Company’s views to change. However, while the Company may elect to update these forward-looking statements at some point in the future, the Company specifically disclaims any obligation to do so. These forward-looking statements should not be relied upon as representing the Company’s views as of any date subsequent to July 26, 2004.


Purpose of Today’s SessionPurpose of Today’s Session� Introduction to SPE (5 Minutes)

� SPE Methodology (15 Minutes)

� Applying SPE to Building Blocks (5 Minutes)

� Performance Analysis Tools and Techniques (5 Minutes)

� Demonstration of Building Block (10 minutes)

� Answer some questions… (10 minutes)

� Introduction to SPE (5 Minutes)

� SPE Methodology (15 Minutes)

� Applying SPE to Building Blocks (5 Minutes)

� Performance Analysis Tools and Techniques (5 Minutes)

� Demonstration of Building Block (10 minutes)

� Answer some questions… (10 minutes)


Introduction to SPEIntroduction to SPEQ: What is software performance?

A: Any characteristic of a software product that you could quantifiably measure?

Q: Why is software performance important?

A: The world we live in today is becoming more digitally sophisticated. We expect our digital transactions to be faster then our paper transactions.

Q: How do we manage performance in the SDLC?

A: Plan, Predict, Prove, Improve performance throughout the SDLC?

Q: What is software performance?

A: Any characteristic of a software product that you could quantifiably measure?

Q: Why is software performance important?

A: The world we live in today is becoming more digitally sophisticated. We expect our digital transactions to be faster then our paper transactions.

Q: How do we manage performance in the SDLC?

A: Plan, Predict, Prove, Improve performance throughout the SDLC?


Introduction to SPEIntroduction to SPE� Performance is at the top of everyone’s mind.

� If Performance is poor, software adoption will decline and potentially cause usage attrition or adoption of alternative products/processes.

� Performance Failures cost institutions a large amount of unplanned money.

� Performance problems can create a huge trust barrier between user and provider of technology.

So then what is SPE?

� Performance is at the top of everyone’s mind.

� If Performance is poor, software adoption will decline and potentially cause usage attrition or adoption of alternative products/processes.

� Performance Failures cost institutions a large amount of unplanned money.

� Performance problems can create a huge trust barrier between user and provider of technology.

So then what is SPE?


Introduction to SPEIntroduction to SPESPE is a methodology that…

� Provides a systematic, quantitative approach to constructing software systems that meet performance objectives.

� Provides a software-oriented approach to architecture, design and implementation choices.

� Prescribes principles and performance patterns for creating responsive software. � Performance antipatterns for recognizing and correcting common problems,

the data required for evaluation, procedures for obtaining performance specifications, and guidelines for the types of evaluation to be conducted at each development stage.

SPE is a methodology that…

� Provides a systematic, quantitative approach to constructing software systems that meet performance objectives.

� Provides a software-oriented approach to architecture, design and implementation choices.

� Prescribes principles and performance patterns for creating responsive software. � Performance antipatterns for recognizing and correcting common problems,

the data required for evaluation, procedures for obtaining performance specifications, and guidelines for the types of evaluation to be conducted at each development stage.


SPE MethodologySPE MethodologySPE is a 7 Step Cyclical Process…

� Assess Performance Risk

� Identify Critical Use Cases

� Select Key Performance Scenarios

� Establish Performance Objectives

� Construct Performance Models

� Determine Software Resource Requirements

� Add System Resource Requirements

SPE is a 7 Step Cyclical Process…

� Assess Performance Risk

� Identify Critical Use Cases

� Select Key Performance Scenarios

� Establish Performance Objectives

� Construct Performance Models

� Determine Software Resource Requirements

� Add System Resource Requirements


SPE MethodologySPE MethodologyAssess Performance Risk

� Distinguish between new development and refactor.

� Work with requirements specialists to understand the problem/domain issues for the work.

� Understand potential inter- versus intra-system integration.

� Apply common sense and basic business logic from past experiences and similar development projects.

Assess Performance Risk

� Distinguish between new development and refactor.

� Work with requirements specialists to understand the problem/domain issues for the work.

� Understand potential inter- versus intra-system integration.

� Apply common sense and basic business logic from past experiences and similar development projects.


SPE MethodologySPE MethodologyIdentify Critical Use Cases

� Most important operations/actions in the feature or system.

� Responsiveness driven.

� Risk driven.� You look for use cases where there is a risk that, if performance goals are

not met, the system will fail or be less than successful.

Identify Critical Use Cases

� Most important operations/actions in the feature or system.

� Responsiveness driven.

� Risk driven.� You look for use cases where there is a risk that, if performance goals are

not met, the system will fail or be less than successful.


SPE MethodologySPE MethodologySelect Key Performance Scenarios

� Unlikely that all critical use cases will be important to performance.

� The key performance scenarios are those that are executed frequently, or those that are critical to the perceived performance of the system.

� Each performance scenario corresponds to a workload.

� Performance scenarios represented through sequence diagrams augmented with some useful extensions.

Select Key Performance Scenarios

� Unlikely that all critical use cases will be important to performance.

� The key performance scenarios are those that are executed frequently, or those that are critical to the perceived performance of the system.

� Each performance scenario corresponds to a workload.

� Performance scenarios represented through sequence diagrams augmented with some useful extensions.


SPE MethodologySPE MethodologyEstablish Performance Objectives

� Identify and define performance objectives.� Specify the quantitative criteria for evaluating the performance

characteristics of the system under development.

� Response time, throughput, or constraints on resource usage

� Identify and define workload objectives.� Specify the level of usage for the scenario.

� They are specified as an arrival rate (e.g., number of Web site hits per hour), number of concurrent users or number of parallel transactions.

Establish Performance Objectives

� Identify and define performance objectives.� Specify the quantitative criteria for evaluating the performance

characteristics of the system under development.

� Response time, throughput, or constraints on resource usage

� Identify and define workload objectives.� Specify the level of usage for the scenario.

� They are specified as an arrival rate (e.g., number of Web site hits per hour), number of concurrent users or number of parallel transactions.


SPE MethodologySPE MethodologyConstruct Performance Models

� Use of Execution Graphs to represent software processing steps in the performance model.

� The sequence diagram representations of the key performance scenarios are translated into execution graphs.

� Reverse Engineer � Reverse Engineer � Reverse Engineer

Construct Performance Models

� Use of Execution Graphs to represent software processing steps in the performance model.

� The sequence diagram representations of the key performance scenarios are translated into execution graphs.

� Reverse Engineer � Reverse Engineer � Reverse Engineer


SPE MethodologySPE MethodologyDetermine Software Resource Requirements

� Understand at an atomic level the types of software resource requirements.

� Method Calls

� SQL Executions

� Data Loading

� Caching

� Understand the effects of this software to inter- and intra-system components.

� Understand at best- and worst-case perspective.

� 90% of the time

� Peak time.

Determine Software Resource Requirements

� Understand at an atomic level the types of software resource requirements.

� Method Calls

� SQL Executions

� Data Loading

� Caching

� Understand the effects of this software to inter- and intra-system components.

� Understand at best- and worst-case perspective.

� 90% of the time

� Peak time.


SPE MethodologySPE MethodologyAdd System Resource Requirements

�Apply Workload Understanding

�Based on findings in the software resource requirements phase you should be able to understand at an atomic level...� CPU Cycles consumed by the method call and/or sql operation.

� Network overhead (packet transfer)

� Memory requirements

� Processing Thread/DB Connection Requirements

Add System Resource Requirements

�Apply Workload Understanding

�Based on findings in the software resource requirements phase you should be able to understand at an atomic level...� CPU Cycles consumed by the method call and/or sql operation.

� Network overhead (packet transfer)

� Memory requirements

� Processing Thread/DB Connection Requirements


Applying SPE to BlocksApplying SPE to BlocksWe chose to create our own building block for this demo…

Santa’s Little Helper

� B2 that provides performance/capacity metrics of sub-system usage within Blackboard.

� Parses an Apache access.log file

� Generate a graphical display of the following metrics – Sub-System Breakdown (Primary Graphical Requirement) – Hits Per Day in terms of Visits (Secondary Graphical Requirement) – Average Length Per Visit (Nice to Have Graphical Requirement)

� Generate a textual summary of the following metrics – Sub-System Breakdown (Primary Graphical Requirement) – Top 50 Click Paths (Primary Graphical Requirement) – Hits Per Day in terms of Visits (Secondary Graphical Requirement) – Average Length Per Visit (Nice to Have Graphical Requirement)

We chose to create our own building block for this demo…


� B2 that provides performance/capacity metrics of sub-system usage within Blackboard.

� Parses an Apache access.log file

� Generate a graphical display of the following metrics – Sub-System Breakdown (Primary Graphical Requirement) – Hits Per Day in terms of Visits (Secondary Graphical Requirement) – Average Length Per Visit (Nice to Have Graphical Requirement)

� Generate a textual summary of the following metrics – Sub-System Breakdown (Primary Graphical Requirement) – Top 50 Click Paths (Primary Graphical Requirement) – Hits Per Day in terms of Visits (Secondary Graphical Requirement) – Average Length Per Visit (Nice to Have Graphical Requirement)


Applying SPE to BlocksApplying SPE to Blocks

Report from Grinder and Study System Resource UsageAdd System Resource Requirements

Review Profiling MetricsDetermine Software Resource Requirements

Sequence Diagram AttachedConstruct Performance Models

1. Ability to Parse up to 10mb in size. (Manage via a log rotation parameter)

2. 1mb file processed in 90 seconds.

3. Conversion of text into graphic 10 seconds or less.

Establish Performance Objectives

Limited only to administrator role access to this sub-system. Expected to be executed daily.

Select Key Performance Scenarios

1. Textual Report (Default Report) of Sub-System Breakdown.

2. Textual Report of Hits Per Day.

Identify Critical Use Cases

Relatively low use case, maybe executed by 1 person on a system a handful of times per day or week. No major database activity, performed in isolation on the application server.

Assess Performance Risk

Application of B2 to SPESPE Phase


Applying SPE to BlocksApplying SPE to Blocks

Example Sequence/Workflow

Diagram


SPE Tools and TechniquesSPE Tools and Techniques� Load Induced Tools for Simulation

� Java Profiling Interface to plug into the JVMPI API.

� SQL Profiler

� Load Induced Tools for Simulation

� Java Profiling Interface to plug into the JVMPI API.

� SQL Profiler


SPE Tools and TechniquesSPE Tools and TechniquesLoad Induced Tools for Simulation

� Commercially Available Tools� Mercury LoadRunner (Primary Tool of BB Perfeng)

� Seque Silk Performer

� Freeware Tools� The Grinder (Primary Tool of BB Dev)

� Volano

� Countless others on Sourceforge

Load Induced Tools for Simulation

� Commercially Available Tools� Mercury LoadRunner (Primary Tool of BB Perfeng)

� Seque Silk Performer

� Freeware Tools� The Grinder (Primary Tool of BB Dev)

� Volano

� Countless others on Sourceforge


SPE Tools and TechniquesSPE Tools and TechniquesJava Profilers

�Commercially Available Tools� JProbe (Used by BB Perfeng)

� Performasure

� OptimizeIt

� Wily Enterprise Application Analyzer

� HyPerformix

�Freeware Tools� HPJmeter

� JProbe Light (Used by BB Dev)

� Eclipse Profiler (Used by BB Dev)

Java Profilers

�Commercially Available Tools� JProbe (Used by BB Perfeng)

� Performasure

� OptimizeIt

� Wily Enterprise Application Analyzer

� HyPerformix

�Freeware Tools� HPJmeter

� JProbe Light (Used by BB Dev)

� Eclipse Profiler (Used by BB Dev)


SPE Tools and TechniquesSPE Tools and TechniquesSQL Profilers

� Commercially Available Tools� Hotsos (Used by BB Perfeng)

� IronEye SQL

� Freeware Tools� Oracle SQL Analyze (Used by BB Perfeng and Dev)

� SQL Profiler for MS SQL Server (Used by BB Perfeng and Dev)

SQL Profilers

� Commercially Available Tools� Hotsos (Used by BB Perfeng)

� IronEye SQL

� Freeware Tools� Oracle SQL Analyze (Used by BB Perfeng and Dev)

� SQL Profiler for MS SQL Server (Used by BB Perfeng and Dev)


Demonstration…Demonstration…

Here is our new Building Block



Click Here to Study Usage

Statistics



Sub-Systems are Categorized and Distributed by Hits

Total System Hits for the Day


Demonstration…Demonstration…Contents of the Grinder.Properties File

grinder.processes=1

grinder.threads=10

grinder.runs=5

grinder.duration=0

grinder.script=tests/usage_analysis.py

grinder.logDirectory=logs

grinder.useConsole=false

grinder.initialSleepTime=5000

grinder.sleepTimeFactor=1

grinder.sleepTimeVariation=1.0

grinder.logProcessStreams=true

grinder.recordTime=true

Contents of the Grinder.Properties File

grinder.processes=1

grinder.threads=10

grinder.runs=5

grinder.duration=0

grinder.script=tests/usage_analysis.py

grinder.logDirectory=logs

grinder.useConsole=false

grinder.initialSleepTime=5000

grinder.sleepTimeFactor=1

grinder.sleepTimeVariation=1.0

grinder.logProcessStreams=true

grinder.recordTime=true



Example Grinder .py file for Load Testing




"Usage Stats"

0NA16566.94106592.2050Test 3

"Admin Panel"

0NA43182.002547.30050Test 2

"My Institution"

0NA36565.00494.32050Test 1

TransactionResponse Errors

Response Bytes Per Second

Mean Response

Length

Mean Transaction Time (ms)

ErrorsIterationsTest










Question and Answer…Question and Answer…

Documents

B2conference performance 2004