Upload
appvance-inc
View
2.964
Download
2
Tags:
Embed Size (px)
DESCRIPTION
This Open Source Test Workshop is for senior IT and business executives needing visibility and management tools and methodology into all the demand for IT. Shows how to bring Open Source Testing into your organization.
Citation preview
Open Source Test Workshop
Frank Cohen, +01 (408) 871-0122, [email protected]
June 8, 2011
Surface and Mitigate Performance Bottlenecks
For CIOs, CTOs, Senior Managers
Wednesday, June 8, 2011
Open Source Test Automation
Slides on Twitter‣http://twitter.com/fcohen
‣Please retweet
‣Please follow me
‣#ost
2
Wednesday, June 8, 2011
Open Source Test Automation
About PushToTest‣Our Mission: We Test, For A More Reliable World
‣Make It Easier To Move From Manual To Automated Testing
‣Support Advanced Testing, including Agile, Load, Integration
‣Open Source Testing (OST) For Everyone
3
Wednesday, June 8, 2011
XML, SOA, and Performance
Web 2.0 Success Driving Business‣Every Organization Wants Modern IT Functions and User Interfaces
‣Rich User Interfaces (RIA, using Ajax, Flash, Flex)
‣SOA Composite Applications
• Faster Time-To-Market
• Increase Revenue
• Cost Savings
‣Service Enabled Applications
• Deliver More Integration Faster
• Always-On
• Less Cost
4
Wednesday, June 8, 2011
XML, SOA, and Performance
The Reality of Web 2.0 Development‣We Build Applications Differently Now
‣Haze of New Protocols, Software Models, Data Formats
‣Agile Often Means Shorter Build-Test-Deploy ‘Waterfall-style’ Cycles
‣Many Still Trying “Test And Trash” Techniques
‣Uptime Depends on Business Service Management (BSM) Testingto Surface Functional and Performance Issues
5
of the Application Changes
Tests Re-recorded
30%Every 8 weeks
30%Wednesday, June 8, 2011
Open Source Test Automation
‣Proprietary Test Tools Offer:
‣QA Labs for 10,000 Users = $35,000 Per Year
‣Does Not Include: Cost of Test Employees, Root Cause Analysis Consultants
‣Does Not Guarantee Success
Testing Costs
6
Functional Test
Functional Test Platform $19,000
Functional Tests $8,000
Protocols: AJAX, SOAP $12,000
Support $5,000
$40,000
Load Test
Load Test Platform $20,000
Simulated User Licenses $120,000
Recoding The Test $35,000
Re-quality The Test $12,000
$187,000
Wednesday, June 8, 2011
Open Source Test Automation
Proprietary Testing Falling Behind‣Adobe Releases Testing APIs for Flex/Flash in 2009
‣JQuery, GWT, YUI, Extjs Each Make 5 Releases in 2010
‣Appcelerator Titanium Makes 3 Releases in 2010
‣Microsoft Draws 8,000 to Mix for StarLight in 2010
‣StarEast, Better Software, PyPy, JavaOne, Devoxx, OSBC, LinuxWorld Feature Keynote Talks On Ajax
7
‣ “HP Refunded Our Payment For The Ajax Protocol Handler”, a PushToTest customer in the automotive industry.
Wednesday, June 8, 2011
Open Source Test Automation
Lots Of Options
Open Source Testing Emerges
8
Selenium soapUI Sahi
HtmlUnit DBUnit Watir
Windmill Hudson TestMaker jMeter
‣Signs of Maturity: Record/Playback, Data Generation/Importer, Iteration Engine, Logger, Recovery Scenario Manager, Object Recognizer
Fitness FIT
FlexMonkeyTG4W
Wednesday, June 8, 2011
XML, SOA, and Performance
Depend On A Service Partner
9
‣Repurposing: One Test Script For Functional Tests, Load and Performance Tests, and Business Service Monitors
‣Components: Tests Run Even After Application Changes
‣Deployment: Run Tests In Your Lab, Ours, or Both!
‣Services: Add Expertise, Training, Tools To Your Organization
‣Enthusiasm: We participate and contribute to Open Source Test Projects
Wednesday, June 8, 2011
Open Source Test Automation
Manual Testing Process
10
Manual
Testing
Explore
1
2
Test Plan
Create Test Plans
Run The Test Steps
3
4
Results
Recommend Changes
Wednesday, June 8, 2011
Open Source Test Automation
Side Effects
11
‣Exhaustion
‣Fatigue
‣Blurry Vision
‣Night Sweats
‣ Insanity
‣Hair Loss
‣Acne
‣Divorce
Wednesday, June 8, 2011
Open Source Test Automation
OST Architecture
12
Functional
Test
Production
Monitor
Load Test
Continuous Integration
Repository Bug TrackerTest
ManagementMonitoring
Test Authoring
SoapUI (Pro)
Sahi (Pro)
JUnit Class Framework
Eclipse Java IDE
Fitness
Languages: Java, Python
Wiki Collaboration
Application Under TestWeb, RIA, BPM, SOA
PTTMonitorCPU, Net, Memory
Test Injector
BundlesMySQL, HtmlUnit
ScriptRunnersSahi, Selenium, SoapUI, Java
Data Production Librariescsv files, rdbms queries
Test Operation Console
RepositoryMySQL RDBMS
Results AnalysisBusiness Intelligence Tool
DesignerRecord/Playback
Grid and CloudTest Deployment
Wednesday, June 8, 2011
Open Source Test Automation
PTTMonitor
Load Injectors
An OST Cloud
13
Application Under Test
Central location from which a tester, developer, IT manager, CI service runs and monitors tests
TestMaker
Console
Thousands of virtual users perform real-life scenarios on the Application Under Test
TestNode
Stores results of every transaction, step, and monitored resource
Results DB
400+ charts identify the root cause to performance bottlenecks, functional issues
Analysis
Wednesday, June 8, 2011
Open Source Test Automation
PTTMonitor
Load Injectors
An OST Cloud
14
Application Under Test
Central location from which a tester, developer, IT manager, CI service runs and monitors tests
TestMaker
Console
Thousands of virtual users perform real-life scenarios on the Application Under Test
TestNode
Stores results of every transaction, step, and monitored resource
Results DB
400+ charts identify the root cause to performance bottlenecks, functional issues
Analysis
Wednesday, June 8, 2011
Open Source Test Automation
Browser Compatibility Tests
15
Application Under Test
TestMaker Console
TestNode
Selenese Test
Selenium RC SeleniumHtmlUnit
Wednesday, June 8, 2011
Open Source Test Automation
16
ScriptRunnersScriptRunner Type Name (langtype)
Designer Script IE, Htmlunit designerscript
Selenium HtmlUnit Browser selenium
Selenium RC IE, Firefox, Chrome seleniumrc
Sahi IE, HtmlUnit sahi
HTTP Archive Page Optimization HAR
Selenium RC IE, Firefox, Chrome seleniumrc
SoapUI SOAP and REST soapui
TestGen4Web Web 1.0 Tests testgen4web
Mozmill Mozilla XUL Tests mozmill
Unit Tests jUnit TestCase Java, Jython, JRuby, PHP, Perl, Groovy
DotNet .NET Tests dotnet
Visual Basic VB tests vb
Command Line Desktop and utils commandline
Wednesday, June 8, 2011
Open Source Test Automation
Component Approach
17
Data Production
Log-In
Component
setUp
runTest
tearDown
ApplicationTestScenario
Test Use Case
Product Search
Order Product
Log-Out
RDBMS DPL
CSV DPL
Custom DPL
Load Test
100 Users 500 Users 1000 Users
Log-In
Component
Component
Development
Database
QA Regression
Database
Load Test
Database Confirm Order
Production
Database
Wednesday, June 8, 2011
Open Source Test Automation
You’re Such A Geek!
Lots of Possibilities
18
Seattle, DatacenterTestNode
TestMaker
Dallas, TexasDatacenter
Generating Load
Test Sent
Wednesday, June 8, 2011
Open Source Test Automation
TestMaker Demonstration‣Fun And Easy Test Script Creation
‣Selenium and TestMaker
‣Repurposing A Single Test
‣Functional Test, Load and Performance Test, Production Monitor
‣Results Analysis
‣Chart Engine
19
Wednesday, June 8, 2011
Open Source Test Automation
Root Cause Analysis
20
‣PTTMonitor - Agent Based Monitor Gateway
‣Glassbox - Performance Monitoring for Java Applications
‣dynaTrace - Analysis In Application and DB Tiers
Wednesday, June 8, 2011
Open Source Test Automation
‣Correlate Test Operation To Resource Usage
‣Deploy as Web Application (WAR) and Service
‣Observes CPU, Network, Memory Utilization As Test Runs
‣Gateway to Glassbox, dynaTrace, Windows PerfMon
21
PTTMonitor
Wednesday, June 8, 2011
Open Source Test Automation
Results Analysis Engine
22 And 350 Additional ChartsWednesday, June 8, 2011
Open Source Test Automation
‣Correlate Test Operation To Application Tier Issues
‣Memory Leaks
‣Thread Deadlocks
‣Slow DB Connections
‣No Instrumentation of Applications
‣Live Performance Alert View
‣Run In Production
23
Glassbox
Wednesday, June 8, 2011
Open Source Test Automation
Glassbox Troubleshooter
24
Wednesday, June 8, 2011
Open Source Test Automation
‣PurePath from TestMaker Virtual Users To Application & DB Tier
‣Answers: What Are The Transactions and Why Are They Slow?
25
dynaTrace PurePaths
Application Server
dynaTrace monitors
applications for
functional issues
and performance
bottlenecks.
Test RunTime
PTTMonitor
dynaTrace Agent
TestNode
test
TestNode
test
TestNode
test
TestMaker Console
Database dynaTrace Agent
PurePath Token
Wednesday, June 8, 2011
Open Source Test Automation
Pinpoint The Problem‣Down to the Line of Source Code
26
Wednesday, June 8, 2011
Open Source Test Automation
‣Correlate PerfMon Statistics with Test Operation
‣PTTMonitor Gateways Data from PerfMon or MS SQL Server
.NET Monitoring
27
.NET Application Tier
Test RunTime
PTTMonitorTestNode
test
TestNode
test
TestNode
test
TestMaker Console
MS SQL ServerPerfMon
SOAP HTTP REST
Wednesday, June 8, 2011
Next let’s talk about
Open Source Test Automation
PushToTest Options
‣OnDemand
‣QuickStart
‣POC Option
28
Wednesday, June 8, 2011
Open Source Test Automation
Immediate Benefits‣Product Solutions
‣Better, Lower-Cost, More Efficient Test Tool
‣Business Service Monitoring at the Application Tier (SLA Compliance)
‣Methodology Solutions
‣ Improved Test Methodology for my Organization and Team
‣Services
‣Turnkey Load Testing
‣ Integrate TestMaker Into My Build Environment
‣Add A Protocol, Chart, Task to TestMaker To Meet My Needs29
Wednesday, June 8, 2011
Open Source Test Automation
Dual License
‣ Functional Testing with TestMaker Object Designer
‣ Repurposing Selenium, soapUI Sahi and unit tests into data-driven functional tests, load and performance tests, and business service monitors
‣ Load Testing To 50 Virtual Users
‣ Data-driven tests using CSV files.
‣ PTTMonitor Correlates CPU, Network, Memory Utilization As Tests Operate.
‣ 25 Charts and Reports
‣ Community Support and access to all documentation
30
‣ TestMaker Enterprise expands on the success of TestMaker Community
‣ Load Testing to Millions of Virtual Users
‣ Cloud and Grid Testing
‣ Data-driven tests using Data Production Libraries for CSV files, RDBMS, Data Generating Objects
‣ Root cause analysis and mitigation using Glassbox
‣ Mock services for agile service development
‣ Flex AMF Protocol Handler for Flex/Flash testing
‣ 1000+ Charts and Reports
‣ Business service monitors, including email notification
‣ Support, QuickStart, Training, Consulting
Wednesday, June 8, 2011
Open Source Test Automation
Pricing‣Annual Subscription Model Based On:
‣Seats for TestMaker Console Installation
‣Seats for SOAP and REST Test Authoring
‣Virtual User Levels for Load Testing
‣Count of Business Service Monitor
‣Technical support: 1 business day or faster response to emails, phone calls, instant messages
31
Wednesday, June 8, 2011
Open Source Test Automation
32
‣As-Needed, On-Demand Basis
‣Test In Your Environment, Or Our Cloud Test Environment, Or Both
‣ Ideal For Consulting Organizations Where Test Needs Are Unknown
‣AMI Distribution
‣Pay Only For What You Use
‣Annual Cap On Total Costs
Wednesday, June 8, 2011
Open Source Test Automation
The QuickStart Option
33
‣Your First Test, Built and Operated by PushToTest Experts
‣ Ideal For Organizations With Little or No Load Testing Experience
‣PushToTest Experts Build Your First Test
‣PushToTest Experts Operate The Test, Provide Root Cause Analysis and Mitigation
‣PushToTest Trains Your Team To Build Their Own Tests
‣$9,750 USD
Wednesday, June 8, 2011
Open Source Test Automation
The POC Option‣PushToTest Experts Help Your TestMaker Evaluation
‣Ensures TestMaker Works In Your Application/System
‣PushToTest Issues Price Proposal
‣Applies Engineers To Build and Operate Your Test
‣PushToTest Trains Your Team To Use TestMaker
‣Success of POC Confirms Intention To Purchase
‣Cost: $034
Wednesday, June 8, 2011
Open Source Test Automation
PushToTest Global Services‣Put Our Experts To Work For Your Projects
‣Methodology, Solutions, Support, Enthusiasm
‣Offices: USA (Texas and California,) Bangalore, London
‣The Near Shore Option
‣150 Test Experts
‣Costa Rica: CST, English, Inexpensive, On Site in 3-5 Hours
‣http://www.pushtotest.com/services
35
“PushToTest Uniquely Built Load Tests For Our Customers, Including Pepsi, And Worked With Our
Engineers To Surface and Solve Performance Bottlenecks.” Alan Rodriguez, Director of Testing, Tribal DDB Dallas
Wednesday, June 8, 2011
Open Source Test Automation
Slides on Twitter‣http://twitter.com/fcohen
‣Please retweet
‣Please follow me
‣#ost
36
Wednesday, June 8, 2011
Open Source Test Automation
Where To Go From Here‣Please Complete The Evaluation Form
‣And We Will Send You The Slides
‣Watch A Screencast and Tutorial
‣http://www.pushtotest.com/screencastcentral
‣Attend An Open Source Test Workshop
‣http://workshop.pushtotest.com
‣Ask For A Proposal on Licenses, Training, Support, Consulting
‣Call +01 408 871 0122, [email protected]
37
Wednesday, June 8, 2011