Upload
dr-ganesh-iyer
View
524
Download
5
Embed Size (px)
Citation preview
© 2015 Progress Software Corporation. All rights reserved.4
Manual Testing vs Automated Testing
Manual software testing• performed by a human sitting in front of a computer carefully going through application
screens, trying various usage and input combinations, comparing the results to the expected behavior and recording their observations
• Manual tests are repeated often during development cycles for source code changes and other situations like multiple operating environments and hardware configurations
Automation Testing (using a tool)• An automated testing tool is able to playback pre-recorded and predefined actions, compare
the results to the expected behavior and report the success or failure of these manual tests to a test engineer
• Once automated tests are created they can easily be repeated and they can be extended to perform tasks impossible with manual testing
• Because of this, savvy managers have found that automated software testing is an essential component of successful development projects
© 2015 Progress Software Corporation. All rights reserved.5
Automation
Functional Automation (Black box testing)• Windows applications – Winrunner, QTP• Web applications – Silk Test, Selenium, Telerik Test Studio• Regression Testing – Same set of tools
Performance/Load Testing• Load Runner, Jmeter
Security Testing (black box and White box)• Veracode, ZAP
© 2015 Progress Software Corporation. All rights reserved.8
WinRunner
Designed to help customers save testing time and effort by automating the manual testing process
Manual process• Perform operations by hand, • visually check results, and • log results by hand
Automated process: • Create a test script that will perform the same operations as a Human operator• check the same information, and • create a summary report showing the test status
© 2015 Progress Software Corporation. All rights reserved.9
Recording Modes
• Two Modes• Context-sensitive mode• Analog mode
• Tests can combine both recording modes
• Context-Sensitive is the default mode
• Switch between modes using same record key (F2)
© 2015 Progress Software Corporation. All rights reserved.10
Context-Sensitive Mode• Object-based
• Unaffected by minor UI changes
• Maintainable (readable/editable)
• Generally used with GUI applications
• Portable script
© 2015 Progress Software Corporation. All rights reserved.11
Context-Sensitive Mode
set_window ("Save As");edit_set(“File Name”,“output14”);
output14
Set focus to the window
Set the edit field content
Button press
button_press(“OK”);
© 2015 Progress Software Corporation. All rights reserved.12
Analog Mode• Position-dependent
• Works with any application
• UI changes force test script changes
• Usually drives tests with • mouse, • keyboard and • other such manual user inputs
• Less maintainable
y
x
© 2015 Progress Software Corporation. All rights reserved.13
Analog Mode
move_locator_track (1);
mouse drag
mtype (" <T55> <kLeft>-<kLeft>+"); mouse click
output14
type (" <t3>output14" );
keyboard
move_locator_track (2); mtype (" <T35><kLeft>-<kLeft>+ ");
timing
© 2015 Progress Software Corporation. All rights reserved.14
Recording Modes Context-Sensitive mode statements can be recorded or
programmed• record: button_press, win_activate• program: list_get_num_items, edit_get_text• recommended for most situations due to greater robustness
Analog mode statements are rarely programmed, mostly recorded and edited• record: move_locator, type, mtype• program: move_locator_abs, move_locator_rel, click• Analog statements are useful for literally describing the keyboard, mouse, and mouse button
input of the user
© 2015 Progress Software Corporation. All rights reserved.15
Run Modes Debug
• debug is good to use while the test script is being debugged• these test results are overwritten with each new run
Verify• Corresponds to actual results• Generally used when executing testing sessions where results need to be stored
Update• Corresponds to “expected” results. Expected results are the benchmarks used to verify
test results• Test runs in “Update” mode generate the expected results for future runs to compare
back against• These test results become the expected results for subsequent test runs in “Verify”
mode
© 2015 Progress Software Corporation. All rights reserved.16
Synchronization Enhances a test script to ensure reliable replay
• accounts for delays in order to prevent the automated script from running faster than the tested application
• critical for successful test automation implementation• among the main reasons why record-n-playback is not reliable
In Context-Sensitive mode• Examples: (operations)
– wait for a window to appear– wait for a bitmap to refresh– wait for an object property– wait for a specific amount of time
In Analog mode• Examples: (operations)
– wait for a window bitmap to appear / refresh– wait for a specific amount of time
© 2015 Progress Software Corporation. All rights reserved.17
Window Synchronization
invoke_application(“Notepad”,””, “c:\\temp”, SW_SHOW);set_window (”Login”, 10);
edit_set(“User ID:”, “guest”);edit_set(“Password:”, “mercury”);
button_press(“OK”);set_window Waits for the specified window to appear onscreen. If the
window appears before the timeout, the script immediately proceeds to the next line.
© 2015 Progress Software Corporation. All rights reserved.20
TSL (Test Script Language) TSL is a C-like language
• High-level proprietary programming language designed for test automation• procedural language
Full programming support• variables, arrays, functions• regular expressions• control flow, decision logic, looping
© 2015 Progress Software Corporation. All rights reserved.21
Built-in TSL Functions TSL provides a comprehensive library of hundreds of built-in functions to
simplify test creation• window/object functions• environment functions• reporting functions• database query functions• file/spreadsheet functions• Win32 functions
WinRunner provides a graphical function browser to assist you • Function Generator
© 2015 Progress Software Corporation. All rights reserved.23
Language Syntax
*** Same syntax as in standard C ***
© 2015 Progress Software Corporation. All rights reserved.24
Variables Basic Rules
• do not need to be declared / defined• specific data types are not explicitly
defined• case sensitive• first letter must be a character or
underscore• cannot be a reserved word• by default all variables are local
(static)• can also be public and/or const
Arrays• single dimension: cust[1], cust[2],
cust[3]• multi-dimension: address[1,1],
address[1,2]• Can be indexed with number
– address[1], address[2]• Can be indexed with strings
(associative)– address[“John”], address[“Mary”]
© 2015 Progress Software Corporation. All rights reserved.25
Operators Math
+ - * / ^ % ++ -- Logical
&& || ! Relational
== != >= <= > < Assignment
= += -= *= /= ^= %= Concatenation
&
© 2015 Progress Software Corporation. All rights reserved.26
Test Verification Enhancing a test script to verify data onscreen
• check objects’ values / states• check images• check text• check the database
Context-Sensitive verification
Analog verification
© 2015 Progress Software Corporation. All rights reserved.27
Exception/Recovery Handling (*)
Unexpected errors during replay• unlike error-handling, these can appear at any time when running a script
WinRunner provides a mechanism to trap and handle exceptions• popup exceptions
– popup windows
• object exceptions– object property value changes
• TSL exceptions– TSL error codes
© 2015 Progress Software Corporation. All rights reserved.28
Functions and Libraries simplifies building test frameworks
• application-specific functions• general-purpose functions• greater modularity
can be stored in a script• compiled module (function library)• can be loaded as part of startup or initialization script and available globally
facilitates data-driven testing• data-driven testing is where data retrieved externally from the test being executed drives
the test rather than using hard-coded data within each test case. Using application specific custom functions and scripts helps further the benefits of data-driven testing.
© 2015 Progress Software Corporation. All rights reserved.30
What is SilkTest ?
A powerful tool for running automated test cases on the front end
• A tool for testing Web based applications across different browsers
• Very very stupid – you have to tell it everything in its own language (4test)
• Inflexible when it comes to interpreting your commands• It cannot guess at what you mean• It requires a certain syntax with certain words
– This can be quite maddening
© 2015 Progress Software Corporation. All rights reserved.31
What can you do with it ?
Mostly anything you can do in a Web browser, you can do with SilkTest• Enter text in text fields• Click buttons• Check boxes• Get files• Clear your cache• etc.
© 2015 Progress Software Corporation. All rights reserved.32
Using a browser
Though SilkTest is its own product, you still have to use a browser to access your application
You have to tell SilkTest what kind of browser you are going to use for your testing so that it knows what it is looking at.
However, you can only run SilkTest on one instance of a browser at a time• One IE browser going at a time
© 2015 Progress Software Corporation. All rights reserved.33
Architecture
Project .vtp (for maintenance) Testplan .pln (organizing the tests) Test Frame* .inc (AUT Window details) Scripts .t (test scripts) Data driven tests .g.t (scripts with many data) Include file .inc Suite .s (for batch runs) Result .res
© 2015 Progress Software Corporation. All rights reserved.34
Test Process
Manage requirements using SilkCentral Test Manager Manage test plans and cases using Silk test Associate test plans/cases to requirements using test manager Declare GUI declarations using SilkTest Develop scripts for atomic activities first so that they can be
reused later Develop scripts using SilkTest and run them Track defects using Defect Tracker
© 2015 Progress Software Corporation. All rights reserved.36
Overview of Selenium
Selenium is a robust set of tools that supports rapid development of test automation for web-based applications.
Selenium provides a rich set of testing functions specifically geared to the needs of testing of a web application.
Selenium operations are highly flexible, allowing many options for locating UI elements and comparing expected test results against actual application behavior.
Supports Cross Browser Testing. The Selenium tests can be run on multiple browsers. Allows scripting in several languages like Java, C#, PHP and Python. Assertion statements provide an efficient way of comparing expected and actual
results. Inbuilt reporting mechanism.
© 2015 Progress Software Corporation. All rights reserved.39
Why Load Test An Application
Does the application respond quickly enough for the intended users?
Will the application handle the expected user load and beyond?
Will the application handle the number of transactions required by the business?
Is the application stable under expected and unexpected user loads?
© 2015 Progress Software Corporation. All rights reserved.40
Functional vs. Load Web Testing
Functional test
Load test
Do business processes function properly after
implementation?Functionality
EXAMPLEOBJECTIVE
Will 2,000 concurrent hits crash the server?
Stability
Is response time acceptable according to specifications?
Performance
Do business processes function properly under
heavy load?
Functionality under load
EXAMPLEOBJECTIVE
© 2015 Progress Software Corporation. All rights reserved.41
Types of Performance Testing
Find out whether the system can handle the expected load upon deployment under real-world conditions.
Load Testing
Find the application's breaking point. Apply testing that measures whether the application's environment is properly configured to handle expected or
potentially unexpected high transaction volumes.
Stress Testing
Find the stability of the system with respect to handling large amounts of data over extended time periods.Volume Testing
Find the behavior and performance of each tier. Component
Testing
© 2015 Progress Software Corporation. All rights reserved.42
Objectives of Performance Testing
How long does it take to complete a task?Application Response Time
Reliability
Configuration Sizing
Capacity Planning
Acceptance
Bottleneck Identification
Regression
Product Evaluation
How Stable is the system under a heavy work load?
Which configuration provides the best performance level?
At what point does degradation in performance occur?
Is the system stable enough to go into Production?
What is the cause of degradation in performance?
Does the new version of Software adversely affect response time?
What is the best server for 100 users?
© 2015 Progress Software Corporation. All rights reserved.43
Testers
Load Generation System Under Test
Do you have the testing resources?• Testing personnel• Client machines
How do you synchronize users?How do you collect and analyze results?How do you achieve test repeatability?
Analysis?123.20
All of you, click the GO button
again
Manual Testing Is Problematic
Web server Database server
Coordinator
© 2015 Progress Software Corporation. All rights reserved.45
System Under Test
The LoadRunner Solution
Web server Database server
Load Generation
Vuser host
Overcomes resource limitations
• Meaningful results with analysis tools• Repeats tests with scripted actions
AnalysisController
• Runs many Vusers on few machines• Replaces testers with “Virtual Users”
• Controller manages the Vusers
© 2015 Progress Software Corporation. All rights reserved.46
Load Testing Tools Available
LoadRunner : HP (Formerly Mercury Interactive) e-Load :Emprix Silk Performer : Borland (Seague) QALoad : Compuware Rational Performance Tester : IBM Rational Web Load : Radview Neo Load : Neotys Open STA : Open Source Jmeter Telerik Test Studio
© 2015 Progress Software Corporation. All rights reserved.47
Introduction to Load Runner• Load Runner is a Mercury Interactive Tool that predicts performance and
behavior of the system
• By creating lots of load, you can see how the system reacts at peak levels or with simultaneous Users
• To test the application, LoadRunner emulates an environment where multiple users work concurrently. While the application is under load, LoadRunner accurately measures and analyzes the system performance, and its functionality
© 2015 Progress Software Corporation. All rights reserved.48
Supporting Environments• Application Deployment Solution - The Citrix protocol. • Client/Server - MS SQL, ODBC, Oracle Web Applications 11i and
DNS protocols. • Custom - C templates, Visual Basic templates, Java templates,
Javascript, and VBScript • Distributed Components - COM/DCOM, Corba-Java, and Rmi-Java
protocols. • E-Business - FTP, LDAP, Palm, Web (HTTP/HTML), Web Services• Enerprise Java Beans -EJB Testing and RMI-Java protocols. • ERP/CRM - Baan, Oracle NCA, Peoplesoft 8, SAP-Web, SAPGUI
© 2015 Progress Software Corporation. All rights reserved.49
Supporting Environments• Legacy
Terminal Emulation (RTE). • Mailing Services
Internet Messaging (IMAP), MS Exchange (MAPI), POP3, and SMTP. • Streaming
MediaPlayer and RealPlayer protocols. • Wireless
i-Mode, VoiceXML, and WAP protocols.
Internet/ Intranet
Web ServersClients App. Servers Database Server
© 2015 Progress Software Corporation. All rights reserved.5050
Supporting Environments
• Platforms• NT, 2000, XP• Sun• HP• IBM• Linux
© 2015 Progress Software Corporation. All rights reserved.51
The LoadRunner Solution
Virtual User Generator
Generates load and collects test results
Creates Scripts as one Single User.
Compiles and displays test results with
graphical and statistical tools
LoadRunner Controller
LoadRunner Analysis
© 2015 Progress Software Corporation. All rights reserved.52
LoadRunner Terminology
• Scenarios• Using LoadRunner, you divide your application performance testing requirements into
scenarios.• A scenario defines the events that occur during each testing sessions.• For example, a scenario defines and controls the number of users to emulate, the
actions that they perform, and the machines on which they run their emulations. • Vusers
• In a scenario, LoadRunner replaces human users with virtual users or Vusers.• When you run a scenario, Vusers emulate the actions of human users—submitting
input to the server. • A scenario can contain tens, hundreds, or even thousands of Vusers.
© 2015 Progress Software Corporation. All rights reserved.53
LoadRunner Terminology
• Vuser Scripts• The actions that a Vuser performs during the scenario are described in a Vuser
script.• When you run a scenario, each Vuser executes a Vuser script. Vuser scripts
include functions that measure and record the performance of the server during the scenario.
• Transactions• To measure the performance of the server, you define transactions. • Transactions measure the time that it takes for the server to respond to tasks
submitted by Vusers.
© 2015 Progress Software Corporation. All rights reserved.54
LoadRunner Terminology
• Rendezvous Points• You insert rendezvous points into Vuser scripts to emulate heavy user load on the server. • Rendezvous points instruct multiple Vusers to perform tasks at exactly the same time. • For example, to emulate peak load on the bank server, you insert a rendezvous point to
instruct 100 Vusers to simultaneously deposit cash into their accounts.
• Controller• You use the LoadRunner Controller to manage and maintain your scenarios. • Using the Controller, you control all the Vusers in a scenario from a single workstation.
© 2015 Progress Software Corporation. All rights reserved.55
LoadRunner Terminology
• Hosts• When you execute a scenario, the LoadRunner Controller distributes each Vuser in the
scenario to a host. • The host is the machine that executes the Vuser script, enabling the Vuser to emulate
the actions of a human user.
• Performance Analysis• Vuser scripts include functions that measure and record system performance during
load-testing sessions. • During a scenario run, you can monitor the network and server resources. • Following a scenario run, you can view performance analysis data in reports and
graphs.
© 2015 Progress Software Corporation. All rights reserved.56
LoadRunner Components
ControllerVuGen
Analysis
LoadRunner
Tuning
© 2015 Progress Software Corporation. All rights reserved.58
TuneSystem Based
on Analysis
LoadRunner Expert Workflow“The Big Picture”
AnalyzeSystem Under Load
Phase 5
LoadRunnerV U G E N
LoadRunnerC O N T R O L L E R &
A N A L Y S I S
RunScenarios
Phase 4
CreateScenarios
Phase 3
Create Web Virtual Users
Phase 2Phase 1
Plan Load Test
© 2015 Progress Software Corporation. All rights reserved.60
Introduction to JMeter
Definition • JMeter is an Apache Jakarta project that can be used as a load testing tool for analyzing and
measuring the performance of a variety of services, with a focus on web applications.
Why ?• Free • JMeter can be used as a unit test tool for JDBC database connection, FTP, LDAP,
WebServices,J MS, HTTP and generic TCP connections. • JMeter can also be configured as a monitor, although this is typically considered an ad-hoc
solution in lieu of advanced monitoring solutions.
© 2015 Progress Software Corporation. All rights reserved.61
Load test Process
System Analysis Creating Virtual User Scripts Defining User Behavior Creating a Load Test Scenario Running Load Test Scenario Analyzing Results
© 2015 Progress Software Corporation. All rights reserved.62
System Analysis
Enable customers to convert their goals and requirements into a successful test script.
you can: -Create your own test Script consistently - Careful UI design allows you to create your own test Easily
© 2015 Progress Software Corporation. All rights reserved.63
Creating Virtual User Scripts
- Tester must emulate the real user by driving the real application as a client. - JMeter support this by adding thread group element, this tell JMeter:- the number of users you want to simulate, - how often the users send requests - how many requests they send.- What request (FTP Request, HTTP Request, and JDBC request). - validate that your application is returning the results you expect.
© 2015 Progress Software Corporation. All rights reserved.64
Defining User Behavior
JMeter allows you to simulate human actions more closely - By controlling how long JMeter engine delays between each sample - So, you can define the way that the script runs
© 2015 Progress Software Corporation. All rights reserved.65
Creating a Load Test Scenario
JMeter offers support for this step by:
Assign scripts to individual virtual user Tester can define any number of virtual users needed to run the tests Allows user to simulate concurrent connections to server application.By
enables user to create multiple threads ( virtual user) executing different test plan.
Increasing the number of virtual users in a controlled fashion.
© 2015 Progress Software Corporation. All rights reserved.66
Running Load Test Scenario
With JMeter you can run your tests in a very easy way
© 2015 Progress Software Corporation. All rights reserved.67
Analyzing Results
JMeter offers support for this step by:
displaying the data visually (Graph Results) . save data in file. allows user to see one multiple views of the data displays the response from the server. shows the URL of each sample taken . listeners will show different sets of data. it can send email based on test results.
© 2015 Progress Software Corporation. All rights reserved.68
Analyzing Results
Showing detailed performance results that can be easily understood and analyzed to quickly pinpoint the root cause of problems.
© 2015 Progress Software Corporation. All rights reserved.70
Souce code Testing Utilities
http://sqa.fyicenter.com/FAQ/Testing-Tools/Other_Source_Code_Testing_Tools.html
© 2015 Progress Software Corporation. All rights reserved.71
Why source code testing?
Code coverage• To determine what proportion of your project's code is actually being tested by coded tests
such as unit tests• code coverage is a measure used to describe the degree to which the source code of a
program is tested by a particular test suite. • A program with high code coverage has been more thoroughly tested and has a lower
chance of containing software bugs than a program with low code coverage
Enables in white box testing• method of testing software that tests internal structures or workings of an application, as
opposed to its functionality (i.e. black-box testing)• also known as clear box testing, glass box testing, transparent box testing, and
structural testing
© 2015 Progress Software Corporation. All rights reserved.72
Advantages of source code testing
Enables to test better
Enables to understand uncovered test scenarios
Ensuring messages for various scenarios are correct• Error messages• Notifications
© 2015 Progress Software Corporation. All rights reserved.73
Source Code Testing Utilities
Splint / LCLint (Lint - static source code analysis tool)Splint (formerly LCLint) is a rewrite of the famous Unix lint utility. It is a static source code analyser, that checks your C program source code for bugs and potential problems. This static source code analysis tool requires a certain amount of configuration before you can use it productively. There are versions for a number of operating systems, including Win32, Linux and other Unix systems.
Ftncheck (Lint utility / Static Source Code Analysis Utility for Fortran)ftncheck, a static source code analysis utility, checks your Fortran programs for semantic errors, leaks and potential bugs. For example, you can use ftncheck to scan your program for the use of uninitialized variables and undeclared variables.
Jlint: Java Program Checker (Lint)Like the lint utility which checks C programs, jlint is a static source code analysis tool that scrutinizes your Java source code for suspicious constructs that might indicate a program bug. Although the program is shareware, it is distributed with source code without any restrictions (and hence is listed here).
http://sqa.fyicenter.com/FAQ/Testing-Tools/Other_Source_Code_Testing_Tools.html
© 2015 Progress Software Corporation. All rights reserved.74
Future of Software Testing and
Test Engineer
© 2015 Progress Software Corporation. All rights reserved.80
What does this mean from a Testing perspective?
Seamless Integration
Sky is the limit for a Software Test Engineer• Internet of Things• Big Data and Analytics• Social• Security and Privacy• Open Hardware
© 2015 Progress Software Corporation. All rights reserved.81
Dimensions of a Quality Analyst
Technical QA
Consultant
Analyst
Coaching
DevOps
© 2015 Progress Software Corporation. All rights reserved.82
Career Opportunities as Software Quality (Test) Engineer
Every software needs to be QAed
Familiarize with Popular tools• Web UI Testing Selenium• Performance Testing JMeter • Security Testing ZAP
© 2015 Progress Software Corporation. All rights reserved.83
ACM Membership Benefits
Computing Information Resources: Learning Center with resources for lifelong learning
• Online courses targeted toward essential IT skills and popular certifications• Online books from Books24x7®, Safari®, Morgan Kaufmann and Syngress• Webinars on hot topics, presented by today's innovators
The option to subscribe to the full ACM Digital Library, which includes over 2 million pages of text A full year (12 issues) electronic subscription to Communications of the ACM A full-year (4 issues) subscription to XRDS: Crossroads, ACM's Student Magazine Complimentary e-mentoring services from MentorNet A full-year electronic subscription to monthly MemberNet and quarterly Student Quick Takes, ACM's
member newsletters. A full-year subscription to TechNews, ACM's tri-weekly email IT news digest A full-year subscription to CareerNews, ACM's twice monthly email career news digest acmqueue website featuring articles, commentary, blogs, roundtables, case studies, mutlimedia and more Access to Ubiquity, ACM's IT opinion magazine and forum, and eLearn, ACM's magazine dedicated to
distance education