43
Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

Embed Size (px)

Citation preview

Page 1: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

1

Automated Software Testing using Open Source Testing Tools

By Elfriede Dustin

Page 2: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

2

http://www.idtus.com

Agenda

• Who is IDT• State of Software Testing• What is Automated Test and Re-test (ATRT) • Approach to ATRT• Advantage of open source• ATRT and open source – Our IDE and solution• ATRT Challenges

Page 3: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

3

http://www.idtus.com

Who is IDT?

• IDT specializes in the design, development, and implementation of Automated Software Testing and Re-Test (ATRT) and Quality Assurance (QA) solutions– Deliver turn key automated test suite

• Automated test strategy and identification of highest payoff areas to apply automation for your project

• Selection of best automation tools for your project

• Using your existing test cases or develop test cases for you

• Test results documented in requirements traceability matrix

– Training and pilot project implementation with companies

Page 4: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

4

http://www.idtus.com

Who is IDT?

• IDT specializes in the design, development, and implementation of Automated Software Testing and Re-Test (ATRT) and Quality Assurance (QA) solutions– Deliver turn key automated test suite

• Automated test strategy and identification of highest payoff areas to apply automation for your project

• Selection of best automation tools for your project

• Using your existing test cases or develop test cases for you

• Test results documented in requirements traceability matrix

– Training and pilot project implementation with companies

•We are currently hiring Java Developers

•Plus we’d like to hire 2 or 3 grad students to work for us part time ( i.e. 20 hours week )

•must be US citizen

Page 5: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

5

http://www.idtus.com

State of Software Testing - Why ATRT?

?

Page 6: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

6

http://www.idtus.com

State of Software Testing - Why ATRT?

Utilize Automated Testing Strategies and Technology to Improve Productivity and Quality

Man

ual

Automated

Reduction in Test Days

IDT Provides Automated Software Testing Solutions

Increased Test

Coverage

Manual vs. Automated

Spiral / Release 1

Spiral / Release 2

Spiral / Release 3

Spiral / Release 4

Siz

e &

Co

mp

lexi

ty o

f S

W B

ase

lin

e

Test Days

Test Days

Test Days

Test Days

50% or More of Overall Development Cost is Typically Spent on Testing

Page 7: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

7

http://www.idtus.com

What is ATRT?

• Our definition of ATRT is:

Application and implementation of software

technology throughout the entire Software

Testing Life Cycle (STL) and QA lifecycle;

with the goal to improve STL efficiencies and

effectiveness

Page 8: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

8

http://www.idtus.com

ATRT Spans the Software DevelopmentLife Cycle – Visual Modeling

Modeling and test case generation Modeling and test case generation

Development Tools

ComponentsUse cases

Design Build Assemble

Round-Trip Engineering

TestTest

Page 9: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

9

http://www.idtus.com

ATRT Spans the Software DevelopmentLife Cycle – Requirements

Organizes, tracks, & controls requirementsOrganizes, tracks, & controls requirements

Requirements Traceability Matrix (RTM)Requirements Traceability Matrix (RTM)

Development Tools

Components

Visual Modeling

Requirements Management andProcess Automation

Page 10: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

10

http://www.idtus.com

Automated Software Testing

ATRT Spans the Software DevelopmentLife Cycle – Automated Test Tools

Development Tools

Components

Visual Modeling

Execution

Management

Development

Requirements Management andProcess Automation

Automates test cases using vendor-provided, Automates test cases using vendor-provided, open-source tools or in-house developmentopen-source tools or in-house development

Automated Testing Tools

Page 11: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

11

http://www.idtus.com

ATRT Spans the Software DevelopmentLife Cycle – Integrated Suite of Tools

Additionally:Additionally:

MiddlewareMiddleware

InfrastructureInfrastructure

Defect TrackingDefect Tracking

Configuration Configuration ManagementManagement

Memory Leak Memory Leak DetectorsDetectors

Performance Performance Testing ToolsTesting Tools

Documentation Documentation ToolsTools

othersothers

Development Tools

Components

Visual Modeling

Automated Testing Tools

Requirements Management andProcess Automation

Software Configuration Management

Defect Tracking

Middleware

Infrastructure

Page 12: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

12

http://www.idtus.com

Types of Software Test Life-cycle Support Tools

• Which tools are you familiar with?

Page 13: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

13

http://www.idtus.com

Types of Automated Test Tools

Life-CyclePhase

Type of Tool Tool Description

BusinessAnalysis

Phase

Business Modeling Records definitions of user needs and automates rapid construction of

flexible, graphical, client-server applications

ConfigurationManagement

Baselines important data repositories

Defect Tracking Manages system life-cycle defects

Technical ReviewManagement

Facilitates communication, while automating the technical review/inspection process

DocumentationGenerators

Automate document generation

RequirementsDefinition

Phase

RequirementsManagement

Manages and organizes requirements; allows for test procedure design and test progress reporting

RequirementsVerifiers

Verify syntax, semantics, and testability

Use CaseGenerators

Create use cases

Page 14: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

14

http://www.idtus.com

Types of Tools (Cont’d)

Programming Phase

Memory Leak and Runtime Error

Detection

Detects runtime errors and memory leaks

Source Code Testing Verifies maintainability, portability, complexity, and standards compliance

Static and Dynamic Analyzers

Depict quality and structure of code

Life-CyclePhase

Type of Tool Tool Description

Analysis and Design Phase

Database Design Develops second generation enterprise client-server systems

Structure Charts, Flowcharts, and

Sequence Diagrams

Manage processes

Test Procedure Generators

Generate test procedures from requirements, design, or data and object models

Syntax Checkers/ Debuggers

Perform syntax checking and have debugging capability; usually available with built-in programming language compiler

Metrics Tools Code (Test) Coverage Analyzers or Code

Instrumentors

Identify untested code and support dynamic testing

Usability Measurements Provide usability testing as conducted in usability labs

Page 15: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

15

http://www.idtus.com

Types of Tools (Cont’d)

Life-CyclePhase

Type of Tool Tool Description

Other Testing Life-Cycle Support Tools

Data Extraction Tool Extract Data from various formats into various formats

Test Data Generators Generate test data

File Compare Utilities Find discrepancies between files that should be identical in content

Simulation Simulates application to measure for scalability, among other tasks

Test Management Tests management

Network Testing Monitors, measures, tests, and diagnoses performance across the entire network

GUI Testing (Capture/Playback)

Conducts automated GUI tests; capture/playback tools record user interactions with online systems so they may be replayed automatically

Load/Performance Testing Conducts load/performance and stress testing

Security Testing Performs security testing and vulnerability scanning at the application or network level; plus debuggers will allow to check for security coding errors (source code checkers)

Page 16: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

16

http://www.idtus.com

Deployment Approach – Adapted ATLM

Collect/Create:• Requirements• Test Cases• Test Procedures• Expected Results• Interface Specifications• System/Component Configuration Description

Identify or Develop & Pilot• Test Manager• Test Scripts• Data Comparators• Performance Test Tools• GUI Record/Playback• Storing/Retrieving Results

Populate Test Manager with Test Cases and Requirements

Develop Test Scripts from Test Procedures

Test Results:• Pass/Fail by Test Case• Pass/Fail Summary• Test Time Required

Page 17: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

17

http://www.idtus.com

Why Open Source

?

Page 18: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

18

http://www.idtus.com

Why Open Source?

• Advantage of Open Source:– no licensing issues; i.e. license cost; maintenance;

etc.– easily modifiable– adaptable– lightweight– flexibility - not tied to one vendor

Page 19: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

19

http://www.idtus.com

ATRTs IDE - Eclipse

Eclipse – this site lists all the plugins

http://www.eclipse-plugins.info/eclipse/plugins.jsp?category=SCM&pager.offset=0&firstItem=1

Eclipse/TPTP – Open Source - Developed by IBM– Eclipse/Test and Performance Tools Platform (TPTP)

offers a common extensible framework for the following functions:

• Requirements Management http://sourceforge.net/projects/osrmt/

• Configuration Management - Subversion

• Testing Tools (http://www.nabble.com/Eclipse-TPTP---Testing-Tools-f2262.html)

• Profiling: default Java applications, but can be extended to other apps

Page 20: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

20

http://www.idtus.com

ATRTs IDE - Eclipse

Eclipse/Test and Performance Tools Platform (TPTP) offers a common extensible framework for the following functions (cont):

• Static Analysis: default for C++ and Java, but can be extended to other apps

• Application monitoring and log analysis

• Build tool – ant

• Xml input and output - using an open source Java package http://jakarta.apache.org/ecs/ , which is a Java library that has an API for constructing and outputting XML. 

• Test Manager – STAF/STAX

Page 21: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

21

http://www.idtus.com

Current Test Tool Implementation – Example and Challenges

Vendor-provided (Capture/Playback) Tool:

Automated test tools mimic actions of the test engineer.

During testing, the engineer uses the keyboard and mouse to perform some type of test or action.

Testing tool captures all keystrokes and subsequent results, which are baselined in an automated test script.

During test playback, scripts compare latest outputs with previous baseline.

Testing tools have built-in, reusable test functions.

Most test tools provide for non-intrusive testing; i.e., they interact with the “application-under-test” as if the test tool was not involved.

Page 22: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

22

http://www.idtus.com

Current Testing Tools – Challenges (cont)

In-House Software Development required:

Capture/Playback tools generate hard-coded values; test scripts are not reusable, nor do they implement software development best practices right out of the box; scripts need to be modified.

Capture/Playback tools don’t necessarily provide all testing features required; code enhancements are often required to meet testing needs.

Capture/Playback tools are not necessarily compatible with system engineering environment, and software testing scripts need to be developed in-house.

Developed testing scripts are not cross-vendor compatible, even if the same scripting language is used

Page 23: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

23

http://www.idtus.com

Linux Capture/Playback Tools IDT Findings

Replay Xcessory http://www.scl.com/products/ics/motif/testing/replay/www.ics.com

QF-Test http://www.qfs.de/en/qftestJUI/index.html

Froglogic http://www.froglogic.com/pg?id=Google&category=squishjava

Redstone Software Eggplant

KD Executor http://www.kdab.net/?page=products&sub=kdexecutor

Jameleon http://sourceforge.net/projects/jameleon/

ShUnit2 for shell scripts testing

GNU Linux Desktop Project http://ldtp.freedesktop.org/wiki/Home

Cantana http://www.ipl.com/pdf/p0003.uk.pdf

crontab Part of Unix

  crontab, cruisecontrol, www.adminschoice.com/docs/crontab.htm

Dogtail http://people.redhat.com/zcerza/dogtail/about.htmlhttp://www.cyberciti.biz/tips/linux-automated-gui-testing-with-python-based-dogtail-tool.html

Expect http://expect.nist.gov/

Page 24: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

24

http://www.idtus.com

Automated Test and Re-Test (ATRT)

• ATRT includes a common framework comprised of re-usable software services– Open Source Software Test Automation Framework

(STAF)

• Provides automated test procedures, test inputs, and test results for SW components– Suitable for use by 3rd parties not associated with

development of the component

Page 25: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

25

http://www.idtus.com

STAX is an automation systemWith STAX you can automate and monitor your entire test environment

ƒ System/product setupƒ Testcase distributionƒ Testcase executionƒ Testcase results collection and analysis

STAX consists ofƒ A programming language designed for automation

–Programs written in this language are called jobsƒ An execution engine (similar to an interpreter for other programming/scripting

languages) which manages the execution and runtime behavior of the jobƒ A GUI application, called the STAX Monitor, which provides a dynamically

updated view of your executing jobsƒ A tool (STAXDoc) that is used to generate documentation for your STAX jobs

What is STAX?

Page 26: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

http://www.idtus.com

    * Windows 95    * Windows 98 (and 98 SE)    * Windows Millenium Edition    * Windows NT Version 4.0    * Windows 2000    * Windows XP    * Windows Server 2003 (IA32, IA64, AMD64)    * Windows Vista (IA32, AMD64)    * Linux (Intel32/64, AMD64, PPC32/64)    * Linux on zSeries (31-bit, 64-bit)    * AIX Version 4.3.3.0 or higher (32-bit, 64-bit)    * Solaris (Sparc) 2.6 and higher    * Solaris AMD Opteron 64-bit    * HP-UX 11.00 and higher (PA-RISC, IA64 32-bit and 64-bit)    * OS/400 V5R2 or higher    * z/OS UNIX V1.4 and higher

STAF can run on:

Page 27: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

27

http://www.idtus.com

STAF Internal Services

• DIAG Provides diagnostics services Internal ("DIAG")• DELAY Provides a means to sleep a specified amount of time Internal ("DELAY")• ECHO Echos back a supplied message Internal ("ECHO")• FILE SYSTEM Allows you to get and copy files across the network Internal ("FS")• HANDLE Provides information about existing STAF handles Internal ("HANDLE")• HELP Provides Help on STAF error codes Internal ("HELP")• MISC Handles miscellaneous commands such as displaying the version of STAF• that is currently running Internal ("MISC")• PING Provides a simple is-alive message Internal ("PING")• PROCESS Allows you to start, stop, and query processes Internal ("PROCESS")• QUEUE Provides a network-enabled IPC mechanism for STAF Programs Internal ("QUEUE")• SEMAPHORE Provides network-enabled named event and mutex semaphores Internal ("SEM")• SERVICE Allows you to list services available on a machine and to examine the• Requests that have been submitted on a machineInternal ("SERVICE")• SHUTDOWN Provides a means to shutdown STAF and register for shutdown• Notifications Internal ("SHUTDOWN")• TRACE Provides tracing information for STAF services Internal ("TRACE")• TRUST Interfaces with STAF's security Internal ("TRUST")• VARIABLEProvides a method for maintaining configuration and runtime data• (variables) Internal ("VAR")           

Page 28: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

28

http://www.idtus.com

STAF External Services

The executable code for external STAF services resides outside of STAFProc, for example in a Java jar file, a C++ DLL file, or a Rexx script file.

• CRON Calls into STAF services at a specified time interval External (Java)• EMAIL Allows you to send email messages External (Java)• EVENT Provides a publish/subscribe notification system External (Java)• EVENTMANAGER Allows you to call STAF services when a specified Event

occurs External (Java)• HTTP Allows you to make HTTP requests which can be grouped together in a

session External (Java)• LOG Provides a full-featured logging facility External (C++)• MONITOR Allows a testcase to publish its current running execution status for

others to read External (C++)• RESOURCE POOL Allows you to manage exclusive access to pools of elements,

e.g. VM UserIDs or Software Licenses External (C++)• STAX Provides an XML-based execution engine External (Java)• ZIP Provides a means to zip/unzip/list/delete PKZip/WinZip compatible archives

External (C++)

Page 29: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

29

http://www.idtus.com

STAFƒ STAF provides the infrastructure on which STAX builds. The full power of

STAF and its services is exposed for use within STAX jobs.XML

ƒ The STAX programming language is based on XML. This provides built-in structure to your jobs, as well as providing a set of existing tools for use in constructing your jobs, such as XML (aware) editors and XSLT.

Pythonƒ The STAX programming language builds on Python to provide a rich and

accessible data modelƒ STAX's Python integration also allows you to access the wealth of existing

Python librariesJava

ƒ The STAX programming language allows you to access existing Java classes/libraries, providing another source for reuse

Technologies used in STAX

Page 30: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

30

http://www.idtus.com

Automation Tasks

Execution

Synchronization

Monitoring

Resource Management

Automation Completion

Testcase Output Analysis

Results Notification

Testcase Cleanup

Testcase Execution

System Setup

Automation Startup

Page 31: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

31

http://www.idtus.com

End-to-End Automation with STAF and STAX

Event, EventManager, Cron

FS, Process

Process, Monitor, Log, Variable, Queue, ResPool

Log, Process

Process

Email, HTTP

STAX

Job

<function><parallel><sequence><process><stafcmd><testcase><timer><block><loop><message><log><import><job><iterate><paralleliterate><hold><release><terminate>Automation Completion

Results Notification

Testcase Cleanup

Testcase Output Analysis

Testcase Execution

System Setup

Automation Startup

Page 32: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

32

http://www.idtus.com

Key ATRT Architecture Requirements

• Support applications running on multiple computers• Support applications developed in different languages• Support applications running on different types of OS’s• Support applications which have GUI and those which do not (for

example Interface testing)• Support applications which use different types of network

protocols such as TCP/IP, DDS, etc• Support integration of multiple commercial testing tools from

different vendors ( allowing as new or better products emerge in the market they can be utilized )

• Support testing w/o having to install ATRT on the samecomputers as the application under test and be able to be

distributed across computers

Page 33: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

33

http://www.idtus.com

Display Application Display Application Analysis & Display Application

Simulation of Application Processing Messages

Simulation of Application Processing Messages

System Under Test

Test ManagerRTMBugzilla

ATRT

EggplantR/P

VNC RobotR/P

STAF/STAX

Prototype – Proof of Concept

WindowsBased

Linux Based

Page 34: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

34

http://www.idtus.com

• Able to successfully execute ATRT for 48 hours without operator intervention

• During the tests:

VNCRobot - Approximately 25,000 operator key strokes were replayed STAF/STAX - More than 1,000,000 messages processed and verified by ATRT

ATRT Prototyping Effort

Page 35: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

35

http://www.idtus.com

Additional Requirement

• Pristine test environment required – automated testing tool cannot be installed on Application-Under-Test PC

Eggplant – allows for remote capture/playback via VNCserver and KVM switch

VNCRobot

Page 36: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

36

http://www.idtus.com

Automated Software Testing

• Software testing paradox

• What is it?

Page 37: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

37

http://www.idtus.com

Effective Automated Software Testing

Current State: Developing software in order to test software

Automate the test support development: automated IDL to code generation; test data generation (using FireEye from NIST); compare utilities; log file analyzers;

Our Automated Software Testing Goal: • Minimize Human interaction

– Possibly get on par with Hardware Automated Testing?

Page 38: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

38

http://www.idtus.com

Hardware Testing: current state

• Many automated test generators for manufacturing/ logic devices, circuit testing, etc., available i.e. SEI/ATG, uMaster, etc.

• Automated testing without human interaction is the norm for hardware testing

Page 39: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

39

http://www.idtus.com

Hardware vs. SoftwareAT

How is hardware testing different?• Software Challenge:

– Changing requirements– GUI interfaces– Code access/availability– Defects uncovered earlier in the software

development lifecycle are cheaper to fix

• Solution flexibility required – problem dependent

Page 40: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

40

http://www.idtus.com

Proposed solutions

Focus on Reusable Components– Test Data / Expected Results– Interfaces – Middleware

Assuming code and models are available:• Model based testing; automated test case generators based on models• Eclipse – effective ATRT IDE• Probes – see Aprobe – Code instrumentation• Building self-testable components• Using standard language MOF to Text to generate automate test casesGUI Testing tool ideasNew tools - other• Working with NIST to produce additional tools

Page 41: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

41

http://www.idtus.com

GUI Testing

• Jemmy - java library to test UI

• http://jemmy.netbeans.org/

• http://abbot.sourceforge.net/doc/overview.shtml

• Perl modules

• run via VNCServer?

Page 42: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

42

http://www.idtus.com

Questions

?Email [email protected] with any follow-up questions

Page 43: Automated Software Testing using Open Source Testing Tools By Elfriede Dustin

43

http://www.idtus.com

IDT

We are hiring…….

Please come see me now if interested or sendemail [email protected] with any follow-up