Embed Size (px)
DESCRIPTION
Software Test automation tools are available under several categories such as commercial, free software, open source software and etc. In this paper Open Source Software Testing Tools will be discussed. Open source software test automation tools may be practical alternatives to popular closed-source commercial applications and some open source tools offers features or performance benefits that exceed their commercial counterparts. The source code is openly published for use and/or modification from its original design, free of charge. And these are usually available under a license defined by the Open Source Initiative.
Citation preview
ICT 3003 – Software Quality Assurance
Assignment I
Open SourceSoftware Quality Assurance Testing Tools
University of Colombo School of Computing
Prepared by
Waruna Kodituwakku
Page | 1
Table of Contents
1. Introduction............................................................................................................................2
1.1 Open Source Software Testing Tools ..............................................................................2
1.2 Types of Software Testing Tools .....................................................................................2
2. Popular Open Source Software Testing Tools.......................................................................3
2.1 Selenium ......................................................................................................................3
2.2 FitNesse........................................................................................................................3
2.3 Watir ............................................................................................................................4
2.4 Apache JMeter .............................................................................................................4
2.5 Robotium......................................................................................................................4
2.6 Bugzilla ........................................................................................................................5
2. Comparison of Selenium, Bugzilla and JMeter .....................................................................5
2.1 Analysis of Strengths and Weaknesses of Selected Tools...............................................6
2.1.1 Selenium ...................................................................................................................6
2.1.2 Bugzilla .....................................................................................................................6
2.1.3 JMeter .......................................................................................................................7
3. Future of Software Testing Tools ..........................................................................................7
4. Conclusion .............................................................................................................................7
5. References..............................................................................................................................8
Page | 2
1. Introduction
Software Quality Assurance (SQA) involves monitoring and improving the software testingprocess, making sure that any agreed-upon standards and procedures are followed, andensuring that problems are found and treated. In today’s context SQA is essential to everysoftware development process. Software Testing is a costly and time consuming process insoftware development. Therefore, software testing tools are often deployed to automate theprocess in order to reduce cost and improve efficiency.
SQA now incorporates more sophisticated tools that enhance software testing capabilities.Testing automation tools enable and help developers and testers to automate the entireprocess of testing in software development which include; find and report defects and issuesfaster; confirm tests meet customer specifications and requirement-traceability; have a fineway for communication; archive the test plan and execute it manually or automatically;maintain reusable tests; and document, verify and audit which tests were run for each project.
1.1 Open Source Software Testing Tools
Software Test automation tools are available under several categories such as commercial,free software, open source software and etc. In this paper Open Source Software TestingTools will be discussed.
Open source software test automation tools may be practical alternatives to popular closed-source commercial applications and some open source tools offers features or performancebenefits that exceed their commercial counterparts. The source code is openly published foruse and/or modification from its original design, free of charge. And these are usuallyavailable under a license defined by the Open Source Initiative.
1.2 Types of Software Testing Tools
Software testing tools can be categorized under several bases. Following table is showssoftware testing tools categorized under the testing activity or the process in which they areused.
Type Description Open SourceExamples
FunctionalTesting
Functional testing is the process of confirming thefunctionality of the application. This type of testingignores the internal parts and focuses on the output anddetermines that it is as per requirement or not.
Selenium, Watir,WatiN, CanooWebTest, Cucumber
Source CodeTesting
Source Code Testing tools are designed to analyze sourcecode and/or compiled version to verify the source codeand help identify the security flaws.
SWAAT, Flawfinder,PMD, Brakeman,RIPS, RATS
PerformanceTesting
To check whether system meets performancerequirements
Apache JMeter,FunkLoad, OpenSTA
DatabaseTesting
Database testing is about checking exact values whichhave been retrieved from the database by the web ordesktop application. Data should exactly match withthose in the records that are stored in the Database
Database OpensourceTest Suite,DBMonster, DBSanity, DBFeeder
Bug TrackingTools
These software applications are designed to help keeptrack of reported software bugs in software developmentefforts.
Bugzilla, Mantis,Trac, Redmine,Request Tracker,
Page | 3
TestManagementTools
Test management tools are used to structure automatedtests and manual test processes, and to manage multipleenvironments easily. Quality assurance teams use thesetypes of tools as a single application for managing testcases, environments, automated tests, defects and projecttasks.
Test Manager,TETware,Requirements andTesting Hub (RTH)
2. Popular Open Source Software Testing Tools
2.1 Selenium
Category : Functional Test Tool, Web Application Testing License: Apache License 2.0
Selenium is composed of multiple software tools. Those are Selenium IDE, SeleniumWebDirver and Selenium Remote Control. Selenium is a portable software testing frameworkfor web applications.. It supports a number of accepted programming languages, includingC#, Java, Groovy, Perl, PHP, Python and Ruby for writing test cases. The tests can then berun against many modern web browsers. Selenium is deployed on Windows, Linux, andMacintosh platforms.
Features
Record and playback Intelligent field selection Auto complete for all common Selenium commands Walk through tests, debug and set breakpoints Save tests as Selenese, Ruby scripts, or other formats Option to automatically assert the title of every page Easy customization through plug-ins
2.2 FitNesse
Category : Test Management Tools License : Common Public License (CPL)
This tool allows users to specify and run acceptance tests for software systems (an agile styleof black-box testing acceptance and regression testing). FitNesse facilitates detailed readabledescription of system function and it supports programming languages such as Java, C++,Python, Ruby, Delphi, C#, etc.
Features
Wiki engine Embedded web server Test engines (FIT, SLIM) Automated input generation and test case creation Additional libraries
Page | 4
2.3 Watir
Category: Functional Test Tool, Web Application Testing License: BSD License
Watir is a free, open-source functional testing tool for automating browser-based tests of webapplications. It works with applications written in any language. Watir project consists ofseveral smaller projects. Some of them are Watir-classic, Watir-webdriver and Watirspec.
Features
Handles multiple browser windows with attach method. Supports JavaScript events Supports frames and nested frames Supports visible and invisible running of test cases Automatic page loading waiting Supports hidden fields and saving images in a page and Screen capturing
2.4 Apache JMeter
Categories: Performance Testing, Load Testing License: Apache License 2.0
Apache JMeter is a 100% pure Java desktop application designed to load test functionalbehavior and measure performance. It was originally designed for testing Web Applicationsbut has since expanded to other test functions. JMeter architecture is based on plugins. Mostof its features are implemented with plugins. Off-site developers can easily extend JMeterwith custom plugins.
Features
Can load and performance test many different server types (Web - HTTP, HTTPS,SOAP, Database via JDBC, LDAP, JMS, Mail - SMTP(S), POP3(S) and IMAP(S),Native commands or shell scripts)
Complete portability and 100% Java purity. Full multithreading framework allows concurrent sampling by many threads and
simultaneous sampling of different functions by separate thread groups. Careful GUIdesign allows faster operation and more precise timings.
Caching and offline analysis/replaying of test results. Highly Extensible
2.5 Robotium
Categories : Mobile Testing tool, Android Application Test Tool License: Apache License 2.0
Robotium is a test framework created to make it easy to write powerful and robust automaticblack-box test cases for Android applications. With the support of Robotium, test casedevelopers can write function, system and acceptance test scenarios, spanning multipleAndroid activities.
Page | 5
Features
Fast test case execution. The framework handles multiple Android activities automatically. Minimal time needed to write solid test cases. Readability of test cases is greatly improved, compared to standard instrumentation
tests. Test cases are more robust due to the run-time binding to GUI components. Integrates smoothly with Maven or Ant to run tests as part of continuous integration
2.6 Bugzilla
Category: Bug Tracking Tool License: Mozilla Public License
Bugzilla is a web-based bug tracker and testing tool. This allows individuals or groups ofdevelopers to keep track of outstanding bugs in their product effectively. Generally Bugzillatracks bugs and code changes, communicates with teammates, submits and reviews patchesand manages quality assurance. Bugzilla is written in Perl, and works on various databasesincluding MySQL and Oracle.
Features
Advanced search capabilities Email notifications controlled by user preferences and integrated email capabilities Bug lists in multiple formats (Atom, iCal, etc.) and reports and charts Automatic Duplicate Bug Detection Time Tracking, Request System, Private Attachments and Comments Automatic Username Completion or Drop-Down User Lists and Patch Viewer Multiple Authentication Methods and Support for Multiple Database Engines
2. Comparison of Selenium, Bugzilla and JMeterSelenium (IDE,
WebDriver and RemoteControl)
BugZilla Apache JMeter
Testing Type Functional Testing,Regression Testing
Bug Tracking Performance Testing,Load Testing
Developed by ThoughtWorks Inc. Mozilla Foundation Apache SoftwareFoundation
Written in Java (ProgrammingLanguage)
Perl (ProgrammingLanguage)
Java (ProgrammingLanguage)
Operating System Cross-platform (Windows,Linux, OS X, Solaris)
Cross-platform Cross-platform
User InterfaceAvailable
Integrated into ALM & IDE,Standalone application,Webbased interface
Web based Interface Batch Mode, StandAlone application
Record andPlayback Option
Available No Available
Page | 6
Application Type It can be used to test onlyweb based applications.
It can be used to testonly web basedapplications
It can test web-basedas well as desktopapplications.
Integration withother tools
Allowed Allowed Allowed
Report GenerationFacility
Available Available Available
Multilingual(Localization)
No Yes No
Plugins, Add-ons,Customizations
Available Available Available
Installation Required Installation Required Installation Required Installation Required
Documentation andCommunity Support
Available Available Available
2.1 Analysis of Strengths and Weaknesses of Selected Tools
2.1.1 Selenium
Strengths
Selenium is one of the most popular open-source automation tools. Selenium provides ahandy unified interface that works with a large number of browsers, and allows writing testsin almost all the languages (E.g. Java, C#, Python, JavaScript, PHP, Erlang and etc).
Weaknesses
Selenium is written in JavaScript which causes a significant weakness: browsers impose apretty strict security model on any JavaScript that they execute in order to protect a user frommalicious scripts. And also it doesn't work directly with Flash.
2.1.2 Bugzilla
Strengths
Main strength of Bugzilla is that it is one of the well established and widely used software inthe industry. Other key strengths of BugZilla can be identified as; easily understood, easilyintegrates with several test management tools (e.g. Test Link, Test Runner, Testopia), verytest friendly and contains almost all the features found in commercial products.
Weaknesses
Main weakness of Bugzilla is its complex installation. And even with improvements;Bugzilla user interface hasn't changed much over the years – a fact that is not attractive forend-users.
Page | 7
2.1.3 JMeter
Strengths
JMeter is an Open Source Tool and it also a Freeware. It supports the distributed testing andhas various target systems and further it is Cross-platform. It has very high extensibility too,because pluggable samplers allow unlimited testing capabilities. Overall JMeter is a stable,easy to comprehend and long standing desktop Java application.
Weaknesses
It’s always advisable not to run JMeter on the same machine running the application to betested. JMeter may use extensive resources that might affect the other application'sperformance. JMeter chart representation looks a bit confusing to some users andterminology appears to be not very clear. Getting "correct" response times from JMeterprocess is very complex. Also remote machines must be declared in a property file beforestarting application.
3. Future of Software Testing Tools
Today, software has changed from a standard, single, windows-based platform to multipleplatforms. When considering about growth of software test automation tools we can see thatthere will be new areas which will enhance user friendliness, integration between tools,combining different tools to fit user needs.
With script-less automation, new tools will be shaped so that users could setup automationtesting in a simple and logical way. It means that technical skills and programmingknowledge will not be required any more. Another aspect is that Visual Test Automation isthe future for test object recognition in the modern software application development andtesting world. It contains algorithms built on OCR (object character recognition), ICR (imagecharacter recognition) and gesture recognition technology (interpreting human gestures).Technology based on visual based object recognition will allow testers to execute scriptscross-client and cross-platform. Automated testing tools will become less complex, easy touse and adaptive. Users will not need weeks to learn how to use testing tools. This kind ofsoftware testing will be even faster and will give exact results.
4. Conclusion
To be successful in the challenging business of software development, it is essential toprovide a quality product to the customers. Such process is known as ‘Software QualityAssurance’ (SQA). Software Testing Tools play a vital role in SQA. Testing is more than justdebugging. Testing is used to locate defects and correct them and also in validation,verification process, and reliability measurement. There are commercial and free testing toolsas well as open source testing tools. Among them, Open Source Testing Tools seems to themost popular. Comparatively it provides more flexibility, more eases of usage and allows toolcustomization also. In this paper several popular and notable open source software testingtools have been categorized, analyzed, compared and their respective strengths andweaknesses discussed. The future of the testing tools is indicative that they will become lesscomplex, more easy to use more adaptive and faster too.
Page | 8
5. References
1. Open Source Testing. 2013. Open source functional testing tools. [ONLINE]Available at: http://www.opensourcetesting.org/functional.php. [Accessed 15 March2013].
2. VTB. 2013. Article - Test Automation Tools. [ONLINE] Available at:http://www.vietnamesetestingboard.org/zbxe/?document_srl=555503. [Accessed 25March 2013].
3. Wikipedia, the free encyclopedia. 2013. Test automation - Wikipedia, the freeencyclopedia. [ONLINE] Available at: http://en.wikipedia.org/wiki/Test_automation.[Accessed 25 March 2013].
4. SparkWiz. 2013. The Types and Benefits of Software Testing Tools - SparkWiz |Spark Wiz. [ONLINE] Available at: http://sparkwiz.com/tips-n-tricks/the-types-and-benefits-of-software-testing-tools/. [Accessed 25 March 2013].
5. OWASP. 2013. Source Code Analysis Tools - OWASP. [ONLINE] Available at:https://www.owasp.org/index.php/Source_Code_Analysis_Tools. [Accessed 25 March2013].
6. Google Project Hosting. 2013. robotium - The world's leading Android™ testautomation framework - Google Project Hosting . [ONLINE] Available at:http://code.google.com/p/robotium/. [Accessed 25 March 2013].
7. Wikipedia, the free encyclopedia. 2013. Bugzilla - Wikipedia, the free encyclopedia.[ONLINE] Available at: http://en.wikipedia.org/wiki/Bugzilla. [Accessed 25 March2013].
8. TheServerSide.com. 2013. Open source software testing tools news, help andresearch - TheServerSide.com. [ONLINE] Available at:http://www.theserverside.com/resources/Open-source-software-testing-tools. [Accessed25 March 2013].
9. IndianGNU. 2013. IndianGNU.org » Blog Archive » Top 5 Open Source SoftwareTesting QA tools. [ONLINE] Available at: http://www.indiangnu.org/2011/top-5-open-source-software-testing-qa-tools/. [Accessed 25 March 2013].
10. Xoriant Software Product Engineering. 2013. Selenium Open Source Test AutomationTool: An Overview | Xoriant Software Product Engineering . [ONLINE] Availableat: http://www.xoriant.com/blog/software-testing-and-qa/selenium-open-source-test-automation-tool-an-overview.html. [Accessed 25 March 2013].
11. Apache JMeter™. 2013. Apache JMeter - Apache JMeter™. [ONLINE] Available at:http://jmeter.apache.org/index.html. [Accessed 25 March 2013].
Page | 9
12. CSS Corp Labs. 2013. CSS Corp Labs: Testing Tools Comparison . [ONLINE]Available at: http://cssinnovations.blogspot.com/2011/10/testing-tools-comparison_27.html. [Accessed 25 March 2013].
13. ProgrammingZ.com. 2013. Open-Source Quality Assurance And PerformanceAnalysis Tools . [ONLINE] Available at:http://www.programmingz.com/b/2435541/Open-Source-Quality-Assurance-And-Performance-Analysis-Tools/#.UU-rkaAtg3I. [Accessed 25 March 2013].
14. Open Source Testing. 2013. Open source test management tools. [ONLINE]Available at: http://www.opensourcetesting.org/testmgt.php. [Accessed 25 March 2013].
15. PHP Unit Manual. 2013. Chapter 8. Database Testing . [ONLINE] Available at:http://www.phpunit.de/manual/3.6/en/database.html. [Accessed 25 March 2013].
16. Mobile QA Zone. 2013. Visual Test Automation Leads the Future of the SoftwareTesting Industry - Mobile QA Zone. [ONLINE] Available at:http://www.mobileqazone.com/profiles/blogs/visual-test-automation-leads-the-future-of-the-software-testing. [Accessed 25 March 2013].
17. Software QA and Testing Resource Center. 2013. Database Testing Tools. [ONLINE]Available at: http://sqa.fyicenter.com/FAQ/Testing-Tools/Database_Testing_Tools.html.[Accessed 25 March 2013].
18. Ryan Schuetzler. 2013. QA Tool Review: JMeter. [ONLINE] Available at:http://www.schuetzler.net/2010/01/02/jmeter/. [Accessed 25 March 2013].
19. Wikipedia, the free encyclopedia. 2013. Comparison of issue-tracking systems -Wikipedia, the free encyclopedia. [ONLINE] Available at:http://en.wikipedia.org/wiki/Comparison_of_issue-tracking_systems. [Accessed 25March 2013].
20. Jon's Technical Notes. 2013. Jon's Technical Notes: Pros and Cons of Selenium .[ONLINE] Available at: http://jdrew33.blogspot.com/2012/02/pros-and-cons-of-selenium.html. [Accessed 25 March 2013].
