Embed Size (px)
Citation preview
marlabsdriving digital agility
WHITEPAPER
Maximizing the Utilization of Test Automation
Automation: New Venues, New challenges
In today’s rapidly changing software world, right testing solutions determine the quality and eventually the success of an application. Customer experiences are critical—brand-reputation, loss of revenue, and eventually loss of business can be the outcomes of bad customer experience.
To meet these quality goals, there has been a reasonable increase in investment on Test Automa-tion. With the increase in investments, there are more expectations set on the returns from Test Automation. While time, money and effort are invested on test automation with great enthusiasm, it often faces utilization setback, wherein test automation is utilized for some time and then kept aside, continuing with manual testing in due course.
This raises a fundamental question: Why is the utilization of Test Automation Low? To answer this question, we need to understand the purpose of Test Automation and the system in which it exists. Hence let’s analyze the underlying mechanisms and issues.
Dynamic Nature of Software Industry
There is a key difference between industrial automation and software automation. In Industrial automation the output is usually fixed as things are physical or tangible in nature. However, in software automation, due to the dynamic nature of software products, the mechanism used to automate software processes also needs to be dynamic in nature. Changing needs of the customer, changes in technology, changes in requirements, and changes in the products or services also lead to dynamic automation processes. As a result of this, automation suites created with a certain environ-ment in mind eventually become outdated, or need frequent updates. The investment in updates sometimes becomes so high that, manual testing is preferable in due course, reducing the utilization of automation suites.
Test Automation Lifecycle
Many managers limit themselves to only software development lifecycle (SDLC) and software test lifecycle (STLC) rather than including test automa-tion lifecycle, a third cycle, which is considerably new to the industry. Creation of a test automation suite is a lifecycle by itself, with its own problems and issues in addition to those posed by SDLC and STLC.
The automation cycle resembles the development cycle, but is assistive to the testing cycle. Automa-tion scripts need to be designed, developed,and tested. The automation resources need some programming background as scripting standards and guidelines need to be formulated.
Many managers find managing activities in all the 3 cycles cumbersome rather than assistive for
1
> W h i t e Pa p e r
various reasons. As a result, they limit themselves to the usual SDLC and STLC ridding them of the benefits of test automation.
Team Structure
In some projects, automation testers are used temporarily, on a need basis, only to develop automation scripts while automated execution is done by manual testers after work is handed over by automation testers.
Projects that follow this kind of model have other reasons for the low utilization of test automation suites. Manual testers’ resistance is an important reason. They find it easy to execute the tests manually than through automation due to lack of technical skills, and lack interest in understanding the intricacies of test automation framework or inertia to change the way tests are executed. They are often deterred by the additional setup work involved in automation.
Another reason for the low utilization of test automation suites in this type of project models is that frequent changes in product demands frequent change in test cases. Automation testers being temporary resources may be oblivious of the
> W h i t e Pa p e r
intricacies of these changes. Moreover, in many cases automation consultants are brought in to fix problems from previous instances of testing instead of employing a planned approach.
In the absence of automation testers, manual tester may not be able to modify automation scripts to suite their changing needs due to their lack of required skills. This again poses a serious threat to the test automation suite utilization and maintenance.
Lacking sense of ownership is one more reason due to which automation suite may not utilized exten-sively. Maintaining automation suite under this type of team-structure becomes a challenge, due to the intricacies of communication and reflects the changes in automation suites.
Short-term goal
Many of the test automation suites are created with a short-term goal in mind i.e., just to automate the pre-existing test cases with a certain set of test data to work with. As Application-Under-Test (AUT) changes, so does the test cases and test data. Eventually, these changes are to be reflected in the automation suite. If the estimated time and effort for incorporating these changes is signifi-cant, manual execution of tests is preferred.
Pitfalls of Test Automation
The disadvantages of test automation itself at times can be the reason for its low utilization. Following are some of the drawbacks:
•Script errors: Debugging the test scripts can be a major issue. These errors can create serious issues, or create a false sense of security if automation suite is unable to detect defects.
•Skills: Automation scripting requires a higher skill levels w.r.t programming and implementation.
•Root-cause analysis: Complex analysis may be required when an automated test fails—Is the failure due to script, or application?
•Maintenance: Maintenance of test scripts and test data can be difficult, if the suite/data size is huge, or the AUT changes frequently.
Tools and Techniques
Each test automation tool comes with its own set of challenges and limitations (compatibility, pricing etc.). In order to overcome these challenges and limitations, various techniques/frameworks are used, which in turn may have certain drawbacks themselves,thereby restraining the coverage/utili-zation of test automation.
To summarize, the fundamental problem for low utilization of test automation suites revolves around the triad—goals/objectives, solution, maintenance ownership. To find the solution, we need to answer the following questions:
•What is the purpose of building test automation suites?
•How are suites usually built/designed?
•Who uses test automation suites?
•Who maintains test automation suites?
•Why do test automation suites get outdated?
The Solution
To address all the challenges faced, we should go to the root of the problem. It may not ensure a sure-shot answer to our problems, but it can definitely take us in the right direction, wherein most of the problems mentioned above are addressed to some extent.
The Purpose
The purpose of building test automation suites is not only to execute manual test cases faster using a tool but also to make the entire STLC effective, efficient, and shorter while assisting the release of a defect-free product. Automation can provide a lot of other significant benefits like elimination of human errors, and test high-risk areas, which need
> W h i t e Pa p e r
to be mandatorily tested; any investment in terms of time and money that’s gone in test automation phases—automation feasibility analysis, framework design, automation script development, script execution, review, rework, and documentation; and should ensure that automation is assisting in one way or the other to meet the identified goals.
Before getting into test automation engagement, ensure that proper ground work is done—feasibili-ty study, determination of automated testing ROI, identification of automation candidate, and consideration of other benefits offered by automa-tion.
Automation Design
Automation tester should keep various factors in mind before designing automation framework, such as those given below:
•Plan ahead, research about the various tools and techniques of automation and find/select an architecture that properly fits the environment and testing needs
•Create robust and fast executing test automation scripts for unchanging modules of AUT
•Ensure flexible and intuitive automation scripts for frequently changing modules of AUT
•Inculcate data-driven capabilities for updating test-data set
•Inculcate keyword-driven capabilities for modify-ing/controlling execution flow
• Identify candidates for automation based on ROI to reap benefits.
This approach will ensure that testers can update the suite with the changing AUT, while exploiting the retesting capabilities for any bug-fixes. Consid-er the following prerequisites for automation:
• Application stability: Application needs to be stable enough for automation-feasibility
• Availability of well-designed manual test cases: well-designed test cases with good requirement coverage can have higher error-detection rates. Automating such test cases ensures better test
coverage
• Maintenance of test cases: Test cases need to be up-to-date when selected as automation candi-dates. Otherwise any failure might lead to confu-sion whether it can be attributed to application or the automation script.
Automation Execution
Manual testers need to understand that automa-tion is to assist them so that they can focus on other important areas/functionalities/test activi-ties than spending time on clicks, keystrokes and other repetitive tasks. Automation can take care of the routine/repetitive tests. The defect detection capacity of test scripts is highly dependent on the quality of test cases and is written primarily by manual testers. They are the custodians of the knowledge of the application-under-test, and can therefore focus on excavating scenarios for functional testing, while automation can focus on the execution of same. In cases where the automa-tion testers develop the suite and the manual testers execute the suite on a regular basis, train manual testers on its usage, as they are the ones who will utilize and maintain it for a longer duration. Their strength is derieved from an understanding the AUT, creation of test scenarios, acumen to break the system, coupled with the swiftness of the automation suite, which will double testing team’s benefits.
Long Term Vision
Envision a long term strategy by not limiting only to work in hand.
Considering test automation as a one-off task to run manual test cases using a test automation tool will be a mistake. It is important to think beyond the conventional and see the big picture of innovation to advance automation to next level. Automation can be seen as a process benefitting the entire business at both macro and micro levels by using automation beyond executing test cases and seeing the business implications with direct and indirect
> W h i t e Pa p e r
benefits to reap from implementation of automat-ed solutions. Keep in mind the following points while envisaging an automated solution:
• Influential factors: Various factors affect the approach that will be implemented. Some of them are ROI, tools, technology, and various other costs the project may incur.
• Feasibility study: Keep in mind that it is not always beneficial to automate every test case or every application out there. Work on the feasibility study and ROI assessment before deep-diving into test automation.
• Requirement traceability: Ensure that automa-tion scripts are properly traced back to the require-ments and are updated accordingly.
• Automation frameworks: Take into account the benefits of various automation frameworks to ensure efficiency and effectiveness in the utiliza-tion of automation suites.
• Automation-script reusability: Modularize scripts with reusability in mind—it is always beneficial if the scripts created for one project can be utilized for another project, even if partially. Script needs to be generic and self-explanatory. This will ensure lower efforts for consecutive projects/test cases/scenarios, resulting in better returns.
• Periodic reviews and upgrades: Ensure that the automation suite is up-to-date with the current set of test cases and product changes.
• Coordination: Ensure coordination between manual and automation testing activities along with periodic reviews and upgrades.
• Training: Fill the gap between teams by training team on the required tools, technologies, applica-tions, and their capabilities.
Automation can be beneficial to QA process, but the lack of knowledge about automation and absence of planning and process can lead to unexpected and unwanted consequences.
Automation Standards
In order to have the best of test automation, there
is a need for certain standards to be laid out and followed. There is a need to set coding standards, guidelines, documents/templates and checklists that will ensure that the automation team sticks to the standards and comes up with a uniform output, in an efficient way. Look out for industry standard frameworks and customize them before applying based on the project needs, to gain maximum out of them. Maintaining standards also have following benefits:
• Integration and maintenance of functions/meth-ods/modules become simpler/comprehensive as they follow the same standards, avoiding any non-standard module to fit into the system
• Smooth communication and coordination for work transition among team members
• Uniform problem solving approach with tried and tested methods to reduce risks
• Minimizes issues arising from mismatch in implementation standards and unnecessary communication arising from such issues
• Reduction in time and money spent in analyzing, reviewing and reworking, thereby saving revenue and minimizing performance pitfalls
• Increase in efficiency and productivity
There are also utility software (apart from the conventional test automation tools) such as code-review tools and file comparators, which at times assist greatly in improving productivity thereby cutting-short the test execution time. Introducing such utilities can benefit the project in the long run.
Conclusion
Success in today’s fast paced software world demands companies to formulate the right testing solution by having a modern testing approach and the right tools for the job. Software market is also flooded with advanced tools and technologies, which are assistive in creating a commendable approach that overcomes the challenges associat-ed with application testing.
Just purchasing a test automation tool won’t solve
> W h i t e Pa p e r
all your STLC problems. It is important to adopt a strong test methodology and devise the right automation framework to suit the organization's unique development environment, keeping in mind the pros and cons of the selected automation tool and the goals of implementing automation. Test automation is an activity to complement manual testing. To maximize the benefits of test automa-tion, utilize it as an assistive process in quality assurance.
To summarize—Key to successful software test automation and maximizing its utilization is attributed to setting the right goals, optimum use of the tool, automation framework, testing process, and the right team composition with specified responsibilities and skills. The required skills include a blend of testing, programming, and tool knowledge to ensure that the right solution is developed and implemented with long term benefits.
Marlabs helps drive digital agility for our clients. We deliver innovative business solutions using digital technologies such as cloud, mobile, analytics, Internet of Things and social. With a dedicated team of over 2,100 associates, a network of delivery centers in USA, Canada, Mexico and India, and strong partnerships with industry leaders, Marlabs offers a wide range of IT services across industries. Through our emphasis on quality driven by CMMi, PCMM, ISO 9001-2000, ISO 27001 and SSAE 16 Type II best practices and a customer-centric client engagement model, Marlabs has achieved a dependable track record of meeting high standards of excellence in every customer engagement. This has resulted in several awards and recognitions, including being consistently ranked in the Deloitte Technology Fast 50 and Fast 500 programs. Marlabs is headquartered in New Jersey, United States.
For more information: please call us at +1(732)-694-1000 or email us at [email protected] • USA | Canada | Latin America | India | Malaysia • www.marlabs.com
© Copyright 2015 Marlabs Inc. All rights reserved.
Marlabs helps drive digital agility for our clients. We deliver innovative business solutions using digital technologies such as cloud, mobile, analytics, Internet of Things and social. With a dedicated team of over 2,100 associates, a network of delivery centers in USA, Canada, Mexico and India, and strong partnerships with industry leaders, Marlabs offers a wide range of IT services across industries. Through our emphasis on quality driven by CMMi, PCMM, ISO 9001-2000, ISO 27001 and SSAE 16 Type II best practices and a customer-centric client engagement model, Marlabs has achieved a dependable track record of meeting high standards of excellence in every customer engagement. This has resulted in several awards and recognitions, including being consistently ranked in the Deloitte Technology Fast 50 and Fast 500 programs. Marlabs is headquartered in New Jersey, United States.
For more information: please call us at +1(732)-694-1000 or email us at [email protected] • USA | Canada | Latin America | India | Malaysia • www.marlabs.com
© Copyright 2015 Marlabs Inc. All rights reserved.
