7
SOFTWARE TESTING New software testing technologies bring new challenges CHAPTER 1 VIRTUAL TEST LABS MAKE MOST OF CLOUD CHAPTER 2 TESTING RIAS FOR SECURITY HOLES March 2009 Volume 1

SOFTWARE TESTING

Embed Size (px)

DESCRIPTION

 

Citation preview

Page 1: SOFTWARE TESTING

SOFTWARE TESTINGNew software testing technologies

bring new challenges

CHAPTER 1VIRTUAL TESTLABS MAKEMOST OF CLOUD

CHAPTER 2TESTING RIASFOR SECURITYHOLES

March

2009

Volume 1

Page 2: SOFTWARE TESTING

p VIRTUALIZED TEST LABS can makeit faster and less expensive to set upand tear down test configurations,better utilize resources, and help toboost overall software quality. Andnow cloud computing brings thepromise of even less infrastructureto worry about, which can help to fur-ther alleviate quality assurance (QA)bottlenecks due to resource issues.“QA and testing organizations areincreasingly embracing virtual testlabs due to the efficiency and costsavings that can result,” said Melinda-Carol Ballou, program director forapplication lifecycle management atIDC. According to Ballou, one of thebarriers to QA is limited access to—and poor management of—physicalinfrastructure for test labs, as well asthe effort involved in setting up andmanaging systems configurations forthe labs.“Groups tend to hoard the physicalresources so that they will haveaccess to them when needed, evenwhen there is no immediate demand,”

she said. “This means that softwarecan languish without being tested dueto poor resource allocation and man-agement.”This bottleneck can be costly, Bal-lou said—an issue that becomes evenmore critical in a down economy. “Vir-tual test lab management can helpaddress these kinds of issues, as wellas cutting the cost of infrastructureby augmenting or replacing physicalsystems.”With the emergence of cloud-basedofferings, the benefits of virtualizationcan be more immediate, Ballou said,“because you don’t have to configurethe software.”

STARTUP GAINS CAPACITY,

SAVINGS WITH VIRTUAL TEST

LAB SOLUTION

For startup Apptio, a provider ofon-demand IT cost transparencysolutions, a virtualized infrastructureprovided both affordability and thecapacity to test more broadly.

4 SOFTWARE TESTING MARCH 2009

aEDITOR’S

LETTER

aCHAPTER 1

VIRTUAL

TEST LABS

MAKE MOST

OF CLOUD

aCHAPTER 2

TESTING RIAs

FOR SECURITY

HOLES

CHAPTER 1

1Virtual test labsmake most of cloudQA and testing organizations are embracing the powersof virtualization and cloud-based computing to performsimpler testing at lower costs. BY COLLEEN FRYE

Page 3: SOFTWARE TESTING

According to Colin Henry, senior soft-ware engineer at the Bellevue, Wash.-based company, Apptio is developinga data integration add-on for its pri-mary product; the add-on will aggre-gate data from disparate datasources.“We realized this portion of theproduct covered a broad range ofsystems—ERP, database, etc.—andthe setup/teardown time for thatwould be costly,” Henry explained.“We knew we needed a virtual infra-structure. Being a startup, we don’twant to spend the world on gettingthree racks of servers or a huge boxto throw five or six [virtual machines]on. I knew we wanted to go for virtual-ized space hosted on the cloud.”Henry said Apptio had performedmanual testing on its primary productand did not have a traditional testingenvironment, “so we were startingfrom scratch.”The company researched on-premise and hosted virtual test labs,“but any kind of managed service ona local machine takes a lot of youradministration time,” he said. Provi-sioning a box can take three to fourhours, according to Henry, and ifyou’re adding application infrastruc-ture on top of that, it could take upto eight. So Apptio decided thatnewcomer Skytap Inc. with its SkytapVirtual Lab product fit the bill. Theresult?“Going from 12 hours to five min-utes,” Henry said. “That’s a hugeincrease in productivity, becauseyou’re not slogged down in installingsoftware.”

Seattle-based Skytap describesitself as “serving up virtual machinesover the Internet.” According to CEOScott Roza, “There is no infrastruc-ture, no upfront expenditures, and it’sdelivered 100% as a service.” SkytapVirtual Lab “is a multitenant model,”

he said. “All you need to access it isa browser. With the browser you canget access to computing resources;you’re not limited by the number ofVMs you can run or the number ofusers you can have accessing [theenvironment].”While QA and testing organizationshave been early adopters of virtualiza-tion, Roza said, “very few QA organi-zations are all virtual” because “largedatabase servers and databases his-torically don’t work all that well in avirtualized environment.” He said Sky-tap offers a hybrid model. “We havesecure bidirectional VPN technology,so you can build part of your lab in thecloud and connect back to your own

5 SOFTWARE TESTING MARCH 2009

aEDITOR’S

LETTER

aCHAPTER 1

VIRTUAL

TEST LABS

MAKE MOST

OF CLOUD

aCHAPTER 2

TESTING RIAs

FOR SECURITY

HOLES

“Being a startup, we

don’t want to spend the

world on getting three

racks of servers or a

huge box to throw five

or six VMs on. I knew

we wanted to go for

virtualized space

hosted on the cloud.”

—COLIN HENRY

Senior Software Engineer, Apptio

Page 4: SOFTWARE TESTING

data center. The hybrid model cantake advantage of the cloud as anextension of an in-house lab.”Roza likens the Skytap service toa family cell phone plan. Henry saidApptio has a plan for about 2,000hours per month. “We don’t have tokeep the infrastructure running full-time,” Henry said, “so we’re not usingunnecessary hours of their time.When you’re done, you’re done. Toget rid of a VM you’re not using any-more, you just delete and within a fewminutes it’s gone from the virtual lab.You can manage it relatively easily.”Roza added that pioneer Softwareas a Service (SaaS) companies likeSalesforce.com broke down barriersof concern that SaaS might not besecure. For Henry, with their testcases, “we’re not dealing with sensi-tive data, so in that regard it’s notmuch of an issue. For long-lead VMsleft on for monthly tests we have itset up over the VPN with Skytap.”

AUTOMATION EASES

TEST CONFIGURATION PROCESS

But testing organizations don’t needa cloud-based service to reap benefitsfrom virtualization, particularly forsetting up and tearing down test con-figurations. “Automated testing took alot of labor out of the process [beforevirtualization], but there was still a lotlabor involved in setting up and tear-ing down the configurations,” saidDave Malcolm, chief technology offi-cer of Surgient Inc., an early pioneerin the virtualization space.Surgient’s QA/Test Solution (for-

merly VQMS) automates the deploy-ment, configuration and teardown ofcomplex software environments. Mal-colm said to get started with virtual-ization, QA organizations need thephysical infrastructure to support avirtual lab, a hypervisor platform likeMicrosoft’s or VMware’s, and testautomation tools. Surgient integrateswith HP’s quality management suiteand IBM Rational. “The testing toolswill make the calls to the Surgientplatform to automate test configura-tions,” he explained.Surgient customers can choose ahosted offering, but most install on-premise, Malcolm said, and buildwhat he called an “internal or privatecloud.”

VIRTUAL TEST LABS

IMPROVE UTILIZATION

Insuresoft, a software provider forthe property and casualty insuranceindustry based in University Park, Ill.,decided to install Surgient’s QA/TestSolution on premise. “We wanted theflexibility to do what we wanted withour images and be able to trouble-shoot ourselves and go through theprocess of learning about virtualiza-tion,” said Hemanth Guttikonda,Insuresoft’s QA manager. “We hadsome expertise in-house to do someof these things; also we’re not a largeenough organization to considerhosting.”Guttikonda said Insuresoft turnedto virtualization to help with testingconsistency first and foremost, andthe company ultimately benefited

6 SOFTWARE TESTING MARCH 2009

aEDITOR’S

LETTER

aCHAPTER 1

VIRTUAL

TEST LABS

MAKE MOST

OF CLOUD

aCHAPTER 2

TESTING RIAs

FOR SECURITY

HOLES

Page 5: SOFTWARE TESTING

from better resource utilization aswell. He said the testing lab prior toSurgient was somewhat ad hoc, andthey would set up and tear down soft-ware as new versions of their Dia-mond product, a hosted application,needed to be tested.“We have 15 to 20 companies weservice and have physical hardwarefor each company. We had about30 to 40 physical servers, which mir-rored our production environment.”The process of installing and unin-stalling software to test for eachcustomer was tedious, he said.While some testing was done inthe lab, Guttikonda said other testingwas done on individuals’ physicalmachines, which he said wasn’t mim-icking what the product environmentlooked like. “The technology we weredeveloping wasn’t being fully tested.”Insuresoft was already utilizingautomated testing tools from HP andsaw Surgient, which supports HP, asa logical choice, he said.Guttikonda said the company’s testlab has been reduced from 30 to 10servers. “In essence, we trimmed ourphysical servers by over 60%, andwe’re getting more utilization out ofthe 10 servers than with the 30 in thepast,” he said. “But it really meansmuch more—it’s the ability to teardown the environment and reuse theservers.” Previously, he said, eachserver was dedicated to one customerand couldn’t be used for any otherpurpose. “Now we can use [the serv-er] for one company, and when we’redone, tear down and deploy it foranother, so hardware utilization is

way up.”Consistency and quality are up aswell, he said. “Now people see thebenefits of how quickly we can get theenvironment available and ready fortesting. In the past it sometimes took

a day or more to get the testing envi-ronment set up, so people wouldabandon the idea of a central environ-ment.” Once the organization startedusing Surgient, “we started noticingthat we were finding more defects.”These were defects that weren’tshowing up when testing on localmachines, he said. “People are start-ing to understand that [with Surgient]we’re actually testing in a true pro-duction environment.”In addition to getting used to thecultural change, Insuresoft had toinvest in more powerful servers thanwhat Guttikonda called the “entry-level” ones they had in order to getthe performance benefits of Surgient.However, Insuresoft is already seeingthe cost savings of better resourceutilization. “These servers are about$5,000, but in the past we wouldhave had to purchase 15 to 20 desk-

7 SOFTWARE TESTING MARCH 2009

aEDITOR’S

LETTER

aCHAPTER 1

VIRTUAL

TEST LABS

MAKE MOST

OF CLOUD

aCHAPTER 2

TESTING RIAs

FOR SECURITY

HOLES

“People are starting

to understand that

[with Surgient] we’re

actually testing in

a true production

environment.”

—HEMANTH GUTTIKONDA

QA Manager, Insuresoft

Page 6: SOFTWARE TESTING

top workstations at $30,000 a piece.Now we can serve the same numberof employees with a lot less hard-ware.”

CLOUD COMPUTING:

THE NEW HYPERVISOR?

This type of capital expense will beone of the effects of cloud computing,according to Ravi Gururaj, CTO ofvirtualization provider VMLogix Inc.in Palo Alto, Calif. “You’ll go from aCapEx to an OpEx,” he said, as organ-izations are billed for usage. “Thecloud is really the new hypervisor.Instead of an ESX [VMware hypervi-sor] box, you will have a cloud box.The cloud will give you the ability toprovision infrastructure on demand,but the tools themselves will be thesame, so the tools you’re using on-premise will be available in the cloud.”Another benefit, he said, is that“the cloud can scale when you havea bursting situation.” For instance,in the final stages of QA, scalabilitymay need to be tested. The cloudwill enable QA to extend its internalresources to provision from the cloudthe amount of infrastructure required.VMLogix is in the process of cloud-enabling its products, Gururaj said.The first stage will enable customersthat have VMLogix on premise toburst to the cloud when necessary,and that will be followed by a purecloud-based offering, he said.For organizations considering acloud-based solution, Skytap’s Rozasaid it will require a mindshift. “We’rein the early adopter phase around

cloud computing. There are going tobe some changes to processes, andyou have to embrace that and look atthe upside of the cost savings versushaving 10 servers next to my desk. Wefind a lot of VPs of IT say it’s a threatto [their] way of life, in terms of head-count, budget, etc. We try to do as

much as we can to make it feel like alab in your data center, with bettercollaboration.”IDC’s Ballou said organizations alsoneed to examine how the vendor hasstructured its cloud strategy. First andforemost, she said, organizationsneed to evaluate the stability and reli-ability of the vendor and the service.For example, she said, is there a planfor disaster recovery? Secondly,organizations need to look at how thepricing model is structured and deter-mine whether it is cost-effective: “DoI have the resources to run this inter-nally, versus do I have the time? Youhave to balance that.” �

COLLEEN FRYE is a freelance writer

and editor in Bridgewater, Mass.

She has been covering the IT industry

for more than 20 years.

8 SOFTWARE TESTING MARCH 2009

aEDITOR’S

LETTER

aCHAPTER 1

VIRTUAL

TEST LABS

MAKE MOST

OF CLOUD

aCHAPTER 2

TESTING RIAs

FOR SECURITY

HOLES

The cloud will

enable QA to extend

its internal resources

to provision from

the cloud the amount

of infrastructure

required.

Page 7: SOFTWARE TESTING