WHITEPAPER ROI Study on Automated Application Deployments ...go.xebialabs.com/rs/xebialabs/images/WP_2012-08_ROI.pdf · Deployment automation will yield a positive ROI within a time

WHITEPAPER

ROI Study on Automated Application Deployments for the Enterprise

Coert Baart

CEO, XebiaLabs


Copyright XebiaLabs – 2011. www.xebialabs.com/roi-calculation 2

Table of Contents 1. Management summary ...................................................................................................................... 3

2. Why deployments are important ....................................................................................................... 4

What is application deployment? ................................................................................................ 5

3. The hidden costs of application deployments ................................................................................... 6

Factors influencing your deployment costs ................................................................................. 6

4. ROI Study at large government agency .............................................................................................. 8

Benchmark assumptions .............................................................................................................. 8

ROI study results .......................................................................................................................... 8

ROI study conclusions ................................................................................................................ 10

5. Non-quantitative benefits of automated deployments ................................................................... 11

Appendix A: ROI calculation sheet ........................................................................................................ 12

Appendix B: Literature used .................................................................................................................. 13



1. Management summary

Within most companies, application deployment is the only semi-automated activity left in an otherwise, entirely automated Application Lifecycle Management process.

If organizations want to avoid error-prone and time-consuming deployments, and eliminate the hidden costs in their IT organization that are the result of manual or semi-scripted processes, end-to-end deployment automation is the way to move forward.

KLM Air France saves 1 million USD annually by having automated their software deployment process. Just as important is their new found ability to rapidly respond to LOB demands for new flight programs, marketing initiatives, customer support, etc. thus increasing their ability to respond to market changes.

Karavel/Promovacances has realized a positive ROI within 5 months time, by saving on hiring external resources.

A large government agency has calculated it saves more than 718k USD annually, while freeing up over 186 man weeks of extra development and test capacity.

Deployment automation will yield a positive ROI within a time period of 6 – 12 months for most organizations. On top of that, it brings many other benefits as well:

Reduced time to market for new applications

Streamlined end-to-end software production and delivery process, supporting continuous deployment workflows

More predictable and reliable deployment process

Provides additional flexibility in resource capacity needs and planning

Better insight into overall software delivery process

To estimate the ROI of an automated software deployment solution for your own organization, an (online) ROI calculation sheet is available for download at www.xebialabs.com/roi-calculation.

http://www.xebialabs.com/roi-calculation



KLM/AirFrance Case Study: Hand-over error rates of 70% KLM/AirFrance performs load tests on all web applications before go-live, in order to preserve the required operational stability.

In most cases, the applications were not deployed to the test systems on time, resulting in idle waiting times and inefficient test cycles. KLM/AirFrance investigated this issue further and found that the late deployment of applications was mainly due to error sensitivity during the hand-over between developer and deployer. Error rates of 70% during hand-over were common. This led to a very high bounce rate of yet-to-be-deployed packages, lots of rework for Development, idle waiting time for both Operations and Development departments and sub-optimal use of the test infrastructure and hardware.

2. Why deployments are important

Until an application is deployed, your software has no real value. It's like a boat that's in the docks: until it has been launched into the open water, it is merely a piece of nice craftsmanship, but doesn’t deliver on its promise. The same goes for software. No matter how well an application has been architected and written, it all comes down to the moment it can be put to work by the intended business user. After all, the proof of the pudding is in the eating.

The logical conclusion should be that application deployment activities are considered to be an integral part of the overall Application Lifecycle Management process. Yet more often than not, software deployments are simply regarded as a set of low-level administrative tasks to be carried out by the Operations team. Looking at software deployment processes like this lowers their importance and makes them hard to plan and difficult to manage. This is further supported by research which shows that on average 28% of all server downtime is caused by configuration changes, upgrades and deployments1.

The unpredictability and possible unexpected downtimes are the main reasons many organizations resist upgrading or replacing existing software applications: Adopting the philosophy of, “If it isn’t broke, don’t fix it”.

Unfortunately, this understandable resistance to upgrading software applications also means that organizations generally work with obsolete software for too long. Even if the new software contains more business functionality, is more efficient and is much more user-friendly, organizations do not readily upgrade because of the unpredictability of the software deployment process.

On the one hand, the business line manager would like to continuously improve his business operations in order to sell more products or services, while keeping costs down. And on the other hand, you have the IT manager responsible for managing a series of complex hardware and software environments who would like to keep the status quo intact, as much as possible, in order to insure continuity of the operational environment.



It’s obvious that organizations that upgrade their software cost effectively and without error, thereby allowing LOBs to stay up-to-date with market shifts, will have a distinct competitive advantage over those that don’t. There has always been a trade-off between keeping things as they are, not breaking the machinery that seems to function well, and improving business operations by supporting the LOB with an optimal set of software applications.

This may be why you often hear business line managers complain that “IT does not know, nor deliver what I need” and why IT managers exclaim “the business departments always demand more without giving me the additional budget needed to support their requirements”.

It is a classic example of two different views within one organization: a need for business agility versus a need for IT stability. But it does not mean that these two views are mutually exclusive, they can be united into one solution. It all starts with making software deployment an integral part of your IT process.

What is application deployment? Software application deployment comprises much more than simply unpacking and installing a piece of software and providing end-users with a link to the newly installed application. This would be a gross over-simplification of all the deployment activities carried out by IT Operations in order to make sure software works properly.

Deploying a piece of software can be quite a delicate task to do right, since it involves several steps that need to be completed correctly.

In fact, carrying out a deployment is a complicated, context-dependent process2.

For many years, deployments have been scripted by senior operations staff. Using some form of a shell script, batch file or Make, the deployment expert ensured a proper deployment. But as applications evolved over time, as well as the target environments to which the applications were deployed, the scripts grew bigger and more complex with every infrastructure modification and software update3. What started as a lean-and-mean script quickly became something big, uncontrollable and unmanageable for the average IT specialist.

A deployment consists of:

Installing an application

Configuring resources like databases

Configuring middleware components

Starting/stopping components

Configuring the installed application for the target environment

These steps are interdependent, can be recursive and need to be carried out in the right order.



3. The hidden costs of application deployments Most IT managers do not realize that a software engineer spends around 200 hours per year on redeploying his own applications. If you take an average salary rate of USD 60/hour, this boils down to around 12,000 USD per developer per year4. This does not yet include the costs of test engineers, build engineers, middleware experts and deployment specialists who frequently are consulted when the software engineer gets stuck. In general, it is safe to say that a deployment error created by one developer involves three other people to help get it solved.

Factors influencing your deployment costs There are many factors that directly or indirectly affect the costs associated with application deployments. These can be split into non-avoidable cost drivers stemming from business requirements, and avoidable cost drivers due to process flaws.

Non-avoidable cost drivers as imposed by business demands:

Business agility demands, like the number of deployments to be carried out per week/month/year;

Stability demands, like the number of intermediate servers to deploy to (think not only production, but also development, test and QA environments);

SLA demands, like high availability requirements for servers and applications, requiring redundant setups.

Avoidable cost drivers due to process flaws:

Number of failed upgrades/installations due to deployment errors;

Debugging time needed per deployment error;

Stand-by time of staff needed, in order to mitigate risks in complex projects;

Slow time-to-market for new applications, resulting in business opportunities lost;

Suboptimal scheduling due to low inter-exchangeability of staff, since not everyone has the same middleware and deployment skills;

Long learning curves for new personnel;

Number of external middleware and deployment experts needed;

Expert deployment skills required, involving additional training for your IT staff;

Delays and idle waiting time due to manual software hand-overs between Development and Operations;

Poor company image when applications are unavailable (“down”) longer than anticipated;

Orchestration meetings taking place within the IT group to coordinate the activities of a deployment;

Development efforts to ensure proper integration of the current toolsets.

On top of these, if an organization is using deployment scripts to support its deployment processes, two additional cost drivers are to be expected:

Development of new deployment scripts for new applications and/or new platforms;

Maintenance of existing deployment scripts due to changes to current applications and/or platform upgrades.

As one can see, the list of avoidable cost drivers is much longer than the non-avoidable cost drivers related to an organization’s demands for agility, stability and reliable service level agreements. Automating the deployment process helps to eliminate or reduce most of these cost drivers. Or as Ronni Colville, vice president and distinguished analyst at Gartner, stated in one of her interviews: “Organizations need to embrace an automated application deployment solution to ensure efficient, repeatable, accurate and reliable application deployments”.



A Hurwitz study1 was conducted amongst 249 companies to determine the impact of complex Java web application development, modification and deployment activities. This survey showed that organizations having a large number of web applications are more likely to have serious problems managing their manually maintained deployment scripts. These organizations faced an increase between 11 and 20 percent in their annual maintenance costs. Based on respondent feedback, the average company is spending over 850,000 USD per year in personnel costs to create, maintain, and support deployment scripts.

One study carried out by KLM Air France and IBM showed that KLM’s WebSystems department typically works on 35 different projects simultaneously, resulting in a total of around 200 application deployments per week5. However, KLM’s deployment system had limited functionality, was inflexible and error prone. KLM faced problems and errors in 70 percent of its software deployments, leading to issues with time to market for their (new) web applications and with overall operations costs. In total, KLM wasted about 1 million USD per year on this process until they decided to fully automate it using Deployit.

Karavel’s IT department manages around 30 web sites in parallel and deploys over 50 Java applications per week6. The growing number of recent acquisitions and the management of white label web sites have significantly increased the number of deployments to be carried out. This led to an increased risk of deployment errors and a number of painful situations. Having invested in fully automating their deployments with Deployit, Karavel achieved a positive ROI within 5 months time, by saving on hiring external resources.

But not only do organizations that develop their own software face enormous hidden deployment costs. Organizations that buy commercial off-the-shelf (COTS) or bespoke software, also pay a significant amount of money each year on hidden application deployment activities.

Just think about the hours of downtime your mission-critical business applications experience when a deployment fails half way, and a rollback is not possible. Based on the same Hurwitz study, companies may lose as much as 72,000 USD for each hour of downtime. This number does not take into account the fact that an hour of not being “open for business” may hurt even more, both in terms of opportunities lost and projecting a poor company image to existing and prospective clients.

Fig 1: Cost savings using fully automated approach.



4. ROI Study at large government agency A large government agency* carried out an internal ROI study after having used Deployit for 6 consecutive months. In their study, they compared the previous status quo, being a manual deployment process supported by several deployment scripts, with their newly installed automated deployment processes running on Deployit. They also made projections for future savings, based on a 3-year time window.

Benchmark assumptions The following assumptions, exclusions and criteria were applied:

The ROI study looks at several cost factors: initial implementation costs, maintenance costs and annual license costs;

Calculations are based on the experiences with Deployit after using it for 6 months, and projected to a 3-year usage period;

There are 30 actively developed applications on average per year, which are (re) deployed on average once per week;

Each deployment is executed four times, since four different environments are used (development, test, acceptance and production);

Historic data on manual deployments showed that the promotion of a deployment package from development to test tot acceptance to production took on average 2 hours;

The costs per hour worked range between $ 100 and $ 125, depending on who needs to be involved in each deployment activity;

Deployit licenses cost $ 80,000; annual support fees are 20% of the purchase price.

ROI study results After 6 months of using an automated deployment approach based on Deployit, the organization calculated that the time needed to complete an end-to-end application deployment through all four environments merely was 40 minutes. Compared to the manual way of handling deployments, this was an enormous improvement as compared to the previous 2 hours of work needed.

Another finding was that the number of wrong deployments (mainly due to initial mis-configurations, incomplete deployment packages and/or non-synchronized target environments) went down from 35% to 6% of all deployments. On average, an unsuccessful deployment took 2.5 man hours to get solved in the old situation based on manual deployments, usually involving a software developer, a deployment expert and a test engineer. In the new situation with Deployit, an error usually was resolved within 1 hour thanks to the automatic logging of each deployment step.

Given the fact that the government organization carries out 30 deployments per week across four environments, a total time of 35% x 120 deployments x 52 weeks x 2.5 hours resulted in 5460 hours per year spent on resolving erroneous deployments. With Deployit, this number was brought down to just 375 hours/year (= 6% x 120 deployments x 52 weeks x 1 hour).

* Due to contractual obligations, the name of this organization cannot be disclosed in this whitepaper.



The tables below give a more detailed view on hours spent and costs involved.

Manual deployments supported by scripts Hours Hourly rate ($) Costs ($) Total costs/year ($)

Maintenance time 400 125 50,000

50,000

Deployment of 30 apps / year

to DEV 1080 100 108,000

to TST 1080 100 108,000

to ACP 720 110 79,200

to PRD 260 110 28,600

323,800

Error resolution time 5460 100 546,000 546,000

Manual deployments Hours/year Total costs/year ($)

Total costs 30 apps 9000 919,800

Table 1: Costs associated with semi-scripted manual deployments.

Carrying out these semi-scripted manual deployments took 9,000 hours per year all in all, or an average of 300 hours per application per year. The majority of these hours are spent on problem resolution, as about one-third of all deployments contain one or more errors that cause them to fail at some point during the deployment process. Total annual deployment costs can (thus) be calculated at more than $ 900,000, of which more than halve the costs were hidden costs in solving errors, rework and waiting times.

Automated deployments using Deployit Hours Hourly rate ($) Costs ($) Total costs/year

† ($)

Initial implementation 120 100 12000‡

4,000

License cost incl 20% maintenance fee

128,000

42,667

Maintenance time 90 125 11,250

11,250

Deployment of 30 apps / year

to DEV 480 100 48,000

to TST 360 100 36,000

to ACP 120 110 13,200

to PRD 80 110 8,800

106,000

Error resolution time 375 100 37,500 37,500

Totals (incl. initial implementation costs split over 3 years)

Hours/year

Total costs/year ($)

Total costs 30 apps 1545 201,417

Table 2: Costs associated with fully automated deployments using Deployit.

† Costs are based on 3-year period

‡ Costs only to be made in first year.



Working with fully automated deployment scenarios based on Deployit, deploying applications has become much more straight-forward. Not only the average deployment time per application was reduced by 80 minutes per full end-to-end deployment, especially the number of broken deployments was brought down. Originally, 35% of all application deployments failed, mainly when the applications were transferred from development to test environments. The major reason behind this was that the development environments were not in-synch with the test environment, as developers had the possibility to install additional software libraries and scripts on their own computers in order to make their software applications work correctly.

In the new situation with Deployit, it has gotten much easier to compare environments before transferring the application from the development to the test environment. The number of broken deployment processes was therefore reduced significantly to only 6% of all deployment processes.

ROI study conclusions The study indicated that the large government organization saves $ 718,383 annually, which equals to a 78% lower annual deployment cost! The total annual deployments efforts were reduced by 7455 hours, or 186 man weeks. This time will now be used to release new versions and new applications faster. Also, due to reducing the number of deployment errors, the entire delivery process has become much more predictable.

These enormous savings include both a reduced deployment time for successful deployments, fewer erroneous deployments and a shorter error resolution time in case a deployment did go wrong.

As a next step, the organization recently decided to implement Deployit within other affiliate departments as well.



5. Non-quantitative benefits of automated deployments

Based on the ROI studies in the previous chapters (KLM, Karavel and the Government Agency) and the Hurwitz study mentioned earlier, it is safe to say that deployment automation yields a positive ROI for most organizations within only a few months’ time.

Given the fact that the average company is spending over 850,000 USD per year in personnel costs only, to create, maintain, and support deployment scripts, it is clear that by automating this approach, enormous cost savings are possible.

However, besides these immediate contributions to the bottom line of your IT organization, other benefits of using an automated deployment approach apply as well.

For example, KLM Royal Dutch Airlines is now able to manage and configure mission-critical applications quickly and confidently, thereby reducing its dependency on specialized technology experts. The deployment automation solution furthermore helped KLM reduce time to market for new applications by about 10 percent. This allowed the manager of the WebSystems department to transform his team from being a notorious production bottleneck into an “enabler of change”.

Non-quantitative benefits of deployment automation include:

Streamlined end-to-end software production and delivery process, supporting continuous deployment workflows:

o Easier to exchange data with other ALM applications o One vocabulary to communicate about deployments between all IT staff o Deployments across different types of middleware involving various

deployment artifacts can be carried out in a consistent, uniform way o Execution based on well-known deployment patterns

More predictable and reliable deployment process: o Prevents ‘forgetting’ items in the deployment package o Higher flexibility and tight security by generating the deployment package's manifest file

without having to hard-code values o Easily integrate automated deployment into your test and build cycle o Fewer human errors by avoiding manual tasks to create a deployment package or update

the CMDB accordingly

Provides additional flexibility in resource capacity planning: o No need to be a deployment expert to carry out deployments in a safe way o Facilitates self-service deployments for developers, saving time for Operations staff o Eliminates steep learning curve for new employees – also junior Operations staff can

carry out non-complicated deployments o Makes it easy to transfer or reschedule work to colleagues

Better insight into overall software (deployment) process: o Facilitates transparency and uniformity for analysis and reporting activities o Overview of what applications have been deployed to what environments o Full insight into all deployment execution logs o Up-to-date status information on all deployments carried out.



Appendix A: ROI calculation sheet

To download a free ROI calculation sheet, please visit www.xebialabs.com/roi-calculation. Based on a short survey, you will get an instant overview of possible cost savings for your own organization when automating your deployment activities. Downloading the calculation sheet is free-of-charge and without any further obligations.

http://www.xebialabs.com/roi-calculation



Appendix B: Literature used

1. The Sources of Web Application Downtime, Hurwitz & Associates, 2009. 2. So what is a deployment really, Robert van Loghem, 2009.

http://blog.xebia.com/2009/07/08/so-what-is-a-deployment-really/ 3. Java Deployments in an Enterprise Environment, Vincent Partington, 2010. 4. Survey results on Java EE Containers: Heaven or Hell?, Zero Turn Around, 2009.

http://www.zeroturnaround.com/blog/java-ee-container-heaven-hell-survey-results/ 5. KLM realizes savings of approximately US$1 million with IBM and XebiaLabs, IBM case study,

2010. ftp://public.dhe.ibm.com/common/ssi/ecm/en/wsc14227usen/WSC14227USEN.PDF

6. Internal study, Karavel, 2010.

http://blog.xebia.com/2009/07/08/so-what-is-a-deployment-really/

http://www.zeroturnaround.com/blog/java-ee-container-heaven-hell-survey-results/

ftp://public.dhe.ibm.com/common/ssi/ecm/en/wsc14227usen/WSC14227USEN.PDF

Documents

WHITEPAPER ROI Study on Automated Application Deployments ...go.xebialabs.com/rs/xebialabs/images/WP_2012-08_ROI.pdf · Deployment automation will yield a positive ROI within a time