TEST AUTOMATION: A PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS · 2019-06-06 · Test automation by no means eliminates all manual testing. Instead, a sound testing plan employs

TEST AUTOMATION:A PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS

TEST AUTOMATION: A PRACTICAL PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS © Santa Rosa Consulting, Inc.

2

TEST AUTOMATION: A PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONSThe time has finally come for healthcare test automation. In fact, a strong

case can be made that test automation is a must-have asset in today’s

healthcare environment. Simply put, best practices in IT system testing have

outpaced almost any organization’s ability to keep up with largely manual

methods. The proof of this reality can be found in the very difficult (and risky)

choices organizations often make when laying out a testing plan for an EHR

implementation:

• What will be tested (and what must be ignored)?

• How often can we test (what is enough or too little)?

• What should we test in the ecosystem outside of our EHR?

Fortunately, test automation has come a long way over the past few years,

making major strides in the areas of deployment and proven effectiveness.

There are many areas of consideration healthcare organizations need

to evaluate when establishing a test automation strategy. The following

addresses many of these major factors.

https://www.santarosaconsulting.com/resources/ehr-implementation/


3

PRIMARY ADVANTAGES OF TEST AUTOMATION

Test automation is the gift that keeps on giving. One of the most time and resource-intensive

activities in an EHR implementation – or any major IT system, such as an ERP or revenue

cycle solution – is the extensive manual testing required before, during and after go-live.

EHR implementations would be far more successful if not for the manual testing needed before, during and after go-live.

The routine is all too familiar. A large group of end users spend multiple days testing keyboard

functions in a classroom setting. The problems identified are fixed, and the cycle starts over

again.

Test automation changes the process, team composition, capacity to test and the monotony

of the effort. A proper test automation program, supported by the necessary tools, allows

healthcare organizations to streamline and strengthen test processes, while providing

tangible advantages and improved employee morale. Simply put: an improved EHR project.

�� The primary benefits of test automation include:

• Greater Test Accuracy/Quality: Even the best manual testers make mistakes. Automation reduces errors while increasing confidence in testing metrics and results.

• Expanded Scope and Test Capacity: The volume of test workflows and iterations/scripts conducted increases significantly with automation and dynamic scripting. This allows for greater capacity and far more departments and functions to be included within the testing scope.

• Reduced Time and Effort: Not only does automation support a greater volume of tests, but tests can be completed in a far shorter period of time as critical tests can be repeated via full regression testing on a daily basis. In the end, you are not performing the actual test but simply validating the error results and repairing the problems identified.

• Cost Savings: Automation means your staff spends far less time away from their primary duties executing test scripts, which decreases backfill and other employee-related costs and significantly increases testing capacity.

• Decreased Risk: Improper testing, or testing shortcuts, can allow small yet critical bugs and problems to slip through into a live patient care setting. Many of these can create significant patient and financial impact if left undetected or without remedy.

• Higher Satisfaction: Test automation results in fewer bugs and challenges discovered ‘the hard way’ by end users in the production environment, which, in turn, minimizes end user (including physicians and clinicians) frustration while maximizing satisfaction and adoption.

• Sustainable Value: While the benefits are considerable during initial EHR implementation, they exponentially increase over time as vendors move toward quarterly (or more frequent) upgrades and the library of tailored test automation scripts consistently grows.


4

KEY STAFFING ASSUMPTIONS

Cost per FTE (Annual and Fully Burdened) for IT/Business/Clinical Test $115,000

Work Hours per Year 2,080

Hourly Fully Burdened FTE Cost $58.20

ANNUAL TEST CYCLE DESCRIPTION

Number of Test Cycles per Year 3

Typical Test Cycle Duration (Weeks) 10

AVERAGE TEST CYCLE DESCRIPTION

Total FTEs Available to Execute Testing 16

Percent FTEs Assigned to Testing + Error Validation (To be Automated)* 60%

Equivalent FTEs Assigned to the Test Cycle 9.6

Percent Work Assigned to Resources Dedicated to Testing 70%

Total Hours Devoted to Testing per Test Cycle 2,688

Total Cost per Test Cycle $148,620

AVERAGE TEST PRODUCTIVITY

Number of Scripts Run (Current) per Test Cycle 100

Average Cost per Script during Test Cycle $1,486

ESTIMATED COSTS

Total Annual Cost under Standard Manual $445,859

3-Year Cost $1,337,576

CURRENT APPROACH COST YEAR 1 YEAR 2 YEAR 3 TOTAL

Total Number of Scripts Run 300 300 300 900

Estimated Annual Cost $445,859 $445,859 $445,859 $1,337,576

ESTIMATED TEST AUTOMATION COSTS YEAR 1 YEAR 2 YEAR 3 TOTAL

Total Scripts Run 15,600 15,600 15,600 46,800

Initial Custom Development + Annual Fee $150,000 $100,000 $100,000 $350,000

One-Time Service Fee $50,000 $50,000

Total $200,000 $100,000 $100,000 $400,000

ANTICIPATED SAVINGS YEAR 1 YEAR 2 YEAR 3 TOTAL

Annual and 3-Year $245,859 $345,859 $345,859 $937,576

Average Annual Savings $312,525

Estimate Current (Standard Manual) Costs

Test Automation - Financial Comparison

THE FINANCIAL IMPACT OF TEST AUTOMATION

*Example for an approximately 300-bed hospital

A Comparison of a Standard Manual Test Effort vs. Full Test Automation

*Testing and validation activities to be automated


5

THE RETURN ON INVESTMENT (ROI) OF TEST AUTOMATION

As with any investment, it is important to have a clear understanding of where and how test automation can deliver

the highest ROI. Healthcare organizations can realize significant returns from relatively modest investments in

test automation within two primary areas.

Regression TestingAs implied above, test automation generates the greatest ROI during repeated regression testing that can be leveraged for each and every EHR update or upgrade, as well as, other applications that intersect with the EHR in your ecosystem. Many vendors are rapidly moving toward releasing new versions on a quarterly (or even on-demand) basis. As such, regression testing becomes even more critical for ensuring EHR usability and safety. In addition, and as the technology ecosystem expands in healthcare, testing your EHR quickly, after additional systems have been introduced to the ecosystem, has exponential value.

Automation allows regression tests to be completed reliably in a few hours, rather than over several weeks or even months if done manually. This efficiency is essential to comprehensive testing of regular software updates and new releases. It doesn’t take long for the cost savings of regression testing to add up to a significant ROI for test automation. In fact, test automation may deliver a one-month ROI if a health system tested everything they would like to with their existing resources or budgets.

Operational EfficienciesHealthcare organizations also reap sizable and tangible ROI results from test automation by allowing staff testers to remain focused on their primary duties and the many needs that surround an EHR implementation. Automation allows them to spend more time addressing/fixing test errors versus merely finding them. However this additional time is used, giving your staff time back in their day results in more efficient workflows and lower operational (and project) costs overall.

These two are the most common areas to yield a financial return from an investment in test automation. However, for many organizations, the non-financial key advantages cited above are often as important (or more important) than an ROI alone.

Test automation by no means eliminates all manual testing. Instead, a sound testing plan employs test automation where it

has the greatest value and reserves manual testing to a select subset of activities.

FINANCIAL ADVANTAGES CURRENT TEST AUTO IMPACT SUMMARY

3-Year Estimated Cost $1,337,576 $400,000 -$937,576 30% of current

Estimated Cost per Script Run $1, 468 $9 -$1,555 0.6% of current

Approximate B/E 9.4 months

TEST SCOPE & CAPACITY CURRENT TEST AUTO IMPACT SUMMARY

Testing Cycles per Year 3 52 +49 17.3 times more

Scripts Run per Cycle 100 300 +200 3 times more

Scripts per Year 300 15,600 14,300 52 times more

Weeks per Cycle 10 1 -9 10% of current

Beneficial Impacts


6

User & DataValidation

FOUNDATION

Validate WorkflowCreation

(Function/Module)

CORE

“End-to-End”Integrated Scripts

IMPROBABLE

User Security & System Access

ImmediateValidation Points(Typically Multiple Steps)

•

•

Automating the Registration Workflow

Positive or NegativeTests Possible

•

•

Data Passage between SystemsApplication or Ecosystem

•

•

01

02

03

PHASES OF IMPLEMENTATIONA proper approach to test automation implementation first

validates foundational “low hanging fruit” before converting manual scripts to E2E automation.


7

TEST PLANNING: THE 80�20 RULE

When deploying test automation, one of the earliest decisions to make is which test activities

are best suited for automation. It is rare for a healthcare organization to automate all

testing as certain activities require personal judgment to determine the best combination

of possible test scenarios. The following guidelines have proven helpful for organizations

preparing initial testing plans.

What to Automate

Generally, automation should cover 80% of all testing activities in an EHR implementation or upgrade, and virtually all activities requiring keyboard or mouse functions.

For example, patient registration is a prime candidate for automated testing. A manual test for registering a patient, with minimal dynamic data, takes approximately eight to twelve minutes to complete. By comparison, an automated test for the same task can be completed nightly in a regressive model in two to three minutes. Couple that with the use of dynamic data and the differences in effort is massive.

Additionally, all orders (medication, labs, radiology, etc.) should be completed via test automation. Doing so can mean starting the test in the EHR, moving into a downstream system to record the result from the order, and going back into the EHR to validate that the result returned as expected. This eliminates the need for experts to be “on-call” and prevents delays in manual script completion.

Beyond registration and orders, patient scheduling and claims processing should also be considered for automation, as the entire end-to-end patient experience is covered.

What Not to AutomateThe remaining 20% of EHR testing activities still should be completed manually when an organization is creating an initial test automation plan. This is considered “single source testing” and covers tests against a specific build that would not be used with each subsequent release. These are random, unexpected tasks, and there is not much benefit to automating their process. It is better to focus automation efforts on the 80% of tasks that are repeatable and will deliver sustainable value through repeatability during implementation and future upgrades.

Again, these guidelines have proven useful to healthcare organizations deploying test automation on a broad scale for the first time. Organizations with test automation experience, bolstered by the steadily growing library of tailored scripts, may tackle increasingly challenging testing through automation and move the 80/20 rule toward 90/10, and beyond. They do so, however, armed with first-hand lessons learned.


8

LESSONS LEARNED IN TEST AUTOMATION

While no two situations are exactly alike, there are some time-proven lessons learned

beyond understanding what to automate (and what not to) in getting maximum value and

benefit from test automation. These include:

1. Tool vs. Talent: A good test automation tool is important but not as important as the experience and skills of the resources wielding the tool. Knowing what to automate, how best to do so, when to run automation and what to do when things go wrong is critical. No tool can make those pivotal decisions.

2. Timeline Expectations: Test automation saves time, but it takes time for the first deployment. The exact timeline depends on the scope of automation and third-party involvement, but, in general, testing begins within a month or so after the test automation system is installed and test patients and end user workflows have been set up. An organization can then expect to see results very quickly.

The entire test automation process—including running automation scripts for 80% of EHR tasks—can be completed over a three- to six-month time period.

3. Testing Design: Automation without good test design may lead to a lot of activity but little value. The primary role of test automation specialists is not only to establish a solid design but to understand when and how that design can be enhanced over time to fine-tune results and expand scope. Deriving increasing value from test automation, over time, is a central and critical part of its upkeep.

4. Staying Current: Speaking of upkeep, it is imperative that test automation design and scripts keep up with the continual updates and releases of the EHR (or whatever system is being tested). Changes to the EHR require at minimum a careful review to determine the extent to which related scripts must be changed. This rather obvious lesson is worth pointing out because over time testing results (automated or manual) become unreliable and even misleading.

These proven and basic lessons learned in test automation can be valuable as a starting point to develop your program. There obviously are many, many more that may be highly relevant to an individual organization’s environment and situation. In the end, it pays to tap into as much available test experience as possible.


9

THE CASE FOR EXPERIENCED HELP

Many healthcare organizations can and will deploy test automation on their own. They

may have the knowledge and resources to do so. Other organizations, however, have found

partnering with a highly experienced consulting firm to pay great dividends. A capable partner

will help prepare an appropriate test automation plan, integrate test automation software

with the EHR(s), automate an entire library of tailored testing scripts, and run the initial wave

of automated tests, if needed. When deciding on the need for experienced, outside help,

most healthcare organizations consider several pivotal factors:

• System Expertise: By working with an experienced partner, healthcare organizations tap into their consultants’ deep system expertise. Consultants have certifications with the EHRs being tested and are knowledgeable of the exact workflow changes required during updates and upgrades. They can provide recommendations around regression testing and help a testing team prioritize scripts for automation.

• Cost Savings: A partner completes automated tests in a far shorter period of time than a hospital’s own testing team, which in turn shortens the overall EHR implementation or upgrade timeline. Less time and fewer internal resources needed for an implementation project results in cost savings for the entire organization, as described above.

• Scripts Library: Most organizations with common EHRs benefit tremendously from the extensive library of ‘starting point’ test scripts that outside help can provide. Many of these scripts can be applied with minimal or no modifications. Reinventing the wheel is a waste of time and money, and the library dramatically shortens time to generate ROI.

• Test Coverage: Organizations without test automation often find it very difficult, if not impossible, to determine with certainty whether they’ve covered 100% or 50% of the most common test cases. They don’t know what they don’t know. A proven and professional test partner has the tools and knowledge to accurately determine test coverage.

• Staff Availability: Most organizations simply do not have the staff required in the near-term to successfully deploy test automation, especially against tight timelines. A partner augments IT and clinical teams, so a hospital can meet implementation deadlines and realize results sooner. Employee morale will also increase when team members can stay focused on their daily jobs without testing interruptions.


10

About Santa Rosa ConsultingSanta Rosa Consulting provides management advisory services and technical consulting across the spectrum of vendor and IT products and systems. We empower better healthcare by designing, creating, staffing and sustaining IT solutions that deliver quantified value. Founded in 2008, Santa Rosa Consulting is managed by industry veterans with an average of 20 years of healthcare information technology experience. Santa Rosa is recognized for delivering world-class services and solutions and has received Modern Healthcare’s Best Places to Work in Healthcare award multiple years in a row.

Visit: santarosaconsulting.com

SUMMARY

Test automation is a true game changer for implementations and

upgrades of major IT systems such as EHRs. The days of making difficult

and risky decisions on testing are over as test automation delivers

tangible advantages while positioning a healthcare organization to have

far greater confidence in, and satisfaction with, new systems.

Santa Rosa Consulting has more experience leading test automation

than any other firm and proven success in helping clients save time and

improve project efficiency. Our team understands the ins and outs of test

automation software, and we know how to seamlessly integrate it with

any EHR on the market. We bring significant experience and expertise,

along with a growing library of test automation scripts, to your table.

Through test automation services, we’re able to provide a proactive, ROI-

driven solution that meets a need all healthcare organizations will face

during EHR implementations.

THE STRENGTH YOU NEED IN EVERY STEP OF THE HEALTHCARE IT JOURNEY

Documents

TEST AUTOMATION: A PREPARATORY GUIDE FOR HEALTHCARE ORGANIZATIONS · 2019-06-06 · Test automation by no means eliminates all manual testing. Instead, a sound testing plan employs