Upload
kristinkgibson
View
221
Download
0
Embed Size (px)
Citation preview
8/8/2019 002343.Online Testing Vendor Landscape
1/12
Making Leaders Successul Every Day
Octoe 20, 2009
The Olie Testig VeoLscpe Joh Lovett
o Cstoe Itelligece Poessiols
http://www.forrester.com/8/8/2019 002343.Online Testing Vendor Landscape
2/12
2009, Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited. Inormation is based on best availableresources. Opinions refect judgment at the time and are subject to change. Forrester, Technographics, Forrester Wave, RoleView, TechRadar,and Total Economic Impact are trademarks o Forrester Research, Inc. All other trademarks are the property o their respective companies. Topurchase reprints o this document, please [email protected]. For additional inormation, go to www.orrester.com.
Fo Cstoe Itelligece Poessiols
ExECuTIVE Summary
Online testing is an underused technology solution in the burgeoning eld o Web site optimization.
Numerous methodologies and testing disciplines exist, each with evidence o success in improved Web
site usability, increased relevance, and elevated revenue. Forrester surveyed 250 Web site decision-
makers to determine current levels o adoption as well as the barriers and benets o online testing. Tis
report uncovers our ndings, untangles a conusing vendor market, and helps customer intelligence
proessionals use testing to improve their online initiatives.
TabLE OF COnT EnTSOli Tsti Still Ss A Adi
Propr Oli Tsti Iprovs Bsiss
Prora
Forrstrs Dirtor O Oli Tsti
Vdors
rECOmmEndaTIOnS
Iplt Tsti As Part O A Opratioal
Pross
nOTES & rESOurCESFoeste iteviewe 14 veo se
copies o this epot.
Rlatd Rsarh Dots
Testig a Optiiztio dieetitos
novee 7, 2007
mltivite Testig age
Octoe 22, 2007
Octoe 20, 2009
The Olie Testig Veo Lscpea mte disciplie Oes The net Fotie I Site Optiiztio
b Joh Lovttwith Clto a. dot, Sesh Vittl, Eil mph
2
3
7
9
mailto:[email protected]://www.forrester.com/http://www.forrester.com/go?docid=53062&src=53637pdfhttp://www.forrester.com/go?docid=52881&src=53637pdfhttp://www.forrester.com/go?docid=52881&src=53637pdfhttp://www.forrester.com/go?docid=53062&src=53637pdfhttp://www.forrester.com/http://www.forrester.com/mailto:[email protected]8/8/2019 002343.Online Testing Vendor Landscape
3/12
2009, Foeste resech, Ic. repoctio PohiiteOctoe 20, 2009
The Olie Testig Veo Lscpe
Fo Cstoe Itelligece Poessiols
2
OnLIne TeSTIng STILL SeekS An AuDIence
Online testing reers to the discipline o using technology to present varied messages, images, or
promotions to consumers in hopes o determining which version resonates best or a specied
audience and purpose. Te practice is as old as dirt, but the prolieration o the Web and digital
channels is giving testing a second wind enabling it to evolve into a science in its own right.
Despite the implicit value and documented success stories, adoption among marketers remains
woeully low. Only 26% o survey respondents currently use testing on their primary Web sites.1
Marketers are still skittish about testing or the ollowing reasons:
Testing methods conuse marketers. esting relies on complex algorithms like aguchi,orthogonal arrays, optimal design, and ull versus ractional actorial. Tese statistical methods
are derived rom design o experiments (DoE) principles and have been applied to digital
marketing over the past decade.2 While some le-brained marketers deploy these methods with
great success, most dont know which techniques to use or what to make o the results, so theyavoid complex testing technologies and treat them with irreverence.
Common challenges continue to stave of adoption. Ironically, the biggest challenge citedby 48% o marketers is proving the return on investment, which testing is inherently designed
to do (see Figure 1). Yet, marketers struggle to adopt online testing because deterrents such as
prioritizing test initiatives, taking action upon test results, and understanding testing outcomes
inhibit action.
Lack o ownership muddies accountability. Online testing lacks a true home in manyenterprise organizations. echnology teams dont want it because it requires creative types
to populate ideas, marketers shy away rom it because it involves page tagging, and analyticsteams dont sign o because most are overwhelmed with their own issues. Furthermore, a urry
o market consolidation in recent years conuses vendor relationship managers.3 As a result,
ownership and accountability o online testing is obscured, thereby limiting its success.
8/8/2019 002343.Online Testing Vendor Landscape
4/12
2009, Foeste resech, Ic. repoctio Pohiite Octoe 20, 2009
The Olie Testig Veo Lscpe
Fo Cstoe Itelligece Poessiols
3
Fir 1 Top Opetiol Chlleges Fo Testig
Source: Forrester Research, Inc.53637
Please indicate your top three challenges in operating your companys
site optimization or A/B or multivariate testing application.
We havent had any challenges in operatingour A/B or multivariate testing application
Attaining enough traffic to achieve statisticalsignificance
Creating test elements (e.g., copy, creative)
Securing funding to execute tests
Establishing testing scenarios (e.g., concepts,areas to test)
Ability to segment/target test participants
Usability of the testing interface
Coding test within administrative interface
Lack of corporate buy-in
Understanding test results
Modifying Web pages for testing
Acting upon testing results
Prioritizing testing initiatives
Demonstrating return on investment (ROI)
Base: 65 Web site decision-makers who currently use A/B or multivariate testing applications
48%
25%
25%
23%
20%
20%
19%
19%
15%
14%
14%
12%
8%
2%
Source: JupiterResearch/e-Rewards Website Spending Executive Survey (12/08)
(multiple responses accepted)
PROPeR OnLIne TeSTIng ImPROVeS BuSIneSS PeRFORmAnce
Despite low adoption and pronounced challenges, 48% o marketers surveyed who currently employ
testing technologies report increases in online conversions as a top benet. We also ound that 46%
o marketers using testing cited increased customer satisaction as another key benet. Overall, the
majority o organizations that are testing preer managing it entirely in-house, ollowed by agency
managed services (see Figure 2). Yet, analysis shows that organizations that leverage managed
testing services produce better results than those that manage in-house. Forrester ound that:
Recurring programs are successul. On average, testers deploy nearly eight distinct A/B ormultivariable tests each month. Our research showed that processes like registration and orm
completion improved with ewer tests, while overall conversions and customer satisaction
benet rom more testing (see Figure 3). Survey results also showed that running more than 12
8/8/2019 002343.Online Testing Vendor Landscape
5/12
2009, Foeste resech, Ic. repoctio PohiiteOctoe 20, 2009
The Olie Testig Veo Lscpe
Fo Cstoe Itelligece Poessiols
4
monthly tests could produce diminishing returns, yet sites that test on a recurring basis exhibit
the greatest success.
Marketers prioritize testing on high-impact areas. Organizations using testing technologiestypically gain the best results when ocusing on specic initiatives such as: streamlining usability,
increasing relevance, or elevating prots (see Figure 4). Heavily trafcked pages or specic
campaign landing pages are eective places to start optimizing Web sites. Why? esting relies
on visitor trafc to determine which test scenarios (sometimes reerred to as recipes) are most
eective and thus requires a certain volume o trafc to reach statistical signicance.
Employing a testing partner pays dividends. While most organizations use sel-service testingmethods ollowed by managed services, our research reveals that organizations working with
managed service vendors, agencies, or consulting partners achieve greater li in most categories
(see Figure 5). While evidence shows that testing can result in improvements regardless o the
method employed, organizations that leverage a managed service model achieve greater gains.
On average these gains are two percentage points higher, which could equate to millions o
dollars or conversion-based Web sites.
Fir 2 Peee metho Fo Testig mgeet
Source: Forrester Research, Inc.53637
Allow vendor to managethe entire process
2%
Work with our vendorsprofessional services
group7%
Allow external agency tomanage the entire process
9%
Work with an externalagency or consultancy
38%
Manage entirelyin-house
45%
Base: 128 Web site decision-makers who currently use or plan to use A/B or multivariate testing applications
Source: JupiterResearch/e-Rewards Website Spending Executive Survey (12/08)
What is your preferred method of performing site optimizationand/or A/B or multivariate testing on your public facing Web site(s)?
8/8/2019 002343.Online Testing Vendor Landscape
6/12
2009, Foeste resech, Ic. repoctio Pohiite Octoe 20, 2009
The Olie Testig Veo Lscpe
Fo Cstoe Itelligece Poessiols
5
Fir 3 Test Feqec a We Site beets
Source: Forrester Research, Inc.53637
Approximately how many A/B or multivariate tests (on average) do you perform each month?
Base: 128 Web site decision-makers who currently use or plan to use A/B or multivariate testing applications
Source: JupiterResearch/e-Rewards Website Spending Executive Survey (12/08)
0%
10%
20%
30%
40%
50%
60%
20 ormore
18-1915-1712-149-116-83512Zero
Number of tests
*Increasedregistrations
*Increasedconversions
Number of tests
*Increased conversions
*Increased registrations
5%
0%
0%
18%
13%
4%
23%
19%
57%
13%
13%
9%
19%
19%
9%
10%
23%
4%
2%
7%
0%
1%
0%
0%
9%
7%
17%
Did you know?
Testers deploy an average of
7.6 tests each month.
What are the benefits your company has realized through siteoptimization or A/B or multivariate testing?*
8/8/2019 002343.Online Testing Vendor Landscape
7/12
2009, Foeste resech, Ic. repoctio PohiiteOctoe 20, 2009
The Olie Testig Veo Lscpe
Fo Cstoe Itelligece Poessiols
6
Fir 4 aes O Focs Fo Olie Testig
Source: Forrester Research, Inc.53637
Off-site display advertising
Product cross-sell/upsell
Product placement/merchandising
Internal promotions/offers
Segment targeting
Site or commerce search results
Email marketing
Landing pages
Multistep processes (e.g., checkout,booking)
Forms (e.g., registrations, surveys)
Site navigation
Site design 66% 19%
65% 21%
56% 29%
52% 20%
59% 23%
56% 20%
51% 23%
45% 32%
56% 21%
53% 27%
46% 25%
43% 26%
Currently use Plan to useUsability
Relevance
Revenue
Base: 128 Web site decision-makers who currently use or plan to use A/B or multivariate testing applications(multiple responses accepted)
Source: JupiterResearch/e-Rewards Website Spending Executive Survey (12/08)
In which of the following areas do you currently use site optimization or A/B or multivariate
testing technologies and in which do you plan to start using it in the next 12 months?
8/8/2019 002343.Online Testing Vendor Landscape
8/12
2009, Foeste resech, Ic. repoctio Pohiite Octoe 20, 2009
The Olie Testig Veo Lscpe
Fo Cstoe Itelligece Poessiols
7
Fir 5 avege Lit Copiso Fo I-Hose Vess Otsoce mgeet
Source: Forrester Research, Inc.53637
0%
3%
6%
9%
12%
Work with an external agency or consultancy
Manage entirely in-house*
Conversionrate
RegistratonCustomersatisfaction
Formcompletion
rate
Averageordervalue
Sitesearch-basedconversions
Landingpage
conversion
4%4%
4%4%
7%
6%
8%
3%
5%5%
7%
5%
8%
10%
Source: JupiterResearch/e-Rewards Website Spending Executive Survey (12/08)
Base: 22 Web site decision-makers who work with an external agency or consultancy*Base: 31 Web site decision-makers who manage entirely in-house
Averagepercentageincrease
Metrics
FORReSTeRS DIRecTORy OF OnLIne TeSTIng VenDORS
Marketers considering testing opportunities have a daunting task o understanding the testing
methodologies available, justiying development o testing programs, and selecting vendor partnersto supply the technology.
o provide insight into vendors oering technology solutions or online testing, we created a
directory o solution providers (see Figure 6). Tis is not a comprehensive list o vendors, nor is
it a ormal evaluation all inormation is sel-reported. Some vendors were not included, while
others oering technologies and services in this area, such as platorm providers, agencies, and
consultancies are not included here.4
8/8/2019 002343.Online Testing Vendor Landscape
9/12
2009, Foeste resech, Ic. repoctio PohiiteOctoe 20, 2009
The Olie Testig Veo Lscpe
Fo Cstoe Itelligece Poessiols
8
Fir 6 Olie Testig Veo mti
Source: Forrester Research, Inc.53637
Methodology Serviceoffering
Fullfactorial
Partialfactorial
Proprietary
Self-service
Managedservice
Fullservice
Maxymiser
Omniture (Test&Target)
Autonomy (Interwoven Optimost)
SiteSpect
SiteTuners
Vertster
Webtrends (Optimize)
Yeartestingfirstoffered
Staffdedicatedtotest
ing
Amadesa
Google Website Optimizer
Averageannualcost
Targetindustries
Conversions,landing pages,
lead generation
Retail,subscription-based
services
Financial services,retail, travel
Financial services,media, retail
Financial services,retail, travel
Agencies, media,retail
Form completions,lead generation,
retail
B2B, B2C
CPG, financialservices, media
YouTube,
Quicken Loans,Obama PresidentialCampaign
Petco, General Mills,New England
Journal of Medicine
ASDA (part of the Wal-Mart group), Santander
Bank, Virgin Media
3M Corp, TDAmeritrade, Real
Networks
Allstate, Avis,Delta Air Lines
Walmart.com,Staples.com,NYTimes.com
CBS Interactive,Facebook, Symantec
The Planet, Gore,Verizon
Microsoft
Marqueeclients
2006
2006
2004
2001
2004
2004
2005
2005
2006
30
Y Y*
Y
Y
Y
Y Y
YY
Y
Y Y
Y Y
Y
YY
Y
Y
Y
Y
Y
Y
Y Y Y
Y
YYY
Y
YYYY Y
Y
Y YY Y
Y
Y
$$
N/A (free)
55 $$$
N/A $$$
N/A $$$
30 $$
15 $$
5 $
28 $$
www.amadesa.com
www.google.com/websiteoptimizer
www.maxymiser.com
www.optimost.com
www.omniture.com
www.sitespect.com
www.sitetuners.com
www.vertster.com
www.webtrends.com/Products/Optimize.aspx
*For AdWords clients.Self Service: Refers to vendors that offer technology and supporting documentation, yet require clients to
develop, manage, and execute tests using their own internal resources.Managed Service: Refers to vendors that aid clients by working collaboratively in activities such as:1) technical support and implementation, 2) testing strategy, and 3) analysis.
Full Service: Refers to vendors that wholly support, design, manage and execute tests on behalf of their clients.Full Service offerings typically include but are not limited to: 1) technical support and implementation, 2)testing strategy, 3) test design, 4) creative development, 5) execution, and 6) analysis.
8/8/2019 002343.Online Testing Vendor Landscape
10/12
2009, Foeste resech, Ic. repoctio Pohiite Octoe 20, 2009
The Olie Testig Veo Lscpe
Fo Cstoe Itelligece Poessiols
9
r E C O m m E n d a T I O n S
ImPLemenT TeSTIng AS PART OF An OPeRATIOnAL PROceSS
Olie testig is citicl copoet o efciet ketig optiiztio soltio. The
coitios o viles possiilities eqie tht ketes levege techolog to
coetl est wht woks olie i tiel e. as sch, testig st ecoe
itegl pt o e-to-e pocess. mketes seekig cotios ipoveet thogh
testig shol:
Bi with a tsti strat. mketes shol evelop iivil test pls tht pck to ovechig testig stteg. Test pls sttegies shol e socilize og
stkeholes to vet gols, epecttios, itios eoe ekig o the st test. a
goo testig pl shol icle: 1) specic hpothesis; 2) gols o the test; 3) sccess
etics. Tal th basis bor oi alorithi. m ketes who ek o testig
eoeosl elieve tht ltivite testig is the ost efciet etho. While mVT oes
epeset the picle o testig sophistictio, ketes shol stt with a/b tests to
ow the el o possiilities eoe eig etils with ltiple viles i pl. This
ppoch eles ketes to ehst low-hgig oppotities with a/b testig eoe
ovig o to oe sophisticte lgoithic optios.
choos tsti aras jdiiosl. all too ote testig eecises ecoe politicl ttleswithi ogiztio. Ogiztios tht stt with visios o ge o optiizig thei
hoe pges ote eet with sttio ile. becse the hoe pge is sch highl
visile ipott piece o olie el estte, eveoe iclig o CEO hs veste iteest i its esig. Whe sttig ot, choose test es tht e less visile t still
iti high-ipct oppotities. Lig pges o olie os ke ecellet povig
gos o testig pogs to illstte sccess eoe tkig o the HiPPOs (highest pi
peoples opiios) thei hoe pges.5
Tst draatiall difrt opts or bst rslts. We ecoe tht ketestke cotol o testig, ct qickl, epeiet o est eslts. Tke isks testig
ticll ieet cocepts the th ce vitios o ies. Wh? big ieeces
i test cocepts will evel oe ot cose setiet th ite vices, which
ote eslt i icoclsive eslts.
Istill tsti as part o a oplt sit optiizatio l. Leig ogiztios viewtestig s st opetig pocee o ew ketig iititive. Fo othe s,
this will eqie cltl chge. Cetive evelopes ee to the hilit to ecogize
tht coses hve gete s i the esig o thei We sites igitl essgig
th the o. yet, s coses pli thei olie itectios, testig povies es to
llow ketes to keep pce with shitig itetios ehvio.
8/8/2019 002343.Online Testing Vendor Landscape
11/12
8/8/2019 002343.Online Testing Vendor Landscape
12/12
Forrester Research, Inc. (Nasdaq: FORR)
is an independent research company
that provides pragmatic and orward-
thinking advice to global leaders in
business and technology. Forrester
works with proessionals in 20 key roles
at major companies providing
proprietary research, customer insight,
consulting, events, and peer-to-peerexecutive programs. For more than 26
years, Forrester has been making IT,
marketing, and technology industry
leaders successul every day. For more
inormation, visit www.orrester.com.
Australia
Brazil
Canada
Denmark
France
Germany
Hong Kong
India
Israel
Japan
Korea
The Netherlands
Switzerland
United Kingdom
United States
Headquarters
Forrester Research, Inc.
400 Technology Square
Cambridge, MA 02139 USA
Tel: +1 617.613.6000
Fax: +1 617.613.5000
Email: [email protected]
Nasdaq symbol: FORR
www.orrester.com
m k i g L e e s S c c e s s l E v e d
For a complete list of worldwide locations,
visit www.forrester.com/about.
Research and Sales Ofces
5363
For inormation on hard-copy or electronic reprints, please contact Client Support
at +1 866.367.7378, +1 617.613.5730, or [email protected].
We oer quantity discounts and special pricing or academic and nonproft institutions.
mailto:[email protected]:[email protected]://www.forrester.com/