How to identify and use the right metrics to delight your customers, engage your audience, and optimize your business.

Building your Company’s Data DNA

Building Your Company’s Data DNA 2

Table of Contents: Section 1: Introducing Data DNASection 2: Finding your Company Metrics Understanding and Measuring Goals

Metrics by Business Type

Section 3: How to Take Action on your Metric Quantitative Data (with contributions from KISSmetrics)

Qualitative Data (with contributions from Qualaroo)

Section 4: Building Data into the Optimization Flywheel Six Steps of Optimization

Craft your Experiment Hypothesis

Assembling your Data DNA

Section 1: Introduction

Building Your Company’s Data DNA 4

There’s no question: data is here to stay. The promise

of data, analytics, and web-based tools make the task

of building a business online accessible and afford-

able. Access has created a new challenge: volume.

Leveraging the entirety of a business’ data on an

ongoing basis is tremendously valuable, but requires

equally tremendous resources, beyond the reach or

scope of many businesses’ technology investments.

Given these challenges, how can you make intelligent

decisions, informed by data, to grow faster?

You can incorporate data to inform decisions you

make about growing your presence online. At each

juncture, you have choices to make—how should I

describe my product to my customers? How can I col-

lect more email addresses to grow my audience? How

should I roll out a new feature and collect feedback?

A clear understanding of metrics and the right type

of data at every level of an organization are essential

for focus. Data is also key to uncovering customer

insights and running a more effective optimization

strategy, but should be properly aligned from the

top-down in order to achieve success.

“The best data-driven companies don’t just passively store and analyze data, they actively generate actionable data by running experiments. The secret to getting value from data is testing, and if you’re looking to grow your online business, implementing well-executed, consistent A/B testing is a necessity.”

—Wyatt Jenkins, Shutterstock*

Access to Data Increases Expectations

In a recent survey, 78% of all marketers

report feeling pressure to become more

data-driven.* The challenge is to choose

which measurements of online perfor-

mance are essential to a data-driven

strategy.

*Teradata Data-Driven Marketing Survey 2013

*”A/B Testing and the Benefits of an Experimentation Culture,” Harvard Business Review.

Building Your Company’s Data DNA 5

How do I know if this guide is for me?

In this guide, we’ll outline an approach to transform-

ing data into action at every level of your organiza-

tion.

You’ll benefit from this guide if:

• You’re already capturing and reporting on data regularly internally

• You have a team or company culture that is open to experi-mentation

• You’re looking for methods to turn your company data into action

• You’re looking to get more meaningful ‘wins’ from your optimization strategy

Here’s what you’ll learn:

• Why becoming data-driven as an organization should start with your optimization programs

• How to align your organization around a failproof guiding light metric

• How to prioritize your optimization strategy against your shared company goals

• How to move past random, ad-hoc A/B tests to an optimi-zation process that produces wins (and if not winning tests, valuable insights derived from a hypothesis)

Being Data-Driven is Tied to Performance

Companies that rate themselves sub-

stantially ahead of their peers in their

use of data are three times more likely to

rate themselves as substantially ahead

in financial performance, according to

findings from the Economist Intelligence

Unit.*

*Tableau Software & Economist Intelligence Report:

“Fostering a Data-Driven Culture.”

“Low hanging fruit tastes great, but the rest is worth the work, too.” —Patrick McKenzie, Kalzemeus Software

Building Your Company’s Data DNA 6

In this guide, we’ll cover many different types of

data. The important point to remember is that

the ideal of being ‘Data-Driven’ is actually to be

‘Data-Informed.’ Data points are an important

set of inputs, but they are not a replacement for

human intuition and judgement. This guide will

provide a list of inputs that are strong positive

indicators for how to improve your company goals

and optimization program. What data points

cannot provide, however, is input on what your

business metrics should be, or what initiatives you

want to prioritize.

What is A/B Testing?

A/B testing is a simple way to test

changes to your page against the cur-

rent design and determine which ones

produce positive results. It is a method to

validate that any new design or change to

an element on your web pages or mo-

bile apps is improving your conversion

rate. It is the assumption that you have

never reached the ‘best’ version of your

website, mobile app, or product, and that

the best method of learning about your

customers is through iterative testing and

experimentation.

What is Optimization?

Optimization begins with broad, sweep-

ing changes and tests that encompass all

of your website traffic, but can become

more granular, by running A/B tests to

understand how changes affect different

types of visitors, and then adapting site

content to eventually deliver a more per-

sonalized experience for each individual.

Section 2: Finding your Company Metrics

Building Your Company’s Data DNA 8

You’ve probably wondered at some point: “What’s

the fastest way for me to create business impact?”

We could frame this question with two others:

1. “What’s the fastest way to make a measurable business impact?”

2. “How do I define success?”

To reach the goal of becoming a data-driven organization

or a data-informed individual, we recommend striving

for one metric that defines the goals of your business.

When you choose your ‘guiding light’ metric, ask

these questions to determine whether it will stand the

test of time, and generate strong optimization results:

• Is it quantifiable? Your metric should be measurable and understandable at-a-glance.

• Is it visible and clearly communicated? Your metric should serve as a reference point in conversations.

• Is it understandable? It should require very little explanation.

• Is it comparative? You should be able to compare the metric to benchmarks, like your competitors’ performance or other points in time.

• Is it a rate or ratio? Showing a rate of change is more illustrative than an absolute number.

• Does it change your behavior? If the metric you choose doesn’t enact change, you need a different one.*

• Is it attributed to revenue? For many people in organiza-tions, this measurement is what communicates above all else. If you have difficulty attributing your efforts to value for the business, you may struggle proving ROI.

Many analytics experts have discussed

the value of identifying a single metric,

which should be carefully chosen and

then rigorously tested. We recommend

Lean Analytics by Ben Yoskovitz and

Alistair Croll.

Use these questions to test and strengthen your metric statement.

Before you take any actions to improve

your metric, ensure that you have tested

it internally to ensure that it will with-

stand scrutiny and testing over time.

*Questions 3-6 reference Ben Yoskovitz and Alistair Croll, Author of Lean Analytics

Building Your Company’s Data DNA 9

Having a singular goal is helpful not only for orga-

nizational focus, but it will be invaluable for more

specific tasks, like your optimization efforts. In order

to run more intentional, ROI-positive A/B tests and

experiments, you need a guiding metric that you can

brainstorm around and prioritize against. We’ll cover

the data inputs that can inform these types of tests in

Section 3.

“What gets measured, gets managed.” —Peter Drucker, Business Management Thought Leader

Building Your Company’s Data DNA 10

Understanding and Measuring Your Goals

What do you do on a day-to- day basis to grow your business?

Use this framework as a starting point for determining what your priority should be for your business, and by association, your online marketing and other customer touch-

points. What parts of your website directly serve the metric you are working to improve? Think of these areas as prime candidates for optimization.

What does this activity help you accomplish?

What is the priority for each of these activities?

What metrics do you currently report on on a regular basis?

If you were to pick one activity and corresponding goal to work on for the next 1-3 months, what would it be?

Which of the metrics from Ques-tion #4 help you to understand the progress you are making on the answer to Question #5?

Which of the metrics from Ques-tion #4 could you live without?

Essential

Important

Nice-to-have

1.

2.

3.

4.

5.

6.

7.

Building Your Company’s Data DNA 11

If you: You might use your website to: And could be measured as:

Sell products or services

(E-Commerce, Platforms)

Collect orders for your product

Distribute and engage your content

Capture, engage, and retain users

Time on siteArticles viewed

Votes or comments generatedSocial traffic

Signed up usersPaying customers

Up-sold customersChurned customers

Number of orders over timeAverage order valueRepeat customers Conversion Rate

Publish content(Media, user-generated content

like Reddit and Wikipedia)

Develop a web-based tool

(SaaS or Mobile app company)

Metrics by Business Type

Good candidates for top metric statements:

“I measure conversion rates from free trials to customers because my most important metric is acquiring new customers.”

“I measure average donation value, because my most important metric is dollars raised during our fundraising campaigns.”

“I measure conversion rate on my homep-age, because that is where the majority of my new leads are acquired.”

Not as strong metric statements:

“I measure pageviews to product pages on my website.”

“I measure social shares.”

“I measure the amount of traffic to my pricing page.”

These statements reflect strong metric statements sup-

ported by a clear rationale. Strive to consolidate and

strengthen your overarching metrics so that everyone can

understand and tailor their priorities to match them.

These metrics are not supported by a strong rationale, and

are more likely to be vanity metrics. They are not enough (on

their own) to justify prioritizing for optimization.

Section 3: How to take Action to Improve your Metric

Building Your Company’s Data DNA 13

You may be wondering: how does eliminating metrics

help me use data? The answer lies in the touchpoints

you have with your customers. Once you have identi-

fied your overarching goal, you will be in a position to

identify touchpoints that can be improved in service

of the corresponding metric.

In order to take action that will improve the outcome

of your metric, it is important to effectively identify

high-value areas of your website, product, or other

customer touchpoint. We will explore the best ways

to gather information about these high-value areas

later in this section.

Each business metric can be broken

down to apply to different levels

and functions within your orga-

nization. A team should plan

to adopt a secondary metric

that supports your top-level

company goal. These levels

may resemble the diagram

to the right:

At the most granular level,

your website and product

goals can be supported by

running experiments in an

attempt to improve the con-

versions that increase perfor-

mance, which in turn affect your

highest-level goals.

Hypothesi ze Tests to Increase AOV

Building Your Company’s Data DNA 14

How do you measure the success of these facets of

your online business? Your efforts to present your

business online and capture interest, engagement, or

revenue require conversions.

Take stock of the conversions that support your

top-priority metric. These areas are prime candidates

for optimization, where you’ll run experiments and

A/B tests in an effort to improve conversions.

Getting started with optimization is not the most

difficult step to success; the challenge will present

itself in the struggle to keep continuously A/B test-

ing to find improvements, uncovering winning tests

consistently.

In the following section, we will explore how to use

a combined qualitative and quantitative approach to

build your Data DNA into your website and app per-

formance. Then, we’ll discuss how to funnel these

data points into experiments that uncover improve-

ments and build your data-driven approach to opti-

mization in Section 4.

Using Quantitative Data to Determine which Areas of your Site to Test

Your website analytics are an indispensable tool for

determining the strengths and weaknesses of your

website. Leverage your website data in order to effec-

tively prioritize and brainstorm the best experiments

to run on your website.

What is a conversion? A desired action

taking place on your at a customer

touchpoint—a click, or registration, or

product action, for instance.

Your conversion rate is the current pace

at which website visitors take action at a

given touchpoint.

Conversion rate optimization and A/B testing describe the practice of experi-

menting with elements to try to improve

a conversion rate.

Mapping Your Company’s Data DNA 15

KISSmetrics Contributed Section:

Before you can find ways to accelerate your acquisition and growth, you’ll need to start tracking your core funnel.

What’s your Core Funnel?

It’s slightly different for each business but it measures how many people move through the major steps of your acquisition. Don’t worry about tracking every little click, page, or action that people take to become a customer. Focus on the core steps.

Let’s look at a few examples.

SaaS (subscription as a service) businesses drive traffic, convert that traffic into free trials, get those free trials to use their prod-uct, and convince people to purchase an ongoing subscription to the software. So a SaaS funnel is built from these steps:

Using Funnels to Find Big Wins and Accelerate Growth

SaaS Funnel

1. Visited site

2. Signed up for free trial

3. Used product

4. Purchased subscription

Mapping Your Company’s Data DNA 16

An ecommerce funnel looks pretty similar:

For whatever business model you have, break your acquisition

down into 4-5 core steps that everyone goes through.

Tracking your Funnel

Ideally, you’ll have a customer analytics tool that can track people as they move through the different steps of your funnel. When you get to the point where you’re doing a lot of optimiza-tions on your funnel, have plenty of traffic, and acquire custom-ers consistently, start looking at customer analytics tools that will help you track the entire funnel.

Even if you’re not there yet, you still want to get a general sense for how your funnel performs. Google Analytics and some inter-nal tracking will get you pretty far.

E-Commerce Funnel

1. Visited site

2. Visited product detail page

3. Added product to cart

4. Entered checkout

5. Finished checkout

Mapping Your Company’s Data DNA 17

Ecommerce companies have it easy by setting up ecommerce tracking in Google Analytics. This tells you how many people purchase and their conversion rates. You’ll also want to config-ure custom events to see how many times products get added to a cart or visitors enter your checkout.

For SaaS, set up Google Analytics and trigger a goal when people start a trial. You’ll also want to build out some internal tracking that tells you how many new accounts you acquired, how many of them started using the product, and how many of them purchased. Add this data to your internal customer data-base and query it every week or so to see how you’re doing.

Don’t worry about having perfect tracking when you get start-ed. Your main goal is to have a general idea for how your funnel performs.

Optimizing your Funnel

As soon as someone’s ready to start optimizing a funnel, they usually pick a random element on a random step. It might be a button color on a call to action, a snippet of copy, an image, or a layout that just feels “off”. Then they’ll launch an A/B test if they have enough traffic or maybe they’ll just launch the change and hope for the best.

After a few rounds of this, you’ll find that your conversions are exactly the same as where they are now. They won’t budge an inch.

Why? Because it’s incredibly difficult to find tests that really make a difference. Most online marketing “best practices” give 2%-3% wins. Not only is it incredibly hard to detect small wins like this (you need a ton of data to test on), small wins aren’t useful to a business. They only matter if you can find a lot of them in a short time period. Which, again, requires large amounts of data.

Mapping Your Company’s Data DNA 18

So how do we find bigger wins consistently?

Start with qualitative data on the step of your funnel that you struggle the most with. Qualitative data is feedback from users or customers that isn’t a metric. Surveys, customer interviews, feedback forms, usability studies, and heatmaps are all qualita-tive data.

It’s common to have one step that is a major roadblock to acquisition. Focus on collecting feedback from your customers at that step of the process. Ask them why they don’t want to purchase, why they didn’t use the trial, or why they didn’t finish the checkout.

Asking for targeted feedback on an underperforming step of your funnel will give you plenty of ideas on what to test. There’s never a 100% guarantee of finding a big win but you’ll definitely find them more frequently than if you test random elements across your funnels.

Mapping Your Company’s Data DNA 19

By examining your website analytics, you will be able

to pinpoint valuable areas of your site that are in need

of improvement. If you are successful in running A/B

tests and experiments to improve these key areas of

your website, you’ll be one or more steps closer to

moving that ‘guiding light’ metric, and shifting your

company towards a more data-driven approach to

solving problems.

Website analytics have a tremendous number of ben-

efits for structuring your optimization strategy:

• Quantifiable, high-value traffic segments

• Common goals that sync across your website analytics to your website optimization platform

• Ability to scale and analyze website data across segments, cohorts, and more

Quantifiable analysis of your website is only one part

of the data equation. To complete the picture of how

your visitors are converting and engaging (and more

importantly, why they aren’t), you’ll need to supple-

ment your quantitative data with qualitative insights

to enhance your data DNA.

Using Qualitative Insights to Determine which Areas of your Site to Test

Qualitative, anecdotal feedback is an effective way to

incorporate data into your business at any stage. This

data is essential because it comes directly from your

visitors and users.

Mapping Your Company’s Data DNA 20

This information is essential to determining why

certain events occur on your website. When quantita-

tive data provides insight into what happens (or isn’t

happening), these qualitative data points can illus-

trate why. Why did that customer buy? What brought

them to your website in the first place? What do your

visitors expect from your product or offering?

Qualitative insights come in many forms, including,

but not limited to:

• User testing: For a fee, a person unfamiliar with your website will com-plete a task on your website while providing their thoughts and feedback. There is a bias inherent in this qualitative data, since the person knows they are being tested.

• On-page surveys: Use these surveys to collect ‘in the moment’ feedback from visitors to a website. The more questions you ask, however, the lower your completion rate is likely to be.

• Heat mapping: These tools will show which areas of a given web page attract the most ‘heat,’ or attention, from clicks and scroll-ing on your site.

• Long-form surveys: Often delivered via email, these surveys can be used to collect in-depth feedback on your company’s positioning, perception from customers, and some of their most valued offerings or products from your business.

“A solid test hypothesis is an informed solution to a real prob-lem – not an arbitrary guess. The more research and data you have to base your hypothesis on, the better it will be.”

—Michael Aargaard, Conversion Rate Optimization Expert

Mapping Your Company’s Data DNA 21

• Feedback from cancellations, returns and complaints: Take your customers’ constructive feedback, and decide what visible changes to your website could help to alleviate pain points and frustrations they might be having.

• Live chat feedback: Uncovering pain points and visitor intent will be key to generating winning experiment ideas.

• Customer feedback: Anecdotal feedback from engagements with customers on calls, at events, and in other formats.

You can leverage these tools (many of which have

affordable and easy-to-use plans for businesses)

to collect feedback directly from the visitors to your

website or customers. You must hone in on motivat-

ing factors, however, in order to collect meaningful

feedback.

Building Your Company’s Data DNA 22

To ask questions of your website visitors and collect more data points about what you should optimize, practice asking the right questions.

Here are two types of questions you should ask:

1. Intent - What did your website visitors come to your website looking to find, or do?

2. Frustration - Could you find what you were look-ing for? What was broken?

Consider the types of questions you could ask to understand key parts of your website funnel:

On your homepage:• What did you come to our website today hoping to accom-

plish?

Before a key conversion:• Do you have any additional questions about this product

or service? Determine what information your visitors are looking for before they purchase.

• Is there anything stopping you from completing this order? Perhaps the visitor needs more information to make their decision. Maybe an unexpected bug with your checkout flow is preventing them from converting.

After a key conversion:• What made you purchase/sign up/donate to us today?

On your content site:• What topics would you like to see us cover more? You could

make this multiple choice or leave as a free response

• Would you like to see more recommended content on this topic? Determine the best way to design recommended content modules for your content.

AAA: Ask your Audience Anything

A few tips to get even better data from your qualitative research:

Be polite: You’re human - so are your

website users. Ask the questions in an

approachable, visitor-friendly manner.

Phrase questions with a “What do you

think of your website? Kindly tell us:”

approach.

Ask for honesty: Your website visitors

are also nice people. It’s sometimes dif-

ficult to provide constructive feedback.

“We’d appreciate your brutally honest

feedback as we work on improving our

website experience for you.”

Keep questions open: Try to avoid ask-

ing questions like, “What was frustrating

about your experience today?” or, “What

do you think of our [product or service]?”

These questions predicate a certain type

of response, and will bias your visitors’

responses.

Ask visitors to self-select: You can

also learn more about your audience by

asking them about their interests, their

business type, or products and services

that they like to use. This will help you to

segment your web traffic and experiment

with running more targeted tests.

Building Your Company’s Data DNA 23

Now, it’s your turn: What are the steps (or clicks) in your website funnel, and what questions would you ask at each step to reduce friction and improve your

visitors’ experience?

?

?

?

?

Mapping Your Company’s Data DNA 24

What do you think is the most common mis-conception about Conversion Rate Optimi-zation and A/B testing?

The number one misconception is that A/B testing is simply about running a test here and there and hoping for improved results. Without an optimization process that focuses on continual improvement A/B testing often fails to live up to it’s promise. Without organizational rigor to make A/B testing a priority, companies give up failing to see early wins, which ultimately costs them the long-term gains that come from a systematic approach to optimization.

Do you think that most companies online are effectively communicating with their customers? Why or why not?

Most companies lack a process for regularly collecting and then taking action on user feedback. Users provide feedback in numerous ways, from bouncing off of web pages, to taking surveys, leaving reviews, filling out customer support tickets and posting on social media. With all of this feedback com-ing in, you’d expect companies to be constantly processing it and using it to improve the visitor experience, and ultimately, their business. But more often, feedback is triaged to manage customer complaints rather than used for true learning and

Online Success and Optimization: Why Quantitative Data isn’t Enough

Q:

Q:

A:

A:

Interview with Startup Marketer and Qualaroo CEO Sean Ellis

Mapping Your Company’s Data DNA 25

business improvement. The most successful companies have a process for collecting, parsing and using customer feedback to improve their business.

What is the best way to get actionable, qualitative data from a website survey?

The single best way to get actionable qualitative data from a website survey is to use them to try to understand specific user behavior. For example, using a website survey on pages that have high bounce rates, or pages within your conversion funnel have high drop-off rates, can give you insights right from the customer that help you understand why they’re leaving. When you understand why a behavior is happening, it’s much easier to take action on the feedback and try to change the behavior.

Do you have any tips on how to choose which feedback should be incorporated into your testing pipeline?

Feedback from qualified people is the most important. The people that are potential customers who aren’t converting are the ones you want to focus on. Ignore the people who aren’t qualified or interested in what you’re offering from the begin-ning. For example asking people who converted a question such as “What almost stopped you from signing up?” or “What made you decide to sign up?” helps you understand the needs of qualified visitors. This qualified feedback will help you sort through the data from exit surveys that include both qualified and unqualified responses.

What would you tell someone who is look-ing for help creating strong hypotheses for their tests?

Q:A:

Q:

A:

Q:

Mapping Your Company’s Data DNA 26

There are two ways to create a hypothesis. In the first case you can look at the data and then spend hours or days with your team trying to interpret what the data means and what you should test next. You can create hypotheses from your interpre-tation and from your team’s opinions. Or you can ask visitors to that page what the actual problem that they’re encountering is. No need to interpret data, just ask visitors and get immediate feedback. I believe asking visitors what issues they’re encoun-tering on your site, conducting user research, is the best way to formulate strong hypotheses that make for valuable tests.

How do you think companies should ap-proach staying focused when it comes to optimization? How do you measure the progress of your CRO program?

Having the organizational rigor to stay focused on conversion optimization is the hardest part of this process. A/B testing is not a one-off project—rather it is a continuous process of improve-ment that needs to be in motion at all times. Econsultancy reports that 87% of companies doing A/B testing run between 1 and 5 tests each month. The best companies run many times that number.

Companies need to commit to A/B testing as a core part of their digital marketing program and invest in it accordingly. Stick to the process and eventually it will become an addictive habit, with the organization constantly trying to outdo its previous test. Systematic, ongoing A/B testing is going from a compet-itive advantage to a competitive necessity. It’s a key require-ment for online success today.

A:

Q:

A:

Building Your Company’s Data DNA 27

At Clearlink, the team asked the question: “How can

we see inside the minds of our customers and be sure

we are offering what they need?” Despite following

testing best practices, the team has frequently en-

countered unusual or contradictory A/B test results.

Utilizing qualitative data gives them insights into

these results and helps them better understand and

explain when met with this challenge.

They hypothesized that using a qualitative insights

tool in conjunction with their optimization software

would enable them to better serve their customer

needs through well-researched and well-executed

tests, resulting in the ability to integrate and better

understand tests’ success and failure, and ultimately

leading the to understand our market better.

While many times quantitative metrics might be interesting, they can-

not tell the entire story when it comes to the customer’s path. In short,

we know the ‘what’, but not the ‘why.’ For these issues we have to turn

to qualitative research and customer insights to make sense of the

customer needs and dive in deeper. This is how we innovate; this is

how we iterate; this is how we make sense of the larger picture.

—Rachel Johnson, Consumer Insights Research Manager, Clearlink

How Clearlink Built Qualitative Insights into their Data DNA Content and conversion services provider, Clearlink,

uses survey software* to foster its data-driven ap-

proach to engaging with its audience and collecting

qualitative feedback from its online audience.

*Clearlink implemented Qualaroo survey software for the purpose of these experiments.

Building Your Company’s Data DNA 28

By using on-page surveys to offer time-sensitive

promotions, the team was better able to serve more

relevant promotions to their website visitors in a

more time-effective manner. This led to overall

higher conversion rates for their sites:

• Stand-Alone vs. Bundle Pricing: The team learned from

survey results that stand-alone pricing was what Clearlink’s customer was looking for. After testing stand-alone pricing messaging against bundled pricing options, stand-alone pricing won with a conversion rate improvement of +5.4%, contributing to a testing win of $41,500 in annualized revenue impact.

• Lead Capture: The team hypothesized that by using the survey tool, they would be able to send the customer directly to their conversion funnel (the sales floor) by engaging them more quickly on the initial landing page. In addition to the phone number, they added promotional messaging (similar to display ads) to the messages to further drive conversion. Using the survey tool bypassed the normal lead time required for a custom overlay to be designed, developed, and pushed to live site, significantly increasing Marketing’s agility in responding to changing customer tastes, preferences, and demands.

Using quantitative and qualitative research meth-

ods together led to several “wow” moments for the

Clearlink team. By combining the two data streams,

qualitative and quantitative, they created an en-

hanced testing cycle, making the most of testing

wins, successes, and failures. Qualitative findings

sped up the testing cycle, and the team gained im-

pactful insights more often. Clearlink’s prevailing

sentiment is that “The more often you fail, the more

often you’ll win; so speed up the frequency at which

you fail.”

Clearlink’s mantra is: “The more often

you fail, the more often you’ll win; so

speed up the frequency at which you

fail.”

Building Your Company’s Data DNA 29

Integrating quantitative and qualitative data into the

optimization cycle creates a more effective testing

program for attaining website goals. Although tak-

ing the time to add consumer research and the cus-

tomer’s point of view can seem like the longer path,

for teams like Clearlink it ultimately improved the

team’s feedback loops and led to insights that previ-

ously might have gone undiscovered.

At this point, you have used data to inform your

strategy from your top-level business goals down to

your website and product-level goals. Now that you

have a sense of what could stand to be improved, let’s

turn to brainstorming and prioritizing your new ex-

periment ideas to complete your Data DNA sequence.

Section 4: Build Data into the Optimization Flywheel

Building Your Company’s Data DNA 31

You’re well on your way to running high-impact,

ROI-positive experiments to improve your customer

touchpoints. The next step is to distill the data you’ve

collected and begin to take action. In this process,

you will funnel your quantitative and qualitative data

points into concepts for A/B tests that can be run as

experiments on your website to improve key conver-

sions and support your business goals. We call this

six-step framework the “Optimization Flywheel.”

In order to see success with this approach, you

should:

• ●Build a rigorous practice of consistently collecting quantita-tive and qualitative data about your website funnels.

• Maintain visibility of your organization’s overarching metric. If your focus as a company changes, your website goals and individual experiment objectives should change accordingly.

• Maintain a sense of experimentation as broadly as possible across your company. Any visible changes to your website help to understand what works best for your customers and visitors. The pursuit of this understanding should support the goals of the company as well, as discussed in Section 2.

Building a Data-Driven Optimization Strategy 32

What is the Optimization Flywheel?

Optimization is a continuous cycle. In order to ensure that you

are responsive to the intent and frustrations experienced by your

audience, you can leverage optimization to run experiments and

discover improvements to your website.

There are six steps of the Optimization Flywheel:

Building Your Company’s Data DNA 33

Step 1: Define Goals

As defined in Section 2, it is important that you begin

your optimization process clearly aligns to your busi-

ness goals. A clear understanding of the metrics you

are optimizing for will help with prioritization and

enables continual iteration and learning from exper-

iments.

Step 2: Determine Optimization Points

Identify a step in your funnel that is a prime candi-

date for optimization. It might be your homepage call

to action (CTA), your campaign landing pages, your

checkout flow, or your recommended content.

Make sure to choose an area for optimization that has

a direct correlation to your business goals. The quan-

titative funnel analysis in Section 3 will provide con-

text for which areas of your website can be improved.

Will optimizing this step of your website experience

create a measurable change to the metric you identi-

fied in Section 2?

Step 3: Hypothesize Improvements

A strong hypothesis about how your experiment will

perform is core to running a winning A/B test. Your

qualitative data collected in Section 3 will provide

clear indicators of what should be tested to better

match your visitors’ intent and solve their frustra-

tions with your product or website. We recommend

you take the time to collaborate on hypotheses with

*Reference “Craft your own Experiment Hypothesis” for more detailed instructions on hypothesis creation.

Building Your Company’s Data DNA 34

your team, and properly document them along with

the data that you used to inform them.

A hypothesis is the following type of statement:

Hypothesize what types of changes to your website

would produce a positive change in conversions.

Step 4: Create Variations

When setting up an A/B or multivariate test, you

must develop a variation of your website feature that

you would like to test against the current version.

Use your hypothesis and your understanding of your

analytics and qualitative data to isolate a variable that

can be changed. The fewer the variables, the more

straightforward your test will be. Changing multiple

variables at once is possible with multivariate testing,

but complicates the test and confounds the hypothe-

sis, making it difficult to simply prove or disprove.

If ________________________ , then _________________ will occur.*

[I make this change to my website]

[the following outcome]

“It’s about coming up with your hypothesis first and then develop-ing a test from there. Rather than saying, ‘I want a test button cre-ated,’ or ‘I want to try two different layouts,’ you create a hypothe-sis first, for example, ‘I think green buttons attract more attention and therefore will have a higher clickthrough rate.”

—Michael Burk, Senior Online Product Manager, Electronic Arts

Building Your Company’s Data DNA 35

Once you have isolated your variable, create varia-

tions of your website based on the modification of

that variable.

Step 5: Run Tests

Prepare an experiment using your A/B testing soft-

ware of choice. For more detailed instructions on how

to set up and run an experiment using Optimizely,

visit our Knowledge Base.

Step 6: Measure the Impact

Allow your experiment to run until it has reached a

statistically significant result. Did your variation win,

lose, or draw even with the current variation?

If the test is a win, congratulations! Your hypothesis

was correct, and you can continue to build upon that

test by applying the data-informed learning to other

areas of your site, or testing another more advanced

hypothesis.

At this point, make sure to extrapolate the value of

your improvement to the website across all of your

traffic over an extended period of time. What will

your lift translate to in sign-ups, orders, et cetera

over the course of a year?

If your test was a draw or your variation lost, investi-

gate why that might be. It could be the case that your

hypothesis needed additional research, or that you

didn’t account for a behavior or event that skewed

your test results. In the event of a losing test or a

Building Your Company’s Data DNA 36

draw, it is still possible to gain additional insights

that you had not anticipated through further analysis

and discussion with your testing team. At this point,

discuss what the data couldn’t account for as you

planned the experiment. What would you do differ-

ently in your next hypothesis and experiment?

Putting it all Together

Here’s an example of Data DNA in action:

• Take an e-commerce company that wants to improve the value of each customer they acquire. This is the top-level metric that they have identified is an excellent indicator of the success of their business.

• To improve this metric, the team uses website analytics to identify the top traffic pages for a high-value customer segment: returning traffic. They identify conversion points for returning traffic that differ from other traffic types.

• To collect qualitative data, they survey their customers to understand why they may not be converting at this point.

• Based on this data, the team hypothesizes a change that could help to improve the conversion rate for these visitors. They run an experiment and to determine whether their hypothesis was correct.

Mapping Your Company’s Data DNA 37

What does a winning test hypothesis look like? It’s a represen-tation of your Data DNA on a hyper-focused level. Your qualita-tive and quantitative data collection will help you formulate a strong, testable prediction.

Hypotheses are statements, not open-ended questions. They address a question with a proposed solution. Crafting a hypoth-esis to address an open question or problem on your website enforces a well-rationalized, thoughtful proposal for how to address that problem.

To take your hypothesis even further, consider what you would learn if your prediction was proven correct or incorrect in an experiment. What would you learn in each scenario?

Craft your own Experiment Hypothesis

“IF _____________ , THEN ___________ DUE TO ____________.”[Variable] [Result] [Rationale]

The Variable: A website element that can be modified, added, or taken away to pro-duce a desired outcome.

Result: The predicted outcome. (More email sign-ups, clicks on a call to ac-tion, or another type of behavior.)

Rationale: Demonstrate that you have informed your hy-pothesis with research: what do you know about your visitors from your qualitative and quantita-tive research that indi-cates your hypothesis is correct?

Building Your Company’s Data DNA 38

Assembling your Data DNA Now, it’s time for you to take action:

• Assess your top metrics with your team and company at large.

• Take stock of your web analytics, and bring together your sources of qualitative data from your customers.

• Most importantly, take action by applying an experimental mentality to your online presence with an intentional, mea-surable approach to continuously optimizing interactions with your audience.

By incorporating data into your metrics and goals at

every level of your business, you’ll move your

organization towards a data-driven process for

making decisions and instrumenting change. More

importantly, your data will become actionable at

the customer touchpoint level, where you’ll better

engage and convert your audience into valuable cus-

tomers, repeat visitors, and users.

Building Your Company’s Data DNA 39

ABOUT THIS GUIDE Building your Company’s Data DNA Written By:Shana RusonisContent Marketing Specialist, Optimizely@srusonis

Designed By:Jon SaquingCommunication Designer, Optimizely@JSaq

Thank you to: Ural Cebeci, KISSmetrics, Sean Ellis and Morgan Brown.

SOURCES• “Teradata Data-Driven Marketing Survey,” Teradata.

• “Fostering a Data-Driven Culture,” Tableau Software &

Economist Intelligence Report.

• “A/B Testing and the Benefits of an Experimentation

Culture,” Harvard Business Review.

• Lean Analytics by Alistair Croll and Benjamin Yoskovitz.

• For more detail from Hiten Shah, Co-Founder of KISS-

metrics: “Increase your Testing Success by Combining

Quantitative and Qualitative Data,” OptiCon 2014.

ABOUT OPTIMIZELY

Optimizely is the world’s leading

optimization platform, providing

A/B testing, multivariate testing,

and personalization for websites

and iOS applications. The platform’s

ease of use empowers organizations

to conceive of and run experiments

that help them make better data-

driven decisions. With targeting

and segmentation using powerful

real-time data, Optimizely meets the

diverse needs of any business looking

to deliver unique experiences to their

visitors.

Amsterdam OfficeNes 76

1012 KE Amsterdam

The Netherlands

San Francisco Office631 Howard Street, Suite 100

San Francisco, CA 94105

OPTIMIZELY.COM/DEMO

To learn more about Optimizely, schedule a live demo today at

To learn more about best practices for people, process, and technology for a

winning optimization strategy, download a copy of our Roadmap to Building a

Data-Driven Optimization Team.

Optimization can be instrumental in improving the performance of your paid

search funnels. Learn how to maximize your marketing ROI by reading

Optimizing your Conversion Engine: Search Engine Marketing.