Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Tim Burgess Justin Bridegan
Web Analytics Manager Senior Marketing Manager
Air Canada MECLABS
Case Study: Air Canada
The Biggest Optimization Mistake I Ever Made
Presenters Tim Burgess
Web Analytics Manager
Air Canada
Justin Bridegan
Senior Marketing Manager
MECLABS
2
3
Where I Came From…
Agile
Selling primarily to new visitors
Experimental with both software and marketing
Failure was accepted as part of the learning process
Don’t ask about the product quality
4
Testing at Air Canada
Unknown or brand new to most individuals and groups
Measurement is done through old database architecture
Company structure is separated into silos
Web analytics and optimization are positioned in IT
5
Web Analytics Maturity Model
2
1.5
2 2
2
3
Resources
Methodology
Tools Management
Objectives
Scope
Tools
1
Management
1
Objectives
0
1
Scope Resources
1
Methodology
1
Jan 2011 Mar 2012
***This has been stolen from Stephan Hammel
6
7
Before I tell you about the worst test I ever had….
…don’t feel bad for me…
• We are installing an enterprise-level testing tool
• We are gaining ground in the marketing community
• My boss still likes me (as far as I know)
8
5 steps for developing a test within Air Canada
1. Develop a strategy – where is the lowest-hanging fruit for profitability?
2. Use the MarketingExperiments concepts of Friction and Value to develop mockups
3. Find a stakeholder whose interest I will work in, and sell them on the concept
4. Set up and run the test
5. Report to stakeholders
1
2
3
4
5
9
Step 1: Strategy
• Test to improve seat selection
– Air Canada’s largest source of ancillary revenue
Develop a strategy – Where is the lowest-hanging fruit for profitability? 1
10
Use the MarketingExperiments concepts of Friction and
Value to develop mockups
Step 2: Analyze Friction and Value
2
Perceived Cost
BPY=Pc(VF>CF)
BPY=Pc[Cl+Cr](VF[Ap+Ex]>CF[Rc+Ef])
Perceived Value
WHY DO PEOPLE SAY “YES”?
11
The Original: Seat selection
The original variation is good from a navigational point of view, but does nothing to emphasize the value of “Advanced” or “Preferred” seat selection.
12
Variation A
This variation focused on bringing Value to the top of the page through bullets, while maintaining clear navigation.
13
Variation B
The goal of Variation B was to bring navigation further above the fold, while still featuring the bullet points.
14
Variation C
Answered the question, “What if they really don’t know the difference between Preferred and Advanced seat selection?” by promoting description in text form above navigation.
15
Find a stakeholder whose interest I will work in, and sell them on the concept
Step 3: Find a stakeholder
3
16
Stakeholder – Revenue Management
The group that manages pricing, loads and network and also responsible for Seat Selection revenue. Loved the idea!
17
Testing
Step 4: Set up and run the test
4 Control
Variable A Variable B Variable C
18
Step 5: Report to stakeholders
5 Report on findings
We ran the test for one week to 95% significance, with a follow-up test for three weeks, that showed that variation C was generating 48.78% more revenue than the original.
Control Variable C
19
We have a winner…
So we celebrated!
Variable C
20
One small hitch… The numbers in the months following the experiment did not reflect the
experiment results
My Mistake: By communicating the results the way I did, I led the stakeholders to believe that we would see those numbers once the test went live.
• Revenue Management did not understand how this could happen
• My boss was disappointed
• I was confused
21
So, what happened?
• Visitors on Aircanada.com are often repeat visitors
• There are many points
in the process where visitors can purchase seat selection (see the email below)
• We cannibalized on
other channels
22
Analytics data
Preferred Seat Revenue/flight order
0
1
2
3
4
5
6
7
Jul 1
, 2011
Jul 6
, 2011
Jul 1
1, 2
011
Jul 1
6, 2
011
Jul 2
1, 2
011
Jul 2
6, 2
011
Jul 3
1, 2
011
Aug
5, 2
011
Aug
10,
201
1
Aug
15,
201
1
Aug
20,
201
1
Aug
25,
201
1
Aug
30,
201
1
Advanced Seat Selection
Revenue/Order
00.5
11.5
22.5
33.5
4
Jul 1
, 201
1
Jul 6
, 201
1
Jul 1
1, 2
011
Jul 1
6, 2
011
Jul 2
1, 2
011
Jul 2
6, 2
011
Jul 3
1, 2
011
Aug 5
, 201
1
Aug 1
0, 2
011
Aug 1
5, 2
011
Aug 2
0, 2
011
Aug 2
5, 2
011
Aug 3
0, 2
011
Even though analytics data showed a change in the amount of revenue for Preferred Seat selection, Revenue Management was used to looking at its own data, which is not slice and trend-able like analytics data. Using conservative numbers the actual impact of the test was an approximate increase of 5% on Preferred Seat selection revenue.
23
The irony…
…is that this is the most successful test I have run to date, in terms of profitability.
24
Key takeaways
Remember the big picture
Manage expectations
Predict scenarios, not results
Keep testing
Remember…
The way in which hypotheses and results are presented is pivotal to building support and making optimization a long-term strategy.
25
Thank you!
Questions?