Data Science at Atlassian: The transition towards a data-driven organisation

Data Science at Atlassian The transition towards a data-driven

organisation

Dr Ilias Flaounas Capital Markets CRC, Sydney

13 May 2015

Atlassian - Trivia• Atlassian is an Australian enterprise software company

specialised in products that support project management and collaboration.

• ~12 years old

• Latest valuation at $US3.3 billion in 2014, (x8 since 2010).

• >1.5K employees

• Headquarters in Sydney

• Best place to work in Australia (The Australian Financial Review, 2014)

• ~43,000 enterprise customers

• Tens of thousands more own $10 starter and free community licenses

JIRA

JIRA

Key Requirements:!

• Support from upper management!

• Some minimum critical mass of customers!

• Serious investmentin infrastructure, people,time

Product Growth Team• Team was formed ~2 years ago

• Main mission: growth!

• Discover value in heaps of behavioural data!

• Champion data-driven decision making

Analytics Infrastructure (v.2)

~3-4TB overall (highly compressed)!

800GB per day produced (uncompressed)

We analyse events, i.e., user actionsTimestamp Instance User Event Attributes

1356958862 mySuperCompany John Login1356968863 mySuperCompany Anne Login1356978862 mySuperCompany John dashboard view1356988862 mySuperCompany Anne dashboard view1357884442 anotherCompany John view issue {issueID = 1234}1357956862 mySuperCompany John create issue {issueID = 1000}1357957862 mySuperCompany Anne view issue {issueID = 1000}1357958862 mySuperCompany Anne comment issue {issueID = 1000}1357959862 mySuperCompany John create issue {issueID = 1001}

… … … … …

The Challenges• Build infrastructure!

- break silos!- replace existing “solutions”!- add instrumentation!- improve data quality!- deal with catch-22…

The Challenges

• Educate!- “analytics is counting”!- help people ask the

correct questions

• Build infrastructure!- break silos!- replace existing “solutions”!- add instrumentation!- improve data quality!- deal with catch-22…

JIRA Agile!Which Agile report is used the most?

However, counting is hard…

Report 1 Report 2 Report 3 Report 4 Report 5 Report 6 Report 7 Report 8 Report 9

No usage

No usage Low usage

No usage Low usage Medium usage High usage

… and beyond counting…

Predicting conversionsHow many variables can you measure?

Top customersnone, n/a, home, self, test,

personal, atlassian, “-”, na, private, abc, personal use, freelance,

my company, student, testing, company, no company,

private, mycompany, a, no, test company, noname, testcompany,

self employed, individual, tester, me, “.” , independent, trial, myself, nocompany, xyz

Predicting conversions• Baseline set at 50% • Using only the sign-up information we have an

accuracy of 65%

The Challenges

• Convince- challenge gut feelings- demonstrate value

• Educate!- “analytics is counting”!- help people ask the correct

questions

• Build infrastructure - replace existing “solutions” - data quality - catch-22…

The argument of common sense

How Common Sense Fails Duncan Watts, 2011

κοινὴ αἲσθησις Aristotle, 384–322 BCE

Source: http://www.tylervigen.com/

Correlation and causation

• Source: P. Chopra, The ultimate guide to A/B testing, http://www.smashingmagazine.com, 2010

A/B Tests

• A/B tests allow a constant improvement of products in a measurable and predictive way

• Last year we deployed ~118 experiments.

• The volume of customers, evaluators, visitors, has become a limited and valuable resource

Subject: %%first_name%%, assemble the bitbucketeers!

Subject: Grab the Bitbucketeers...4, 3, 2, 1

LAUNCH!

Do A/B tests slow down development?

“Move fast with stable infra” Mark Zuckerberg

Facebook F8, 2014

“Pauca sed matura” i.e., “Few, but mature” Carl Friedrich Gauss

1777-1855

“Move Fast and Break Things” “Move Fast and Break Things”

Instead of a Conclusion: We simply recycle the scientific method from other fields, e.g., medicine…

Patient User

Medical doctor Product manager

Illness Bad User Experience

Diagnosis Interviews, surveys…

Treatment UX modification

Medical researchers Growth team

Observations Data mining

Medical trials A/B tests

Ethics Privacy Policy

Evidence-basedmedicine

Data-drivendevelopment

Reed Johnson

Ilias Flaounas

Stephen Lee

Shaun Clowes

Graeme Smith

Tim Garnsey

Fardin Sarker

Theo Voronov

Herman Chow

Leo Balan

John Jiang

Rob Sangster

Andrew Wakeling

Martin Smyrk

Kostas Servis

Houda Chehab

Lakshan Perera

We want YOU!

Thank you!

Dr Ilias Flaounas !

ilias [dot] flaounas [at] atlassian.com