Embed Size (px)
Citation preview
Kirill: This is episode number 39 with Director of Data Science at
VSCO Ruben Kogel.
(background music plays)
Welcome to the SuperDataScience podcast. My name is Kirill
Eremenko, data science coach and lifestyle entrepreneur.
And each week we bring you inspiring people and ideas to
help you build your successful career in data science.
Thanks for being here today and now let’s make the complex
simple.
(background music plays)
Hello everyone. Today we've got the legend, Ruben Kogel,
returning for podcast session number 2. Ruben was my very
first guest on the SuperDataScience show, back in
August/September 2016, and since then it's been about 6
months, and guess what. His podcacst, his session, has
been downloaded and listened to 10,000 times. Over 10,000
people have listened to his session. How incredible is that?
Very excited about all of the knowledge that people gained
from that one episode that we recorded back in 2016.
So it's going to be very exciting to catch up today again, and
we had a fantastic chat. We talked about many different
things, and specifically here is a couple of them. So Ruben
joined the team at VSCO and he built up a data science
capability from the ground up. And he mentions the three
points that he focused on when building the capability, so
this will be very valuable to those of you who are looking
forward towards more senior roles, where you'll be in charge
of some elements of the data science inside the organisation.
Also, we talk in detail about what Ruben is looking for in
people who he is hiring for VSCO. Ruben right now is hiring
data scientists for a team in VSCO and he explains the
actual skillsets and mentalities that he's looking for in
people he is hiring. And apart from all that, of course,
there's lots and lots of other value that we've discussed on
this podcast, and I just want to reiterate again, Ruben is
hiring, so at the very end of the podcast, we speak about
how you can get an opportunity. And VSCO is actually in
Oakland, San Francisco, and I know that a lot of you guys
are in the San Francisco Bay Area, and a huge number of
you are in California.
So if you are interested in a data science position at a
company with an amazing culture with Ruben as your
mentor, this is an opportunity to die for. So if you're
interested in that, if you're open to new opportunities, even if
you're not, I would highly recommend checking out their job
description. You can get it if you go to Ruben's LinkedIn or
the show notes. You'll be able to see the link to it, but in
short, it's bit.ly/VSCOdatajob. So definitely check that out. I
think that there's going to be so much interest in this
opportunity, it's going to be a first come first served basis.
Definitely make sure to jump on top of this.
And on that note, without further ado, I bring to you the
legend, Ruben Kogel, Director of Data Science at VSCO.
(background music plays)
Welcome everybody to the SuperDataScience podcast, and
today, by popular demand, I have our favorite guest Ruben
Kogel back on the show. Hi Ruben, how are you today?
Ruben: I'm doing great. It's great to be back. Thank you for having
me again.
Kirill: Awesome. What an intro, right?
Ruben: Yeah!
Kirill: And your podcast, 10,000 views. It's mind-blowing. 10,000
downloads all over the world, people listening, watching.
How do you feel about that?
Ruben: I'm kind of shocked! I always think like it's not real! Because
it's just like you and I recording this podcast just a few
months ago, so it's kind of weird thinking that it had that
reach. But it's really great and amazing. I think it is also a
testament to how good a job you did at marketing the
podcast and reaching out to your audience, so really great.
Kirill: Thanks man, thanks. But the thing that I think is -- you
didn't tell me before the previous podcast, but you were
actually a radio host yourself! You gave me a little hint about
it afterwards, and I looked you up, and you -- everybody
listening, right away go and run and check out Ruben's old
podcast, which is called "All That Swing", right?
Ruben: Yeah, "All That Swing". Yeah, it was a short-lived podcast!
Kirill: But I listened to it, I couldn't stop listening. It's so fun, you
mixing the music and so on. And no wonder that podcast
started out well, because it was my first session and yet you
had all this experience! So it’s a testament to you, you drove
it home, man.
Ruben: (Laughs) Thanks. You want to hear a funny story? When I
did this podcast, I had zero experience doing a radio show, a
podcast, and I was really doing it out of a basement on some
radio somewhere. I didn’t think anyone was listening to it.
And one day someone calls in, and we had like this old
phone ringing, so you had to wrangle and get the phone, and
the person is like, “I really liked the piece of music you just
played. I like the podcast, but can you stop saying ‘um’ in
between every sentence?” It’s the kind of thing you don’t
realize when you talk on the radio, on a podcast, but all
those fillers, they are very, very visible and people can hear
them on the other end.
Kirill: Yeah. So how did you get rid of them?
Ruben: I don’t think I did. (Laughs)
Kirill: (Laughs) Okay, gotcha. You took that feedback on board,
totally took it on board.
Ruben: You know, I did a lot of editing. That’s what I did.
Kirill: Yeah. I know what you’re saying. Oh, the editing, I
remember the days. When you’re sitting at editing, you’re
thinking “It’s probably a good idea to actually improve my
skills so I spend less time editing.”
Ruben: Exactly, yes. (Laughs) It’s definitely a good experience.
Kirill: Yeah, totally. And it’s good even for data science, in terms of
presentation later on, when you’re talking to people and
presenting data stuff. Like, any kind of public speaking,
whether it’s podcast or anything else, I think it’s a good
experience.
Ruben: Yeah, totally.
Kirill: Speaking of data science, after the last podcast—it was very
interesting. You were at Udemy and we recorded the
podcast. And before I could release it, you had already left
Udemy, so I had to put like a little disclaimer on the page
that “Actually, Ruben is no longer with Udemy.” And that’s a
testament to how popular and valuable data scientists are.
So tell us a little bit about where you went and what have
you been up to for the past six months?
Ruben: Yeah. Shortly after recording our podcast, I accepted a
position at VSCO. VSCO is a platform or mobile app for
editing. It’s very popular amongst people who post photos on
Instagram. They use it to edit photos because it has a really
nice collection of presets and editing tools. But it’s more
than that. It’s also a community of expression. We have a lot
of professionals, so professional photographers posting
photos on VSCO, discovering other people. It’s really a great,
great community of photographers that we are building. I
actually knew the tool and the app from doing some casual
photography and I was really attracted to the challenge of
starting a new data team and really building the data
function at VSCO, not just pulling data and running
analysis, but actually thinking around what kind of data do
we need, how do we structure the data, what is the best way
to look at it, how can I have the most impact on the product
decision, on the strategy of the company. So it’s been a very
interesting challenge.
Kirill: Oh, that’s really cool. Like, building it from the ground up,
when you come in and there’s pretty much zero, or a little
bit, but you have full flexibility and full control. Is that right?
Ruben: Yeah, there was already data and tools in place, but what
was missing is a sense of direction and also we were at a
crossroad where we were using too many vendors, too many
tools, everything was competing, people were looking at
different dashboards, different vendors to get their
information. We just needed to rationalize a little bit how we
were looking at things and also we needed to put in place a
structure that would be scalable with the audience and the
future of the app. So even though there were a lot of tools
and great things that had been put in place, when I came in
I had this mandate to actually take a look at everything, at
all the vendors, all the tools, all the data that we had and
decide which one we kept, which one we got rid of, how we
would structure the data and what would be our 2 or 3-year
strategy for collecting, storing and utilizing data.
Kirill: Okay. That’s really cool. So how did you go about that
challenge? What was your thinking process?
Ruben: The way I went about it was I talked to a lot of people. I set
about to reach out or ask people to connect me with other
people who worked at similar companies that had faced
similar challenges and started just talking to them. And in
the process of talking with them, some themes started to
emerge, and some tools, and I started to form some opinions
about what was better than other solutions. So, to give you
more concrete examples, I spoke with the Head of
Engineering at Slack, Head of Data Analytics at Instacart,
people at Pandora, people at Twitter. I also started reading a
lot of blog posts and it helped me get my thoughts around
“Do we need to go with like a full-on data lay, put everything
in Hadoop and then use Hive or Pig to look at the data and
build dashboards on top of it? Should we use a solution like
Tableau, which seems to be popular with larger
organizations? Should we maybe use Redshift? Should we go
into Google BigQuery?”
So all those questions came and I had to sort of organize
like, “Okay, why do we need to have these different
solutions? What are the needs they are serving and what is
the best architecture that will let us address all of the needs,
that is both cost-effective but also scalable?”
Kirill: Wow, that sounds like a really cool thing. I’m looking
forward to digging into that right now because I think after a
lot of the podcast sessions that we’ve had on
SuperDataScience, people have slowly started to build up a
good overview, understanding of all the tools that exist out
there, and all the types of data science that exist. I think this
could be like a good summary of how to put it all together.
So, you spoke to all those people, you found out what they
do, what kind of tools they use in that and what their
strategic directions are. And how did you then assess your
situation and what is relevant to you and then take those
pieces of advice and implement them in your organization?
Ruben: Yeah, that’s a good question. The other piece I only alluded
to was that I had to look at the needs in my organization and
the fact that I was a very small data team. I was basically a
data team of one, so it was going to be impossible to provide
the entire organization with on-going analysis. I had to find
a solution that was more scalable.
And what I realize is that for companies like us, that have a
mobile app, or have like a platform that is consumer facing,
a lot of the analysis comes down to recording events,
recording taps, clicks, people going from one screen to the
next and counting how many people did a particular action
in a particular week in the U.S. versus not the U.S., on iOS,
Android, etc. And for this type of analysis, there were
already standalone solutions that existed and that were
doing a fairly good job of it and also enabled anyone in the
company to look at the data without being an expert in
SQL—in fact, without knowing SQL at all.
And so I realized that there were really different needs in the
company. There was this need, especially for product people,
designers, engineers, to have a quick look at the data,
knowing over time how many people have published a photo,
how many people have republished a photo, dice it maybe by
country, operating system, and having that tool that let
them explore the data very quickly and draw time series was
extremely valuable. And if we were to rebuild that tool from
the ground up, it would take enormous effort and resources
and we wouldn’t even be able to match the kind of simplicity
and latency that this tool offered.
Clearly, there was a need here that was at the moment filled
by a tool called Mixpanel. We can talk more about Mixpanel
and the alternatives, but there was this need that had been
identified specifically for product engineer people. There was
a second need around creating dashboards and reporting
tools that were very clean and that sort of conveyed a sense
of truth, because oftentimes you come into these
organizations, and people have been looking at data from
different ways and it’s not normalized, so the CEO might be
seeing something, the COO will see something else, the
analyst will see something else entirely. So having a place
where people can agree this is the truth, this is what’s
happening on our app with our audience was extremely
important. So that talks to the dashboards reporting piece.
And the third part was around having a place that let
analysts and data scientists do deeper analysis. That is the
concept of a data warehouse. That’s the concept of you put
all the data in one place, all the data is cleaned up, is
verified, and that led people who manipulate SQL to do
deeper analysis, do maybe some clustering or things that are
deeper than just doing a time series or reporting. So, once I
had identified the three needs – the product event analytics
on one hand, dashboarding, and deeper data analysis – I
think it was a matter of choosing the right technology and
the right tools, making sure it was both cost-effective and
scalable.
Kirill: Yeah. Wow! That’s a really cool overview. I like what you said
that in an organization like this, you need to really think
about how you’re collecting the data. You need to set up the
right data points for event analytics or for other things that
you want to know what’s been happening with your app,
how are people interacting with it. So, you set up those data
points, and then you also look at the different needs in your
company. You spoke about three: you’ve got the product
engineers, you’ve got the dashboards, and you’ve got the
deeper analytics for—
Ruben: —the analysts.
Kirill: Yeah. So, is it like many different people, or does everybody
have access to all of the data? Is it one team predominantly
working on all of these things? For instance, product
engineers, I understand they would have their own access.
But for instance, dashboards and deeper analytics—is that
one team, is that your team working on it? Or do you have
separate teams for each one of these three points that you
identified?
Ruben: In terms of building those tools and data access, it’s actually
different processes. The product analytics suite—we use
Mixpanel, and Mixpanel basically hooks up to your mobile
app and you just have to set up a hook for them and they
automatically download all of the event, clean it up,
augment it with location, add in some anonymous, unique
ID, and put it in a very high-performance database and put
a user interface on top of it to let anyone ask questions
without writing in SQL.
In that particular case, we are buying a service from this
vendor, it’s a very high-performance service, and you’d be
amazed. Like, running a query on billions of rows typically
takes a second or two, so it’s a very, very optimized
database, they actually run it directly in C. It’s not even
SQL, it’s not Java, it’s like directly in C when you enter some
command. So, it’s really optimized both for data reliability
and data performance.
Kirill: Okay, fantastic. How many users do you have so that we
know what numbers we’re talking about?
Ruben: On a monthly basis, we have an audience of about 40
million people worldwide, and basically we have a billion
events flowing every day. And because we like to instrument
everything, we will record any time someone goes onto one
screen, tap on a button, maybe do a quick view on an image,
republish the image, do an editing, what kind of editing they
did, what were the parameters, the sliders, everything. We
have like billions and billions of events flowing, so in order to
manipulate that, you need a very high performance data
pipeline and database system.
Mixpanel deals with all of that as a service. We don’t have to
worry about any of it as long as the events are defined by
our engineering team in the app, so they hardcode the
events. As long as that happens, Mixpanel can collect all of it
at the end of the app. They use an SDK and they organize,
clean up everything, and make it accessible.
Kirill: Gotcha. That’s very interesting that you mentioned how you
collect all the data. Sometimes I hear comments from data
scientists that there’s too much data, there’s too much data
in the world, in their organization, and they’re getting
overwhelmed with it. To be fair, not all data is useful. You
can’t extract information, you can’t get insights from every
single piece of data that you have. Most of the time you have
to be very selective and only some of the data is going to tell
you some stories. What is your view? Is it efficient for you
guys to collect all this data and then go through it and
actually pick out only the important stuff? For somebody
else, would you recommend to instead just collect the data
that is actually necessary? Why did you guys take this
approach of collecting absolutely everything?
Ruben: There are two reasons to collecting all of the events we do.
And by the way, we don’t collect absolutely everything. There
are things that we don’t collect but we do err on the side of
collecting more than less. So, there are two reasons to that.
One is, oftentimes you have product managers that are
launching features and they are interested in understanding
how users interact with a particular feature.
To give you an example, let’s say that we just launched a
subscription product, so we added a little tile on our apps so
that people can click on it, they get brought into the store,
and then in the store they can browse different products,
including the subscription. So we are interested in following
exactly what people are doing. We want to know how many
people saw the tile, clicked on it, got into the store, viewed
the product that is a subscription, and then eventually
purchased it.
Because once you have that entire view, you can start
optimizing the flow, you can start finding maybe there’s a
problem somewhere and that’s why your conversions are not
good. So, it is important that you really have access to all of
the data so you know exactly what’s going on and you know
exactly how to optimize your app to maximize engagement
and revenue. That being said, you may not need to record
everything at all time. There are events that are only
important to record for a certain period of time when you’re
trying to optimize things and after that you can stop
recording them.
But it’s easier to over-instrument than under-instrument
because what’s costly is not the instrumentation itself,
what’s costly is then the storage and parsing through the
data. But the storage itself, you can make a decision later on
of “We are keeping this data and we are not keeping this
data.” And I think that’s where the data scientists come in
because they have an intuition of what is important and
what isn’t important and they can parse the data and sort of
separate the two.
Kirill: Gotcha. Very, very cool. That’s some good advice, just record
as much as you can and then remove some of it. Can you
walk us through a recent example, out of what you can
disclose, of how you went about in a project? I think in a
situation like this, with all this data, it’s very important to
ask the right questions, right? If you have so much data and
then you don’t know exactly which question you are asking,
you can get very lost. So can you walk us through a project
and start off by how the person or whoever was requesting
this project, how you guys went about asking the right
questions and then solving it?
Ruben: Sure. A project that I completed with something was looking
at people who converted for a subscription. There was a
general question of “Who are the people who are buying
subscriptions and are they behaving differently after buying
a subscription than they were beforehand?” And the
question came to me exactly this way, so very general, and it
was on me to start engaging the conversation and start
specifying like, “Okay, what exactly do we mean by ‘Who are
those people?’ Why are we interested in this question?”
Basically, it was a question that came from one of the
investors that got transmitted through our CEO.
And basically I was trying to understand what exactly we’re
trying to get out of it to help me orient the analysis. You
know, it quickly became clear that the question was really a
marketing question. We have all these millions of users that
are potential buyers, but we know they are not all as likely
to purchase, so we’d like to know who’s more likely to
purchase because there are two things: one, that will tell us
what is the size of the opportunity; and two, that will tell us
who should we market this product to to be more effective in
our messaging and be more effective with our marketing
dollars. So once that was clarified—
Kirill: Sorry, if I could interrupt here — what were the other
options? It’s a marketing question. What were the other
options that it could have been, just so we’re up to speed?
What were you pondering? What decisions were you
making?
Ruben: I think it could have been—you know, we just want to know,
generally speaking, what those people look like in terms of
maybe their behaviours in the app, or it could have been a
question of what do those people look like in and of itself,
like descriptive questions. You know, “We have all these
people buying. Tell us a little bit more about them.” But
instead, I took it as comparative questions. “We have all
these people buying. What makes them different from the
other people and what makes them more likely to buy?”
Kirill: Okay, gotcha. So a comparative question, as in an absolute
question. For instance, when you were saying that, it came
to my mind that you could have been asked that question to
identify what they look like, not because you want to sell to
them, but because once they’re in there you want to better
service them, you want to maybe identify which products
they like once they’re in the subscription membership. But
instead, as you say, this indeed is a marketing question, so
two different types of questions.
Ruben: Yeah, exactly.
Kirill: Sorry I interrupted you there.
Ruben: So, once that was clarified, I think it became fairly
straightforward because I know there are only a limited
number of actions that can predict that someone is more
engaged and is more likely to purchase the subscription,
and all I did really was to collect all of those actions into one
big table and then run some sort of analysis. It was not even
a fancy statistical analysis, it was just a propensity analysis.
So what I did is I said to myself, “Well, the things that are
predictive of whether someone will buy a subscription
probably are whether they’ve purchased something in the
past and how much of it they have. Have they been active in
the last 30 days, and how active have they been. What
country they are in, because that might affect their
purchasing power. Whether they are iOS or Android. So the
list is really not that long, and once you have all of that, it’s
really a matter of doing the right comparison.
What I find was the most challenging piece of the analysis
was not collecting the data or knowing which event to look
at, but more like doing an apple to apple comparison
because, for example, you might have a group of people who
purchased the subscription in February, but out of that
group, there are people who sign up to VSCO in February.
So they have no history before February.
You also have people who were completely inactive in the 30
days prior and somehow came back to VSCO because they
saw an ad on Facebook or Google, so they randomly came
back. You had to be very careful because you can’t compare
the people who just signed up in February and bought a
subscription with people who were active in January and
didn’t buy the subscription. So I think it’s important to
establish the right comparison here, which in this case was,
“Let’s take a segment of people who were active in the 30
days prior to February and let’s look at the people who did
purchase and the people who did not purchase.” Because
then we have like a fixed base of the people who did that
same thing, they looked exactly the same in January, but
somehow one portion of them did purchase and one portion
of them did not purchase. So establishing that baseline is
very important, and once you have that, it makes it a lot
easier to compare the two groups.
Kirill: Yeah, very cool. Love it. And you mentioned you do some
propensity modelling on that. Why did you choose
propensity modelling over logistic regression or other types
of classification techniques?
Ruben: Because I think I didn’t want to overwhelm our team with
unnecessary complexity. I didn’t think that the goal here
was to have one formula that would predict the probability
of someone buying. I thought more important was to surface
trends and surface propensity according to different
dimensions. So to be specific, the kind of thing that we
looked at is, “Does the fact that you are on iOS make you
more likely to buy a subscription, and if yes, by how much?”
It doesn’t matter that you’re on iOS and you live in Australia
and maybe you’ve used like three paid presets in the last 30
days. Altogether increase gives you a likelihood of 30% or
whatever it is. What matters is, on each individual
dimension, how much more likely you are to convert
because then we can start narrowing down our marketing
effort to those people. And we can also separate out the
effect of all these different variables. So, in a way, it’s a
much more natural and clear analysis to communicate to
the marketing manager rather than coming up with a long
logistic regression forming at the end.
Kirill: Yeah, got it. And that’s been a very interesting kind of trend
that I’ve seen in this podcast. A lot of the time, people like
yourself who have the skills, who have the access to the
tools and methodologies, you sometimes (but not always)
choose to use a simpler approach simply because you have a
further agenda after your analysis is complete. You’re not
just delivering a consultant report. You’re part of a team.
You’re building something. You want the company to
succeed. You want the marketing manager to understand
what’s going on rather than having this convoluted formula
which might add an extra 10% or 2% accuracy, but overall
it’s not going to add that much more value because they
don’t understand how to use it properly.
Ruben: Exactly. I think here the key is that you want your
recommendation to be actionable. You want people to
understand what the analysis says and to be actionable, so
telling someone that being in the U.S. gives you like a 50%
lift over other people to convert or having purchased a paid
preset in the last 30 days gives you like a 60% lift, that is
much more clear and actionable than saying “The overall
probability is blah-blah-blah.”
Kirill: Yeah, gotcha. Very interesting example. Thanks for that. The
other thing I wanted to ask you and really pick your brain
on, being in your position, as we discussed at the start, just
before the podcast, you are a Director of Data Science, but a
lot of the stuff that you’re doing is also strategic. It’s also
which database systems are we going to implement, who’s
going to have access to what and so on. You’re performing
the role of Chief Technology Officer in combination with
Director of Data Science, as I understand it.
A big part of that is spreading the culture of data across the
organization. And that’s something I’ve personally faced in
different roles I’ve been in, and it’s often tough to get that
culture or appreciation across to people so that they become
data advocates, that they help you out, that they look for
these things in their own roles, how they can apply data and
so on. So, how have you been dealing with that? What kind
of challenges have you faced and the opposite – what wins
have you had in the space of data culture in your
organization?
Ruben: Yeah, it’s a good question because I think you touched on
something that’s much more practical than running a
machine learning algorithm or regression, it’s like how do
you have an impact on your organization and how can you
be effective? I would say that generally speaking, people are
open-minded and everybody wants to be data-driven. The
problem is that sometimes data is not accessible, or it’s too
complicated, or people don’t have the time or patience to
figure out exactly what’s going on.
And so my role as Head of Data at VSCO is to sit down with
people and make sure that one, they have access to the data
that they need, but also that when they are trying to make a
decision, and they always want to make a data-driven
decision, but when they are making that decision, that they
are looking at the right data, they are looking at it the right
way, and they understand the trade-off of the data and they
understand the limitation of what the data can say. So, to
give you a concrete example, we actually have a weekly
meeting between my team and the product team where it’s
kind of like a freewheeling meeting, but more often than not
we talk about some of the analysis that I came up with, or
we talk about some of the A/B tests that the PMs run.
It’s actually a forum where I get a chance to educate the
team on how to interpret data, how to interpret the results of
an A/B test and what the data says and what the data
doesn’t say. It’s been really the source of great
conversations. In particular, we’ve had great conversations
around how to interpret the results of an A/B test, what is
significant, what is not significant, whether one week of data
is sufficient, should we look at just one metric, should we
look at more metrics and retention, etc. I think the key here
is really having an on-going conversation and trying to serve
the needs of the people who are making decisions and are
trying to use data to make decisions.
Kirill: Yeah, that’s great. I think that’s a very valuable thing as
well, training up the staff and having that opportunity. It’s
good that they’re very open to meeting up with you on a
weekly basis and having this discussion because that’s a
first step, for people to be open with that. And what’s your
vision for that? What’s your vision for spreading the data
culture in the organization? I know that you’re still in the
early days, just six months there. But say in two or three
years, how would you like to see every employee in the
company? To what level would you like them to be able to
interact with data?
Ruben: Yeah, I think there are already tools in place that let people
access the data. We talked about Mixpanel. We also have a
dashboarding tool that lets people see the key metrics and
play with the key metrics a little bit. I think the key is that I
want to be able to automatically answer 90% or 95% of
people’s general questions. I think the role of the data team
is not to do data pools and do like a SQL query to find out
the number of users who did that or the number of people
who’ve purchased in China or in South Korea in the last
month. All of this should be answered automatically with
self-service analytics, something that we talked about in our
previous podcast. So there is that.
The second part is also that people who make decisions
around data, particularly product managers, I want to make
sure that we have a very solid framework for setting up and
analysing A/B tests. Basically it means that we have a
playbook of “Okay, these are the rules of how you design an
A/B test, how many people you put in each bucket. Do not
touch your A/B test for like two weeks so that you don’t
mess up the bucket. And then, when you analyse the
results, I want to you to go into this dashboard and to not
just a particular funnel, but look at bunch of other metrics
and check that they are not affected, or check that the
retention is also there.”
It’s basically a combination of making sure that everybody
has access to the tools and is comfortable with using the
tools, so they go through training, but also setting up for
more specific use cases, like funnel analysis, A/B test
analysis, setting up like a playbook and making sure that
the people who are going to own those features know how to
run the analysis, because at the end of the day, I think even
doing an A/B test analysis is something that’s pretty
programmatic, you know, you can follow a set of rules. I
don’t have to redo the analysis every time. I don’t have to
have my team redo the analysis. I can just set up the
dashboard, set up the guidelines, and have the owner of the
product do the analysis on their own. So that’s kind of like
my vision for how I would educate the team around the use
of data and how I would empower the team to use data.
Kirill: Okay. That’s pretty cool. What it reminds me of is, there’s a
company in San Diego called Digital Marketer. It’s a large
company for marketers and how to do online ads and stuff
like that. But I like how they’ve got it set up. They’ve got like
a set of onboarding training which, actually, they have been
so successful within the company that they have made them
available. You can subscribe to them, and in
SuperDataScience we use some of their trainings as well.
You can subscribe to them and your staff can go through
this training and some of them are like A/B tests and
statistical analysis for marketers, of course. But at the same
time, it’s pretty valuable and I like the structured approach.
It kind of reminded me when you said that a lot of it has to
do with education, so when somebody new starts, or maybe
once in a while employees can go through these trainings
and educate themselves on how to use these tools properly.
That’s very important.
The other thing I wanted to ask you while we’re on this topic,
you’ve touched on A/B tests several times, so maybe you
can educate us a little bit. What is your rule of thumb for an
A/B test? What should the sample size be for an A/B test to
most likely be statistically significant? Do you have a rule of
thumb?
Ruben: Obviously, it really depends on the size of your audience, the
number of people who will see the particular thing that
you’re changing, and the conversion rate. There are like
pretty standard calculators online that will tell you, you
know, “You need that many people.” Typically, in our case
we need like 5,000-10,000 people in each bucket because we
are interested in looking at meaningful changes.
One of the things that I warn people against is running a lot
of A/B tests just trying to optimize the button or an image
and trying to see small changes because I don’t think that’s
the right way of running an A/B test because one is, you will
always have false positives. Even when you run a test at
95% confidence, it still means like 1 in 20 you’ll have a false
positive. So if you run five tests in parallel, all of a sudden
you have a much higher chance of just hitting a false
positive.
And second is because I think it’s important to, even when
you run an A/B test, to have a hypothesis behind it. You’re
not just testing random combinations of things. You are
testing a hypothesis. You are saying, “I’m going to change
the conversion flow on my store from A to B” or “I’m going to
make the shopping cart much more prominent and therefore
I think that people will see it and I think people will convert
at a higher rate.” So it’s important to test the hypothesis and
it’s also important to hold yourself accountable to a big
change. You don’t want to optimize things and move like
conversion from 10% to 10.1%. You want something that
will move it meaningfully, that will move it from 10% to 11%
or 12% or 15%. I think that’s the right way of running A/B
tests. And when you do that, it also means that you don’t
need as big of a sample because you are really looking for
big changes. And if you don’t see a big change, then maybe
it’s not that meaningful of a test.
Kirill: Yeah, gotcha. And last time you were on the podcast, you
recommended a book by Nate Silver, “The Signal and the
Noise.” I actually have it here. I’m reading it and I know I
probably should have read it years ago, it’s so good, but it
talks a bit about what you mentioned now, and I want to
reiterate this point for our listeners. Your tests that you’re
running, they have to be meaningful and they have to make
sense, right? Because in the book, Nate Silver does a good
comparison of Bayes, who was 18th century or something
like that, and Fisher, who was 20th century or maybe he
was 19th century; Bayesian probabilities and inference
versus Fisherian, as they call it, probability and inference,
and how Fisher didn’t really like Bayes for this concept. A lot
of the stuff that we do, especially in this just pure statistical
testing, is actually developed from the concept of the—what
were we talking about—
Ruben: Frequentist statistics?
Kirill: Yeah, frequentist statistics, exactly. And therefore, we forget
sometimes about the whole purpose, the whole idea behind
the test. He has a good example in the book that if you use
this approach, of just standard A/B testing, standard
frequentist statistics, you can kind of prove that frogs can
predict earthquakes just because there’s some correlation
there, but correlation doesn’t always imply causation. So
that’s a good example of you have to think about that what
you’re testing makes sense. There’s no point even thinking
about testing that frogs can predict earthquakes because it
just doesn’t make sense. Even if you have a positive result,
it’s most likely going to be a false positive, you know in
advance. I think that kind of stands to the point which you
raise.
Ruben: Yeah, I totally subscribe to that. I think it’s important to be
thoughtful when you set up experiments and when you
analyse A/B tests.
Kirill: Yeah, exactly. All right, moving on to my next area of
discussion, your LinkedIn says you’re hiring. It seems like
this is a credo that you carry through life. (Laughs) Wherever
you go, you are hiring data scientists. I don’t think we talked
much about this last time, but this will really benefit people
listening. What are the skills—not just skills, what are the
qualities that you want in your ideal candidate when you are
hiring data scientists?
Ruben: That’s a great question because I think there’s a lot of
misconception about what it takes to be a data scientist and
how do you get hired. There are a couple of things that are
important. The first thing is sort of like a general business
intuition and communication skills. I think it’s easy for
people to get mired into complicated analysis, trying to spin
up Python and run this crazy regression, maybe boosted
trees or whatever, but not being able to one, bring in the
right ingredients in the analysis, and two, explain what
comes out of the analysis.
Let me be a bit more specific. Let’s say that you were tasked
with predicting churn of your customer base. I think
probably one of the most important pieces of this work
would be to come up with the right variables, the right
features in your analysis that are most likely to predict
churn. In order to do that, you really need some business
sense and intuition. You need some data intuition so that
you construct the right features in your analysis. And when
you have the right feature, usually that’s like 80% of the job.
It almost doesn’t matter what techniques you use to identify
the most important features, having them in your dataset is
the most important part and they will always come out at
the end. So that’s very, very critical.
And the second thing is then being to explain to a lay
audience what does the result of the analysis mean because,
again, if you’re just writing a technical note that no one will
read, then you have zero impact in your company. You want
to be able to talk to the CEO, to the product manager, to the
marketing manager, and tell them like, “I looked at the data
and the data says these are the five things that predict
churn and these are the rank order of the things that predict
churn.”
Kirill: Yeah, totally. Break it down to them and make it digestible,
accessible, the insights. And actionable, as you say.
Ruben: Exactly. So that basically speaks to both business intuition
and communication skills. I think what’s important on top of
that is also structured thinking. Like what you just said, the
ability to take a problem that is very vague and complex and
breaking it down into more manageable pieces and coming
up with a data question that corresponds to that business
problem. Earlier we talked about subscription and the
business question was like, “What do our people who
subscribe look like?” You have to be able to turn this into an
actual data question that will yield a meaningful answer,
and the data question is like, “Oh, let’s look at all the people
that were active in the month of January. Let’s look at their
activity in terms of editing and purchasing, etc., and let’s
look at what makes them more likely to convert than not.”
So this ability to critical thinking, structured thinking, is
also extremely important. And all of that has nothing to do
with skills and knowledge. It’s really more about a way of
thinking and grappling with problems. Now, obviously, there
are important skills and knowledge. Number one, at least in
my line of work, is SQL. You can’t really go very far without
knowing SQL. Fortunately, SQL is a fairly simple language,
so I would never hold it against a candidate if they are not
experts in SQL because I think anyone can learn it. But
having that baseline is important, and also having a very
good understanding of statistics is crucial, I think. By that, I
don’t mean to know every distribution and every statistical
test on the Earth. I mean understanding a few statistical
concepts and understanding them very, very well. Because
people will come up with edge cases and this and that, that
unless you’re very well steeped into your statistics you will
not be able to answer.
Kirill: Gotcha. That’s very interesting. I want to get back to that in
a second, but before, I want to comment on what you said
about it’s not about the skills. I totally agree with that. For
instance, even when we’re hiring at Super Data Science, we
not as much look at what the person can do in terms of how
experienced they are or what skills they have.
Personally, I look for passion. I look for drive. I look for
purpose, meaning. I look for people who are eager, who can’t
sit still. They want to know everything, and even before
you’ve spoken to them, they already know everything about
your company, they already looked everywhere and they
know what they can improve, they come with value add
propositions and so on. Those are the people that add value
to the bottom line. I just wanted to comment on your point
that it’s not only about the skills. In fact, it’s less about the
skills than it’s about the type of person you are.
Ruben: Yeah, I totally agree. In fact, there are a lot of skills that can
be learned. I spoke about SQL being an easy language. You
know, if someone can demonstrate that they have the ability
to learn, they are very curious and they want to learn, then I
wouldn’t at all hold it against them that they are not experts
in SQL. Because I know that they will be able to master it
and there are things that are more important to me than
their pure technical skills on a programming language.
Kirill: Yeah, I totally agree. And speaking of statistics, can you give
us a few concepts, hypothesis testing, A/B testing, things
that you think are important for a person to know, just
examples, and then our listeners can pick and choose from
there what they would want to focus on.
And I’m asking because I very often get this question. You
know, people start understanding the skills and the
techniques and methodologies and tools in data science, but
then they are still left with this bit of uncertainty that there’s
so much statistics out there. Which concepts should they
focus on in order to, as you say, when they come in to an
organization, be well-grounded and be able to stand their
ground in terms of statistical discussions and presenting
their findings and things like that?
Ruben: That’s a great question. I think you’ve mentioned a few. So,
hypothesis testing is absolutely fundamental. And there are
subtleties in hypothesis testing. It’s not something that’s
very intuitive—in fact, it’s not intuitive at all, so really
mastering the language and the logic behind hypothesis
testing is very important, in part because that’s what people
use to prove or disprove a hypothesis. That’s what people
use to accept or reject an A/B test, so understanding that is
very critical.
Also, understanding the key statistics that are used to
accept or reject an A/B test, whether it’s like a chi-squared
test or whether it’s just a t-test between two variables is
important. One other thing that I think is also important is
to understand the concept around the central limit theorem.
The fact that when you have individual events or individual
users and you group them together, you always end up
having some sort of a normal distribution.
If you take—let’s say that we were looking at the average
number of photos edited per cohort on VSCO, so we start
with the January cohort, the February cohort, the March
cohort, and we just take the average of number edited per
cohort, it will obey a normal distribution, even though
people individually are certainly not editing photos on a
normal distribution. If you just took the distribution of the
number of photos edited per person in a given month, that is
an exponential distribution. You have more people editing
one, then two, then three, then four. It’s monotonous and
it’s decreasing. However, when you start taking averages, it
always follows a normal distribution.
So being able to move in and out of this concept and
understanding that is actually pretty important, because few
people really understand that. Another thing is regression
analysis. I think it’s still pretty key. It’s a simple tool, but it
can be pretty powerful when you have a few high-powered
features or predictors. So being able to correctly interpret
regression analysis, and also knowing the trade-off and the
problems with regression analysis is important.
Kirill: Awesome. Yeah, the limitations of linear and multiple linear
regression.
Ruben: Yeah, exactly.
Kirill: Gotcha. That’s been a great overview. So, hypothesis testing,
mastering the language and logic, A/B testing, key statistics
used for this, so it’s either chi-squared test or a t-test,
central limit theorem, being able to jump in and out of
understanding how things become normalized or turn into
normal distributions when you do certain statistical changes
to them, and regression analysis. I want to add one more,
my favourite one which I always hold dear to my heart – the
law of large numbers. It’s very basic, but the more times you
do something, the closer your average is going to be to the
expected value. You’ve got to know that one. (Laughs)
Ruben: Yeah, totally. I totally agree with that.
Kirill: All right. Cool. Okay, time flies with you, Ruben. We’re
getting close to the end. I’m like, “We need a third podcast.
Where is this going?” But I wanted to do a big shout-out to
you, to your hiring in VSCO. I think about 30%-40%—I
might be lying here, so I won’t say the statistic, but a huge
percentage of our listeners are in San Francisco, not just in
California, but specifically in the San Francisco Bay Area.
So, guys, VSCO is hiring, and whether you have a job or you
don’t have a job already, get in touch with Ruben and first
come, first served. Ruben, you’ll probably get bombarded.
(Laughs)
Ruben: (Laughs) No, I love the promotion. Thanks for the shout-out.
Kirill: Yeah. So what’s the best way they can get in touch with you
and see if this position is right for them?
Ruben: I think they can check it on my LinkedIn. I have a link to the
position itself. I encourage people to check it out, see
whether it vibes with their interest, with their qualifications.
Feel free to either send me a message or apply directly on
the website. I guarantee that we look at every single resume.
Guarantee it! It’s not a black box, so either one of them
works fine.
Kirill: Awesome. Gotcha. And I’m just looking at your LinkedIn. If
somebody is listening to this in their car or something, it’s
very easy to remember – bit.ly/VSCOdatajob. Check it out. If
I was looking for a job, I would love to work in a team like
that. I’m looking at some of your photos on the VSCO
LinkedIn page. It looks like you guys have a really cool office,
right? Is the corporate culture there pretty good?
Ruben: Yeah. It’s actually one of the most amazing companies I’ve
worked for. It’s actually really incredible. It’s a bunch of very
smart, but very humble people that are extremely
collaborative. It’s never about who did what or who forgot to
do what or who takes credit for a particular project. It’s
always, “Let’s solve this as a team.” If anyone sees any bug
or any problem, that person will immediately signal it to the
rest of the team and everybody will work to solve it. When
there’s a big achievement, the entire team is recognized, not
just the one person who drove the project. It’s a very relaxed
but also very accountable environment. People know the job
that they have to do and they do it because they know that
other people rely on them, not because their boss asked
them to do it. I really love it for that. In fact, it’s a very flat
organization. Some day you might be working with the COO
on a project, and another day you might be put in a team
with a couple of engineers and product managers working
on it. It really doesn’t matter who’s in the room and who
asked to do it, everybody will perform the same.
Kirill: Fantastic! Whoever gets this job or jobs, I’ve got an idea. Let
me know that you applied from the podcast and you got the
position and then next time I’m in San Francisco, you,
Ruben and I will all go out for dinner and have a fun chat
about how all this happened and came to be. (Laughs)
Ruben: Absolutely. Dinner is on me.
Kirill: Deal. Thanks a lot for coming on the podcast once again and
sharing all of this overwhelm of knowledge. Yeah, incredible.
Ruben: Thanks for having me again. It was again a pleasure to talk
to you and to your listeners.
Kirill: So there you have it. I hope you enjoyed today’s session.
Lots and lots of valuable information. So diverse, right? We
talked about so many different things.
We talked about the three points that Ruben focused on
when he came into the organization. Ruben actually gave us
a case study breakdown of how he went about analysing a
project and answering questions and actually posing the
right questions to answer in the first place. I think that was
a very valuable example. Then we talked about the statistical
concepts that Ruben is looking for in his ideal candidate and
I think that was very valuable.
I’m kind of torn between the two, what is the most valuable
for me, the example that he gave of the case study, where
there’s these two types of ways that you could pose a
question, or the statistical breakdown. They both have
merits, but I’ll probably lean towards the statistical concepts
that he outlined. That is very valuable, just having that
summary of the five statistical points that you guys have
been asking me for so many times, like “What do you learn
in statistics?” Well, there you go. We’ve outlined them in the
podcast and you can always reference these when you’re
researching statistics for your career.
And speaking of careers, don’t forget that Ruben is hiring, so
definitely check out his link. You can get all of the links and
show notes at www.superdatascience.com/39 and there
you’ll see the link to the job he’s posted. I don’t think it’ll be
up for long, especially with this podcast coming out, you
know, thousands of people listening to it per day, or
especially in the first few days. So jump on it early, get in
touch with Ruben, and I look forward to all of us catching
up and having dinner together in San Francisco when I’m
there next time. So make sure you get that job and I’ll see
you there soon. And thank you so much for being on this
show. I really appreciate you spending an hour with us
today. And I can’t wait to see you next time. Until then,
happy analyzing.
