Embed Size (px)
Citation preview
Managing Machine LearningDavid Murgatroyd - VP, Engineering @dmurga
3
4
5
Problem: I can’t fully specify the behavior I want.
6
Problem: I can’t fully specify the behavior I want.Solution: Machine Learning
Where does machine learning fit in the technology universe?
Valuable
... a star of the Data Science orchestra. - John Mount, Win-Vector
Central
... the new algorithms ... at the heart of most of what computer science does.
- Hal Daumé III, U. Maryland Professor
Last Resort
… for cases when the desired behavior cannot be effectively expressed in software logic without dependency on external data.
- D. Sculley et al., Google
7
Where does machine learning fit in developing technology?
8
Stuff to do Demonstrable ValueStuff to do now
How does machine learning affect value demonstration?
Distill business goal into a repeatable,
balanced metric.
Measure on the most representative data you
can get.
Distinguish intrinsic errors from
implementation bugs.
Let your customer override the model when
they absolutely must get some answer.
9
Demonstrable Value
Distill business goal into a repeatable, balanced metric.
10
Demonstrable Value
Business goals in our example:
● fewer incorrect candidates sent to
analysts for review
● no increased volume of work for
analysts
● confidence to help analysts prioritize
Example metric: area under an error trade-off
curve based on confidence, constrained to
max volume. Sometimes called an ‘overall
evaluation criteria’ (OEC).
Note that the more skewed the OEC (e.g., if #
of positives varies by day and season) the
more samples are required to be sure of
statistical significance.
Measure on the most representative data you can get.
11
Demonstrable Value
Considerations when selecting data:
● online v offline: A/B test in production with
feature flags (one or two variables at a time,
agile-y) vs. stable data set
● implicit v explicit: implicit can correlate
more with value but omits unseen states
● broad v targeted: if explicitly annotating
consider targeting based on diagnostic
value or where systems disagree
Resist the temptation to ‘clean’ data -- you may
kill it. Instead include normalization in your
model.
Distinguish intrinsic errors from implementation bugs.
12
Demonstrable Value
Distinction
● Error: incorrect output from a model
despite the model being correctly
implemented.
● Bug: incorrect implementation, doing
something other than what was
intended
Useful to manage expectations about quality
and effort required to improve/fix.
Providing an explanation for output can help
make this distinction. Bug Error
Let your customer override the model when they absolutely must get some answer.
13
Demonstrable Value
Varieties of overrides:
● Always give this answer.
● Never give this answer.
Can apply for sub-models or overall.
Beware of potential toward ‘whack a mole’.
Feel sad every time they use it.
Where does machine learning fit in developing technology?
14
Stuff to do Demonstrable ValueStuff to do now
How does machine learning affect team organization?
15
Machine Learning Expert
Spectrum of options between:
Integrate machine learning expertise in every
team that needs it.
Separate it in an independent, specialist
team.
Option 1: integrated teams with cross-team interest groups
16
Encourages alignment with
business goals.
Challenges machine learning
collaboration, depth and reuse.
Best for small, diverse products.
Option 2: independent machine learning team delivering models
17
Encourages machine learning
collaboration, depth and reuse.
Challenges alignment with
business goals.
Best for products with large, complex model(s).
How does machine learning affect iteration structure?
18
Pros for shorter:
● More simple experiments are better
than fewer complex ones
● The value of machine learning leads to
high cost of delay
Pros for longer:
● Innovation takes deep thinking
● More time to control technical debt
creation
Where does machine learning fit in developing technology?
19
Stuff to do Demonstrable ValueStuff to do now
How does machine learning affect chunks of work?
Focus on experiments following the
scientific-method: hypothesis, measurement
and error analysis.
Continuously test for regression versus
expected measurements.
Decouple functional tests from model
variations.
20
Stuff to do now
Focus on experiments with hypothesis, measurement and analysis.
21
Stuff to do now
Continuously test for regression versus expected measurements.
22
Stuff to do now
With machine learning’s dependence on data
changing anything changes everything. This
makes it the “high-interest credit card of
technical debt”.
Determine what’s a significant change,
including looking at aggregate effect across
different data sets.
Decouple functional tests from model variations.
23
Stuff to do now
Options:
Black-box style: enforce “can’t be wrong”
(“earmark”) input/output pairs. Might lead to
spurious test failures.
Clear-box style: use a mock implementation
of the model that produces expected answers.
Decouple functional tests from model variations.
24
Stuff to do now
Options:
Black-box style: ensure “can’t be wrong”
(“earmark”) input/output pairs. Might lead to
spurious test failures.
Clear-box style: use a mock implementation
of the model that produces expected answers.
Decouple functional tests from model variations.
25
Stuff to do now
Options:
Black-box style: ensure “can’t be wrong”
(“earmark”) input/output pairs. Might lead to
spurious test failures.
Clear-box style: use a mock implementation
of the model that produces expected answers.
42
Where does machine learning fit in developing technology?
26
Stuff to do Demonstrable ValueStuff to do now
How does machine learning affect prioritization?
27
Stuff to do
Do we need more training data?
Do we need a richer representation of our
data?
Do we need a combination of models?
How much could improving a sub-component
of the model help?
What development milestones should we
target?
Do we need more training data?
28
Stuff to do
The learning curve implies adding training
data should bring down the test error closer
to the desired level.
Do we need a richer representation of our data?
29
Stuff to do
The learning curve implies adding data won’t
help but a richer data representation may.
Could be more features identified by
someone with domain expertise analyzing
errors. Though remember more features
often means less speed.
Could require a new model if the domain
information identified is not representable in
the existing one.
Do we need a combination of models?
30
Stuff to do
The learning curve implies the model is
overfitting the training set.
Consider training multiple models on random
subsets of the data and combine them at
runtime to decrease the variance while
retaining a low bias. Presuming you can spend
the compute.
How much could improving a sub-component of the model help?
31
Stuff to do
Build an ‘oracle’ for the sub-component --
something that takes perfect output from
data.
Annotate to get that perfect output on some
test data to feed the oracle.
Measure the overall system with the oracle
turned on.
What development milestones should we target?
32
Stuff to do
Make it…
● Glued-together with some rules
(Prototype)
● Function (Alpha)
● Measurable & inspectable (early Beta)
● Accurate, not slow, nice demo,
documented & configurable (late Beta)
● Simple & fast (GA)
● Handle new kinds of input (post-GA)
Questions?
33
Stuff to do Demonstrable ValueStuff to do now
Suggested questions:
Say more about integrating domain expertise?
Say more about online vs. offline testing?
How to manage acquiring data?
How to recruit machine learning folks?
What bad habits can ML enable?
Where can I try your stuff? api.rosette.com
You hiring? Yes - basistech.com/careers/
@dmurga
Appendix @dmurga
Manage Technical Debt
35
Data is even less visible than code -- blurs boundaries.
Recruiting machine learning experts
36
who
◦ expertise in sequence models > in domain
◦ depth in specific model > breadth over many
where to find them
◦ local network: meet-ups, LinkedIn
◦ academic conferences
◦ communities (e.g., Kaggle, users of ML tools)
how to attract them
◦ explain purpose & uniqueness of the problem
Online vs. offline evaluation
37
Online (e.g., A/B)
● Individual decisions need to not be mission critical● Enough use to get sufficient statistics in short time● Helps motivate aligning production and development environments● If the model is updated online, validate it against offline data periodically to
watch out for drift● Usually focused on extrinsic or distant measures
Offline
● Always have some of this to for long-term protection against regression● May be required for intrinsic measurement
38
Epistemology Exact sciences
Experimental sciences
Engineering Art
Example ... Theoretical C.S. Physics Software Management
Deals with ... Theorems Theories Artifacts People
Truth is ... Forever Temporary “It works” In the eye of the beholder
Parts of machine learning fit all four...
Learning theory Model & measure
Systems Users
This is great, as long as we don’t confuse one kind of work for another.(This table is an expansion of one in Bottou’s ICML 2015 talk.)
