Northwestern data visualization - why why why

Northwestern Data Visualization| @gabegaster | 2015 may

what is data science?





who is a data scientist?




review of literature
















who is a data scientist?“a scientist who can code”




• lower barrier to attack new problems




• lower barrier to attack new problems • repeatable analysis




• lower barrier to attack new problems • repeatable analysis • freedom to think about problems new ways





using emerging technologies to approach problems scientifically




which were difficult to answer before




computing has progressed



1950computing has progressed


1950

cost of new analysis



1950

cost of new analysis years



1950


todaycomputing has progressed


1950


today

v



1950


today

hoursv

v



1950


today

same person thinking about the problem can conduct experiments to answer it

hoursv

v





open-source code



open-source code

standing on shoulders of giants



open-source code




open-source code




open-source code


reinventing the wheel



open-source code


reinventing the wheel









knowing what is possible







doing something useful







HOW







HOW WHY













usingnew good

the righttools







usingnew good

the rightasking whytools







usingnew good








usingnew good








usingnew good

the rightasking whytools WHY







usingnew good

the rightasking whytools WHY

WHY


why why whywhat is data science?



science is about asking why



science is about asking whystart there


an anecdote





an examplefrom the real world


• e

an example


• what

what data do you have? how can we make a network?


• what

what data do you have? how can we make a network?




goal: save money


goal: save money


goal: save money


goal: save money


goal: save moneytask: find needle in the haystack (without poking yourself)


abou

t pat

ent

not

abou

t pat

ent



abou

t pat

ent

not

abou

t pat

ent

turn over to plaintiffdon’t

turn over to plaintiff

adverse inference



abou

t pat

ent

not

abou

t pat

ent



adverse inference

give away trade secrets



abou

t pat

ent

not

abou

t pat

ent



adverse inference

give away trade secrets










goal: save moneyprototype


goal: save moneyprototype — design for lawyers


Sexier. Less nerdy. Tailored. design for transparency


http://www.daegis.com/judicial-acceptance-of-tar/


another examplecontests


another example



task:


classify schizophrenia w MRItask:


why?classify schizophrenia w MRItask:



improve understanding of disease



improve understanding of diseasehow?



improve understanding of diseasehow? … outside contest purview


why? outside contest purview





kaggle



kagglegetting data

& making usable



kagglegetting data

& making usable

WHY


timeline of contest

Accuracy of Classification


timeline of contest

AUC



what is AUC?AU

C


AUCwhat is AUC? Area Under Curve



what curve?



what curve? Receiver Operating Characteristic








balances:




balances: True Positive RateFalse Positive Rate








AUCwhat is AUC?


Area Under Curve



why?AUCwhat is AUC?


Area Under Curve



why?…

AUCwhat is AUC?


Area Under Curve



why?…

upshot:

AUCwhat is AUC?


Area Under Curve



why?…

choice of metric matters a LOT

upshot:

in practice

AUCwhat is AUC?


Area Under Curve



timeline of contest


AUC


timeline of contest


AUC

random guess


timeline of contest


AUC

random guess

basic SVM


timeline of contest

goal?


AUC

random guess

basic SVM


timeline of contest

goal: depends on why


AUC

random guess

basic SVM


random guess

basic SVM

timeline of contest


AUC


me

timeline of contest


AUC


me

timeline of contest


AUC turned out to place 9th — because overfitting


me

timeline of contest


AUC turned out to place 9th — because overfitting

very common problem


timeline of contest


worth it?

AUC











We need to reduce the costs of Service Requests. They are too expensive.

!!!!

Thousands of engineers around the world, 24-7 read through emails and hardware log files to determine

the cause of failure of a server. This is an expensive process. We've tried to automate it. We can now

automatically resolve 7% of new Service Requests. But we want more. That's why we bought a few

super computers with TBs of memory.

client

an example !

from the industrial internet


Why? Why do you need to set up a hadoop architecture to do clustering? What will this help you achieve? !!!!!How do you handle Service Requests? !


!!!!





client


Why? Why do you need to set up a hadoop architecture to do clustering? What will this help you achieve? !!!!!!!


!!!!





client




!!!!





client




!!!!





client


clienttools are not everything



but it is important to know the right tool for the job










don’t start w hadoop unless you have to. !




don’t start w hadoop unless you have to. !

probably you don’t have to.


client

How did you automate resolving Service Requests? !!!!!!!!!!!


client


A group of senior engineers thought about different use cases and came up with a list of conditions that, if any

are met, lead to predetermined solutions.


client



are met, lead to predetermined solutions. !

Took a year to create. !


client





We’ve been keeping track of every solved request for several years now.


client





We’ve been keeping track of every solved request for several years now.

from sklearn import naive_bayes as nb!nb.GaussianNB().fit(historical_requests,!! ! ! ! ! ! historical_decisions)


client

This works really well! But we can’t use it. !!!!!!


client


Oh. Why is that?


client


Engineers don’t trust the predictions.

Oh. Why is that?


client


Engineers don’t trust the predictions.

Oh. Why is that?


an example

just for fun


an example

just for fun

(a bit more depth this time)


a typical trip for me


Bus transit times = a LIE


Chicago is a grid city


Difficult

Public Transit on the grid

=+ Diagonals


Difficult

Public Transit on the grid

=+ Diagonals

2+ buses = FAIL


Adding bikes to public transit

= win


show how has divvy changed where people

can go

viz Goal:


show how has divvy changed where people

can goshow where people

actually go

viz Goal:


How to show where people go from here?


one way is with an arrow

A B

How to show where people go from here?


good for abstract relationships

one way is with an arrow


hard to decipher


lines between pts?

@flowingdata


emphasizes traffic

@flowingdata

lines between pts?(the lines superimpose)


emphasizes traffic

@flowingdata

lines between pts?

beautiful map(the lines superimpose)


emphasizes traffic

@flowingdata

lines between pts?

beautiful map(the lines superimpose)

— but not suited for this goal


lines between pts?


lines between pts?


lines between pts?

how to represent stations?


lines between pts?

on a maphow to represent stations?


but how?

lines between pts?

on a maphow to represent stations?


can use gradient — to show gradual differences

between stationsLondon transit map

@mySociety


@mbostock

or use natural borders?

London transit map

@mySociety


what regions?


each point is related to the closest station

what regions?



what regions?

—> Voronoi



what regions?

—> Voronoihuh?



what regions?

—> Voronoihuh? http://alexbeutel.com/webgl/voronoi.html

http://alexbeutel.com/webgl/voronoi.html



what regions?

—> Voronoihuh? http://alexbeutel.com/webgl/voronoi.html

Find the closest station — that’s my region!

http://alexbeutel.com/webgl/voronoi.html


Czech beer


Czech beer

starbucks

@gabegaster | http://bit.ly/1pdP2Tb

recap:

http://bit.ly/1pdP2Tb


recap: stations are voronoi tiles


@gabegaster | http://bitly.com/bundles/gabegaster/1


http://bitly.com/bundles/gabegaster/1



bubbles







too many edges to show…




too many edges to show… !

what about using color?



how touse color?



how touse color?

colors v colors



colors !

v!

colors

@mySociety

@mbostock



how touse color?

• two color scale

colors v colors



how touse color?

binned v gradient

colors v colors

• two color scale



• hard to read • differences subtle

gradient



how touse color?

• two colors not many • binned not gradient

binned v gradient

colors v colors



how touse color?

• two colors not many • binned not gradient

binned v gradient

colors v colors

binned



how touse color?

• two colors not many • binned not gradient • transparent empty bin

binned v gradient

colors v colors

binned



how touse color?

• two colors not many • binned not gradient • transparent empty bin • iterate

binned v gradient

colors v colors

binned



• dispersion (where people can / do go)

with these choices, the viz emphasizes:

• exploration



what can we learn from this?

urban vs lake frontcommute vs party

vs train

there’s me, too.hipster vs yuppie



Great! This looks great! !




But it’s too much at once.





make it interactive





make it interactive

divvy.datasco.pe


http://divvy.datasco.pe


new question


How are stations different?



who uses it



when is the station usedwho uses it



when is the station usedhow it used

who uses it



when is the station usedhow it used

who uses it

use the time signature of a station


Time Signature of a station


Time Signature of a station

http://divvy.datasco.pe/multiline/

http://divvy.datasco.pe/multiline/




questions?


in conclusion







thanks!@gabegaster

Data & Analytics

Northwestern data visualization - why why why