• Home

  • Documents

  • D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì...
80
flyingpenguin Great Disasters of Machine Learning: Predicting Titanic Events in Our Oceans of Math Davi Ottenheimer flyingpenguin 2016

D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

Embed Size (px)

Citation preview

Page 1: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Great Disasters of Machine Learning:Predicting Titanic Events in Our Oceans of Math

Davi Ottenheimerflyingpenguin

2016

Page 2: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Agenda• About Me• We Easily Believe Machines Will Be Better Than Us• But They Repeat The Same Awful Mistakes (Faster)• For Perhaps Obvious Reasons• Let’s Fix Sooner Rather Than Later

GT 2016GT 2016

Page 3: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

ABOUT ME

@daviottenheimerflyingpenguin

GT 2016GT 2016

http://www.flyingpenguin.com/?p=9621

Page 4: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

Page 5: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

It’s “Learning”

11:32 PM - 23 Mar 2016

GT 2016GT 2016 https://twitter.com/daviottenheimer/status/712889915533500416flyingpenguin

Page 6: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinhttp://www.foilingweek.com/blog/2016/02/safety-is-only-an-excuse/

Sailing

GT 2016GT 2016flyingpenguin

Page 7: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

“No matter how far away, if you see it, act on it now…NOW”

Page 8: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

“No matter how far away, if you see it, act on it now…NOW”

Visibility: Fair

flyingpenguin

Page 9: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

Visibility: Not Fair “No matter how far away, if you see it, act on it now…NOW”

flyingpenguin

Page 10: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

WE EASILY BELIEVE MACHINES WILL DO THINGS BETTER THAN US

GT 2016GT 2016

Page 11: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Dangerous to Give Impression of “Passing” Grade for Failures

• No Weather • No Unmarked Lanes

– Roundabout– Crossings– Unpaved

• No Judgment Zones– School – Shopping Center

GT 2016GT 2016

http://publications.lib.chalmers.se/records/fulltext/168996/local_168996.pdf

https://www.theguardian.com/technology/2016/feb/29/google-self-driving-car-accident-californiahttp://spectrum.ieee.org/transportation/advanced-cars/how-googles-autonomous-car-passed-the-first-us-state-selfdriving-test

Page 12: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Dangerous to Give Impression of “Passing” Grade for Failures

GT 2016GT 2016 https://twitter.com/AmyJBrittain/status/750796150396096512

Page 13: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

“Data collection so cheap everyone expected to dive in and play with machine learning”

Page 14: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

“Computers increasingly bear the burden of making our decisions”

Page 15: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

World Class Expert on Risk Margins Decides to Transfer Burden To Car…

GT 2016GT 2016flyingpenguin

Page 16: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Brown on April 17th: “I actually wasn’t watching…”

GT 2016GT 2016

https://www.youtube.com/watch?v=9I5rraWJq6E

”It was a mistake on the other driver's part. He did not even know I was there”

20 Seconds Before Merge Risk Detected

“I became aware of the danger when Tessy alerted me with the ‘immediately take over’ warning chime”

Page 17: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Me on April 17th: “human far earlier detects what Tesla blind to”

GT 2016GT 2016 https://twitter.com/daviottenheimer/status/721875721946202112

See Something, Do Something

”I actually wasn't watching…It was a mistake on the other driver's part. He

did not even know I was there”

Page 18: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

“The Autopilot system allows the car to keep itself in a lane, maintain speed and operate for a limited time without a driver doing the steering. Autopilot is by far the most advanced driver assistance system on the road, but it does not turn a Tesla into an autonomous vehicle and does not allow the driver to abdicate responsibility.”

GT 2016GT 2016

• Adaptive Cruise Control• Lane Keep Assist

“Steering to Avoid

Collision”

https://twitter.com/daviottenheimer/status/721875721946202112

Known Capabilities

CEO Description

flyingpenguin

Page 19: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Was Joshua Brown “Victim of an Innovation Geared Precisely to

People Like Him”?

GT 2016GT 2016

Page 20: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Victim of an Innovation Geared Precisely to People Like Him

GT 2016GT 2016

• Didn’t See Toddler– “…meant for

observing and reporting only”

– Knocked Him Down– Ran Over Him– Weighs 300lbs

• Second Incident

http://abc7news.com/1423093/http://www.fastcoexist.com/3049708/meet-the-scary-little-security-robot-thats-patrolling-silicon-valley

Page 21: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Tesla: Otto Has Arrived!

GT 2016GT 2016

https://www.teslamotors.com/blog/your-autopilot-has-arrived

He “relieves drivers of the most tedious and potentially dangerous aspects of road travel”

flyingpenguin

Page 22: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

“For something to catch Elon Musk’s eye, I can die and go to heaven now”

GT 2016GT 2016

http://www.nytimes.com/2016/07/02/business/joshua-brown-technology-enthusiast-tested-the-limits-of-his-tesla.html

Brown Victory

Page 23: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Others Had a Different Tone“…filming this just so you can see scenarios where the car does not do well…”

GT 2016GT 2016

Page 24: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

https://www.nytimes.com/2016/07/01/business/self-driving-tesla-fatal-crash-investigation.htmlhttps://www.google.com/maps/@29.4105804,-82.5394011,385m/data=!3m1!1e3

https://www.reuters.com/article/us-tesla-autopilot-dvd-idUSKCN0ZH5BW

7 May 16:40

flyingpenguin

Page 25: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Tesla: “Neither Autopilot nor thedriver noticed the white side ofthe tractor trailer against abrightly lit sky, so the brake wasnot applied.”

GT 2016GT 2016 https://www.teslamotors.com/blog/tragic-loss

Page 26: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

Plausible Theory

flyingpenguinGT 2016GT 2016

Page 27: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

Actual Location

60-0 Tesla Brake Test = 108 ft

“I don’t know why he went over to the slow lane…”

flyingpenguinGT 2016GT 2016

Page 28: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Tesla: “Sensor that did spot truck interpreted as overhead sign.”

GT 2016GT 2016 https://www.teslamotors.com/blog/tragic-loss

Page 29: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

http://www.ntsb.gov/investigations/AccidentReports/Pages/HWY16FH018-preliminary.aspx

“..overhead sign” (more likely a moving bridge)

GT 2016GT 2016flyingpenguin

Page 30: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Truck driver: “Saw him at the top of the hill”

65 mphSpeed Limit

95 fps

1000ft/95fps = 11 seconds

Going 74 mph

“Traffic-Aware Cruise Control and Autosteer lane keeping assistance”

flyingpenguinGT 2016GT 2016

Page 31: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

“Learning” Expectation Gap Kills

GT 2016GT 2016

Actual Features Learned

Expected “Collision Avoidance” Features Learned

Aggressive feature marketing + non-transparency in “beta” OTA programs = ZERO SAFETY MARGIN EVEN FOR EXPERTS

OTA rollback

Page 32: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Segnet for Near Sight

GT 2016GT 2016

“currently labels more than 90% of pixels correctly, according to the researchers…

It’s remarkably good…”

http://mi.eng.cam.ac.uk/projects/segnet/

https://www.youtube.com/watch?v=CxanE_W46ts

Page 33: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

“It’s remarkably good…”The people who made it

Page 34: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

Page 35: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

Holy @#$%!Is this disaster really called...

…more than 90% correct?

“It’s remarkably good”

We’re not in United Kingdom anymore

Page 36: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

MACHINES REPEAT SAME AWFUL MISTAKES FASTER

BUT BUT BUT

GT 2016GT 2016

Page 37: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Basic ML Tool Market DriversLower Cost of Ingredients Increases Adoption

Faster Hardware

Bigger Sets of “Public” Data

Better Algorithms

GT 2016GT 2016

Page 38: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

“…we make technology as brain-dead easy to use as possible…”

-- Alan Eagle

GT 2016GT 2016http://www.nytimes.com/2011/10/23/technology/at-waldorf-school-in-silicon-valley-technology-can-wait.html

Page 39: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Past Automation Tool ImpactMaxim, an egomaniacal draft dodger, gave the world the first true automatic weapon (Patent No

3493 1883). Used by British in Colonial Africa and by Germans in WWI to turn earth into hell. Died proud.

-- C. J. Chivers

GT 2016GT 2016

https://www.worldcat.org/title/gun/oclc/535493119

Page 40: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Past Automation Tool Impact

GT 2016GT 2016

Page 41: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

“Swordsmen and their [cyber] swords”

GT 2016GT 2016

https://www.kaggle.com/tonyliu/d/bmkramer/101-innovations-research-tools-survey/swordsman-and-their-swords-a-tree-model

Page 42: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

/Anyone/ Can Pull The ML Trigger

GT 2016GT 2016https://medium.com/@ageitgey/machine-learning-is-fun-part-4-modern-face-recognition-with-deep-learning-c3cffc121d78

Page 43: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Uhhhhh

“…we have no idea [how it works]. It doesn’t really matter to us…”

GT 2016GT 2016

Page 44: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Shouldn’t We Care A LOT About Transparency?

GT 2016GT 2016

http://www.vocativ.com/345313/hunting-syrian-war-crimes-from-5000-miles-away/http://www.bbc.com/news/world-middle-east-35110877

“…military photographer, using USB sticks hidden inside his shoes, smuggled out 55,000 photos depicting systematic torture of more than 10,000 prisoners by Assad regime”

Page 45: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Facebook “Non-Lethal” Analysis ofTrajectory…Targeting by Prediction

https://www.kaggle.com/c/facebook-v-predicting-check-insGT 2016GT 2016

The goal of this competition is to predict which place a person would like to check in to. For thepurposes of this competition, Facebook created an artificial world consisting of more than 100,000places located in a 10 km by 10 km square. For a given set of coordinates, your task is to return aranked list of the most likely places. Data was fabricated to resemble location signals coming frommobile devices, giving you a flavor of what it takes to work with real data complicated by inaccurateand noisy values. Inconsistent and erroneous location data can disrupt experience for services likeFacebook Check In.

Page 46: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Uber “Non-Fraud” Data Analysis• “We do have access to a

tremendous amount of data…found you'd accept up to 9.9x surge price if your battery's low”

• “…more likely to offer you a journey that costs 2.1X normal than 2X…”

GT 2016GT 2016

http://www.npr.org/2016/05/17/478266839/this-is-your-brain-on-uberhttp://sfist.com/2014/12/15/uber_commands_100_surge_fares_durin.phphttp://www.independent.co.uk/life-style/gadgets-and-tech/news/uber-knows-when-your-phone-is-about-to-run-out-of-battery-a7042416.html

“Uber Commands $100 Surge Fares

During Sydney Hostage Standoff,

Apologizes”

blame our algorithm. no person in charge here. didn’t know what a disaster is...

Page 47: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Delivery “Non-Fatality” Data Analysis

GT 2016GT 2016

Amazon

https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812269

Page 48: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Amazon “Non-Racist” Data Analysis

GT 2016GT 2016 http://www.bloomberg.com/graphics/2016-amazon-same-day/

“Berman says ethnic composition of neighborhoods isn’t part of data Amazon examines when drawing maps.”“historical racial divide”

Page 49: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Pokemon (Don’t) Go

GT 2016GT 2016

“crowdsourced its areas of interest from its players, but its players weren't diverse”

http://www.bnd.com/news/nation-world/national/article89562297.html

Page 50: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

Police Algorithms to Police the Police?

Page 51: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

ProPublica: “Machine Bias”“…compared predicted recidivism to actual recidivism. We found the scores were wrong 40% of the time and biased against black defendants, who were falsely labeled future criminals at almost twice the rate of white defendants.”

GT 2016GT 2016

https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

Page 52: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

SO MANY FAILURES….

GT 2016GT 2016

Page 53: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Ok, Maybe They Got This One Right

GT 2016GT 2016 Face recognition with deep neural networks. http://cmusatyalab.github.io/openface/

Page 54: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Failure

GT 2016GT 2016

Detection Result:Error: “0 face detected”

Face recognition with deep neural networks. http://cmusatyalab.github.io/openface/

Page 55: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

http://www.datasciencecentral.com/profiles/blogs/how-to-lie-with-visualizations-statistics-causation-vs

Page 56: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

HOUSTON, WE HAVE A PROBLEM

GT 2016GT 2016

Page 57: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

http://www.datasciencecentral.com/profiles/blogs/how-to-lie-with-visualizations-statistics-causation-vs

Page 58: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

F@#$%ng Disaster

GT 2016GT 2016 https://twitter.com/jackyalcine/status/615329515909156865https://twitter.com/yonatanzunger/status/615586442413146112

Page 59: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

SOME OBVIOUS REASONSIT’S THE ECONOMICS…

GT 2016GT 2016

Page 60: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Artificial Intelligence: Human Effects

GT 2016GT 2016

“…in general situations AI cannot help at all…”

Page 61: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Example: Google “ML Ethics” Guide• Avoiding Negative Side Effects: How can we ensure that an AI system will not disturb its

environment in negative ways while pursuing its goals, e.g. a cleaning robot knocking over a vase because it can clean faster by doing so?

• Avoiding Reward Hacking: How can we avoid gaming of the reward function? For example, we don’t want this cleaning robot simply covering over messes with materials it can’t see through.

• Scalable Oversight: How can we efficiently ensure that a given AI system respects aspects of the objective that are too expensive to be frequently evaluated during training? For example, if an AI system gets human feedback as it performs a task, it needs to use that feedback efficiently because asking too often would be annoying.

• Safe Exploration: How do we ensure that an AI system doesn’t make exploratory moves with very negative repercussions? For example, maybe a cleaning robot should experiment with mopping strategies, but clearly it shouldn’t try putting a wet mop in an electrical outlet.

• Robustness to Distributional Shift: How do we ensure that an AI system recognizes, and behaves robustly, when it’s in an environment very different from its training environment? For example, heuristics learned for a factory workfloor may not be safe enough for an office.

https://research.googleblog.com/2016/06/bringing-precision-to-ai-safety.htmlGT 2016GT 2016

Page 62: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Google “ML Ethics” Perspective

1. Avoid Cost (Negative Side Effect)2. Avoid Cost (False Finish)3. Avoid Cost (Feedback Burden)4. Avoid Cost (Self Harm)5. Avoid Cost (Expensive Training)

https://research.googleblog.com/2016/06/bringing-precision-to-ai-safety.html

1. Privacy2. Fairness3. Security4. Abuse5. Transparency6. Policy

Google Guide(Formerly Known As “Do No Evil”)

Imperative Categories Missing

GT 2016GT 2016

If You Kant Get It Right…

Page 63: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Tesla Miles Claim Doesn’t Add Up• Given 10.2 deaths per 100,000 people and 1.08

deaths per 100 million vehicle miles• Distracted driver 10% of all fatal crashes

– Visual: taking your eyes off the road;– Manual: taking your hands off the wheel; and– Cognitive: taking your mind off of driving

• Tesla claim of 130M miles is intentionally misleading; doesn’t qualify autopilot oversight

GT 2016GT 2016 http://www.iihs.org/iihs/topics/t/general-statistics/fatalityfacts/state-by-state-overviewhttp://www.cdc.gov/Motorvehiclesafety/Distracted_Driving/index.html

Page 64: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

There’s Data and Then There’s Data

GT 2016GT 2016

NTSB investigations in 2013• 1,750 done for 429 Aviation fatalities (100%)• 40 done for 32,719 Road fatalities (0.0007%)

http://www-fars.nhtsa.dot.gov/Main/index.aspx

2015 1960

Page 65: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Tesla Waited 9 Days to Report

GT 2016GT 2016

“…continue to find parts of the car in their yard eight weeks after the crash”

http://www.dailymail.co.uk/news/article-3677101/Tesla-told-regulators-fatal-Autopilot-crash-nine-days-happened.htmlhttp://www.ntsb.gov/investigations/AccidentReports/Pages/HWY16FH018-preliminary.aspx

Page 66: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Absence of Regulatory Expertise

GT 2016GT 2016

40 States with “too slow” laws

http://www.mit.edu/~jfc/right.html

Page 67: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

NHTSA Automation Levels0. No Automation1. Function Specific Automation

Cede A Control (Foot = Speed)2. Combined Function Automation

Cede Dual Controls (Hand + Foot)3. Limited Self-Driving Automation

Cede Full Control of Safety for Certain Conditions 4. Full Self-Driving Automation

Driver Not Expected to Be Available

GT 2016GT 2016 http://www.nhtsa.gov/staticfiles/rulemaking/pdf/Automated_Vehicles_Policy.pdf

Page 68: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

ZF TRW Automation Levels

GT 2016GT 2016

http://safety.trw.com/autonomous-cars-must-progress-through-these-6-levels-of-automation/0104/

Page 69: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Science: ML Cars Struggling At Level 1

Adversarial Environment Levels1. Freeways (same direction/speed)2. Boulevards (separated)3. Residential (high stakes because people live)4. Urban (compact, pedestrian)

GT 2016GT 2016

Page 70: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

NHTSA Auto Level Environmental Adversity Levels

Freeway Boulevard Residential Urban

0 None

1 Function Specific(Cede a control)

Cruise Control

2 Combined Function(Cede dual control)

Speed Control + Lane Assist

3 Limited Self-Driving (Cede full safetycontrol for certain conditions)

4 Full Self-Driving(Driver not expected)

Tesla “Autopilot”

Page 71: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Google “Disengagements by Location”

GT 2016GT 2016

37 Freeway

Source: Google Self-Driving Car Testing Report (on Disengagements of Autonomous Mode), December 2015, pg. 11

= 90% Street(70% Due to Perception, Hardware

or Software “Discrepancy”)

Page 72: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Google False Victory Celebrations“Driverless 2, Humans 0”

GT 2016GT 2016 https://www.youtube.com/watch?v=Uj-rK8V-rik&feature=youtu.be&t=24m54s

Page 73: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

LET’S FIX THIS

GT 2016GT 2016

Page 74: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Don’t Do This

GT 2016GT 2016

http

s://

twitt

er.c

om/c

at_b

elta

ne/s

tatu

s/58

8359

3541

3640

3969

flyingpenguin

Page 75: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguinGT 2016GT 2016

1637: “Cogito, ergo sum”

1693: Reflective Process, Articulated StepsJohn Locke

(1632-1704)

Rene Descartes(1596-1650)

Trusted Learning System History

http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002106

Page 76: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Require Reflective (Due) Process

GT 2016GT 2016 http://eur-lex.europa.eu/legal-content/EN/ALL/?uri=CELEX%3A32016R0679

Be the Grain of Sand in the ML Oyster

• Accept feedback on both strengths and weaknesses• Question underlying values and beliefs• Recognize bias or discrimination• Challenge assumptions and express fears• Admit possible inadequacies and areas for improvement

Page 77: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Choose Augmentation not Authorimation

GT 2016GT 2016

May 23, 2016 “Caught Sleeping With Autopilot” and not Charged with Unauthorized Transfer of Responsibility

https://imgur.com/E3joXpL

Page 78: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Hold “Algowner” Accountable

GT 2016GT 2016

• Improve the world, not repeat it; science to deal with promotion and preservation of security

• Conditions and practices to promote or preserve security– “Tools” must be properly treated, used & controlled– Only applied to appropriate circumstances– Responsibility (authentic, authorized, accountable)

Page 79: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Trusted Reflective Learning Model• US Doctor a Pioneer in Water Safety

– Educates humans to take control of destiny– Assigns humans to the sensors to keep watch

• Peering/empathy approach replaces “I am from [Fancy Tech Big Corp]. And I am here to tell you what to do, you ignorant people”

GT 2016GT 2016 https://twitter.com/daviottenheimer/status/732131830770491393

Page 80: D Z ] v > v ] v P W - flyingpenguin.com · Z W l l Á Á Á X ( } Æ ] X } u l ï ì ð õ ó ì ô l u r Z r Ç r o ] o r µ ] Ç r } } r Z r } o o ] v P r ] o ] } v r À o o Ç

flyingpenguin

Great Disasters of Machine Learning:Predicting Titanic Events in Our Oceans of Math

Davi Ottenheimerflyingpenguin


µ Ç } v Ç v u ] } v } ( u ] r µ u ] o ( o } ] v P Á ] v µ ...€¦ · µ Ç } v Ç v u ] } v } ( u ] r µ u ] o ( o } ] v P Á ] v µ ] v } v ] ] v P } u ] v Á À v µ v o

µ Ç } v Ç v u ] } v } ( u ] r µ u ] o ( o } ] v P Á ] v µ ...€¦ · µ Ç } v Ç v u ] } v } ( u ] r µ u ] o ( o } ] v P Á ] v µ ] v } v ] ] v P } u ] v Á À v µ v o

Documents
R E F R I G E R A Ç Ã O

R E F R I G E R A Ç Ã O

Documents
6287+ .25($ - goldendeluxe.com.my · Ç / ] v Ç D o , } o í ] À µ v ~ ó ñ í ô t d ì ô ð ñ r D P Ç o } o } µ P µ ] XD0.R&/*û=7^Í2¤

6287+ .25($ - goldendeluxe.com.my · Ç / ] v Ç D o , } o í ] À µ v ~ ó ñ í ô t d ì ô ð ñ r D P Ç o } o } µ P µ ] XD0.R&/*û=7^Í2¤

Documents
µ o ] ] } v ( } W ] v v P ] À } µ W } r µ v t Z Ç d Z Ç D · {W r Ks/ r í õ Z ^E& Z ( Á À ] o o X / v Z Ks/ r í õ U ^E& u Ç Z À o ] u ] ] } v } v Á Z ] v Z Ç v l X

µ o ] ] } v ( } W ] v v P ] À } µ W } r µ v t Z Ç d Z Ç D · {W r Ks/ r í õ Z ^E& Z ( Á À ] o o X / v Z Ks/ r í õ U ^E& u Ç Z À o ] u ] ] } v } v Á Z ] v Z Ç v l X

Documents
< o o Ç < v n u Ç ( i } µ v Ç X } u o / v r ^ u ] v o v ~ î ì î ì h ( } } v o ... · 2020-05-24 · ( } } v o µ } v o Ç Title: Microsoft Word - Clear Case Insert - Stampin

< o o Ç < v n u Ç ( i } µ v Ç X } u o / v r ^ u ] v o v ~ î ì î ì h ( } } v o ... · 2020-05-24 · ( } } v o µ } v o Ç Title: Microsoft Word - Clear Case Insert - Stampin

Documents
r ] l µ d o P Z ^ X r ^ o ( } o o ] } v v µ v Z } u Ç Æ ] r K Z Á o l r ] v l Á Ç · 2020-04-07 · î W Ç u v Ç W ÇE } Áh E W î ì í ô ì ñ î ó í ~h } v µ o µ

r ] l µ d o P Z ^ X r ^ o ( } o o ] } v v µ v Z } u Ç Æ ] r K Z Á o l r ] v l Á Ç · 2020-04-07 · î W Ç u v Ç W ÇE } Áh E W î ì í ô ì ñ î ó í ~h } v µ o µ

Documents
Common Approach Welcome Webinar â Copy of PowerPoint … · î ì í õ r ì ó r í î í ò } v } o } P Ç r u v ] ] v } ] o ] Ç o ( } u

Common Approach Welcome Webinar â Copy of PowerPoint … · î ì í õ r ì ó r í î í ò } v } o } P Ç r u v ] ] v } ] o ] Ç o ( } u

Documents
t Z P Ç Á Z ( } P } o ( } v & ] Ç U } o Ç } v } ( Z Ç } o

t Z P Ç Á Z ( } P } o ( } v & ] Ç U } o Ç } v } ( Z Ç } o

Documents
< o o Ç < v n u Ç ( i } µ v Ç X } u o / v r î ì î í / v

< o o Ç < v n u Ç ( i } µ v Ç X } u o / v r î ì î í / v

Documents
Settlement and Locality Names - calico.scot · Z ] v } µ ( o Ç ( } Ç o o Ç o } µ v Z Ç

Settlement and Locality Names - calico.scot · Z ] v } µ ( o Ç ( } Ç o o Ç o } µ v Z Ç

Documents
W v ] v P Á ] Z W v } W Z Ç ] o o Ç v u } ] } v o o Ç ......v P o & ] Z r^ } o } u } v U E d> ... í ñ r í õ Ç t Z Ç } Ç } µ Z ] v l Z À Ç Ç } µ v P Z ] o v u } À µ

W v ] v P Á ] Z W v } W Z Ç ] o o Ç v u } ] } v o o Ç ......v P o & ] Z r^ } o } u } v U E d> ... í ñ r í õ Ç t Z Ç } Ç } µ Z ] v l Z À Ç Ç } µ v P Z ] o v u } À µ

Documents
&/y / Xd X · ä ä } u µ } U Ç } µ o L Z u Ç æ ä ä ë &/y / Xd X Á Á ] o o } À ] Ç } µ Á ] Z } v r } r } v

&/y / Xd X · ä ä } u µ } U Ç } µ o L Z u Ç æ ä ä ë &/y / Xd X Á Á ] o o } À ] Ç } µ Á ] Z } v r } r } v

Documents
& u ] o Ç r } r & u ] o Ç î ì í õ d Z d ] v ] v P o ... · & u ] o Ç r } r & u ] o Ç î ì í õ d Z d ] v ] v P o ] ] } v P u v dZ /E/E' >K d/KE W z l ] u î r z dZ /E/E

& u ] o Ç r } r & u ] o Ç î ì í õ d Z d ] v ] v P o ... · & u ] o Ç r } r & u ] o Ç î ì í õ d Z d ] v ] v P o ] ] } v P u v dZ /E/E' >K d/KE W z l ] u î r z dZ /E/E

Documents
d}µ W ] }À }o}u ] r oµÆ ~ W W îí^ u îìíóUíí Ç Çí W Çî · ... P} ~^µv Ç ,}o] Ç d} ÇÇ}µÁ]oo l ]v Z ] o}À]v Æ o} Z ] Ç}( }P} Ç ]l XdZ ^ ] o}À] _ l o }v^µv

d}µ W ] }À }o}u ] r oµÆ ~ W W îí^ u îìíóUíí Ç Çí W Çî · ... P} ~^µv Ç ,}o] Ç d} ÇÇ}µÁ]oo l ]v Z ] o}À]v Æ o} Z ] Ç}( }P} Ç ]l XdZ ^ ] o}À] _ l o }v^µv

Documents
v o Ç ] } ( Z o } } µ ] v v / D } o ] Z Ç o Z o } } µ ] v Ç > …...v o Ç ] } ( Z o } } µ ] v v / D } o ] Z Ç o Z o } } µ ] v Ç > rD^ lD^ µ ] v P < ] v Æ } r^

v o Ç ] } ( Z o } } µ ] v v / D } o ] Z Ç o Z o } } µ ] v Ç > …...v o Ç ] } ( Z o } } µ ] v v / D } o ] Z Ç o Z o } } µ ] v Ç > rD^ lD^ µ ] v P < ] v Æ } r^

Documents
} v Ç ] o Ç d ì ï ( } z o z í î Z zD ] o } v Ç ] o Ç d ì ð ......} v Ç ] o Ç d ì ï ( } z o z í î Z zD ] o } v Ç ] o Ç d ì ð ... ... 3, , 3d, , &

} v Ç ] o Ç d ì ï ( } z o z í î Z zD ] o } v Ç ] o Ç d ì ð ......} v Ç ] o Ç d ì ï ( } z o z í î Z zD ] o } v Ç ] o Ç d ì ð ... ... 3, , 3d, , &

Documents
ZKK& dKW >Ky ^W /&/ d/KE^ZKK& dKW >Ky ^W /&/ d/KE^ Z Ç o W } o Ç } Ç o v } } o Ç u } Ç D ] o o l hs ] o ] Ì W } o Ç } Ç o v } } o Ç u X í ì ì } X í ï ñ Á o o Z ] l

ZKK& dKW >Ky ^W /&/ d/KE^ZKK& dKW >Ky ^W /&/ d/KE^ Z Ç o W } o Ç } Ç o v } } o Ç u } Ç D ] o o l hs ] o ] Ì W } o Ç } Ç o v } } o Ç u X í ì ì } X í ï ñ Á o o Z ] l

Documents
7ROOWDULII WLO (8 - Catch Certificate · r r } v } Z Ç v Z µ Z } P K v } Z Ç v Z µ Z Ç } P ô r r v í î r o ~ v P µ ] o o X W

7ROOWDULII WLO (8 - Catch Certificate · r r } v } Z Ç v Z µ Z } P K v } Z Ç v Z µ Z Ç } P ô r r v í î r o ~ v P µ ] o o X W

Documents
VebraAlto.com - Agency Cloud · W } Ç^µuu Ç } } µv] Ç } µÇu} v Z }}u Z Z}µ ]v Z] }µPZ ( µ oÀ]ooP vi}Ç]vP o v µor r o} ]}vUÁZ]o ]vP o} }} v

VebraAlto.com - Agency Cloud · W } Ç^µuu Ç } } µv] Ç } µÇu} v Z }}u Z Z}µ ]v Z] }µPZ ( µ oÀ]ooP vi}Ç]vP o v µor r o} ]}vUÁZ]o ]vP o} }} v

Documents
t >> î ñ r s , Z' Z · ô ò X W } ] } v µ v ] v } l t >> ] µ ] Á ] Z ( Ç ( µ µ ] ( } o ] o ( Ç X / ] ] v v o o Ç }

t >> î ñ r s , Z' Z · ô ò X W } ] } v µ v ] v } l t >> ] µ ] Á ] Z ( Ç ( µ µ ] ( } o ] o ( Ç X / ] ] v v o o Ç }

Documents
d Z ] u ] o Á Ç t s ] P ] v ] D ] o / v µ v ] ] ] µ Ç , o ... · G D^ } o ] Ç X W µ o ] } v E } X r í í^Kt r X í r í î ì ð î ì í ð r ì î HHQI’s Progressive Cardiovascular

d Z ] u ] o Á Ç t s ] P ] v ] D ] o / v µ v ] ] ] µ Ç , o ... · G D^ } o ] Ç X W µ o ] } v E } X r í í^Kt r X í r í î ì ð î ì í ð r ì î HHQI’s Progressive Cardiovascular

Documents
^ Z } } µ } v y r Ç ] ( ( ] } v v o Ç ] } ( Z ] v ( ] o u v µ ( W Courses... · 2019-04-11 · y r Ç v o Ç ] } ( Z ] v ( ] o u v µ ( ] v & d^ / v µ u v WyZ d&yZ W& yZZ '/^

^ Z } } µ } v y r Ç ] ( ( ] } v v o Ç ] } ( Z ] v ( ] o u v µ ( W Courses... · 2019-04-11 · y r Ç v o Ç ] } ( Z ] v ( ] o u v µ ( ] v & d^ / v µ u v WyZ d&yZ W& yZZ '/^

Documents
o ] P ] ] o ] Ç...Á l } P µ o r ] u } i } Z u o } Ç Z µ o } Á } l o î ì Z } µ Á l { ^ ] u X _ / u u v Ç } µ } ] ] } v ] o ] u ] v Ç Z } u v Ç U Z

o ] P ] ] o ] Ç...Á l } P µ o r ] u } i } Z u o } Ç Z µ o } Á } l o î ì Z } µ Á l { ^ ] u X _ / u u v Ç } µ } ] ] } v ] o ] u ] v Ç Z } u v Ç U Z

Documents
À Ç v ] } Ç v Z ] } ( o } Ç Ç o Ç } v ( } u Z } o ] P v r ... · 8/10/2020  · +3/& h[dplqdwlrq ri hwk\o dfhwdwh h[wudfwv ri &ur fxowxuh uh yhdohg wkh suhvhqfh ri vhyhudo shdnv

À Ç v ] } Ç v Z ] } ( o } Ç Ç o Ç } v ( } u Z } o ] P v r ... · 8/10/2020  · +3/& h[dplqdwlrq ri hwk\o dfhwdwh h[wudfwv ri &ur fxowxuh uh yhdohg wkh suhvhqfh ri vhyhudo shdnv

Documents
B ' Ì ] ] P r < o Ç ( ] l i µ Ï Ç v · /// B ' Ì ] ] P r < o Ç ( ] l i µ Ï Ç v ^ Ì Ì P s Ç. Title: Kopia SumaCzasDrużynSzczegóŠy1 Author: Robert Created Date

B ' Ì ] ] P r < o Ç ( ] l i µ Ï Ç v · /// B ' Ì ] ] P r < o Ç ( ] l i µ Ï Ç v ^ Ì Ì P s Ç. Title: Kopia SumaCzasDrużynSzczegóŠy1 Author: Robert Created Date

Documents
d Z E Æ D i } / v ] v & ] o ] Ç t ] o o D /E WK/Ed í K µ Ç ... · r r r r r r r r/ u u µ v Ç u r r r r r r ^ Z o } v o X U ] } o X Z } X ô í W í ì î ñ ~ î ì ì õ ,QFLGHQFH

d Z E Æ D i } / v ] v & ] o ] Ç t ] o o D /E WK/Ed í K µ Ç ... · r r r r r r r r/ u u µ v Ç u r r r r r r ^ Z o } v o X U ] } o X Z } X ô í W í ì î ñ ~ î ì ì õ ,QFLGHQFH

Documents
s o ] Ç ~Dd lW o ] À Ç lW K Á v...2020/03/02  · s o ] Ç ~Dd lW o ] À Ç lW K Á v ^ l d v W > í ò } o o Ç Y µ Ç · ì ó r ì î U ^ ] v P } ì ð õ ï í ô d o W =

s o ] Ç ~Dd lW o ] À Ç lW K Á v...2020/03/02  · s o ] Ç ~Dd lW o ] À Ç lW K Á v ^ l d v W > í ò } o o Ç Y µ Ç · ì ó r ì î U ^ ] v P } ì ð õ ï í ô d o W =

Documents
} } o ^ Z^ r }s r î ] v Z } ] o v o } v P r u Z } u µ ] v P Z Ks/ r í õ v ......2021/05/31  · ô î E ] } v o D ] } ] } o } P Ç > } } Ç U W µ o ] , o Z P v Ç } ( v U t ]

} } o ^ Z^ r }s r î ] v Z } ] o v o } v P r u Z } u µ ] v P Z Ks/ r í õ v ......2021/05/31  · ô î E ] } v o D ] } ] } o } P Ç > } } Ç U W µ o ] , o Z P v Ç } ( v U t ]

Documents
zE u ^ ] } µ v Ç ^ ] d Ç d v } v } µ v Ç , ^€¦ · o l & } Æ o u v Ç o Ç } µ v Ç rZ ] l ( Z } } o v & ] À ] Á } Ç v ' ] o o µ ^ } } µ v Ç rZ ] l ( Z } } o v

zE u ^ ] } µ v Ç ^ ] d Ç d v } v } µ v Ç , ^€¦ · o l & } Æ o u v Ç o Ç } µ v Ç rZ ] l ( Z } } o v & ] À ] Á } Ç v ' ] o o µ ^ } } µ v Ç rZ ] l ( Z } } o v

Documents
LVW 2FWREHU ^ ] E u } µ v Ç µ Ç ] v P ' } µ ~ } d h v ... · ^ ] E u } µ v Ç µ Ç ] v P ' } µ ~ } d h v ] } v D o & u o d } o r î ó r ' ] }/ o Ç s ] ] v U ð ð U ' ]

LVW 2FWREHU ^ ] E u } µ v Ç µ Ç ] v P ' } µ ~ } d h v ... · ^ ] E u } µ v Ç µ Ç ] v P ' } µ ~ } d h v ] } v D o & u o d } o r î ó r ' ] }/ o Ç s ] ] v U ð ð U ' ]

Documents