29
Software Complexity Steven M. Bellovin https://www.cs.columbia.edu/~smb 1

Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

  • Upload
    others

  • View
    40

  • Download
    0

Embed Size (px)

Citation preview

Page 2: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

WhatHappened?

• Whyisatrainarrivingin-2minutes?• Isthe10:26runningaheadofthe9:38?• (We’llignorethefactthatthey’rebothquitelate.)

2

Page 3: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

ATrainStatusDisplay

3

Page 4: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

AndinWashington…

4

Page 5: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

Optimism

“Theprogrammer,likethepoet,worksonlyslightlyremovedfrompurethought-stuff.Hebuildshiscastlesintheair,fromair,creatingbyexertionoftheimagination.Fewmediaofcreationaresoflexible,soeasytopolishandrework,soreadilycapableofrealizinggrandconceptualstructures.”

FredBrooks,TheMythicalMan-Month

5

Page 6: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

RealityCheck

“[O]nemustperformperfectly.Thecomputerresemblesthemagicoflegendinthisrespect,too.Ifonecharacter,onepause,oftheincantationisnotstrictlyinproperform,themagicdoesn’twork.Humanbeingsarenotaccustomedtobeingperfect,andfewareasofhumanactivitydemandit.”

FredBrooks,TheMythicalMan-Month

6

Page 7: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

RealSoftware

• Softwareisbuggy• Softwareisalwaysbuggy• Thebiggertheprogram,thebuggierthesoftware—always

7

Page 8: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

Why?Interactions

8

Page 9: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

ComplexityKills…

Withncomponents,thereareroughlyn2interactions

9

Page 10: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

So?

• Therearelimitstohowgoodoursoftwarecanbe• Therearethereforethingswecan’tdo• Moreprecisely,whenweincreasecomplexitywe

a) Increasethebugratedramaticallyb) Increasethedevelopmentcostdramaticallyc) Both!

10

Page 11: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

ComplexityandCurrentEvents

• Webankonline• Webuythingsonline• Wecommunicateonline• Whycan’twevoteonline?

11

Page 12: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

ProbablyJustaBug

(PhotobyEdFelten) 12

Avotingmachinetapefromthe2008presidentialprimaryinaNewJerseyprecinct.

Page 13: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

EntertheAdversary

13NYTimes

Page 14: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

GoingDark

“Asaresult,althoughthegovernmentmayobtainacourtorderauthorizingthecollectionofcertaincommunications,itoftenservesthatorderonaproviderwhodoesnothaveanobligationunderCALEAtobepreparedtoexecuteit.”

ValerieCaproni,GeneralCounseloftheFBI

14

Page 15: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

TheFBI’sSolution

• Allcommunicationsystemsneedsomeformofaccessforlawenforcement• Allencryptionsystemsneeda“backdoor”(whichtheycalla“goldenkey”)• Canwedoit?

15

Page 16: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

WiretapInterfacesareHard

• Someyearsago,theNSAevaluatedthestandardizedwiretapinterfaceon26differentphoneswitches• Allhadsecurityflaws

• Someone(probablyanintelligenceagency)hackedacellphoneswitchinAthensandabusedthewiretapinterface• About100phoneswereillegallytapped,includingthePrimeMinister’s

16

Page 17: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

CryptographyisHard

“Finally,protocolssuchasthosedevelopedherearepronetoextremelysubtleerrorsthatareunlikelytobedetectedinnormaloperation.”

RogerNeedhamandMichaelSchroeder,“UsingEncryptionforAuthenticationinLargeNetworksofComputers”

17

Page 18: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

From“KeysUnderDoormats”

“Wehavefoundthatthedamagethatcouldbecausedbylawenforcementexceptionalaccessrequirementswouldbeevengreatertodaythanitwouldhavebeen20yearsago.Inthewakeofthegrowingeconomicandsocialcostofthefundamentalinsecurityoftoday’sInternetenvironment,anyproposalsthatalterthesecuritydynamicsonlineshouldbeapproachedwithcaution...Thecomplexityoftoday’sInternetenvironment,withmillionsofappsandgloballyconnectedservices,meansthatnewlawenforcementrequirementsarelikelytointroduceunanticipated,hardtodetectsecurityflaws.”

Abelsonetal.

18

Page 19: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

WhyTechnologistsOpposeGoldenKeys

• IthasnothingtodowithdislikeoftheFBIortheNSA• Technologistscanbevictimsofcriminalsandterrorists,too• Rather,it’saquestionofcrimeprevention—thesoftwarenecessarytopermitlawenforcementaccesshasahighprobabilityofopeningupnewsecurityholes• Therootcauseisthecomplexityofsoftware

19

Page 20: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

TheInternetofThings

• We’reconnectingmoreandmore“things”totheInternet• Theserunonsoftware;thissoftwareisoftenpoorlywrittenandneverpatched

20

Page 21: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

Self-DrivingCars

• Almostcertainly,wewillseesomecrashesduetobuggycode• Possibly(thoughnotcertainly),therewillbecrashesduetohacking

• Eventoday’s“dumb”carscontain50-75networkedcomputers• Amoderncarisactuallyamobiledatacenter!

• But—self-drivingcars,flawsandall,willalmostcertainlybesaferthanhuman-drivencars• Carsdon’tgetdrunk,sleepy,distracted,etc.

21

Page 22: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

UsersDon’tSeeMostoftheComplexity

• Goodsoftwareoftenhideshowcomplexitis• But—thecomplexityisstillthere• Often,it’sthepartsyoudon’tknowaboutthatcancausethemosttrouble

22

Page 23: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

SoWhatDoWeDo?

• Giveup?

23

Page 24: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

SoWhatDoWeDo?

• Giveup?• No;thatsacrificesthebenefitsofcomputers.Therearereasons(andgenerallygoodones)whywerelyonsoftware

24

Page 25: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

SoWhatDoWeDo?

• Giveup?• No;thatsacrificesthebenefitsofcomputers.Therearereasons(andgenerallygoodones)whywerelyonsoftware• Often,somesmallrateoffailureisquiteacceptable—nothingelseisperfect,either

25

Page 26: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

SoWhatDoWeDo?

• Giveup?• No;thatsacrificesthebenefitsofcomputers.Therearereasons(andgenerallygoodones)whywerelyonsoftware• Often,somesmallrateoffailureisquiteacceptable—nothingelseisperfect,either• Thetrickisknowinghowtodecide.Wewantmajorbenefits,comparativelylowrisks,andacceptableconsequencesifthereisafailure

26

Page 27: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

“Thecompetentprogrammerisfullyawareofthestrictlylimitedsizeofhisownskull;thereforeheapproachestheprogrammingtaskinfullhumility…”

EdsgerDijkstra,“TheHumbleProgrammer”

27

Page 28: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

SomeSuggestions

Good

• Self-drivingcars• Communicationsapps

• Thesmartgrid?

Bad

• (Residential)lightbulbs• Bikelocks• Anti-missilesystems• Votingmachines

• Networkedsextoys?

28

Page 29: Software Complexitysmb%c2%a0%c2%a0%c2%a0%c2%a0%c2%a0/tal… · Why Technologists Oppose Golden Keys • It has nothing to do with dislike of the FBI or the NSA • Technologists can

Questions?

29