Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
AlgorithmicSpeed
BBM101- Introduction toProgramming I
Hacettepe UniversityFall2015
FuatAkal,AykutErdem,Erkut Erdem,Vahid Garousi
1Slides basedonmaterialpreparedbyE.Grimson, J.Guttag andC.Terman inMITx 6.00.1x
Measuringcomplexity
• Goalsindesigningprograms1. Itreturnsthecorrectansweronalllegalinputs2. Itperformsthecomputationefficiently
• Typically(1)ismostimportant,butsometimes(2)isalsocritical,e.g.,programsforcollisiondetection
• Evenwhen(1)ismostimportant,itisvaluabletounderstandandoptimize(2)
2
Computationalcomplexity
• Howmuchtimewillittakeaprogramtorun?• Howmuchmemorywillitneedtorun?
• Needtobalanceminimizingcomputationalcomplexitywithconceptualcomplexity– Keepcodesimpleandeasytounderstand,butwherepossibleoptimizeperformance
3
Howdowemeasurecomplexity?
• Givenafunction,wouldliketoanswer:“Howlongwillthistaketorun?”
• Couldjustrunonsomeinputandtimeit.• Problemisthatthisdependson:
1. Speedofcomputer2. SpecificsofPythonimplementation3. Valueofinput
• Avoid(1)and(2)bymeasuringtimeintermsofnumberofbasicstepsexecuted
4
Measuringbasicsteps
• Usearandomaccessmachine(RAM)asmodelofcomputation– Stepsareexecutedsequentially– Stepisanoperationthattakesconstanttime• Assignment• Comparison• Arithmeticoperation• Accessingobjectinmemory
• Forpoint(3),measuretimeintermsofsizeofinput
5
Butcomplexitymightdependonvalueofinput?
def linearSearch(L, x): for e in L:
if e==x: return True
return False
• IfxhappenstobenearfrontofL,thenreturnsTruealmostimmediately
• IfxnotinL,thencodewillhavetoexamineallelementsofL
• Needageneralwayofmeasuring
6
Casesformeasuringcomplexity• Bestcase:minimumrunningtimeoverallpossibleinputsofagivensize– ForlinearSearch – constant,i.e.independentofsizeofinputs
• Worstcase:maximumrunningtimeoverallpossibleinputsofagivensize– ForlinearSearch – linearinsizeoflist
• Average(orexpected)case:averagerunningtimeoverallpossibleinputsofagivensize
• Wewillfocusonworstcase– akindofupperboundonrunningtime
7
Example• Numberofsteps– 1 (forassignment)– 5*n (1fortest,plus2forfirstassignment,plus2forsecondassignmentinwhile;repeatedn<mesthroughwhile)
– 1 (forreturn)• 5*n+2steps• Butasngetslarge,2isirrelevant,sobasically5*nsteps
8
def fact(n): answer = 1 while n > 1:
answer *= n n -= 1
return answer
Example
• Whataboutthemultiplicativeconstant(5inthiscase)?
• Wearguethatingeneral,multiplicativeconstantsarenotrelevantwhencomparingalgorithms
9
Exampledef sqrtExhaust(x, eps):
step = eps**2 ans = 0.0 while abs(ans**2 - x) >= eps and ans <= max(x, 1):
ans += step
return ans
• Ifwecallthison100and0.0001,willtakeonebillioniterationsoftheloop– Haveroughly8stepswithineachiteration
10
Exampledef sqrtBi(x, eps):
low = 0.0high = max(1, x) ans = (high + low)/2.0 while abs(ans**2 - x) >= eps:
if ans**2 < x: low = ans
else: high = ans
ans = (high + low)/2.0 return ans
• Ifwecallthison100and0.0001,willtakethirtyiterationsoftheloop– Haveroughly10stepswithineachiteration
• 1billionor8billionversus30or300– itissizeofproblemthatmatters
11
Measuringcomplexity
• Giventhisdifferenceiniterationsthroughloop,multiplicativefactor(numberofstepswithinloop)probablyirrelevant
• Thus,wewillfocusonmeasuringthecomplexityasafunctionofinputsize–Willfocusonthelargestfactorinthisexpression–Willbemostlyconcernedwiththeworstcasescenario
12
Asymptoticnotation
• Needaformalwaytotalkaboutrelationshipbetweenrunningtimeandsizeofinputs
• Mostlyinterestedinwhathappensassizeofinputsgetsverylarge,i.e.approachesinfinity
13
Exampledef f(x):
for i in range(1000): ans = i
for i in range(x): ans += 1
for i in range(x): for j in range(x):
ans += 1
Complexityis1000+2x+2x2,ifeachlinetakesonestep
14
Example
• 1000+2x+2x2
• Ifxissmall,constanttermdominates– E.g.,x=10then1000of1220stepsareinfirstloop
• Ifxislarge,quadratictermdominates– E.g.x=1,000,000,thenfirstlooptakes0.000000005%oftime,secondlooptakes0.0001%oftime(outof2,000,002,001,000 steps)!
15
Example• Soreallyonlyneedtoconsiderthenestedloops(quadraticcomponent)
• Doesitmatterthatthisparttakes2x2 steps,asopposedtosayx2 steps?– Forourexample,ifourcomputerexecutes100millionstepspersecond,differenceis5.5hoursversus2.25hours
– Ontheotherhandifwecanfindalinearalgorithm,thiswouldruninafractionofasecond
– Somultiplicativefactorsprobablynotcrucial,butorderofgrowthiscrucial
16
Rulesofthumbforcomplexity
• Asymptoticcomplexity– Describerunningtimeintermsofnumberofbasicsteps
– Ifrunningtimeissumofmultipleterms,keeponewiththelargestgrowthrate
– Ifremainingtermisaproduct,dropanymultiplicativeconstants
• Use“BigO”notation(akaOmicron)– Givesanupperboundonasymptoticgrowthofafunction
17
Complexityclasses
• O(1)denotesconstantrunningtime• O(logn)denoteslogarithmicrunningtime• O(n)denoteslinearrunningtime• O(nlogn)denoteslog-linearrunningtime• O(nc)denotespolynomialrunningtime(cisaconstant)
• O(cn)denotesexponentialrunningtime(cisaconstantbeingraisedtoapowerbasedonsizeofinput)
18
Constantcomplexity
• Complexityindependentofinputs• Veryfewinterestingalgorithmsinthisclass,butcanoftenhavepiecesthatfitthisclass
• Canhaveloopsorrecursivecalls,butnumberofiterationsorcallsindependentofsizeofinput
19
Logarithmiccomplexity
• Complexitygrowsaslogofsizeofoneofitsinputs
• Example:– Bisectionsearch– Binarysearchofalist
20
Logarithmiccomplexity
21
def intToStr(i): digits = '0123456789' if i == 0:
return '0' result = '' while i > 0:
result = digits[i%10] + result i = i/10
return result
Logarithmiccomplexity• Onlyhavetolookatloopasnofunctioncalls
• Withinwhileloopconstantnumberofsteps
• Howmanytimesthroughloop?– Howmanytimescanonedividei by10?
– O(log(i))
22
def intToStr(i): digits = '0123456789' if i == 0:
return '0' result = '' while i > 0:
result = digits[i%10] + result
i = i/10 return result
Linearcomplexity• Searchingalistinordertoseeifanelementispresent
• Addcharactersofastring,assumedtobecomposedofdecimaldigits
def addDigits(s): val = 0 for c in s:
val += int(c)
return val
• O(len(s))23
Linearcomplexity• Complexitycandependonnumberofrecursivecalls
def fact(n): if n == 1:
return 1 else:
return n*fact(n-1)
• Numberofrecursivecalls?– Fact(n),thenfact(n-1),etc.untilgettofact(1)– Complexityofeachcallisconstant– O(n) 24
Log-linearcomplexity
• Manypracticalalgorithmsarelog-linear• Verycommonlyusedlog-linearalgorithmismergesort
• Willreturntothis
25
Polynomialcomplexity
• Mostcommonpolynomialalgorithmsarequadratic,i.e.,complexitygrowswithsquareofsizeofinput
• Commonlyoccurswhenwehavenestedloopsorrecursivefunctioncalls
26
Quadraticcomplexity
27
def isSubset(L1, L2): for e1 in L1:
matched = False for e2 in L2:
if e1 == e2: matched = True break
if not matched: return False
return True
Quadraticcomplexity
• Outerloopexecutedlen(L1)times
• Eachiterationwillexecuteinnerloopuptolen(L2)times
• O(len(L1)*len(L2))• WorstcasewhenL1
andL2samelength,noneofelementsofL1inL2
• O(len(L1)2)28
def isSubset(L1, L2): for e1 in L1:
matched = False for e2 in L2:
if e1 == e2: matched = True break
if not matched: return False
return True
Quadraticcomplexity
Findintersectionoftwolists,returnalistwitheachelementappearingonlyonce
29
def intersect(L1, L2): tmp = [] for e1 in L1:
for e2 in L2: if e1 == e2:
tmp.append(e1) res = [] for e in tmp:
if not(e in res): res.append(e)
return res
Quadraticcomplexity
• Firstnestedlooptakeslen(L1)*len(L2)steps
• Secondlooptakesatmostlen(L1)steps
• Lattertermoverwhelmedbyformerterm
• O(len(L1)*len(L2))
30
def intersect(L1, L2): tmp = [] for e1 in L1:
for e2 in L2: if e1 == e2:
tmp.append(e1) res = [] for e in tmp:
if not(e in res): res.append(e)
return res
Exponentialcomplexity
• Recursivefunctionswheremorethanonerecursivecallforeachsizeofproblem– TowersofHanoi
• Manyimportantproblemsareinherentlyexponential– Unfortunate,ascostcanbehigh–Willleadustoconsiderapproximatesolutionsmorequickly
31
Exponentialcomplexity
32
def genSubsets(L): res = [] if len(L) == 0:
return [[]] #list of empty list smaller = genSubsets(L[:-1]) # get all subsets without last element extra = L[-1:] # create a list of just last element new = [] for small in smaller:
new.append(small+extra) # for all smaller solutions, add one with last element return smaller+new# combine those with last element and those without
Exponentialcomplexity
• Assumingappendisconstanttime
• Timeincludestimetosolvesmallerproblem,plustimeneededtomakeacopyofallelementsinsmallerproblem
33
def genSubsets(L): res = [] if len(L) == 0:
return [[]] smaller = genSubsets(L[:-1]) extra = L[-1:] new = [] for small in smaller:
new.append(small+extra) return smaller+new
Exponentialcomplexity
• Butimportanttothinkaboutsizeofsmaller
• Knowthatforasetofsizekthereare2kcases
• Sotosolveneed2n-1 +2n-2 +...+20 steps
• MathtellsusthisisO(2n)
34
def genSubsets(L): res = [] if len(L) == 0:
return [[]] smaller = genSubsets(L[:-1]) extra = L[-1:] new = [] for small in smaller:
new.append(small+extra) return smaller+new
Complexityclasses
• O(1)denotesconstantrunningtime• O(logn)denoteslogarithmicrunningtime• O(n)denoteslinearrunningtime• O(nlogn)denoteslog-linearrunningtime• O(nc)denotespolynomialrunningtime(cisaconstant)
• O(cn)denotesexponentialrunningtime(cisaconstantbeingraisedtoapowerbasedonsizeofinput)
35
Comparingcomplexities
• Sodoesitreallymatterifourcodeisofaparticularclassofcomplexity?
• Dependsonsizeofproblem,butforlargescaleproblems,complexityofworstcasemakesadifference
36
ConstantversusLogarithmic
37
Constant*versus*logarithmic*
Observations
• Alogarithmicalgorithmisoftenalmostasgoodasaconstanttimealgorithm
• Logarithmiccostsgrowveryslowly
38
LogarithmicversusLinear
39
Logarithmic*versus*Linear*
Observations
• Logarithmicclearlybetterforlargescaleproblemsthanlinear
• Doesnotimplylinearisbad,however
40
LinearversusLog-linear
41
Linear*versus*LogFlinear*
Observations
• Whilelog(n)maygrowslowly,whenmultipliedbyalinearfactor,growthismuchmorerapidthanpurelinear
• O(nlogn)algorithmsarestillveryvaluable
42
Log-linearversusQuadratic
43
LogFlinear*versus*Quadra/c*
Observations
• Quadraticisoftenaproblem,however.• Someproblemsinherentlyquadraticbutifpossiblealwaysbettertolookformoreefficientsolutions
44
QuadraticversusExponential• Exponentialalgorithmsveryexpensive– Rightplotisonalogscale,sinceleftplotalmostinvisiblegivenhowrapidlyexponentialgrows
• Exponentialgenerallynotofuseexceptforsmallproblems
45
Quadra/c*versus*Exponen/al*
• Exponen/al*algorithms*very*expensive*– Right*plot*is*on*a*log*scale,*since*leD*plot*almost*invisible*given*how*rapidly*exponen/al*grows*
• Exponen/al*generally*not*of*use*except*for*small*problems**