Lexical Analysis, III - Rice University · source IR IR code target code … Chapter 2 in EaC2e Ignore § 2.4.4 in EaC2e. Read the replacement sec?on ... NFA → DFA (Subset construcon)

LexicalAnalysis,III

Comp412

COMP412FALL2017

Copyright2017,KeithD.Cooper&LindaTorczon,allrightsreserved.StudentsenrolledinComp412atRiceUniversityhaveexplicitpermissiontomakecopiesofthesematerialsfortheirpersonaluse.

FacultyfromothereducaMonalinsMtuMonsmayusethesematerialsfornonprofiteducaMonalpurposes,providedthiscopyrightnoMceispreserved.

FrontEnd OpMmizer BackEnd

IR IRsourcecode

targetcode

…

Chapter2inEaC2e

Ignore§2.4.4inEaC2e.Readthereplacementsec?onpostedonthecoursewebsite.

ThePlanforScannerConstrucMon

RE→NFA(Thompson’sconstruc.on)✔ü•  BuildanNFAforeachtermintheRE•  CombinetheminpaSernsthatmodeltheoperators

NFA→DFA(Subsetconstruc.on)✔ü•  BuildaDFAthatsimulatestheNFA

DFA→MinimalDFA←•  HopcroV’salgorithm•  Brzozowski’salgorithmMinimalDFA→Scanner•  See§2.5inEaC2eDFA→RE•  Allpairs,allpathsproblem•  Uniontogetherpathsfroms0toafinalstate

COMP412,Fall2017 1

minimalDFARE NFA DFA

TheCycleofConstruc.ons

Scanner

DFAMinimizaMonTheBigPicture•  DiscoversetsofbehaviorallyequivalentstatesintheDFA•  RepresenteachsuchsetwithasinglenewstateTwostatessiandsjarebehaviorallyequivalentifandonlyif:

•  ∀c∈Σ,transiMonsfromsi&sjoncleadtoequivalentstates•  Thesetofpathsleadingfromsi&sjareequivalent

Apar??onPofasetS:•  AcollecMonofsubsetsofPsuchthateachstatesisinexactlyonepi∈P•  ThealgorithmiteraMvelyconstructsparMMonsoftheDFA’ssetofstates

WewantaparMMonP={p0,p1,p2,…pn}ofDthathastwoproperMes:1.  Ifdi&dj∈psandctakesdi➝dxanddj➝dy,thendx&dy∈pt,∀c,i,j,s,t2.  Ifdi&dj∈psanddi∈Fthendj∈F

COMP412,Fall2017 2DisthesetofstatesfortheDFA:(D,Σ,δ,s0,DA)

Recursivedefini.on

Detailsofthealgorithm•  Groupstatesintomaximally-sizediniMalsets,op.mis.cally•  IteraMvelysubdividethosesets,basedontransiMongraph•  Statesthatremaingroupedtogetherareequivalent

Ini?alpar??on:P0hastwosets:{DA}&{D–DA}

Property1providesthebasisforrefining,orspliOng,thesets•  Assumesi&sj∈ps,andδ(si,a)=sx,&δ(sj,a)=sy•  Ifsx&syarenotinthesamesetpt,thenpsmustbesplit

–  COROLLARY:sihastransiMonona,sjdoesnot⇒asplitsps

•  AsinglestateinaDFAcannothavetwotransiMonsona–  EachpswillbecomeaDFAstate

finalstates

otherstates

Maximallysizedsets⇒minimalnumberofsetsDFAMinimizaMon

COMP412,Fall2017 3

(property1)

D=(D,Σ,δ,s0,DA)

(property2)

DFAMinimizaMonAlgorithm(Worklistversion)

Worklist←{DA,{D–DA}}Par..on←{DA,{D–DA}}While(Worklist≠∅)do

selectasetSfromWorklistandremoveitforeachα∈Σdo

Image←{x|δ(x,α)∈S} foreachq∈Par..ondo p1←q∩Image p2←q–p1 ifp1≠∅andp2≠∅then removeqfromPar..on Par..on←Par..on∪p1∪p2 ifq∈Worklistthen removeqfromWorklist Worklist←Worklist∪p1∪p2

elseif|p1|≤|p2| thenWorklist←Worklist∪p1 elseWorklist←Worklist∪p2

ImageisthesetofstatesthathaveatransiMonintoSonα:δ-1(S,α)

“splitq”

adjustWorklist

COMP412,Fall2017 4

p1isthesubsetofqthattransi.onstoSonαp2istherestofq

KeyIdea:SpliqngQAroundTransiMonsonα

sα

Asthealgorithmconsiderssandα ,itwillsplitq.

Par??oningQaroundS

α

α

Assumethatq,r,s,&taresetsinthecurrentapproximaMontothefinalparMMonqhastransiMonsonα tor,s,&t,soitmustsplitaroundα

r

t

COMP412,Fall2017 5

q

KeyIdea:Spliqngqaroundsandα

p1 (Image(s,α) ∩q)

s α

p2musthaveanα-transi.ontooneormoreotherstatesinoneormoreotherpar..ons(e.g.,r&s),orstateswithnoα-transi.ons.Otherwise,qdoesnotsplit!

Findmaximalsubsetofq(p1)thathasanα-transi?onintos

q

Thinkofp1astheimageofsintoqundertheinverseofthetransiMonfuncMon:

p1 ←δ –1(s,α)∩q

p2 = q – p1

COMP412,Fall2017 6

7






p1isthesubsetofqthattransi.onstoSonαp2istherestofq

“splitq”

adjustWorklist

COMP412,Fall2017

ProjecMonisthesetofstatesthathaveatransiMonintoSonα:δ-1(S,α)

And,asanimplementaMonnit,ifwejustsplitS—thatis,Swasq&itsplit—weneedanewS






Ifqisasingleton,wecanskipthebodyoftheloopbecauseasingletoncannotsplit.

Onelasthack…

8COMP412,Fall2017

ADetailedExample

TheDFAfor(a|b)*abb

•  DeterminisMcversionofNFAfromlastlecture•  SpecificallynottheminimalDFA•  Usesamecodeskeletonasbefore

s0a

s1

b

s3b

s4

s2

a

b

b

a

a

a

b

Character

State a b

s0 s1 s2

s1 s1 s3

s2 s1 s2

s3 s1 s4

s4 s1 s2

COMP412,Fall2017 10

ADetailedExample

SpliOngaPar??on

•  Thealgorithmstartsoutwith{{s0,s1,s2,s3},{s4}}•  Howdoes{s4}split{s0,s1,s2,s3}?

–  Ona,noedgesrunfrom{s0,s1,s2,s3}to{s4},sonothingsplits

s0a

s1

b

s3b

s4

s2

a

b

b

a

a

a

b

COMP412,Fall2017 11

ADetailedExample

SpliOngaPar??on


–  Onb,{s0,s1,s2,s3}hasedgesintoboth{s4}and{s0,s1,s2,s3},so{s4}splits{s0,s1,s2,s3}into{s0,s1,s2}and{s3}➝  {s0,s1,s2}→{s0,s1,s2}onb➝  {s3}→{s4}onb

s0a

s1

b

s3b

s4

s2

a

b

b

a

a

a

b

COMP412,Fall2017 12

ADetailedExample

SpliOngaPar??on


–  Onb,{s0,s1,s2,s3}hasedgesintoboth{s4}and{s0,s1,s2,s3},so{s4}splits{s0,s1,s2,s3}into{s0,s1,s2}and{s3}➝  {s0,s1,s2}→{s0,s1,s2}onb➝  {s3}→{s4}onb

s0a

s1

b

s3b

s4

s2

a

b

b

a

a

a

b

Now,everystatein{s3}hasthesametransiMononb•  Singletonset⇒sametransiMon•  Neither{s3}nor{s4}canbesplit•  {s4}causesnomoresplits•  {s3}willsplit{s0,s1,s2}into{s0,s1}and{s2}

Notethatwhenwesplit{s0,s1,s2,s3}around{s4},weleVbehindmorework—theresulMngset,{s0,s1,s2},couldbesplitfurther.

Inthealgorithm,{s3}endsupontheworklist,whereitwilllatersplit{s0,s1,s2}

COMP412,Fall2017 13

DetailedExample

s0a s1

b

s3b s4

s2a

b

b

a

a

a

b

CurrentPar..on Worklist s Splitona Splitonb

0 {s4}{s0,s1,s2,s3} {s4}{s0,s1,s2,s3}

Exampleinthistabularformatisfortheworklistversionofthealgorithm.

COMP412,Fall2017 14

DetailedExample

s0a s1

b

s3b s4

s2a

b

b

a

a

a

b


0 {s4}{s0,s1,s2,s3} {s4}{s0,s1,s2,s3} {s4} none

COMP412,Fall2017 15

DetailedExample

s0a s1

b

s3b s4

s2a

b

b

a

a

a

b


0 {s4}{s0,s1,s2,s3} {s4}{s0,s1,s2,s3} {s4} none {s3}{s0,s1,s2}

COMP412,Fall2017 16

17

DetailedExample

s0a s1

b

s3b s4

s2a

b

b

a

a

a

b



1 {s4}{s3}{s0,s1,s2} {s3}{s0,s1,s2}

COMP412,Fall2017

DetailedExample

s0a s1

b

s3b s4

s2a

b

b

a

a

a

b



1 {s4}{s3}{s0,s1,s2} {s3}{s0,s1,s2} {s3} none

COMP412,Fall2017 18

DetailedExample

s0a s1

b

s3b s4

s2a

b

b

a

a

a

b



1 {s4}{s3}{s0,s1,s2} {s3}{s0,s1,s2} {s3} none {s1}{s0,s2}

COMP412,Fall2017 19

DetailedExample

s0a s1

b

s3b s4

s2a

b

b

a

a

a

b




2 {s4}{s3}{s1}{s0,s2} {s1}{s0,s2}

COMP412,Fall2017 20

DetailedExample

s0a s1

b

s3b s4

s2a

b

b

a

a

a

b




2 {s4}{s3}{s1}{s0,s2} {s1}{s0,s2} {s1} none none

COMP412,Fall2017 21

DetailedExample

s0a s1

b

s3b s4

s2a

b

b

a

a

a

b





3 {s4}{s3}{s1}{s0,s2} {s1}{s0,s2} {s0,s2} none none

Emptyworklist⇒done!

COMP412,Fall2017 22

DetailedExample





3 {s4}{s3}{s1}{s0,s2} {s1}{s0,s2} {s0,s2} none none

s0a s1

b

s3b s4

s2a

b

b

a

a

a

b

s0,s2a s1

b

s3b s4

ba

a

a

b

20%reduc?oninnumberofstatesCOMP412,Fall2017 23






Whydoesthisalgorithmhalt?•  Fixed-pointalgorithm•  DFAhasfinitenumberofstates•  Startwith2setsinParMMon•  Spliqngbreaks1setinto2smalleronesbutnevermakesasetlarger→ Monotonebehavior

•  Simple,finitelimiton|Par..on|;itcannotbe>|States|

•  Finite#steps,monotoneincreasingconstrucMon⇒algorithmhalts

COMP412,Fall2017 24

DFAMinimizaMon

Whatabouta(b|c)*?

First,thesubsetconstrucMon:

q0 q1a ε

q4 q5b

q6 q7c

q3 q8q2 q9

ε

ε

ε ε

ε ε

ε ε

s3

s2

s0 s1c

ba

b

b

c

c

States ε-closure(Move(s,*))

DFA NFA a b c

s0 q0 s1 none none

s1q1,q2,q3,q4,q6,q9 none s2 s3



Fromlastlecture…COMP412,Fall2017 25

DFAMinimizaMon

Then,applytheminimiza?onalgorithm

ItsplitsnostatesaVertheiniMalparMMon

⇒ TheminimalDFAhastwostates

⇾  Onefor{s0}

⇾  Onefor{s1,s2,s3}

s3

s2

s0 s1c

ba

b

b

c

c

Spliton

CurrentPar..on a b c

P0 {s1,s2,s3}{s0} none none none

COMP412,Fall2017 26

DFAMinimizaMon

Then,applytheminimiza?onalgorithm

ItproducesthisDFA

s3

s2

s0 s1c

ba

b

b

c

c

s0 s1a

b|cEarlier,IsuggestedthatahumanwoulddesignasimplerautomatonthanThompson’sconstrucMon&thesubsetconstrucMondid.

MinimizingthatDFAproducesexactlytheDFAthatIclaimedahumanwoulddesign!

Spliton

CurrentPar..on a b c

P0 {s1,s2,s3}{s0} none none none

COMP412,Fall2017 27

AbbreviatedRegisterSpecificaMon

Startwitharegularexpressionr0|r1|r2|r3|r4|r5|r6|r7|r8|r9

Registernamesfromzerotonine

minimalDFA

RE NFA DFA


COMP412,Fall2017 28


Thompson’sconstruc?onproducesr 0

r 1

r 2

r 8

r 9

… …

s0 sf

ε

ε

ε

ε

ε

ε

εεε

ε

ε

ε ε

εε

ε

ε

ε

εε

…

Tomaketheexamplefit,wehaveeliminatedsomeoftheε-transiMons,e.g.,betweenrand0 minimal

DFARE NFA DFA


COMP412,Fall2017 29


Applyingthesubsetconstruc?onyieldsThisisaDFA,butithasalotofstates…

r

0

s2

s1s0

s31s42

s11s10

…

98

minimalDFA

RE NFA DFA


COMP412,Fall2017 30


Hopcrob’salgorithm

Fdoesnotsplit.SincenotransiMonsleaveit,therearenostatestosplitit.

minimalDFA

RE NFA DFA


COMP412,Fall2017 31

r

0 s2

s1s0

s31

s42

s11

s10…

98

F

S—F

Ini?alsets

Technically,thisedgeshowsupas10transiMons,whicharecombinedbyconstrucMonofthecharacterclassifier…



{S—F}doessplitAnycharacterwillsplititinto{s0},{s1}

minimalDFA

RE NFA DFA


COMP412,Fall2017 32

r

0 s2

s1s0

s31

s42

s11

s10…

98

F

S—F

Ini?alsets




{S—F}doessplitAnycharacterwillsplititinto{s0},{s1}ThisparMMonisthefinalparMMon

minimalDFA

RE NFA DFA


COMP412,Fall2017 33

r

0 s2

s1s0

s31

s42

s11

s10…

98

F

Ini?alsets



Hopcrob’salgorithmBecomes,throughminimiza?on

minimalDFA

RE NFA DFA


COMP412,Fall2017 34

r

0 s2

s1s0

s31

s42

s11

s10…

98

Ini?alsets

r s1s0 sf

0,1,2,3,4,5,6,7,8,9


TheCri?calTakeawayPoints:•  TheconstrucMonwillbuildaminimalDFA

•  ThesizeoftheDFArelatestothelanguagedescribedbytheRE,notthesizeoftheRE

•  TheresultisaDFA,soithasO(1)costpercharacter

•  Thecompilerwritercanusethe“mostnatural”or“intuiMve”RE

ThePlanforScannerConstrucMon

RE→NFA(Thompson’sconstruc.on)✔ü•  BuildanNFAforeachtermintheRE•  CombinetheminpaSernsthatmodeltheoperators

NFA→DFA(Subsetconstruc.on)✔ü•  BuildaDFAthatsimulatestheNFA

DFA→MinimalDFA←•  HopcroV’salgorithm✔•  Brzozowski’salgorithmMinimalDFA→Scanner•  See§2.5inEaC2eDFA→RE•  Allpairs,allpathsproblem•  Uniontogetherpathsfroms0toafinalstate

COMP412,Fall2017 35



Scanner

Brzozowski’sAlgorithmforDFAMinimizaMon

TheIntui?on•  ThesubsetconstrucMonmergesprefixesintheNFA

s0

s10s9s8

s5 s7s6

s3s2s1 s4εε

ε

a

b

a

b

c

d

c

abc|bc|ad

Thompson’sconstrucMonwouldleaveε-transiMonsbetweeneachsingle-characterautomaton

s0

s6

s4 s5

s2s1 s3a

b

b

c

d

c SubsetconstrucMoneliminatesε-transiMonsandmergesthepathsfora.Itleavesduplicatetails,suchasbc,intact.

COMP412,Fall2017 36

Brzozowski’sAlgorithm

Idea:UseTheSubsetConstruc?onTwice•  ForanNFAN

–  Letreverse(N)betheNFAconstructedbymakinginiMalstatefinal,addinganewstartstatewithanε-transiMontoeachpreviouslyfinalstate,andreversingtheotheredges

–  Letsubset(N)betheDFAproducedbythesubsetconstrucMononN–  Letreachable(N)beNaVerremovinganystatesthatarenotreachablefrom

theiniMalstate

•  Then,reachable(subset(reverse(reachable(subset(reverse(N)))))

isaminimalDFAthatimplementsN[Brzozowski,1962]

Noteveryonefindsthisresulttobeintui.ve.Neitheralgorithmdominatestheother.

COMP412,Fall2017 37


Step1•  ThesubsetconstrucMononreverse(NFA)mergessuffixesinoriginalNFA

s11

εε

ε

ReversedNFAs0

s10s9s8

s5 s7s6

s3s2s1 s4εε

ε

a

b

a

b

c

d

c

s11s9s8

s3s2s1 a

a

b

d

c

subset(reverse(NFA))

COMP412,Fall2017 38


Step2•  Reverseitagain&usesubsettomergeprefixes…

Reverseit,agains11

s9s8

s3s2s1 a

a

b

d

c

s0 εε

ε

Andsubsetit,agains11s3

s2a

b

d

cs0

b

MinimalDFA Brzozowski



COMP412,Fall2017 39


Startwitharegularexpressionr0|r1|r2|r3|r4|r5|r6|r7|r8|r9

Registernamesfromzerotonine

minimalDFA

RE NFA DFA


COMP412,Fall2017 40


Thompson’sconstruc?onproducessomethingalongtheselinesr 0

r 1

r 2

r 8

r 9

… …

s0 sf

ε

ε

ε

ε

ε

ε

εεε

ε

ε

ε ε

εε

ε

ε

ε

εε

…

Tomaketheexamplefit,wehaveeliminatedsomeoftheε-transiMons,e.g.,betweenrand0 minimal

DFARE NFA DFA


COMP412,Fall2017 41


Applyingthesubsetconstruc?onyieldsThisisaDFA,butithasalotofstates…

r

0

s2

s1s0

s31s42

s11s10

…

98

minimalDFA

RE NFA DFA


COMP412,Fall2017 42


ApplyingBrzozowski’salgorithm,step1

r

0

s2

s1s0

s31s42

s11s10

…

98

minimalDFA

RE NFA DFA


COMP412,Fall2017 43

s12

εε

ε

εεReversedNFA

r s1s0 s12

0,1,2,3,4,5,6,7,8,9

AVerSubsetConstrucMon

Technically,thisedgeshowsupas10edges,whichneedtobecombined…


Brzozowski,step2reversesthatDFAandsubsetsitagainAskilledhumanmightbuildthisDFA

rs1s0 s12

0,1,2,3,4,5,6,7,8,9

minimalDFA

RE NFA DFA


COMP412,Fall2017 44

TheCri?calPoint:•  TheconstrucMonwillbuildaminimalDFA•  ThesizeoftheDFArelatestothelanguagedescribedbytheRE,notthesizeoftheRE

•  TheresultisaDFA,soithasO(1)costpercharacter•  Thecompilerwritercanusethe“mostnatural”or“intuiMve”RE

REBacktoDFAKleene’sConstruc?on

fori←0to|D|-1; //labeleachimmediatepathforj←0to|D|-1;R0ij←{a|δ(di,a)=dj};if(i=j)thenR0ii=R0ii|{ε};

fork←0to|D|-1; //labelnontrivialpathsfori←0to|D|-1;forj←0to|D|-1;

Rkij←Rk-1ik(Rk-1kk)*Rk-1kj|Rk-1ij

L←{} //unionlabelsofpathsfromForeachfinalstatesi //s0toafinalstatesiL←L|R|D|-10i

Rkijisthesetofpathsfromitojthatincludenostatehigherthank

minimalDFA

RE NFA DFA


COMP412,Fall2017 45

OneLastAlgorithm

Adapta?onofallpoints,allpaths,lowcostalgorithm

TheWikipediapageon“Kleene’salgorithm”ispreSygood.ItalsocontainsalinktoKleene’s1956paper.ThisformofthealgorithmisusuallyaSributedtoMcNaughtonandYamadain1960.

LimitsofRegularLanguages

NotalllanguagesareregularRL’s⊂CFL’s⊂CSL’s

YoucannotconstructDFA’storecognizetheselanguages•  L={pkqk} (parenthesislanguages)

•  L={wcwr|w∈Σ*}Neitheroftheseisaregularlanguage (noranRE)

But,thisisaliSlesubtle.YoucanconstructDFA’sfor•  StringswithalternaMng0’sand1’s

(ε|1)(01)*(ε|0)

•  Stringswithandevennumberof0’sand1’s

RE’scancountboundedsetsandboundeddifferences

COMP412,Fall2017 46

LimitsofRegularLanguages

AdvantagesofRegularExpressions•  Simple&powerfulnotaMonforspecifyingpaSerns•  AutomaMcconstrucMonoffastrecognizers

–  O(1)costperinputcharacter•  ManykindsofsyntaxcanbespecifiedwithREs

DisadvantagesofRegularExpressions•  ManyinteresMngconstructsarenotregular

–  Balancedparentheses,nestedif-thenandif-then-elseconstructs•  TheDFArecognizerhasnorealnoMonofgrammaMcalstructure

–  Givesnohelpwithmeaning

COMP412,Fall2017 47

Documents

Lexical Analysis, III - Rice University · source IR IR code target code … Chapter 2 in EaC2e Ignore § 2.4.4 in EaC2e. Read the replacement sec?on ... NFA → DFA (Subset construcon)