compiler-lexical analysis

Lexical Analysis

OutlineRole of lexical analyzerSpecification of tokensRecognition of tokensLexical analyzer generatorFinite automataDesign of lexical analyzer generator

Compiler Construction

Lexicalanalyzer

ParserSource program

read char

put back char

pass tokenand attribute value

get next

Symbol TableRead entireprogram into

memory

id

The role of lexical analyzer

Lexical Analyzer ParserSource

program

token

getNextToken

Symboltable

To semanticanalysis

Other tasks…Stripping out from the source program

comment and white space in the form of blank, tab and newline characters .

Correlate error messages with source program (e.g., line number of error).

Lexical analyzer scanning (simple

operations)

lexical analysis(complex)

Why to separate Lexical analysis and parsing1. Simplicity of design

1. Improving compiler efficiency

1. Enhancing compiler portability

Eliminate the white space and comment lines before parsing

A large amount of time is being spent on reading the source program and generating the tokens

The representation of special or non standard symbols can be isolated in the lexical analyzer

Tokens, Patterns and LexemesPattern: A rule that describes a set of stringsToken: A set of strings in the same patternLexeme: The sequence of characters of a token

ExampleToken Informal description Sample lexemes

ifelse

comparison

idnumberliteral

Characters i, fCharacters e, l, s, e

< or > or <= or >= or == or !=

Letter followed by letter and digits

Any numeric constant

Anything but “ sorrounded by “

if

else<=, !=

pi, score, D2

3.14159, 0, 6.02e23

“core dumped”


E = C1 * 10

Token Attribute

ID Index to symbol table entry E

=

ID Index to symbol table entry C1

*

NUM 10

Lexical Error and RecoveryA character sequence that can’t be scanned into any valid

token is a lexical error.Lexical errors are uncommon, but they still must be

handled by a scanner. We won’t stop compilation because of minor error.

fi(a==f(x))... //mistyped if lex will give valid token Delete one character from the remaining input. Insert a missing character into the remaining input. Replace a character by another character. Transpose two adjacent characters from the

remaining input

Input bufferingSometimes lexical analyzer needs to look

ahead some symbols to decide about the token to returnIn C language: we need to look after -, = or <

to decide what token to returnWe need to introduce a two buffer scheme to

handle large look-aheads safely

E = M * C * * 2 eof


Specification of TokensRegular expressions are an important notation for

specifying lexeme patterns. While they cannot express all possible patterns, they are very effective in specifying

those types of patterns that we actually need for tokens.

Specification of tokensIn theory of compilation regular expressions

are used to formalize the specification of tokens

Regular expressions are means for specifying regular languages

Example: Letter(letter| digit)*

Each regular expression is a pattern specifying the form of strings

Regular expressionsƐ is a regular expression, L(Ɛ) = {Ɛ}If a is a symbol in ∑then a is a regular

expression, L(a) = {a}(r) | (s) is a regular expression denoting the

language L(r) ∪ L(s) (r)(s) is a regular expression denoting the

language L(r)L(s)(r)* is a regular expression denoting (L(r))*

R* = R concatenated with itself 0 or more times

= {ε} ∪ R ∪ RR ∪ RRR ∪(r) is a regular expression denoting L(r)

ExtensionsOne or more instances: (r)+Zero of one instances: r?Character classes: [ abc ]

Example:letter_ -> [A-Z , a-z_]digit -> [0-9]id -> letter(letter | digit)*

Regular definitionsd1 -> r1

d2 -> r2

…d n -> r n

Example:letter -> A | B | … | Z | a | b | … | z | _digit -> 0 | 1 | … | 9id -> letter(letter| digit)*

Operations on Languages

Example: Let L be the set of letters {A, B, . . . , Z, a, b, . . . , z ) and let D

be the set of digits {0,1,.. .9). L and D are, respectively, the alphabets of uppercase and lowercase letters and of digits. other languages can be constructed from L and D, using the operators illustrated above


Terms for Parts of Strings

Specification of TokensAlgebraic laws of regular expressions 1) |= |

2) |(|)=(|)| () =( ) 3) (| )= | (|)= | 4) = = 5)(*)*=*

6) *= ＋ | ＋ = *7) (|)*= (* | *)*= (* *)*

Recognition of TokensTask of recognition of token in a lexical

analyzer Isolate the lexeme for the next token in the

input buffer Produce as output a pair consisting of the

appropriate token and attribute-value, such as <id,pointer to table entry> , using the translation table given in the Fig in next page

Recognition of TokensTask of recognition of token in a lexical

analyzer

Regular expression

Token Attribute-value

if if -id id Pointer to

table entry< relop LT

Recognition of TokensMethods to recognition of token

Use Transition Diagram

Recognition of tokensStarting point is the language grammar to

understand the tokens:Grammar for branching statement

stmt -> if expr then stmt | if expr then stmt else stmt | Ɛexpr -> term relop term | termterm -> id | number

Recognition of tokens (cont.)The next step is to formalize the patterns:

digit -> [0-9]digits -> digit+number -> digits(.digits)? (E[+|-]? digits)?letter -> [A-Z a-z_]id -> letter (letter | digit)*If -> ifThen -> thenElse -> elseRelop -> < | > | <= | >= | = | <>

We also need to handle whitespaces:ws -> (blank | tab | newline)+


Ex :RELOP = < | <= | = | <> | > | >=

0

1

5

6

2

3

4

7

8

start<

=

=

=

>

>

other

other

return(relop,LE)

return(relop,NE)

return(relop,LT)

return(relop,GE)

return(relop,GT)

return(relop,EQ)

#

# # indicates input retraction


Ex2: ID = letter(letter | digit) *

9 10 11start letter

return(id)

# indicates input retraction

other #

letter or digitTransition Diagram:

Transition diagrams (cont.)Transition diagram for whitespace

9 10 11start letter

return(id)other

letter or digit

switch (state) {case 9:

if (isletter( c) ) state = 10; else state = failure();break;

case 10: c = nextchar(); if (isletter( c) || isdigit( c) ) state = 10; else state 11case 11: retract(1); insert(id); return;

Architecture of a transition-diagram-based lexical analyzerTOKEN getRelop(){

TOKEN retToken = new (RELOP)while (1) { /* repeat character processing until a

return or failure occurs */switch(state) {

case 0: c= nextchar(); if (c == ‘<‘) state = 1; else if (c == ‘=‘) state = 5; else if (c == ‘>’) state = 6; else fail(); /* lexeme is not a relop */ break;

case 1: ……case 8: retract();

retToken.attribute = GT; return(retToken);

}

Lexical Analyzer Generator - Lex

Lexical Compiler

Lex Source programlex.l

lex.yy.c

Ccompiler

lex.yy.c a.out

a.outInput stream Sequence of tokens

Structure of Lex programs

declarations%%translation rules%%auxiliary functions

Pattern {Action}

Example%{

/* definitions of manifest constantsLT, LE, EQ, NE, GT, GE,IF, THEN, ELSE, ID, NUMBER, RELOP */

%}

/* regular definitionsdelim [ \t\n]ws {delim}+letter [A-Za-z]digit [0-9]id {letter}({letter}|{digit})*number {digit}+(\.{digit}+)?(E[+-]?{digit}+)?

%%{ws} {/* no action and no return */}if {return(IF);}then {return(THEN);}else {return(ELSE);}{id} {yylval = (int) installID(); return(ID); }{number} {yylval = (int) installNum();

return(NUMBER);}…

Int installID() {/* funtion to install the lexeme, whose first character is pointed to by yytext, and whose length is yyleng, into the symbol table and return a pointer thereto */

}

Int installNum() { /* similar to installID, but puts numerical constants into a separate table */

}

35

Finite AutomataRegular expressions = specificationFinite automata = implementation

A finite automaton consists ofAn input alphabet A set of states SA start state nA set of accepting states F SA set of transitions state input state

36

Finite AutomataTransition

s1 a s2

Is readIn state s1 on input “a” go to state s2

If end of inputIf in accepting state => accept, othewise =>

rejectIf no transition possible => reject

37

Finite Automata State GraphsA state

• The start state

• An accepting state

• A transitiona

38

A Simple ExampleA finite automaton that accepts only “1”

A finite automaton accepts a string if we can follow transitions labeled with the characters in the string from the start to some accepting state

1

39

Another Simple ExampleA finite automaton accepting any number of

1’s followed by a single 0Alphabet: {0,1}

Check that “1110” is accepted but “110…” is not

0

1

40

And Another ExampleAlphabet {0,1}What language does this recognize?

0

1

0

1

0

1

41

And Another ExampleAlphabet still { 0, 1 }

The operation of the automaton is not completely defined by the inputOn input “11” the automaton could be in either

state

1

1

42

Epsilon MovesAnother kind of transition: -moves

• Machine can move from state A to state B without reading input

A B

43

Deterministic and Nondeterministic AutomataDeterministic Finite Automata (DFA)

One transition per input per state No -moves

Nondeterministic Finite Automata (NFA)Can have multiple transitions for one input in a

given stateCan have -moves

Finite automata have finite memoryNeed only to encode the current state

44

Execution of Finite AutomataA DFA can take only one path through the

state graphCompletely determined by input

NFAs can chooseWhether to make -movesWhich of multiple transitions for a single input

to take

45

Acceptance of NFAsAn NFA can get into multiple states

• Input:

0

1

1

0

1 0 1

• Rule: NFA accepts if it can get in a final state

46

NFA vs. DFA (1)NFAs and DFAs recognize the same set of

languages (regular languages)

DFAs are easier to implementThere are no choices to consider

47

NFA vs. DFA (2)For a given language the NFA can be simpler

than the DFA0

10

0

01

0

1

0

1

NFA

DFA

• DFA can be exponentially larger than NFA

48

Regular Expressions to Finite AutomataHigh-level sketch

Regularexpressions

NFA

DFA

LexicalSpecification

Table-driven Implementation of DFA

49

Regular Expressions to NFA (1)For each kind of rexp, define an NFA

Notation: NFA for rexp A

A

• For

• For input aa

50

Regular Expressions to NFA (2)For AB

A B

• For A | B

A

B

51

Regular Expressions to NFA (3)For A*

A

52

Example of RegExp -> NFA conversionConsider the regular expression

(1 | 0)*1The NFA is

1C E0D F

B

G

A H 1I J

53

Next

Regularexpressions

NFA

DFA

LexicalSpecification

Table-driven Implementation of DFA

54

NFA to DFA. The TrickSimulate the NFAEach state of resulting DFA

= a non-empty subset of states of the NFAStart state

= the set of NFA states reachable through -moves from NFA start state

Add a transition S a S’ to DFA iffS’ is the set of NFA states reachable from the

states in S after seeing the input a considering -moves as well

55

NFA -> DFA Example10 1

A BC

D

E

FG H I J

ABCDHI

FGABCDHI

EJGABCDHI

0

1

0

10 1

56

NFA to DFA. RemarkAn NFA may be in many states at any time

How many different states ?

If there are N states, the NFA must be in some subset of those N states

How many non-empty subsets are there?2N - 1 = finitely many, but exponentially many

57

ImplementationA DFA can be implemented by a 2D table T

One dimension is “states”Other dimension is “input symbols”For every transition Si a Sk define T[i,a] = k

DFA “execution”If in state Si and input a, read T[i,a] = k and

skip to state Sk

Very efficient

58

Table Implementation of a DFA

S

T

U

0

1

0

10 1

0 1S T UT T UU T U

59

Implementation (Cont.)NFA -> DFA conversion is at the heart of

tools such as flex or jflex

But, DFAs can be huge

In practice, flex-like tools trade off speed for space in the choice of NFA and DFA representations

ReadingsChapter 3 of the book

Documents

compiler-lexical analysis