Compiler ConstructionCompiler Construction

Lexical Analysis Lexical Analysis

Rina Zviel-Girshin and Ohad ShachamRina Zviel-Girshin and Ohad ShachamSchool of Computer ScienceSchool of Computer Science

Tel-Aviv UniversityTel-Aviv University

22

AdministrationAdministration Project Teams Project Teams

Send me your groupSend me your group Send me an email if you can’t find a teamSend me an email if you can’t find a team

First PA – for two weeksFirst PA – for two weeks

There are no recitations next weekThere are no recitations next week November 26 – Schreiber 07November 26 – Schreiber 07

9:00 – 10:009:00 – 10:00 13:00 – 14:0013:00 – 14:00

November 27 – Schreiber 07November 27 – Schreiber 07 10:00 – 11:0010:00 – 11:00

33

Generic compiler structureGeneric compiler structure

Executable

code

exe

Source

text

txt

Semantic

Representation

Backend

(synthesis)

Compiler

Frontend

(analysis)

44

Compiler

ICProgram

ic

x86 executable

exeLexicalAnalysi

s

Syntax Analysi

s

Parsing

AST Symbol

Tableetc.

Inter.Rep.(IR)

CodeGeneration

IC compilerIC compiler

55

Lexical AnalysisLexical Analysis

converts characters to tokensconverts characters to tokens

class Quicksort { int[] a; int partition(int low, int high) { int pivot = a[low]; ...}

1: CLASS1: CLASS_ID(Quicksort)1: LCBR2: INT2: LB2: RB2: ID(a)

. . .

2: SEMI

66

Lexical AnalysisLexical Analysis TokensTokens

ID – _size, _numID – _size, _num Num – 7, 5 , 9, 4926209837Num – 7, 5 , 9, 4926209837 COMMA – ,COMMA – , SEMI – ;SEMI – ; ……

Non tokensNon tokens Comment – // Comment – // WhitespaceWhitespace Macro Macro ……

77

ProblemProblem

InputInput Program textProgram text Tokens specificationTokens specification

OutputOutput Sequence of tokensSequence of tokens

class Quicksort { int[] a; int partition(int low, int high) { int pivot = a[low]; ...}

1: CLASS1: CLASS_ID(Quicksort)1: LCBR2: INT2: LB2: RB2: ID(a)

. . .

2: SEMI

88

SolutionSolution

Write a lexical analyzerWrite a lexical analyzerToken nextToken(){

char c ;loop: c = getchar();switch (c){

case ` `:goto loop ;case `;`: return SemiColumn;case `+`: c = getchar() ;

switch (c) { case `+': return PlusPlus ; case '=’ return PlusEqual; default: ungetc(c);

return Plus; }

case `<`:case `w`:

… }

99

Solution’s ProblemSolution’s Problem

A lot of workA lot of work Corner casesCorner cases Error pruneError prune Hard to debugHard to debug ExhaustingExhausting BoringBoring Hard to reuseHard to reuse Switch parser’s code between peopleSwitch parser’s code between people …….. ……..

1010

JFlexJFlex

Off the shelf lexical analysis generatorOff the shelf lexical analysis generator InputInput

scanner specification filescanner specification fileOutputOutput

Lexical analyzer written in JavaLexical analyzer written in Java

JFlex javacIC.lex Lexical analyzer

IC text

tokens

Lexer.java

1111

JFlexJFlex

SimpleSimple Good for reuseGood for reuse Easy to understandEasy to understand Many developers and users debugged the generatorsMany developers and users debugged the generators

"+" { return new symbol (sym.PLUS); }"boolean" { return new symbol (sym.BOOLEAN); }“int" { return new symbol (sym.INT); }"null" {return new symbol (sym.NULL);}"while" {return new symbol (sym.WHILE);}

"=" {return new symbol (sym.ASSIGN);}

……

1212

JFlex Spec FileJFlex Spec File

User codeUser code Copied directly to Java fileCopied directly to Java file

%%

JFlex directivesJFlex directives Define macros, state namesDefine macros, state names

%%

Lexical analysis rulesLexical analysis rules How to break input to tokensHow to break input to tokens Action when token matchedAction when token matched

Possible source of

javac errors down the

roadDIGIT= [0-9]

LETTER= [a-zA-Z]

YYINITIAL

{LETTER}({LETTER}|{DIGIT})*

1313

User codeUser code

package IC.Parser;import IC.Parser.Token;

…any scanner-helper Java code…

1414

JFlex DirectivesJFlex Directives

Control JFlex internalsControl JFlex internals %line %line switches line counting onswitches line counting on %char %char switches character counting onswitches character counting on %class class-name%class class-name changes default name changes default name %cup %cup CUP compatibility modeCUP compatibility mode %type token-class-name%type token-class-name %public %public Makes generated class public (package by default)Makes generated class public (package by default) %function read-token-method%function read-token-method %scanerror exception-type-name%scanerror exception-type-name

1515

JFlex DirectivesJFlex Directives

State definitionsState definitions%state %state state-name state-name %state %state STRINGSTRING

Macro definitionsMacro definitionsmacro-name = regexmacro-name = regex

1616

Regular ExpressionRegular Expression

rr $$match reg. exp. match reg. exp. rr at end of a line at end of a line

. . any character except the newlineany character except the newline"...""..."stringstring{name}{name}macro expansionmacro expansion**zero or more repetitions zero or more repetitions ++one or more repetitionsone or more repetitions??zero or one repetitions zero or one repetitions

(...) (...) grouping within regular expressionsgrouping within regular expressions

aa||bbmatch match aa or or bb

[...][...]class of characters - any class of characters - any oneone character enclosed in brackets character enclosed in brackets

aa––bbrange of charactersrange of characters

[^…] [^…] negated class – any one not enclosed in bracketsnegated class – any one not enclosed in brackets

1717

Example macrosExample macros

ALPHA=[A-Za-z_] ALPHA=[A-Za-z_]

DIGIT=[0-9]DIGIT=[0-9]

ALPHA_NUMERIC={ALPHA}|{DIGIT}ALPHA_NUMERIC={ALPHA}|{DIGIT}

IDENT={ALPHA}({ALPHA_NUMERIC})*IDENT={ALPHA}({ALPHA_NUMERIC})*

NUMBER=({DIGIT})+NUMBER=({DIGIT})+

NUMBER=[0-9]+NUMBER=[0-9]+

1818

RulesRules

[states] regexp {action as Java code}[states] regexp {action as Java code}

PrioritiesPriorities Longest matchLongest match Order in the lex fileOrder in the lex file

Rules should match all inputs!!!Rules should match all inputs!!!

Breaks Input to Tokens Invokes when

regexp matches

breakbreakdown int

identifier or integer ?

The regexp should be evaluated ?

1919

Rules ExamplesRules Examples

<YYINITIAL> {DIGIT}+ DIGIT}+ {

return new Symbol(sym.NUMBER, yytext(), yyline);

}

<YYINITIAL> "-" {

return new Symbol(sym.MINUS, yytext(), yyline);

}

<YYINITIAL> [a-zA-Z] ([a-zA-Z0-9]) * {

return new Symbol(sym.ID, yytext(), yyline);

}

2020

Rules – ActionRules – Action

ActionActionJava codeJava codeCan use special methods and varsCan use special methods and vars

yylineyylineyytext()yytext()

Returns a token for a tokenReturns a token for a tokenEats chars for non tokensEats chars for non tokens

2121

Rules – StateRules – State

StateStateWhich regexp should be evaluated?Which regexp should be evaluated?yybegin(stateX)yybegin(stateX)

jumps to stateXjumps to stateX

YYINITIALYYINITIAL JFlex’s initial stateJFlex’s initial state

2222

Rules – StateRules – State

<YYINITIAL> "//" { yybegin(COMMENTS); }

<COMMENTS> [^\n] { }

<COMMENTS> [\n] { yybegin(YYINITIAL); }

YYINITIAL COMMENTS

‘//’

\n

^\n

2323

Lines Count ExampleLines Count Exampleimport java_cup.runtime.Symbol;

%%%cup%{ private int lineCounter = 0;%}

%eofval{ System.out.println("line number=" + lineCounter); return new Symbol(sym.EOF);%eofval}

NEWLINE=\n%%

<YYINITIAL>{NEWLINE} {lineCounter++;

} <YYINITIAL>[^{NEWLINE}] { }

2424

Lines Count ExampleLines Count Example

JFlex

javac

lineCount.lex

Lexical analyzer

text

tokens

Yylex.java

Main.java

JFlex and JavaCup must be on CLASSPATH

sym.java

java JFlex.Main lineCount.lex

javac *.java

2525

Test BedTest Bedimport java.io.*;

public class Main { public static void main(String[] args) { Symbol currToken; try { FileReader txtFile = new FileReader(args[0]); Yylex scanner = new Yylex(txtFile); do { currToken = scanner.next_token(); // do something with currToken } while (currToken.sym != sym.EOF); } catch (Exception e) { throw new RuntimeException("IO Error (brutal exit)” +

e.toString()); } }}

2626

Common PitfallsCommon Pitfalls

ClasspathClasspathPath to executablePath to executableDefine environment variablesDefine environment variables

JAVA_HOMEJAVA_HOMECLASSPATHCLASSPATH

2727

Programming Assignment 1Programming Assignment 1 Implement a scanner for ICImplement a scanner for IC class Tokenclass Token

At least – line, id, valueAt least – line, id, value Should extend java_cup.runtime.SymbolShould extend java_cup.runtime.Symbol Numeric token ids in Numeric token ids in sym.javasym.java

Will be later generated by JavaCupWill be later generated by JavaCup class Compilerclass Compiler

Testbed - calls scanner to print list of tokensTestbed - calls scanner to print list of tokens class LexicalErrorclass LexicalError

Caught by CompilerCaught by Compiler Don’t forget to generate scanner and recompile Java sources Don’t forget to generate scanner and recompile Java sources

when you change the specwhen you change the spec You need to download and install You need to download and install bothboth JFlex and JavaCup JFlex and JavaCup

2828

sym.javasym.java

public class sym {public class sym {

public static final int EOF = 0;public static final int EOF = 0;

public static final int ID = 1;public static final int ID = 1;

......

}}

Defines symbol constant ids Communicate between parser and scanner Actual values don’t matter

Unique value for each tokes

Will be generated by cup in PA2

2929

Token classToken class

import java_cup.runtime.Symbol;import java_cup.runtime.Symbol;

public class Token extends Symbol {public class Token extends Symbol {

public int getId() {...}public int getId() {...}

public Object getValue() {...}public Object getValue() {...} public int getLine() {...} public int getLine() {...}

......

}}

3030

JFlex directives to useJFlex directives to use

%cup%cup (integrate with cup)(integrate with cup)

%line%line (count lines)(count lines)

%type Token%type Token (pass type Token)(pass type Token)

%class Lexer%class Lexer (gen. scanner class)(gen. scanner class)

3131

%cup%cup

%implements java_cup.runtime.Scanner%implements java_cup.runtime.Scanner Lex class implements java_cup.runtime.ScannerLex class implements java_cup.runtime.Scanner

%function next_token %function next_token Returns the next tokenReturns the next token

%type java_cup.runtime.Symbol%type java_cup.runtime.Symbol Return token ClassReturn token Class

3232

StructureStructure

JFlex javacIC.lexLexical analyzer

test.ic

tokens

Lexer.java

sym.javaToken.java

LexicalError.javaCompiler.java

3333

DirectionsDirections

Download JavaDownload JavaDownload JFlexDownload JFlexDownload JavaCupDownload JavaCupPut JFlex and JavaCup in classpathPut JFlex and JavaCup in classpathEclipseEclipse

Use ant build.xmlUse ant build.xml Import jflex and javacupImport jflex and javacup

Apache AntApache Ant

3434

DirectionsDirections

Use skeleton from the websiteUse skeleton from the websiteRead AssignmentRead AssignmentUse ForumUse Forum