Tutorial Lexer and Parser

Compiler Construction 1

S C I E N C E P A S S I O N T E C H N O L O G Y

u www.tugraz.at

Georg Hinteregger, Christopher Liebmann

Institute for Software Technology

Tutorial 1: Lexer & Parser

Tutorial Lexer and Parser

Compiler Construction 2

Overview

ANTLR

Lexical and Syntax Analyzer

Tutorial Lexer and Parser

Compiler Construction 3

Procedure

Source

Code

(.calc file)

Abstract

Syntax Tree

(AST)

Lexical and

syntactical errors

Debugging Output

(Lexer and Parser)

ANTLR Lexer

and Parser

Tutorial Lexer and Parser

Compiler Construction 4

ANother Tool for Language Recognition

(ANTLR)

framework for constructing

interpreters

compilers

translators from grammatical descriptions

supports

tree construction (AST only up to version 3.5)

tree walking

translation

error recovery / reporting

Tutorial Lexer and Parser

Compiler Construction 5

ANTLR - Getting started

Download

• ANTLR v3.5 (http://www.antlr3.org/download.html)

• Set classpath

Compile Example

• java org.antlr.Tool Calculator.g

• javac Calculator*.java

Run Example

• java CalculatorParser CalcExample.txt

Tutorial Lexer and Parser

Compiler Construction 6

Example: Simple Calculator

Parser for a simple calculator (Calculator). It should

handle the 4 basic arithmetical operations.

ignore whitespaces.

read the input-string from a file.

Valid input

13 * 2

3.5 + 4.0

Invalid input

17 a + 3

4 + / 2

4.0 + 3

42

Tutorial Lexer and Parser

Compiler Construction 7

ECLIPSE DEMO

Tutorial Lexer and Parser

Compiler Construction 8

Calculator.g 1 – Lexer Rules

grammar Calculator; INT : (DIGIT DIGIT0*) | '0'; FLOAT : ((DIGIT DIGIT0*) | '0') '.' (DIGIT0)+; fragment DIGIT : '1'..'9'; fragment DIGIT0 : '0'..'9'; SIGN : '+' | '-' ; MULOP : '*' | '/' ; WHITESPACE : ( '\t' | ' ' | '\r' | '\n') { $channel =

HIDDEN; } ;

Tutorial Lexer and Parser

Compiler Construction 9

Calculator.g 2 – Parser Rules

program : main_function EOF ; main_function : 'main' '(' ')' fct_body ; fct_body : '{' expression* return_stmt '}' ; expression : float_expr | int_expr; float_expr : FLOAT ( (MULOP FLOAT) | (SIGN FLOAT)) ; int_expr : INT ( (MULOP INT) | (SIGN INT)) ; return_stmt : 'return' ;

Tutorial Lexer and Parser

Compiler Construction 10

Call of Lexer / Parser

public static void main(String[] args) throws Exception{ CalculatorLexer lex = new CalculatorLexer(new ANTLRFileStream(args[0])); CommonTokenStream tokens = new CommonTokenStream(lex); CalculatorParser parser = new CalculatorParser(tokens); try { parser.program(); } catch (RecognitionException e) { e.printStackTrace(); } }

Tutorial Lexer and Parser

Compiler Construction 11

General Remarks

Arithmetic Rule: MUL before SIGN

Generate a structured AST

Tutorial Lexer and Parser

Compiler Construction 12

Questions?