Review 1.Lexical Analysis 2.Syntax Analysis 3.Semantic Analysis 4.Code Generation 5.Code...

Review

1. Lexical Analysis

2. Syntax Analysis

3. Semantic Analysis

4. Code Generation

5. Code Optimization

Syntax Analysis

• Often called parsing• Groups tokens of source program into grammatical

phrases that are used by the compiler to check for correct syntax and to help in generating code

• Creation of a hierarchical structure called a syntax tree– Tree helps us determine if program is syntactically correct

– Also aids in the translation of source program to target language

Grammar Example

San Francisco

Seattle Lexical Ogay orthnay eightay undrendhay ilesmay

Syntactical Miles go hundred north eight

Logical Go eight hundred miles(facing south)

Run-Time Go eight hundred miles (facing West)

Grammars – Defining the Language Rules

• Terminals (tokens)• Non-terminals - Syntactic variable. Contains groups

of tokens that define some part of the language– Example: expression, if statement, etc

• Start symbol - A special non-terminal (i.e. a program)• Productions

– The manner in which terminals and non-terminals are combined to form statements

– A non-terminal in LHS and a string of terminals and non-terminals in RHS

Example

• Variable Declaration– A type followed by one or more comma separated

identifiers that end with a semi-colon.<Var> -> <Indent>

• <Var> and <Ident> are non-terminals

• The comma is a terminal

• The two lines are called productions

Grammars

• We will be defining a grammar for the entire JO99 programming language.

• We will then have JCUP produce for us a parser that detects if the JO99 program is syntactically correct.

• In addition, the parser will create for us a tree that represents the program and allows us to be able to do other things like semantic checks and code generation.

Example

• Simple arithmetic expressions with + and *– 8.2 + 35.6– 8.32 + 86 * 45.3 – (6.001 + 6.004) * (6.035 * -(6.042 + 6.046))

• Terminals (or tokens)– num for all the numbers– ‘+’, ‘-’, ‘*’, ‘(‘, ‘)’

• What is the grammar for all possible expressions?

Example

<expr> ( <expr> )

<expr> num

<op> +

<op> *

Categories of Parsers

– L - parse from left to right– R - parse from right to left

– L - leftmost derivation– R - rightmost derivation

– Number of look ahead characters

– Examples: • LL(0) – Parse Left to Right, Derive the tree using a

leftmost derivation (top down), no look ahead characters• LR(1) – Parse Left to Right, Derive the tree using a

rightmost derivation (bottom up), 1 look ahead character.

– Each category of parsing handles a different type of language.

– We will be learning about LR(k) parsers and will implement an LR(k) parser.

Why LR(k)?

• Virtually all programming language grammars can be parsed using a LR(k) technique

• Most general parsing method for programming grammars

• Can build a very efficient parser engine given just the syntax rules of the language.

• Can detect a syntactic error as soon as it is possible to do so

• Because its so general, programs have been written (JCUP) that produce the parser instead of writing it from scratch.

LR(k) Parser implementation

• Sometimes called a Shift-Reduce Parser

• Parse from left to right (get the tokens from left to right)

• Bottom up parsing (same as rightmost derivation)

Actions of a Shift-Reduce Parser

ParseTree

ParseTreeParseTree

ParseTree

• How do we build this tree?

• As productions are recognized, a portion of the tree is created.

• This portion of the tree will be needed later to build bigger portions of the tree and therefore must be saved for future use.

• This requires a stack.

• The stack plus the next token read from the source program determines the action.

Actions of a Shift-Reduce ParserS

Current Symbol

Parser A

ParserEngine

Actions of a Shift-Reduce Parser• Shift

– Shift the current element into top of the stack– Move the current pointer (next token)

• Reduce– Apply a production (we recognize a part of the

program)– Top of the stack should match the RHS of the

grammar– Remove those symbols from the stack– Add the LHS non-terminal

• Accept– End of stream reached and stack only has the start

symbol• Reject

– End of stream reached but stack has more than the start symbol

Shift-Reduce Parser Example

* ( + num )numnum

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

* ( + num )numnum

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

* ( + num )numnum

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *num

* ( + num )numnum

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *num

* ( + num )num

<expr>

<expr> ( <expr> )

<expr> num

<op> +

<op> -

( + num )num

<expr>

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

( + num )num

<expr>

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

( + num )num

<expr>

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

+ num )num

<expr>

num * (

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

* + num )

<expr>

num ( num

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

* + num )

<expr>

num ( num

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

* + num )

<expr>

num ( num

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

* num )

<expr>

num ( num +

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

* num )

<expr>

num ( num +

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

* num )

<expr>

num ( num +

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

num ( num + num

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

num ( num + num

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

num ( num + num

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

num ( num + num

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

num ( num + num

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

num ( num + num

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

num ( num + num

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

num ( num + num

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

num ( num + num

<expr>

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

num ( num + num

<expr>

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr>

num ( num + num

<expr>

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

What does the parser engine do?

• If the top symbols of the stack match the RHS of a production then do the reduction – Pop the RHS from the top of the stack– Push the LHS symbol onto the stack

• If no production is found do the shift– Push the current input into the stack

• If the input is empty– Accept if only the start symbol is on the stack– Reject otherwise

ParserEngine

This is not that simple!

• Many choices of reductions if there are multiple RHS.

• Which LHS do we put on stack in its place?

• Choice between shift and reduce– Stack matches a RHS– But that may not be the right match– May need to shift an input onto stack and later find

a different reduction

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

• Change in the Grammar

<expr> ( <expr> )

<expr> num

<op> +

<op> -

<op> *

<expr> ( <expr> )

<expr> <expr> -<expr> num

<op> +

<op> -

<op> *

<expr> ( <expr> )

<expr> <expr> -<expr> num

<op> +

<op> -

<op> *

- numnum

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum

<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum

<expr>

-<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr>

-<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

We have a choice!!!<op><expr>

-<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

But not the right thing to do!!<op><expr>

<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr>

But not the right thing to do!!

<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

But not the right thing to do!!<expr>

<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

But not the right thing to do!! No more actions!!!

<expr>

• But this is perfectly valid input for the grammar

• We chose the wrong production and thus the wrong LHS

• Lets see what happens when we choose the right production and the right LHS

-<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

We have a choice<op><expr>

The step before we went wrong

-<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

Use the other production

<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr>

<op> Use the other production

<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr>

< <expr> <expr> <op> <expr><expr> ( <expr> )<expr> <expr> -

<expr> num<op> +

<op> - <op> *

numnum -

<expr>

<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr>

Shift-Reduce Parser

• Parser Engine is far more complicated that it appears.

• Requires it to know all possible productions that would match the top of the stack and the given input symbol.

Constructing a LR(k) Parser

• What is in the parse engine– decide between shift and reduce– decide on the right reduction

Constructing a LR(k) Parser

• Create a DFA – Encodes all the possible states that the parser can be in– DFA state transition occurs on terminals and non-

terminals

• Create a Parser Table – From the DFA create a transition table that stores

what action should be taken for the current state and current input character

• Maintain a stack of states in parallel with the stack of symbols

LR(k) Parser Engine

Current Symbol

Parser A

LR(k)ParserEngine

ACTION GotoState ( ) $ Xs0 shift to s2 error error goto s1s1 error error accept s2 shift to s2 shift to s5 error goto s3s3 error shift to s4 error s4 reduce (2) reduce (2) reduce (2) s5 reduce (3) reduce (3) reduce (3)

Parser Tables

• Look-up [top of state stack] [ input symbol] in the parser table

• Carry-out the described action

Parser Tables

• Shift to sn– Push input token into the symbol stack– Push sn into state stack– Advance to next input symbol

Parser Tables

• Reduce (n)– Pop both stacks as many times as the number of symbols

on the RHS of rule n – Push LHS of rule n into symbol stack– Lookup [top of the state stack][top of symbol stack]– Push that state (in goto k) into state stack

Parser Tables

• Accept– Stop parsing and report success

Parser Tables

• Error– Stop parsing and report failure

LR example

• The grammar

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

Question

• The grammar

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

Parser Table in Action• The grammar

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

Parser Table in Action

• Parser Table• The grammar

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)ACTION Goto

State ( ) $ Xs0 shift to s2 error error goto s1s1 error error accept s2 shift to s2 shift to s5 error goto s3s3 error shift to s4 error s4 reduce (2) reduce (2) reduce (2) s5 reduce (3) reduce (3) reduce (3)

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( )( $

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( )( $

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( )( $

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

))( ( $

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

))( ( $

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

))( ( $

s2 (s2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

))( ( $

s2 (s2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s2 (s2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s2 (s2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s5 )s2 (s2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s5 )s2 (s2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s5 )s2 (s2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s5 )s2 (s2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s5 )s2 (s2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s5 )s2 (s2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( )

s5 )s2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s3 Xs2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s3 Xs2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s3 Xs2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s3 Xs2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s4 )s3 Xs2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s4 )s3 Xs2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s4 )s3 Xs2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s4 )s3 Xs2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s4 )s3 Xs2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

s4 )s3 Xs2 (

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

Parser Table in Action• The grammar

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

)( ( ) $

Accept

Parser Table

• The table (DFA) and the stacks are called Push Down Automaton (PDA).

• There is an algorithm for converting a grammar to a PDA.

• Parser generators – given a grammar, produce a parser table with a stack (PDA)

• We will not study how this is done because its somewhat complicated.

• JCUP will convert our grammar into a shift-reduce parser.

Review 1.Lexical Analysis 2.Syntax Analysis 3.Semantic Analysis 4.Code Generation 5.Code...

Documents

Lexical Analysis Dragon Book: chapter 3. Compiler structure Lexical analyzer Syntax analyzer Semantic analyzer Intermediate code generator Code optimizer

Lexical Analysis

Midterm Review€¦ · The Structure of a Modern Compiler Lexical Analysis Syntax Analysis Semantic Analysis IR Generation IR Optimization Code Generation Optimization Source Code

1 Lecture 10: Code Generation. You are here 2 Executable code exe Source text txt Compiler Lexical Analysis Syntax Analysis Semantic Analysis Inter. Rep

Lexical Analysis - Computer Science and Engineeringweb.cse.ohio-state.edu/~rountev.1/5343/pdf/LexicalAnalysis.pdfImplementing a Lexical Analyzer • Do the code generation automatically,

Lexical analysis is the first phase of compilation: The file is ... Passes Analysis of input program (front -end) character stream Lexical Analysis Code Generation Optimization Intermediate

1. The output of lexical analyser is a) A set of RE b ... · c) Lexical analysis, syntax. Analysis and code generation d) None of the mentioned 8. The lexical analyser takes_____as

Lecture Notes on Programming Languages Elvis C. Foster€¦ · · 2017-04-04Source Code Lexical Analysis Symbol Table Syntax Analysis Syntax ... These are called lexical items or

The Lexical Analysis - DMCSneo.dmcs.pl/cc/02-lexical.pdf · Lexical Analysis Process ... Preprocessed source code, read char by char if ( b == 0 ) a = b ; 3 ... •Non-deterministic

Syntax Analysis (parsing) · Lexical Analysis Syntax Analysis Semantic Analysis IR Generation IR Optimization Code Generation Optimization Source Code Machine Code Flex-pert Monday,

Lexical Analysis - Stanford University We Are Lexical Analysis Syntax Analysis Semantic Analysis IR Generation IR Optimization Code Generation Optimization Source Code Machine Code

Syntax Analysis - Stanford University We Are Lexical Analysis Syntax Analysis Semantic Analysis IR Generation IR Optimization Code Generation Optimization Source Code Machine Code

Topic: Lexical Analysis€¦ · The Lexical Analysis Scanner Token-StreamProgram code ATokenis a sequence of characters, which together form a unit. Tokens are subsumed inclasses

Lecture 10: Code Generation 1. You are here 2 Executable code exe Source text txt Compiler Lexical Analysis Syntax Analysis Semantic Analysis Inter. Rep

Basic Compiler Functions Grammars Lexical Analysis Syntactic Analysis Code Generation

Code Generation II. 2 Compiler IC Program ic x86 executable exe Lexical Analysis Syntax Analysis Parsing…

Lexical Analysis - Computer Science and Engineering are we? Where We Are Lexical Analysis Syntax Analysis Semantic Analysis IR Generation IR Optimization Code Generation Optimization

Lecture 8: Intermediate Code CS 540 Spring 2009. CS 540 GMU Spring 20092 Compiler Architecture Scanner (lexical analysis) Parser (syntax analysis) Code

Lexical Analysis (Scanning) Lexical Analysis (Scanning)

Plan for Lexical Analysis with Jlex and One Pass Code Gen ...cs453/yr2013/Slides/...CS453 Lecture Lexical Analysis with JLex 1 Plan for Lexical Analysis with Jlex and One Pass Code