Cse321, Programming Languages and Compilers 1 6/13/2015 Lecture #6, Jan. 31, 2007 Notes about homework Project #1 More about ML-LEX –String lexers –File

Cse321, Programming Languages and Compilers

104/18/23

Lecture #6, Jan. 31, 2007•Notes about homework•Project #1•More about ML-LEX

–String lexers

–File lexers

–Handling exceptions

–Using named REs

•Library functions•Anonymous functions•The compile manager


204/18/23

Survey

• How many intend to take CS322 next quarter?

• How many people would like to see the class in the evening as it is this quarter (6:00 - 7:50 pm)

• How many would like to see the class earlier in the day. Say 2:00 – 3:50 pm

• How many would like to see the class later in the afternoon, but not in the evening. Say 4:00 – 5:50


304/18/23

A note about handing in homework• When you hand in homework, please hand in the

following:1. The complete file as you wrote it, with no extra text, etc. I should be able

to load the file (as is) into SML.

2. A trace of your test of the program. This should include commands to load the file above, and a sequence of tests that test the code you wrote. Many times I ask you to extend some code. Be sure and test the code you wrote, not my code. My code has enough errors I don’t want to know about new ones, unless it affects your code.

3. You should include enough tests to convunce me your program works. You should include enough tests so that every statement in your program is exercised at least once in some test. 3 tests per function is the minimum, some functions may require more if they have many paths.

4. An optional cover sheet where you provide any additional information I need to grade your assignment.

– Be sure that your name is clearly written on the top left hand side of what you hand in.

– If your program doesn’t load, a trace of the errors may help me figure out what went wrong, so I can suggest a fix.


404/18/23

Project 1

• Project 1 is assigned today (Wed. Jan 31, 2007)

• It is Due in 2 weeks: Wed. Feb. 15, 2007• It can be downloaded off the web page.• Other useful links can also be found there.

• There will be no homework assigned next Monday or Wednesday. But there is homework assigned today.

• Today’s homework is practice to get you started on using sml-lex.


504/18/23

Example lex filetype lexresult = unit;

type pos = int;

type svalue = int;

exception EOF;

fun eof () = (print "eof"; raise EOF);

%%

%%

[\t\ ]+

=> ( lex() (* ignore whitespace *) ) ;

Anne|Bob|Spot

=> ( print (yytext^": is a proper noun\n"));

a|the

=> ( print(yytext^": is an article\n") );

boy|girl|dog

=> ( print(yytext^": is a noun\n") );

walked|chased|ran|bit

=> ( print(yytext^": is a verb\n") );

[a-zA-Z]+

=> ( print(yytext^": Might be a noun?\n") );

.|\n

=> ( print yytext (* Echo the string *) );

lexresult must be defined in every lex

program

The function eof must be defined in every lex program


604/18/23

Running ml-lex• Cd to the directory where the foo.lex file resides (or else

use the full pathname of the file).• E.g.

..LexYacc\english> ml-lex english.lexNumber of states = 43Number of distinct rows = 33Approx. memory size of trans. table = 4257 bytes

• This creates the file english.lex.sml

• Start SML and then load the file.…LexYacc\english> smlStandard ML of New Jersey v110.57 [built: Mon Nov 21 21:46:28 2005]- use "english.lex.sml";[opening english.lex.sml]structure Mlex : sig val makeLexer : (int -> string) -> unit -> Internal.result exception LexError structure Internal : <sig> structure UserDeclarations : <sig> endval it = () : unit

This is a synonym for the type lexresult

defined in the lex file.


704/18/23

Building the lexer• Consider the function: Mlex.makeLexer

Mlex.makeLexer : (int -> string) -> unit -> lexresult

• It takes a function as an argument. This function feeds the lexical analyzer the input “n” characters at a time.

val testString = ref "the boy chased the dog";

fun feed n = let val ss = !testString in if String.size ss < n then ( testString := ""; ss ) else ( testString := String.extract(ss,n,NONE); String.extract(ss,0,SOME n) ) end;

val lex = Mlex.makeLexer feed;


804/18/23

Running the lexer- val lex = Mlex.makeLexer feed;val lex = fn : unit -> Mlex.Internal.result- lex();the: is an articleval it = () : Mlex.Internal.result- lex();boy: is a nounval it = () : Mlex.Internal.result- lex();chased: is a verbval it = () : Mlex.Internal.result- lex();the: is an articleval it = () : Mlex.Internal.result- lex();dog: is a nounval it = () : Mlex.Internal.result- lex();unexpected end of fileuncaught exception EOF raised at: english.lex.sml:9.53-9.56


904/18/23

Exceptions

exception Error of string;

fun error s = raise (Error s);

fun ex3 b =

(if b then error "true branch"

else "false branch")

handle Error message => message

| other => raise other;

A new exception is declared, it carries a string as error information. Exceptions

can carry any kind of data.

Exceptions can be raised, to short-circuit normal

evaluation.

Main computation

returns a string

Handler also returns a stringA handler, like a “case” has multiple clauses, each can

handle a different kind of error. Note “|” separating clauses.

Keyword handle


1004/18/23

Syntax of “case” vs “handle”• “case” is before the computation being

analyzed

case (revonto (g x) 4) of

[] => true

| (x:xs) => false

• “handle” is after the computation that might fail

(compute (g y) (length zs))

handle Error s => g s

| BadLevel n => n+1

| other => raise other


1104/18/23

Catching Exceptions in the Lexer

val lex =

let val f = Mlex.makeLexer feed

fun lex () =

(f ()) handle

Mlex.UserDeclarations.EOF =>

print "\nReached end of file\n"

| other => print "Lex Error"

in lex end;

Things defined in the first section of a lex

file appear in the inner library


1204/18/23

A String Lexer• For testing purposes it would be nice to

generate lexers that lex a string given as input (rather than one fixed in a variable like teststring).

fun makeStringLexer s = let val testString = ref s fun feed n = let val ss = !testString in if String.size ss < n then ( testString := ""; ss ) else ( testString := String.extract(ss,n,NONE); String.extract(ss,0,SOME n) ) end in Mlex.makeLexer feed end;


1304/18/23

Testing a lexer interactively- val f = makeStringLexer "the boy sang";val f = fn : unit -> Mlex.Internal.result

- f ();the: is an articleval it = () : Mlex.Internal.result- f ();boy: is a nounval it = () : Mlex.Internal.result- f ();sang: Might be a noun?val it = () : Mlex.Internal.result- f ();unexpected end of fileuncaught exception EOF raised at: english.lex.sml:9.53-9.56


1404/18/23

Exercise

• Can we make a string lexer that catches exceptions?

• Try it in class …


1504/18/23

Running from a file

• To lex a file, rather than an explicit string we need to define a function that reads and returns “n” characters of a file at a time.

fun inputc h =

(fn n =>

TextIO.inputN(h,n):string);

val lex = let val h = TextIO.openIn “test.english”

in Mlex.makeLexer (inputc h) end;

Opens a file and returns a handle

Reads n characters from a file pointed to

by handle


1604/18/23

Running lexval lex =

Mlex.makeLexer

(inputc "test.english");

- lex();

the: is an article

val it = () : unit

- lex();

9val it = () : unit

- lex();

9val it = () : unit

- lex();

chased: is a verb

val it = () : unit

the 99 chased the dog

The file test.english


1704/18/23

Named REs in SML-Lextype lexresult = Tokentype pos = int;type svalue = int;fun eof () = EOF; %%digit=[0-9];number={digit}+ ;%%

[\t\ ]+ => ( lex() ) ;

Anne|Bob|Spot| {number} => ( ProperNoun yytext );

a|the => ( Article yytext );

boy|girl|dog => ( Noun yytext );

walked|chased|ran|bit => ( Verb yytext );

We can name a REName = RE Semicolon

Use a name by surrounding it in

{ }s

english2.lex

Note that all unrecognized

input will raise the default exception.

Not sure why we need pos and svalue


1804/18/23

A driver programdatatype Token = ProperNoun of string | Noun of string | Verb of string | Article of string | EOF;

use "english2.lex.sml";

fun makeStringLexer s = let val testString = ref s fun feed n = let val ss = !testString in if String.size ss < n then ( testString := ""; ss ) else ( testString := String.extract(ss,n,NONE); String.extract(ss,0,SOME n) ) end val f = Mlex.makeLexer feed fun lex() = (f ()) handle Mlex.LexError => (print "Lex error\n"; EOF) in lex end;

Must come after the definition of token or there will be errors.


1904/18/23

Test it out- val f = makeStringLexer "Anne chased the dog";

val f = fn : unit -> Token

- f();

val it = ProperNoun "Anne" : Token

- f();

val it = Verb "chased" : Token

- f();

val it = Article "the" : Token

- f();

val it = Noun "dog" : Token

- f();

val it = EOF : Token


2004/18/23

More SML

• In SML we use library functions all the time.– Int.toString– List.exists

• The list library functions are particularly useful.– These library functions often take a function as an argument– List.map : ('a -> 'b) -> 'a list -> 'b list– List.find : ('a -> bool) -> 'a list -> 'a option– List.filter : ('a -> bool) -> 'a list -> 'a list– List.exists : ('a -> bool) -> 'a list -> bool– List.all : ('a -> bool) -> 'a list -> bool– List.foldr : ('a * 'b -> 'b) -> 'b -> 'a list -> 'b

• It is worth studying these functions closely


2104/18/23

List.map captures a pattern

• Add one to every element of a list– fun addone [] = []– | addone (x::xs) = (x + 1) :: addone xs

addone [2,3,4] val it = [3,4,5] : int list

• Turn a list of ints into a list of strings– fun stringy [] = []– | stringy (x::xs) = (Int.toString x) :: stringy xs

stringy [2,5,9] val it = ["2","5","9"]:string list

• Negate every element of a list– fun negL [] = []– | negL (x::xs) = (not x) :: negL xs

negL [true,3 > 4] val it = [false,true] : bool list


2204/18/23

Patternfun addone [] = [] | addone (x::xs) = (x + 1) :: addone xsfun stringy [] = [] | stringy (x::xs) = (Int.toString x) :: stringy xsfun negL [] = [] | negL (x::xs) = (not x) :: negL xs

fun map f [] = [] | map f (x::xs) = (f x) :: (map f xs)

val ex1 = map (fn x => (x+1)) [2,3,4];val ex1 = [3,4,5] : int list

val ex2 = map Int.toString [2,5,7];val ex2 = ["2","5","7"] : string list

val ex3 = map not [true, 3 > 4];val ex3 = [false,true] : bool list


2304/18/23

Anonymous functions• Study: (fn x => (x+1))

– It is an anonymous function. A function without a name.– It has one parameter “x”– It adds one to its parameter, and returns the result.

(fn x => (x+1)) 4;val it = 5 : int

• Any non-recursive function can be written anonymously.– (fn x => x = 5)

» Tests if its parameter is equal to 5map (fn x => x=5) [1,4,5,3,5];val it = [false,false,true,false,true] : bool list

– (fn x => fn y => (x,y))» Has two parameters» Returns a pair

– (fn (x,y) => (not y, x+3))» What is the type of this function?


2404/18/23

List.find

• Used for searching a list.– List.find : ('a -> bool) -> 'a list -> 'a option

• Uses a function as a parameter to determine if the search is successful.

• E.g. Is there an even element in a list?List.find even [1,3,5];

val it = NONE : int option

List.find even [1,3,4];

val it = SOME 4 : int option


2504/18/23

List.find and anonymous functions

List.find (fn x => x = "Tim")

["Tom", "Jane"];

val it = NONE : string option

List.find (fn x => even x andalso x>10)

[2,4,5,12];



2604/18/23

List.filter

Filter keeps some elements, and throws away others.– List.filter : ('a -> bool) -> 'a list -> 'a list

It uses a function (p) as a parameter to decide which elements to keep (p x = true), and which to throw away (p x = false)

val ex6 = List.filter even [1,2,3,4,5,6];

val ex6 = [2,4,6] : int list


2704/18/23

List.filter and anonymous functionsval people = [("tim",22),("john",18),("jane",25),("tim",8)];

val ex7 = filter

(fn (nm,age) => nm <> "tim" orelse age>10)

people;

val ex7 =

[("tim",22),("john",18),("jane",25)]

: (string * int) list


2804/18/23

List.exists

• “exists” is like “find” in that it searches a list– but rather than the element that completes the search it is only

interested in if such an element exists.– List.exists : ('a -> bool) -> 'a list -> bool

• Uses a function as a parameter to determine if the search is successful.

val ex8 = List.exists even [2,3,5];

val ex8 = true : bool

• Note that even if only 1 element in the list causes the function to be true, exists returns true.


2904/18/23

List.all

• List.all tests elements in a list for a property. It returns true only if every element has that property.– List.all : ('a -> bool) -> 'a list -> bool

• Uses a function as a parameter to perform the test.

val ex9 = List.all even [2,4,5];

val ex9 = false : bool

• List.exists and List.all are related functions. They are duals.– not(List.all p xs) = List.exists (fn x => not(p x)) xs


3004/18/23

List.foldr captures a pattern

• Add up every element in a list.

fun sum [] = 0

| sum (x::xs) = x + (sum xs);

• Compute the maximum element in a list of natural numbers (Integers >= 0).

fun maximum [] = 0

| maximum (x::xs) = Int.max(x,maximum xs);

• Compute if every element in a list of boolean is true.

fun allTrue [] = true

| allTrue (x::xs) = x andalso (allTrue xs);


3104/18/23

Patternfun sum [] = 0


fun maximum [] = 0




fun foldr acc base [ ] = base

| foldr acc base (x::xs)

= acc(x,foldr acc base xs);


3204/18/23

See the pattern in use.fun sum [] = 0


fun sum xs = foldr (op +) 0 xs;

fun maximum [] = 0


fun maximum xs = foldr Int.max 0 xs;



fun allTrue xs =

foldr (fn (a,b) => a andalso b) true xs;


3304/18/23

Take another lookWhat does this function do?

fun ok [] = false

| ok xs = not(exists (fn ys => xs=ys) (!old))

andalso

not(exists (fn ys => xs=ys) (!worklist))


3404/18/23

The Option Library- open Option;opening Option datatype 'a option = NONE | SOME of 'a exception Option val getOpt : 'a option * 'a -> 'a val isSome : 'a option -> bool val valOf : 'a option -> 'a val filter : ('a -> bool) -> 'a -> 'a option val join : 'a option option -> 'a option val app : ('a -> unit) -> 'a option -> unit val map : ('a -> 'b) -> 'a option -> 'b option val mapPartial : ('a -> 'b option) -> 'a option -> ‘ b option


3504/18/23

Interesting functions that use Options• Int.fromString: string -> int option

Int.fromString "234";


Int.fromString "abc";


• String.extract: string * int * int option -> stringString.extract("abcde",1,SOME 3);

val it = "bcd" : string

String.extract("abcde",1,NONE);

val it = "bcde" : string


3604/18/23

More option functions• List.find: ('a -> bool) -> 'a list -> 'a option

List.find even [1,3,5];


List.find (fn x => x="tim") ["tom","tim","jane"];

val it = SOME "tim" : string option

• List.getItem: 'a list -> ('a * 'a list) option– List.getItem [1,2,3,4];– val it = SOME (1,[2,3,4])

– List.getItem [];– val it = NONE


3704/18/23

Using While Loopsfun ident c cs = let val xs = ref cs val x = ref c val ans = ref [] in while (not(null(!xs)) andalso Char.isAlpha (hd (!xs))) do ( ans := !ans @ [!x] ; x := hd(!xs) ; xs := tl(!xs) ); (Id (String.implode (!ans @ [!x])), !xs)

end

Don’t forget to test for empty list


3804/18/23

The Compile-manager

• The compile manager is a “make” like facility for SML.

• It has some documentation found here.– http://www.smlnj.org/doc/CM/index.html

• Peter Lee’s notes also contains a brief (but out of date) introduction – http://www.cs.cmu.edu/~petel/smlguide/smlnj.htm

• The basic approach is that a single file contains a list of all the pieces that comprise a project.– A complete scan of all the pieces determines a dependency graph

of which pieces depend on which other pieces.

– It determines the time-stamp of each piece

– It recompiles all the pieces that are out of date

– It links everything together


3904/18/23

Structures and Libraries

• The compile manager compiles whole compilation units called libraries or structures.

•Break program into files where each file contains 1 library.

• If a file needs stuff from another library, open that library inside of the file.

• Create a compile manager source file that lists all the libraries.


4004/18/23

structure TypeDecls = struct

datatype Token = ProperNoun of string | Noun of string | Verb of string | Article of string | EOF;

end

File: TypeDecls.sml

Name of the library, can be different from the name of the file.

Libraries are bracketed by“structure name = struct”

and “end”


4104/18/23

structure Driver = struct

open TypeDecls

fun makeStringLexer s = let val testString = ref s fun feed n = let val ss = !testString in if String.size ss < n then ( testString := ""; ss ) else ( testString := String.extract(ss,n,NONE); String.extract(ss,0,SOME n) ) end val f = Mlex.makeLexer feed fun lex() = (f ()) handle Mlex.LexError => (print "Lex error\n"; EOF) in lex:(unit -> Token) end; end

File: Driver.smlOpens libraries with needed components


4204/18/23

open TypeDecls;

type lexresult = Token;type pos = int;type svalue = int;fun eof () = EOF; %%digit=[0-9];number={digit}+ ;%%

[\t\ ]+ => ( lex() (* ignore whitespace *) ) ;

Anne|Bob|Spot| {number} => ( ProperNoun yytext );

a|the => ( Article yytext );

boy|girl|dog => ( Noun yytext );

walked|chased|ran|bit => ( Verb yytext );

File: english2.lex

Opens libraries with needed components


4304/18/23

The sources file

group (sources.cm)is

typeDecls.sml driver.sml english2.lex $/basis.cm $/smlnj-lib.cm

File: sources.cm

The name of the file where this list

resides

All the user defined pieces

System libraries

The extension “.lex” tells the

manager to run ml-lex!


4404/18/23

Putting it all together.- CM.make "sources.cm";[scanning sources.cm][D:\programs\SML110.57\bin\ml-lex english2.lex]

Number of states = 39Number of distinct rows = 33Approx. memory size of trans. table = 4257 bytes[parsing (sources.cm):english2.lex.sml][compiling (sources.cm):english2.lex.sml][code: 13069, data: 5565, env: 1235 bytes][New bindings added.]val it = true : bool- open Driver;opening Driver val makeStringLexer : string -> unit -> Token datatype Token = Article of string | EOF | Noun of string | ProperNoun of string | Verb of string- val f = makeStringLexer "Anne bit Bob";val f = fn : unit -> Token- f ();val it = ProperNoun "Anne" : Token- f();val it = Verb "bit" : Token- f();val it = ProperNoun "Bob" : Token- f();val it = EOF : Token

Note how it calls ml-lex to create

english2.lex.sml

It compiles the “*.sml” files

I open the Library Driver to get at the function makeStringLexer


4504/18/23

Assignments• Programming exercise 6 is now posted on the website.

You should download it.• It requires you to

– Create a simple lexical analyzer using SML-lex

– write short functions using library functions and anonymous functions to answer questions about.

datatype Gender = Male | Female;

datatype Color = Red | Blue | Green | Yellow | Orange;

val people =

[("tim",25,Male,Red) ,("mary",30,Female,Orange)

,("john",14,Male,Yellow) ,("bob",55,Male,Green)

,("jane",19,Female,Blue) ,("alexi",25,Male,Green)

,("joan",31,Female,Blue) ,("jill",16,Female,Green)];

Documents

Cse321, Programming Languages and Compilers 1 6/13/2015 Lecture #6, Jan. 31, 2007 Notes about homework Project #1 More about ML-LEX –String lexers –File