[ 1 ] Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009 ETAPS/LDTA, York Banana Algebra: Jacob...

[ 1 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Banana Algebra:Banana Algebra:

Jacob Andersen[ jacand@cs.au.dk ]

Aarhus University

Claus Brabrand[ brabrand@itu.dk ]

IT University of Copenhagen

Syntactic Language Extension via an Algebra of Languages and Transformations

[ 2 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

AbstractAbstract

We propose an algebra of languages and transformations as a means for extending languages syntactically. The algebra provides a layer of high-level abstractions built on top of languages (captured by CFGs) and transformations (captured by constructive catamorphisms).

The algebra is self-contained in that any term of the algebra specifying a transformation can be reduced to a constant catamorphism, before the transformation is run. Thus, the algebra comes "for free" without sacrificing the strong safety and efficiency properties of constructive catamorphisms.

The entire algebra as presented in the paper is implemented as the Banana Algebra Tool which may be used to syntactically extend languages in an incremental and modular fashion via algebraic composition of previously defined languages and transformations. We demonstrate and evaluate the tool via several kinds of extensions.

[ 3 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Introduction: "What is a Banana?"

Bananas for Language Transformation

Language Extension Pattern

Banana Algebra

Examples

Implementation

Related Work

Conclusion

OutlineOutline

[ 4 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

What is a 'Banana' ?What is a 'Banana' ?

Datatype; "list":

Banana ("sum-of-list"):

Separation of recursion and evaluation

Implicit recursion on input structure

bottom-up re-combination of intermediate results

list = Num N | Cons N * list

[Num n] = n[Cons n l] = n + [l]

list N

(aka. "Catamorphism" )

(| n.n , (n,l ).n+l |)

[ 5 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Language TransformationLanguage Transformation

Bananas (statically typed):

Source language: 'LS'

Target language: 'LT'

Nonterminal-typing: ' '

Reconstructors: 'c '

list = Num N | Cons N * list

tree = Nil | Leaf N | Node N * tree * tree

[Num n] = Leaf n[Cons n l] = Node n (Nil) [l]

[list -> tree]

(| LS -> LT [] c |)

Type-check'able!

LS -> LT

[ 6 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Banana properties:Simple (corresponds to: “simple recursion”)Safe (syntactically safe + always terminate)Efficient (linear time in size of input + output)(Expressive) (…enough for interesting extensions)

Banana Algebra “for free” (16 banana ops):ModularIncrementalSimpleSafeEfficient(Expressive)

Statically reduce: Banana Algebra (term) Banana (term)

Banana PropertiesBanana Properties"The metafront System: Safe and Extensible Parsing and Transformation" [ Claus Brabrand | Michael Schwartzbach ] ( LDTA 2003, SCP J. 2007 )

"Growing Languages with Metamorphic Syntax Macros" [ Claus Brabrand | Michael Schwartzbach ] ( PEPM 2002 )

[ 7 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Introduction: "What is a Banana?"

Bananas for Language Transformation

Language Extension Pattern

Banana Algebra

Examples

Implementation

Related Work

Conclusion

OutlineOutline

[ 8 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Language Extension PatternLanguage Extension Pattern

[var V] = var V

[lam V E] = lam V [E ]

[app E1 E2] = app [E1] [E2]

[zero] = lam z (var z)

[succ E] = lam s [E ]

[pred E] = app [E ] (lam z (var z))

Numeral extension: Lambda-Calculus:

Nonterminal typing:

Reconstructors:

[Exp -> Exp]

Exp : var Id : lam Id * Exp : app Exp * Exp : zero : succ Exp : pred Exp

Exp : var Id : lam Id * Exp : app Exp * Exp

'LT''LS'

' '

'c '

(| LS -> LT [] c |)Catamorphism:

Using very simple numeral encoding

[ 9 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Algebraic SolutionAlgebraic Solution

Exp : var Id : lam Id * Exp : app Exp * Exp

Exp : zero : succ Exp : pred Exp

(| ln -> l [Exp -> Exp]

[zero] = lam z (var z) [succ E] = lam s [E ] [pred E] = app [E ] ...|)

idx

+

lnl

ln l

ln+l l

l l

[ 10 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Banana AlgebraBanana Algebra

Languages (L):

lv

L \ L

L + L

src ( X )

tgt ( X )

let v = L in L

letx w = X in L

Transformations (X):

xw

X \ L

X + X

X X

idx ( L )

let v = L in X

letx w = X in X

(| L -> L [] c |){ CFG }

[ 11 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Algebraic LawsAlgebraic Laws

Idempotency of '+':

Commutativity of '+':

Associativity of '+':

Source-identity:

…

L L + L

L1 + L2 L2 + L1

L1 + (L2 + L3) (L1 + L2) + L3

L src(idx(L)) L tgt(idx(L))Target-identity:

[ 12 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Introduction: "What is a Banana?"

Bananas for Language Transformation

Language Extension Pattern

Banana Algebra

Examples

Implementation

Related Work

Conclusion

OutlineOutline

[ 13 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Example RevisitedExample Revisited

{ Id = [a-z] [a-z0-9]* ;

Exp.var : Id ; Exp.lam : "\\" Id "." Exp ; Exp.app : "(" Exp Exp ")" ;}

{ Exp.zero : "zero" ; Exp.succ : "succ" "(" Exp ")" ; Exp.pred : "pred" "(" Exp ")" ;}

--- "l.l" --- --- "ln.l" ---

let l = "l.l"in let ln = "ln.l"in idx(l) + (| ln -> l [Exp -> Exp] Exp.zero = '\z.z' ; Exp.succ = '\s.$1' ; Exp.pred = '($1 \z.z)' ; |)

--- "ln2l.x" ---

[ 14 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Numerals + BooleansNumerals + Booleans

l

lb l idx

+lb+l l

l l

lb l

ln l idx

+ln+l l

ln

+l+ln+lb l

…with Nums …with Bools

…with Nums & Bools?

[ 15 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Java + RepeatJava + Repeat

let java = "java.l"in let repeat = "repeat.l"in idx(java) + (| repeat -> java [Exp -> Exp, Stm -> Stm]

Stm.repeat = 'do $1 while (!($2));' ; |)

{ Stm.repeat : "repeat" Stm "until" "(" Exp ")" ";" ; }

7 lines !

{ Java ... "try" Stm "catch" ... Name.id : Id ; }

575 lines--- "java.l" ---

--- "repeat.l" ---

--- "repeat2java.x" ---

[ 16 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Concrete vs. Abstract SyntaxConcrete vs. Abstract Syntax

Exp.or : Exp1 "||" Exp ; .exp1 : Exp1 ;Exp1.and : Exp2 "&&" Exp1 ; .exp2 : Exp2 ;Exp2.add : Exp3 "+" Exp2 ; .exp3 : Exp3 ; Exp7.neg : "!" Exp8 ; .exp8 : Exp8 ;Exp8.par : "(" Exp ")" ; .var : Id ; .num : IntConst ;

Exp (with explicit assoc./prec.):Stm.repeat = Stm.do(<1>, Exp.exp1( Exp1.exp2( Exp2.exp3( Exp3.exp4( Exp4.exp5( Exp5.exp6( Exp6.exp7( Exp7.neg( Exp8.par(<2>) ))))))))) ;

Abstract syntax:

Stm.repeat = 'do $1 while (!($2));' ;

Concrete syntax:

NB: Tool supports BOTH ! (unambiguous: concrete abstract)

[ 17 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

"FUN" Example"FUN" Example

Lambda Calculus

Nums→

Unsigned arithmetic + booleans + definitions + pairs

Bools→ Defs→ Pairs→+ + +

Fun

Literals

Literals→Nums

Fun grammar transform

The "FUN" Language: used for Teaching Functional Programming

Basically The Lambda Calculus with…: numerals, booleans, arithmetic, boolean logic, local definitions, pairs, literals, lists, signs, comparisons, dynamic types, fixed-point combinators, …

(at Aarhus University)

[ 18 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Literals

Literals→Nums

"FUN" Example"FUN" Example

Lambda Calculus

Nums→

Unsigned arithmetic + booleans + definitions + pairs

Bools→ Defs→ Pairs→+ + +

Fun

Fun grammar transform

Signed arith→Nums

Literals→Nums

Fun grammar transform FunSigned GT+

Fun FunSigned+Component re-use

[ 19 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

"FUN" Example"FUN" Example

Lambda Calculus

Nums→

Unsigned arithmetic + booleans + definitions + pairs

Bools→ Defs→ Pairs→+ + +

Fun GT FunSigned GT+

Fun FunSigned+ FunCompare FunTypesafe+ +

FunCompare GT+ FunTypesafe GT+

245x Banana Algebra ops 4 MB Banana !

[ 20 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

"FUN" Usage Statistics"FUN" Usage Statistics

Usage statistics (245x operators) in "FUN":58x { …cfg… } Constant languages51x "file.l" Language inclusions28x L + L Language additions23x v Language variables17x (|L L [] c|) Constant transformations17x X + X Transformation additions14x "file.x" Transformation inclusions10x let-in Local definitions 9x idx(L) Identity transformations 8x X X Compositions 4x L \ L Language restriction 4x w Transformation variables 2x src(X) Source extractions

[ 21 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Other ExamplesOther Examples

Self-Application (The tool on itself!):

SQL embedding (in <bigwig>):

My-Java (endless variations):

[L1 << L2] = '(L1 \ L2) + L2' [X1 << X2] = '(X1 \ src(X2)) + X2'

Stm.select = 'factor (<2>) { if (<3>) return ( # \+ (<1>) ); }'

java( + sql) ( \ loops) o syntaxe_francais

[ 22 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

ImplementationImplementation

[ http://www.itu.dk/people/brabrand/banana-algebra/ ]

The 'Banana Algebra' Tool:(3,600 lines of O'Caml)

Uses (underlying technologies): 'dk.brics.grammar': for parsing, unparsing, and ambiguity analysis ! 'XSugar': for transformation: "concrete syntax abstract XML syntax" 'XSLT': for transformation: "XML XML"

[ 23 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Introduction: "What is a Banana?"

Bananas for Language Transformation

Language Extension Pattern

Banana Algebra

Examples

Implementation

Related Work

Conclusion

OutlineOutline

[ 24 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Related Work (I/III)Related Work (I/III)

Macro Systems:

"The metafront System: Safe and Extensible Parsing and Transformation" [ Claus Brabrand | Michael Schwartzbach ] ( LDTA 2003 , SCP J. 2007 )

"Growing Languages with Metamorphic Syntax Macros" [ Claus Brabrand | Michael Schwartzbach ] ( PEPM 2002 )

[ 25 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Related Work (II/III)Related Work (II/III)

Attribute Grammars:Language transformation (and extension)…

…via computation on AST's

(using "inherited" or "synthesized" or … attributes)

E.g., Eli, JastAdd, Silver, …

Rewrite Systems:Language transformation (and extension)…

…via syntactic rewriting, using encodings…:

gradually rewrite "S-syntax" to "T-syntax"

gradually rewrite "S-syntax" to "T-syntax"

E.g., Elan, TXL, ASF+SDF, Stratego/XT, …

S T

S T

Both; compared to bananas: More ambitious (expressivity) No termination guarantees (safety) Transformation "indirect" (simplicity)

[ 26 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Related Work (III/III)Related Work (III/III)

Functional Programming:Catas mimicked by "disciplined style" of fun. programming

…aided by:Traversal functions (auto-synthesized from datatypes)

Combinator libraries

"Shortcut fusion" (to eliminate ' ' at compile-time)

Category Theory:A lot of this work can be viewed as Category Theory:

Basically yeolde issue:GPL vs. DSL

[ 27 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

ConclusionConclusion

IF bananas are sufficiently:(Expressive)

THEN you get…:

Banana Algebra “for free” (16 banana ops):IncrementalModularSimpleSafeEfficient

"Niche"

Statically reduce: Banana Algebra (term) Banana (term)

[ 28 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

BONUS SLIDESBONUS SLIDES

- Reduction Semantics -

"Syntactic Language Extension via an Algebra of Languages and Transformations" [ Jacob Andersen | Claus Brabrand ] ( ITU Technical Report, Dec. 2008 )

If you want all the details:

[ 29 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Reduction SemanticsReduction Semantics

Environments:

Reduction relations:

Abbreviations: ...as a short-hand for:

...as a short-hand for:

ENVL = VARL EXPL

ENVL ENVX EXPL EXPL 'L'

ENVX = VARX EXPX

ENVL ENVX EXPX EXPX 'X'

, |- L L l (,,L,l) 'L'

(,,X,x) 'X', |- X X x

environment of languages

environment of transformations

[ 30 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Semantics (L)Semantics (L)

, l L l[CONL] , v L (v)

[VARL]

, L \ L' L l l'[RESL]

, L' L l', L L l

, L + L' L l l'[ADDL]

, L' L l', L L l

l

ll ~ l'l

lwfl

[ 31 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Semantics (L)Semantics (L)

, src (X) L lS

[SRCL], X X (| lS -> lT [] c |)

, tgt (X) L lT

[TGTL], X X (| lS -> lT [] c |)

, let v=L in L' L l'[LETL]

[v=l], L' L l', L L l

[ 32 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Semantics (X)Semantics (X)

, X \ L X x l[RESX]

, L L l, X X x

, X + X' X x x'[ADDX]

, X' X x', X X x

x

xx ~ x'x

, (| LS -> LT [] c |) X (| lS -> lT [] c |)[CONX]

, LS L lS , LT L lT(| lS -> lT [] c |)wfx

, w X (w)[VARX]

[ 33 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Semantics (X)Semantics (X)

, X' X X (| lS -> lT' [' ] c' c |)[COMPX]

, X' X (| lS' -> lT' ['] c' |)

, idx (L) X (| l -> l [id(l)] idc(l) |)[IDXX]

, letx w=X in X' X x'[LETL]

,[w=x] X' X x', X X x

, X X (| lS -> lT [ ] c |)

, L L l

lT lS'l

[ 34 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

BONUS SLIDESBONUS SLIDES

- More Examples -

[ 35 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Numeral & Boolean ExtensionNumeral & Boolean Extension

Numeral Extension (catamorphism):

Boolean Extension (catamorphism):

[var V] = var [V ]

[lam V E] = lam [V ] [E ]

[app E1 E2] = app [E1] [E2]

[zero] = lam z (var z)

[succ E] = lam s [E ]

[pred E] = app [E ] (lam z (var z))

[var V] = var [V ] [lam V E] = lam [V ] [E ] [app E1 E2] = app [E1] [E2] [true] = lam a (lam b (var a)) [false] = lam a (lam b (var b))[if E1 E2 E3] = app (app [E1] [E2]) [E3]

Exp : var Id : lam Id * Exp : app Exp * Exp : zero : succ Exp : pred Exp

Exp : var Id : lam Id * Exp : app Exp * Exp

Exp : var Id : lam Id * Exp : app Exp * Exp : true : false : if Exp Exp Exp

Exp : var Id : lam Id * Exp : app Exp * Exp

[ 36 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Lambda with BooleansLambda with Booleans

Exp : true : false : if Exp Exp Exp

(| lb -> l [Exp -> Exp]

[true] = '\a.\b . a' [false] = '\a.\b . b'[if E1 E2 E3] = '(([E1] [E2]) [E3])'

|)

idx

+

lbl

lb l

lb+l l

l l

Exp : var Id : lam Id * Exp : app Exp * Exp

[ 37 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Incremental DevelopmentIncremental Development

let l = "l.l" in idx(l) + (| "ln.l" -> l [Exp -> Exp] Exp.zero : '\z.z' ; Exp.succ : '\x.$1' ; Exp.pred : '($1 \z.z)' ; |)

let l = "l.l" in idx(l) + (| "li.l" -> l [Exp -> Exp] Exp.id : '\z.z' ; |)

{ Exp.zero : "zero" ; Exp.succ : "succ" Exp ; Exp.pred : "pred" Exp ; }

{ Id = [a-z] [a-z0-9]* ; Exp.var : Id ; Exp.lam : "\\" Id "." Exp ; Exp.app : "(" Exp Exp ")" ; }

--- "ln.l" ---

--- "ln2li.x" ---

--- "l.l" ---

--- "ln2l.x" ---let l = "l.l" in idx(l) + (| ln -> l+"li.l" [Exp -> Exp] Exp.zero : 'id' ; Exp.succ : '\x.$1' ; Exp.pred : '($1 id)' ; |)

{ Exp.id : "id" ; }--- "li.l" ---

--- "li2l.x" ---

"li2l.x" o "ln2li.x"

--- "ln2l.x" ---

[ 38 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Example cont'dExample cont'd

Both statically reduce to same catamorphism:(|

Exp.app : Exp.app($1, $2) ; Exp.lam : Exp.lam($1, $2) ; Exp.pred : Exp.app($1, Exp.lam(Id("z"), Exp.var(Id("z")))) ; Exp.succ : Exp.lam(Id("x"), $1) ; Exp.var : Exp.var($1) ; Exp.zero : Exp.lam(Id("z"), Exp.var(Id("z"))) ;|)

{ Id = [a-z] [0-9a-z]* ;

Exp.app : "(" Exp Exp ")" ; Exp.lam : "\" Id "." Exp ; Exp.var : Id ; }

{ Id = [a-z] [0-9a-z]* ;

Exp.app : "(" Exp Exp ")" ; Exp.lam : "\" Id "." Exp ; Exp.pred : "pred" Exp ; Exp.succ : "succ" Exp ; Exp.var : Id ; Exp.zero : "zero" ; }

->

[Exp -> Exp, Id->Id]

[ 40 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Usage ScenariosUsage Scenarios

Programmers:May extend existing languages (~ syntax macros)

Developers:May embed DSLs into host languages (SQL in Java)

Developers (and teachers):May incrementally specify multi-layered languages

Compiler writers:May rely on tool and implement only a small core

(and then specify the rest externally as extensions)

[ 41 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

BONUS SLIDESBONUS SLIDES

- Parsing & Error Reporting -

[ 42 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

ParsingParsing

Parsing (XSugar):Variant of Earley's algorithm: O ( ||3 )

Can parse any context-free grammar

Closed under union of languages

Support for production priority

Tool easily adapts to other parsing algorithms

[ 43 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Unparsing:Canonical whitespace

Ambiguity: parsingunparsingAmbiguity: parsingunparsing

ASTL / ~L

.

L

. Parsing:Grammar ambiguity

ASTL / ~L

.

L

.

.

[ 44 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Ambiguity AnalysisAmbiguity Analysis

Ambiguity Analysis:

Using implementation ( ) on:Source language;

Target language; and/or

…all intermediate languages (somewhat expensive)

(Note: Ambiguity analysis comes with XSugar tool)

"Analyzing Ambiguity of Context-Free Grammars" [ Claus Brabrand | Robert Giegerich | Anders Møller ] ( CIAA 2007 )

"dk.brics.grammar"[ by Anders Møller ]

[ 45 ]Claus Brabrand, ITU BANANA ALGEBRA March 28, 2009ETAPS/LDTA, York

Error ReportingError Reporting

Error reporting:Static parse-error (O'Caml-lex):

Static transformation error (XSugar):

(is actually a parse-error in a cata reconstructor)

Dynamic parse-error (XSugar):

Dynamic transformation error:impossible :-)

*** In ln2l.x (4,4)-(4,7): Parse error at "Exp"

*** Parse error at character 6 (line 1, column 7) in /tmp/shape84e645.txt

*** Parse error at character 23 (line 1, column 24) in /dev/stdin

Could beimproved

Prototype