Center for Software Sciences Northeastern University A domain specific language for Traversal Specification Johan Ovlinger Mitchell Wand Northeastern

Center for Software SciencesNortheastern University

A domain specific language for Traversal Specification

Johan Ovlinger

Mitchell WandNortheastern University

Center for Software Sciences

Talk Outline

• Introduction– introduce visitor pattern– point out some of its shortcomings– introduce our motivating, and running,

example– show how it is unsatisfactorily solved by

visitor pattern, despite the obvious applicability.



Introducing the Visitor Pattern

• Visitor pattern is useful for – removing structural assumptions from

programs, by separating navigational and behavioral concerns

– adding behavior over classes independently of their definition.


Critiquing Traversals

• Visitor is guided around object graph by traversal (iterator, strategy)

• Unfortunately, the traversal serializes the hierarchical structure of the object being traversed into a series of visits.

• One global traversal is typically hard-coded into classes, restricting its interface to Visitors to LCD.


Motivating Example

• Taken from real world experience, writing a compiler.

• We want to verify that all variables have been declared before use.

• Obvious use of visitor pattern (suggested in “Design Patterns”).


Useless language

• Our example is a very simple language that only has the ability to bind and reference variables.

• Lets are non recursive• No Shadowing (for simplicity) • Epitomizes a common task: Finding

undeclared variables.• Leave grammar up to imagination.


AST Class DiagramProg

Let Ref Fun

StringBindEmpty

Exp

BindList


A small AST object

Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

LET foo = FUN (x) {y} bar = fooIN LET grok = barIN grok

Prog


Visitor Considerations• Strategy:

– Calculate new bindings for one level and also analyze their bodies at same time

– Remember which variables are in scope when entering a Let, and reset on exiting.

– Since body of binding may also include Lets, must also remember List of new bindings to be added to scope.

– Calculate list of bindings to add from let and process their bodies simultaneously.


Visitor’s view of Traversal

Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog

before Progvisit in scope to add

•administrator:

•these are missing the Bind middle visit

•administrator:

•these are missing the Bind middle visit



Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog

before Progbefore Let visit in scope to add


before Progbefore Let before Bindvisit in scope to add


Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog


before Progbefore Let before Bindbefore Fun xvisit in scope to add


Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog


before Progbefore Let before Bindbefore Fun xbefore Ref xvisit in scope to add


Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog


before Progbefore Let before Bindbefore Funbefore Ref xafter Ref xvisit in scope to add


Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog


before Progbefore Let before Bindbefore Funbefore Ref xafter Ref xafter Funvisit in scope to add


Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog


before Progbefore Let before Bindbefore Funbefore Ref xafter Ref xafter Funbefore Bindvisit in scope to add


Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog


...before Bindbefore Funbefore Ref xafter Ref xafter Fun xbefore Bindbefore Refafter Refbefore Emptyafter Emptyafter Bind barvisit in scope to add


Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog


...before Funbefore Ref xafter Ref xafter Fun xbefore Bindbefore Refafter Refbefore Emptyafter Emptyafter Bind barafter Bind foo,barvisit in scope to add


Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog


...before Ref xafter Ref xafter Fun xbefore Bindbefore Refafter Refbefore Emptyafter Emptyafter Bind barafter Bind foo,barmiddle Let foo,barvisit in scope to add


Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog


...after Bind barafter Bind foo,barmiddle Let foo,barbefore Let foo,barbefore Bind foo,barbefore Ref foo,barafter Ref foo,barbefore Emptyfoo,barafter Empty foo,barafter Bind Letfoo,bar grokmiddle Let foo,bar,grokvisit in scope to add


Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog


...middle Let foo,barbefore Let foo,barbefore Bind foo,barbefore Ref foo,barafter Ref foo,barbefore Emptyfoo,barafter Empty foo,barafter Bind Letfoo,bar grokmiddle Let foo,bar,grokbefore Ref foo,bar,grokbefore Ref foo,bar,grok visit in scope to add


Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog


...before Let foo,barbefore Bind foo,barbefore Ref foo,barafter Ref foo,barbefore Emptyfoo,barafter Empty foo,barafter Bind Letfoo,bar grokmiddle Let foo,bar,grokbefore Ref foo,bar,grokbefore Ref foo,bar,grokafter Let foo,barvisit in scope to add


Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog


...before Ref foo,barafter Ref foo,barbefore Emptyfoo,barafter Empty foo,barafter Bind Letfoo,bar grokmiddle Let foo,bar,grokbefore Ref foo,bar,grokbefore Ref foo,bar,grokafter Let foo,barafter Letafter Prog visit in scope to add


Let Bind

Bind

Empty

Let

Ref

foo

bar

Fun

Ref

x Ref y

foo

Bind

Empty

grok Ref bar

grok

Prog


Implementation in Java

• We assume a “typical” traversal distributed through classes which calls appropriate accept method on Visitor.

• Visitor is a superclass of all visitors, and has default do-nothing accept methods.


Visitor in Javaclass UDVisitor extends Visitor { Vector inscope,boundbylet; Stack oldscope, oldbound;

void before(Ref host) { if (!inscope.contains(host.str)) complain(host); } void after(Bind host) { boundbylet.addElement(host.str); } void before(Fun host) { inscope.addElement(host.str); } void after(Fun host) { inscope.removeElement(host.str); }


Visitor Cont void before(Let host) { oldscope.push(inscope.copy()); oldbound.push(boundbylet.copy()); boundbylet = new Vector(); } void middle(Let host) { inscope.add_from(boundbylet); } void after(Let host) { inscope = oldscope.pop(); boundbylet = oldbound.pop(); }


Visitor Interface Critique

– Since all traversal code is outside our control, visitors have to pass parameters to subsequent visits by side-effecting own variables. Very imperative.

– We assume a more intricate interface than normally accepted -- rather than just accept, we have before, middle and after. Otherwise, it would be hard to know when to add new bindings to scope.

•administrator:

•We could do with just accept by adding bound at empty and checking on the way down.

•administrator:

•We could do with just accept by adding bound at empty and checking on the way down.


Structural Critique

– The structure cannot be ignored -- the programmer needs to know when to save/restore the parameter passing variables.

– Programmer needs to save and restore state manually

– Restoring state depends on our non-standard visitor interface.

– The order of traversing parts is vitally important.

•administrator:

•the fact that Binding’s body may contain lets forces Let to s/r bound. Also imp is when not to restore vars

•administrator:

•the fact that Binding’s body may contain lets forces Let to s/r bound. Also imp is when not to restore vars

•administrator:

•if we s/r around Bindings, then all is lost

•administrator:

•if we s/r around Bindings, then all is lost

•administrator:

•to restore state with only accept would be hard. We know after will close before

•administrator:

•to restore state with only accept would be hard. We know after will close before

•administrator:

•who ever said that it could?

•administrator:

•who ever said that it could?

•administrator:

•Bindings before exp

•administrator:

•Bindings before exp


Visitor Critique Summary

– Visitors want to return results and accept arguments like recursive functions

– The visitor needs to control the order in which parts are traversed.

– A more elaborate traversal/visitor interface is desirable. accept is least common denominator.

•administrator:

•it is natural to expect that complex behavior will need more complicated behavior than before / middle / after

•administrator:

•it is natural to expect that complex behavior will need more complicated behavior than before / middle / after

Talk Outline

• Introduction

• Traversal Specifications– compare to std traversal



Traversal Specification

– Define traversals with domain specific language.

– Traversal is external to classes, and can be tailored to just one visitor, or be generic.

– Traversal is explicit about traversing all parts. Assume only local knowledge of the object structure. Non-local knowledge cannot be expressed in Traversal Spec.

•administrator:

•surprisingly, we don’t have the faults we pointed out with visitor pattern.

•administrator:

•surprisingly, we don’t have the faults we pointed out with visitor pattern.


Features of our Language• Separate Behavior, Navigation, and

Class Definition

• Specify which parts of an object are to be traversed, and which order

• Traversals and Visitors return results

• Traversal may bind intermediate results

• Behavior may influence Navigation

• Iterate over collections automatically

•administrator:

•There is a lot of talking about this slide

•administrator:

•There is a lot of talking about this slide


traversal everywhere() = Program => visit before ()

traverse exp () visit after ()

Ref => visit before () visit after()

Let => visit before () traverse bindings() visit middle () traverse exp() visit after () Binding => visit before () traverse exp () visit middle () traverse bindings()

visit after () ...

Go Everywhere Traversal

•administrator:

•this makes an interface everywhere_visitor that looks just like the Visitor class. Interface, not class, because we will have many traversals, with different requirements, while the std model has just one traversals, or if several, all w/ subsets of one big interface.

•administrator:

•this makes an interface everywhere_visitor that looks just like the Visitor class. Interface, not class, because we will have many traversals, with different requirements, while the std model has just one traversals, or if several, all w/ subsets of one big interface.

•administrator:

•introduce concept of subtraversal

•administrator:

•introduce concept of subtraversal


The Go Everywhere Traversal

• Will generate an interface with before, after, middle methods, just like the traversal will impose its p

• Each traversal imposes different requirements on visitors it is used with

• By generating Java interfaces instead of abstract classes, a visitor may be used with several traversals.

Talk Outline

• Introduction

• Traversal Specifications– compare to std traversal– overview



Syntax of the Language– Variables are declared with C/C++/Java

syntax, with optional types– A Traversal is a named set of

TraversalEntries, named after the class of objects they traverse.

– TraversalEntry describes how to traverse objects of a certain class, using a list of Actions including visit and traverse.


More syntax

– On entering an object, the most specific TraversalEntry for that object is executed, by performing the Actions in order and returning the result of the last one.

– Visit actions name the visitor method to invoke, and which arguments to pass (current object is always passed).

– Traversal actions specify the part of the current object to traverse, along with arguments to the traversal.

Talk Outline

• Introduction

• Traversal Specifications– compare to std traversal– overview– running example

• detailed interface to visitor• recursion



Our Strategy

• Write a traversal that calls specific methods on visitor, at appropriate time.

• Visits return their results to traversal.

• Generate a Java Interface from the traversal to be implemented by visitor.

• Traversal uses recursion to save/restore visitor variables automatically.


Traversal Spec for ASTtraversal unbound(Vector scope) = Program => traverse exp (scope) Ref => visit checkref(scope) Let => Vector bound = traverse bindings(scope) Vector newscope = visit concat(scope,bound)

traverse exp(newscope) Binding => Vector bound = traverse bindings(scope)

visit addtobound(bound) traverse exp(scope) bound

Empty => visit emptybound Fun => visit addvar(scope) traverse exp(scope) visit remvar(scope)


Interface for AST Traversalinterface unbound_visitor { void checkref(Ref host, Vector scope); Vector concat(Let host,Vector a,Vector b); void addtobound(Binding host, Vector bound); Vector emptybound(Empty host, Vector bound); void addvar(Fun host,Vector scope); void remvar(Fun host,Vector scope); }


Visitor Definition

class UBVisitor implements unbound_visitor { Vector concat(Let host,Vector a,Vector b) {...} void checkref(Ref host, Vector inscope) { if (!inscope.contains(host)) complain(host); } void addtobound(Binding host, Vector bound) { bound.addElement(host.var); } Vector emptybound(Empty host, Vector bound) { return new Vector(); } void addvar(Fun host,Vector scope) { scope.addElement(host.var); } void remvar(Fun host,Vector scope) { scope.removeElement(host.var); } }


Compare to Visitor pattern

• Surprisingly, we much prefer the separation of our version.

• Navigation and Behavior are specified separately, but are tailored

Talk Outline

• Introduction

• Traversal Specifications– compare to std traversal– overview– running example– influencing navigation



Behavior influences Navigation

• In several cases, it is necessary to decide dynamically the navigational details.

• The only way to do this with the visitor pattern is to hard code another traversal both in the classes, and even worse, in the visitor.


Example: Recursive Lets

• Recursive Let adds bound variables to scope before checking bodies.

• Let has boolean variable indicating whether it is recursive.

• Strategy: – First calculate variables bound by Let– Add these to scope at appropriate time

during processing


In Java

• Separate traversal, launched from before(Let) collects bound vars, with separate visitor.

• The new traversal doesn't enter bodies of Let or Bindings.

•administrator:

•so this example needs two traversals, as if we enter the bodies of bindings, we’re hosed.

•administrator:

•so this example needs two traversals, as if we enter the bodies of bindings, we’re hosed.


Recursive Lets in Javaclass UDVisitor extends Visitor { ... void before(Let host) { oldscope.push(inscope.copy()); oldbound.push(boundbylet.copy()); GBVisitor gbv = new GBVisitor(); host.traverse_bindings(gbv); boundbylet = gbv.boundbylet; if (host.rec) inscope.add_from(boundbylet); } void middle(Let host) { if (!host.rec) inscope.add_from(boundbylet); } void after(Let host) { inscope = oldscope.pop(); boundbylet = oldbound.pop(); }


GBVisitor

class GBVisitor extends Visitor { Vector boundbylet = new Vector(); void before(Bind host) { boundbylet.addElement(host.var); }}


Java Critique

• The traversal_bindings is hard-coded in visitor, mixing navigation and behavior.

• We need another visitor, as we do not want to calculate bindings twice, and the before, middle, after interface is to simple to allow two different modes of use for visitor

•administrator:

•and although we would get the same result with just two travs (by duplicating binding collection), that would be inefficient.

•administrator:

•and although we would get the same result with just two travs (by duplicating binding collection), that would be inefficient.

Talk Outline

• Introduction

• Traversal Specifications– compare to std traversal– overview– running example– influencing navigation

• Done right with Traversal Specifications– Thunks – Calling other traversals



In Traversal Specs• Named visits make it easy to use the

same (extended) visitor with both traversals.

• Auxiliary traversal is called from main traversal, not visitor.

• Thunks make objects out of Traversal Snippets, allowing the visitor to decide which option to use, without knowledge of navigational details.


Traversal Spectraversal unbound(Vector scope) = Program => traverse exp (scope) VarRef => visit checkvarref(scope) Let => Vector bnd = othertrav genbnd bindings() Vector newscope = visit concat(scope,bnd) Thunk_void reccase = Thunk { Vector traverse bindings(newscope) } Thunk_void normcase = Thunk { Vector traverse bindings(scope) } visit choosecase(reccase,normcase) traverse exp(newscope) Binding => Vector bnd = traverse bindings(scope)

traverse exp(scope) Empty => visit emptybound Fun => visit addvar(scope) traverse exp(scope) visit remvar(scope)


Traversal Spec, contand getbnd() = Binding => Vector bnd = traverse bindings() visit addtobound(bnd) bnd Empty => visit emptybound class UBVisitor implements unbound_visitor, getbnd_visitor { ... void choosecase(Let Host, Thunk_void reccase, Thunk_void normcase) { if (host.rec) reccase.invoke(); else normacase.invoke(); } }


Dynamically Controlling Behavior

• Thunks encode current object and a list of actions for later invocation.

• Thunks are first class objects that can be passed as arguments to visitors.

• The return type of the Thunk's actions determines the type of the Thunk.


The interfaces generated

interface getbnd_visitor { Vector emptybound(Empty host, Vector bound); void addtobound(Binding host, Vector bound); }interface unbound_visitor { void checkvarref(VarRef host, Vector scope); Vector concat(Let host,Vector a,Vector b); void addvar(Fun host,Vector scope); void remvar(Fun host,Vector scope); void choosecase(Let Host, Thunk_void reccase, Thunk_void normcase); }


Thunks

• Close over (for reading only) traversal variables at definition site.

• Can be nested, and can be returned from traversals and other thunks.

• Are typed by return value.

• Are first class objects.


Uses of Thunks

• Allow complicated behaviors to be programmed– Traversals over cyclic structures – Fixpoint iterations– Searches in Binary Trees


Calling Other traversals

• Could just invoke a traversal in the standard way from a visitor, but then navigational details enter the visitor.

• othertrav allows mutually recursive traversals to be written to share the same visitor.

• Used this to collect the bound variables


Encoding State in Traversals

• We have mutually recursive traversals

• This allows us to encode traversal state in traversal name– When we entered Let, we were in “check”

state, then entered “collect” state to get bindings, and then exited that to “check” state again.

– We don’t have tail recursion of course.

Talk Outline

• Introduction

• Traversal Specifications– compare to std traversal– overview– running example– influencing navigation– Wrapping up

• modifying the class graph• Miscellania



Class Graph Modifications

• For all but the most trivial modifications to the class graph, the traversal specification must be modified as well.

• Traversals can only express local knowledge, so the modifications will be local as well.


Other abilities– Traversing Enumerations

• Visitor must provide combine method to combine traversal results for each element to one result for whole traversal.

– Traversing Results• Can invoke traversal on values returned from

visits or traversals, not just parts of the host object

– Traverse TraversalEntries for superclasses.

• Extend behavior

Talk Outline

• Introduction

• Traversal Specifications

• Conclusion– future work– summary



Conclusion• Future work

– Investigate Multiple Extension Dimensions• Visitors can be subclasses• TraversalEntries for superclasses• Traversal should be extensible

• Summary– We allow Traversals and Visitors to

• have more detailed interface to visitors• be more recursive -- return results, pass args• Behavior needs to influence Navigation

Documents

Center for Software Sciences Northeastern University A domain specific language for Traversal Specification Johan Ovlinger Mitchell Wand Northeastern