True Higher-Order Modules, Separate Compilation, andSignature Calculi∗

George Kuan

January 15, 2009

Abstract

In the past three decades, the ML module system has been the focal point of tremen-dous interest in the research community. The combination of parameterized modulesand fine-grain data abstraction control have proven to be quite powerful in practice.Mainstream languages have slowly adopted features inspired by the ML module system.However, programmers have run into various limitations and complexities in implemen-tations of the ML module system. In the presence of common extensions such as truehigher-order modules, true separate compilation becomes a problem. This conflict re-flects a fundamental tension in module system design. Module systems should bothpropagate as much type information across module boundaries as is unconstrained bythe programmer and be able to separately typecheck modules.

1 Background

Modularity in programming dates back at least to Parnas and his information hiding criteria[44]. Linguistic support for program modularity took many forms. There are two main linesin the development of module systems. First, the Modula [61] and Cedar/Mesa [21] lineemphasized approach based on ad hoc and extralinguistic schemes for modeling linking ofprogram modules,i.e., such languages used separate tool, an object file format sensitive link-loader, to compose separately compiled modules. Second, the ML line of modularity [35–37]described modularity in terms of a small functional programming language where applicationis the main form of linkage. Common to both of these approaches is the idea that theseparation of interface and implementation leads to better program structure.

The most basic form of modularity is namespace management. Simple namespace man-agement only requires a means to declare namespaces at the top-level of program structure(namespace N), functions that reside inside those namespaces, and a notation for projectingthose functions (N.f). Each namespace is a module in such a system. One can neither ma-nipulate nor reference a namespace outside of the projection notation. A programmer can

∗Draft v.4

1

call functions by projecting out the desired components from these rudimentary modules.Namespace management, however, is only a convenience for the programmer, providing noadditional expressive flexibility or power in the language. In particular, namespace man-agement by itself does not support separate compilation, which is crucial for independentdevelopment of components of a large software project and efficient compilation. In fact,without separate compilation, some programs would have been impossible to compile withthe limited resources on early machines. Even on modern machines, large software compila-tion can be quite taxing. Many languages did not gain even a rudimentary module systemuntil the late 1980s and early 1990s. For example, the venerable FORTRAN did not includea rudimentary module system until FORTRAN 90.

A record of functions and variables can be used as a slightly richer form of modularity. Infunctional languages, regular records may also contain functions as fields. Records aggregatelabeled fields that can be projected. Unlike namespaces, records are typically first-class, sofunctions on records enable a degree of generic programming especially in conjunction withrow polymorphism [47]. Unfortunately, records normally do not support data abstraction.Pierce’s model of object-oriented languages leverages records as its core construct [45].

Modula-2 pioneered many of the major ideas behind module systems. It supportedhierarchical composition (i.e., nesting), type components, and an exception handling mech-anism [61]. The exception handling mechanism consisted of exception declarations as com-ponents of modules and a module-level exception handler. Compositions of modules werelimited to explicit imports and exports of definite modules (as in a specific concrete modulewith all its components filled out) and their components.

Ada is an example of an imperative and object-oriented language that supports mod-ularity in the form of basic and generic packages [6]. Packages are specified in two parts,specifications (akin to interfaces or signatures) and bodies that define the implementation ofthe package. Generic packages map basic packages to basic packages, thus providing a formof parameterization. Basic packages can be nested.

Object-oriented languages use the class or object system to provide a degree of modular-ity for functions and data. Most object-oriented languages do not support type components,thus limiting their expressiveness. Scala’s object system [42] appears to be the only OOlanguage that supports type components as members of a class. Beyond simple namespacemanagement, OO languages support information hiding via private data members. Unfortu-nately, this is a rather coarse-grain encoding even with multiple levels of privacy, visibility toan external client is all-or-nothing. Scala’s structured types offer another approach to mod-ularity. There has also been a considerable amount of interest in Bracha’s mixin approachto modularity [4, 5, 16] and traits [18, 51].

Type classes [59] can be modeled using a very stylized use of modules coupled with adictionary-passing inference [14]. Where type classes apply, this inference scheme makescode significantly more succinct. However, as the set of instances of classes at any one pointin the program is global, type classes sometimes interfere with modular programming.

2

2 The ML Module System

The ML module system consists of structures, signatures, functors (parameterized mod-ules), and functor signatures. Structures are sequences of type, value, structure, and functorbindings. In Standard ML, structures can be named or anonymous. Signatures are sets ofspecifications for types, the type of values, structure component signatures, and functor com-ponent signatures. Similar to structures, signatures can be either named and anonymous.Anonymous signatures are either written inline (i.e., sig ... end) or inferred or extractedfrom a structure. There is a many-to-many relationship between structures and signatures.Ascribing the signature to the structure verifies that that structure satisfies the signature.Alternatively, one can use a signature SIG to seal a structure M which makes types in Mabstract or transparent as required by SIG and hides omitted components. Type specifica-tions in signatures can be transparent (type t = unit), abstract types (type u), and relativelydefined in terms of abstract types (type v = u ∗ int), usually considered a kind of transpar-ent specification. The possibility that a signature contains both transparent and abstractspecification is called translucency.

The distinguishing characteristics of the ML module system are functors and the interac-tion of the module system and core language type inference. Ada supports type componentsin modules (called packages) and “generic” modules parameterized on a single restricted typeand some associated operator definitions. The module system of ML can express much ofSystem F. Indeed, the SML/NJ compiler compiles source into a System F-like core language.Although the Definition [40] does not require nor define support for higher-order modules,they can be found in many implementations of Standard ML including SML/NJ, MoscowML, and MLton. Apart from implementations of ML, higher-order module systems are notfound elsewhere. In the ML module system, functors can be typechecked independently fromtheir uses (i.e., applications) unlike related programming constructs such as C++ templates.Separate typechecking is necessary for true independent development of large software sys-tems. It also makes it possible to gain confidence in the type safety of the functor beforeeven beginning to write functor arguments and applications of functors, thus revealing typeerrors earlier in the development process when they can be resolved more easily. The MLmodule system also supports type abstraction via opaque ascription. With this feature, pro-grammers can define abstract data types and data structures with their corresponding suiteof operations.

Another important property of ML module systems is the phase distinction [25]. If alanguage respects phase distinction, then the static and dynamic parts of the program canbe separated such that that former does not depend on the latter. Because the ML modulesystem can be compiled into System Fω, it naturally respects the phase distinction. Atfirst glance, it may appear that types and values are entangled in type definitions such astype t = M.u where M is a module contains value as well as type components. Becausetype components always only refer to other type components in modules, one can split anymodule into one that only contains type components and the other only value components.The proviso to this argument is that modules cannot be first-class. Because in ML, themodule and core languages are stratified, this proviso holds.

3

2.1 Principles for evaluating module system design

Module system designs are often evaluated subjectively and qualitatively. However, one candefine our design goals around more objective criteria. One goal of the module system canbe to achieve as much raw performance as is possible by admitting as many different kindsof optimizations as is practical. Although it is true that performance attributable modulesystem design is difficult to isolate, If I consider type-based optimizations, this goal wouldmean that I would like to have types propagate across module boundaries after compilationas far as is possible. True higher-order functors propagate types across transparent higher-order functor applications. The performance benefit of such type propagation across moduleboundaries can be measured at least in terms of synthetic benchmarks.

We can consider the notational convenience of the module system design. Minimizing theamount of syntactic overhead would help programmers in rapid prototyping and softwaremaintenance. In some sense, the very principle of module systems runs against the spirit ofrapid prototyping. The popular misconception is that requiring the programmer to confrontinterfaces and types stifles prototyping. Type inference mitigates this concern for the corelanguage somewhat. A new module system design ought to optimize and balance the syntaxfor brevity and readability. Syntactic overhead is always tricky to measure accurately. Rawlines of code for a series of realistic programs give a very rough and potentially misleading ideaof overhead. Verbosity may improve readability and maintainability. Moreover, if automatedtools such as IDEs and preprocessors can produce with little or no programmer interventionsome of the syntax, then the actual significant overhead may be minimal. Alternatively, Imay derive syntactic overhead by comparing the best practice encodings of very commonprogramming patterns to gain a relative measure of notational convenience. Again, one fallsinto the trap of leaving open the possibility of only solving very contrived problems.

The software engineering discipline has devised a number of additional metrics for measur-ing program source complexity including McCabe’s cyclomatic complexity (MCC), coupling,cohesion, and Martin’s software package metrics. MCC measures complexity of a program’scontrol flow graph. Of these, coupling, cohesion, and software package metrics were devisedexpressly to measure complexity in the presence of a some modularity mechanism, thoughespecially in the case of software package metrics, the measures tend to be designed forclass-based object-oriented languages. More fundamentally, most of these metrics were in-tended to measure code quality for software projects and not the complexity of programminglanguages in which they are written.

A related design criterion is the predictability and transparency of module system design.Because compilers are becoming more sophisticated and complex, it is sometimes difficultto figure out what a compiler would generate for a given source. Although much of thecomplexity resides in the compiler optimization phases, it is important that elaborationand compilation for the module system are not overly sensitive, producing wildly differentcode for otherwise roughly semantically equivalent source. A trained programmer should beable to easily understand the source of errors and performance bottlenecks. Although thiscriterion is somewhat qualitative, one can perhaps measure the complexity of elaborationand compilation by quantifying how much information must be reconstructed in order to

4

predict the general outcome of elaboration or compilation. If one considers programmerunderstanding of elaboration as an abstract interpretation of program source, our metric ofpredictability might be analogous to how precise such an abstraction interpretation will haveto be to identify the source of elaboration errors.

One goal of module systems is to support extensible software design. Thus the modulesystem must promote ease of extensibility in multiple dimensions. The Expression Problemis a particularly well-known instance of this issue [60]. While a good module system de-sign might not necessarily produce a solution to the expression problem, it will provide abest practice that permits programmers to extend software without touching parallel cases.Again, this criterion can be loosely quantified as the number of expressions or statementsthat must be changed to extend source in a certain direction. A closely related objective isto permit programmers to safely compose program modules together in a flexible way.

2.1.1 Garcia et al.

Garcia et al. [20] evaluated how well 8 major programming languages support the imple-menting type-safe polymorphic containers by generic programming techniques. Their chiefcriticisms of the ML module system are three-fold. They argue that the signature languageshould permit semantic compositions of signatures beyond the syntactic include inheritancemechanism similar to Ramsey’s signature language design [46]. The authors also remark thatthe module system would be impractical for programming-in-the-small because the syntacticoverhead is excessive even when compared to other languages that require explicit instan-tiation, another criticism. The lack of “functions with constrained genericity aside fromfunctions nested within functors” was also cited as a disadvantage. In other words, Garciaet al. suggests that some bounded polymorphism would be desirable.

3 Principal Research Problems

This dissertation will address three main subjects: the formalization of the true higher-ordermodule system in SML/NJ, a formal investigation of the relationship between true higher-order modules and true separate compilation, and the development of a rich foundationalsignature calculus that solves some of the shortcomings of the module system without rad-ically altering it. The implementation of the higher-order module system has evolved sinceMacQueen and Tofte’s paper [38]. The proposed dissertation will also formalize, simplify,extend, and improve the static semantics implicit in the SML/NJ elaborator. I will focus onchanges in module representations and improve elaboration algorithms such as the instanti-ation algorithm for solving type sharing constraints. This dissertation will study what thisintuition means precisely and formally by giving a modern formal account of true higher-order modules, possible signature languages, the incompleteness of signature languages, andthe relationship to separate compilation. These results will open the way to exploration ofthe design space of full transparency (i.e., exactly what can and should propagate throughhigher-order functor applications) and separate compilation.

5

module type T = sig type t endmodule Id = functor (X:T) −> X

module type FS = functor (X:T) −> T

module Apply =functor (X: sig

module F : FSmodule M:T

end) −>X.F(X.M)

module M0 = struct type t = uni t endmodule M1 = Apply ( struct

module F=Idmodule M=M0

end)

l et x : M1. t = ( ) ; ;

Figure 1: Apply functor example in OCaml

3.1 Higher-order module system

Higher-order modules extend the ML module system with higher-order functors, a naturalextension. There are many different applications and three general approaches for higher-order modules [2, 11, 31, 32, 34, 38, 48, 56, 57] which differ mainly on how they handle typesharing in the apply functor (fig. 1).

In the apply functor example, the higher-order functor apply simply applies its firstargument, functor F, to its second, a module M. The main problem of interest is how onetypes the apply functor and the application of the apply functor on line 14. Tofte [56]introduced the first cut at the semantics for higher-order modules with non-propagatingfunctors. Under Tofte’s semantics, the apply functor does not propagate the type M0.tthrough the functor application X.F(X.M), thus M1.t 6= M0.t = unit. It assigns the signaturein fig. 2. to the apply functor. Notice that the signature for the body does not say anythingmore about type t.

MacQueen-Tofte [38] argues that the type sharing M1.t = M0.t = unit should hold.The strong sums model of modules also predicts this behavior of full type propagation [37].The semantics in MacQueen-Tofte, true higher-order functors or fully transparentgenerative functors, re-elaborates the functor body of apply given the contents of X.M atthe point of the functor application on line 14. Although this re-elaboration has the desiredeffect of propagating types, MacQueen-Tofte assigns exactly the same functor signature asthe Tofte semantics. Leroy [31] offered an alternative approach, applicative functors, that

6

functor Apply (X: sigfunctor F : (X: T) : Tstructure M : T

end) :sig type t end

Figure 2: Signature assigned to Apply functor under Tofte and MacQueen-Tofte semantics

functor Apply (X : sigfunctor F : FSstructure M : T

end) :sig type t = X.F(X.M) . t end

Figure 3: Signature assigned to Apply functor under applicative functor semantics

enriched the notion of type paths in functor signatures with functor applications such asF(M).t. Applicative functor semantics assigns the functor signature in fig. 3.

However, applicative functors only solve the type propagation problem under certain cir-cumstances and loses the generative semantics of functors, i.e., functor applications do notgenerate fresh abstract types to enforce abstraction. To address this shortcoming, MoscowML and Dreyer’s module system [13] combined applicative and non-propagating generativehigher-order functors in the same language. Fully transparent generative functors both solvethe type propagation problem under all circumstances and do not have to compromise ongenerative functor semantics. In the last decade, researchers have gained significant expe-rience in engineering non-fully transparent generative functors and transparent applicativefunctors in compilers such as Moscow ML and OCaml. Although they differ internally, bothSML/NJ and MLton compilers support some variant of true higher-order functors semantics.I take OCaml and SML/NJ as representatives of the former and latter groups respectively.

Under both OCaml and SML/NJ compilers, the apply functor example typechecks. Fur-thermore, although applicative functors cannot directly handle applications to nonpaths,lambda lifting the offending nonpath generally solves that issue. Applicative functor seman-tics lifts this restriction in the case where m1(m2) such that m1 does not have a dependenttype,i.e., the formal parameter does not occur free in the signature of m2. Together with sig-nature subtyping, such a technique can remove the restriction on nonpaths for dependentlytyped functors in certain cases but only at the cost of principality. However, if I complicatethe apply functor slightly by applying a formal functor F to struct type t = int end as infig. 4, then applicative functors fail to propagate enough type information. When the type-checker gets to line 3 in fig. 4, it does not have enough information about F to give a strongersignature for ApplyToInt’s functor body. Neither is there a path to struct type t = int endto construct an applicative functor path. Because the typechecker must give the functorbody a signature immediately in order to give HO functor ApplyToInt a complete signature,

7

module ApplyToInt =functor (F : functor (X:T) −> T)−> F( struct type t = in t end)

module R = ApplyToInt ( Id )l et x : R. t = 5 ; ;

Figure 4: ApplyToInt functor example

OCaml functor signature for H

module ApplyToInt :functor (F : functor (X : T) −> T)−> sig type t end

OCaml type error:

This expression has type unit but is here used with type R.t = ApplyToInt(Id).t

SML/NJ functor signature for ApplyToInt

functor ApplyToInt ( functor F :(X: sig type t end): sig type t end end)

: sig type t end

SML/NJ types x as R.t

Figure 5: A higher-order ApplyToInt functor fails to properly propagate types under Leroy’s ap-plicative functor semantics. Consequently, the last line fails to typecheck.

it can only give the weakest signature, sig type t end (fig. 5). A-normalization gets theprogram to typecheck but unnecessarily clutters up the code. Thus, in this sense, applica-tive functors cannot be said to be fully transparent. The SML/NJ compiler has no suchrestrictions.

The example in figs. 4 and 5 illustrates the fundamental problem with applicative func-tors. They work well as long as the relationship between functor parameter and body issimple, i.e., can be captured in the extended notion of a path with functor application ora lambda lifted path. However, not all possible functors fit into this mold. Having to A-normalize functor applications in itself is an unnecessary shortcoming. In contrast, truehigher-order functors propagate types across all transparent functor applications with nochange the source. The solution that MacQueen and Tofte [38] advocates is a re-elaborationof the functor body given the actual argument module.

In instances such as the SymbolTable functor (fig. 6), applicative functors admit toomuch sharing as noted by Dreyer [11]. Applicative functors would permit the symbols fromone SymbolTable ST1 to be used to index another ST2 despite the sealing of the functor

8

signature SYMBOL TABLE =sig

type symbolval s t r ing2symbol : s t r i n g −> symbolval symbol2st r ing : symbol −> s t r i n g. . .

endfunctor SymbolTable ( ) =struct

type symbol = in tval t ab l e : HashTable . t =

(∗ a l l o c a t e i n t e r n a l hash t a b l e ∗)HashTable . c r e a t e ( i n i t i a l s i z e , NONE)

fun s t r ing2symbol x =(∗ l ookup ( or i n s e r t ) x ∗) . . .

fun symbol2st r ing n =( case HashTable . lookup ( tab le , n ) of

SOME x => x| NONE => raise ( Fa i l ”bad symbol” ) )

. . .end :> SYMBOL TABLEstructure ST1 = SymbolTable ( )structure ST2 = SymbolTable ( )

Figure 6: SymbolTable functor example from Dreyer [11]

body to signature SYMBOL TABLE. Both OCaml and SML/NJ elaborators will make thesymbol type abstract, but it is only abstract with respect to external clients and not otherinstances of SymbolTable. Because SymbolTable is applied to the same argument for bothST1 and ST2, the two instances also share the same symbol type according to applicativefunctor semantics. This behavior breaks an important abstraction. Generative functors aremore appropriate for enforcing the exact kind of abstraction desired. In contrast, applicativefunctor semantics are appropriate for some purposes such as the Set functor in fig. 7 wherethe type sharing of Item.item is desirable and it is acceptable to use items in Sets of the sametype interchangeably. However, it is debatable whether the Set functor is a common case.I will argue that the set of programs for which true higher-order functors propagate typesis exactly those one would want to propagate types. In contrast, one should not propagatetypes in the set Applicative functors - True HO functors. Dreyer [11] noted correctly thatapplicative functors and non-fully transparent generative functors are incomparable. Neitherapplicative nor true higher-order functors can subsume the other. However, there remainsthe question whether the programs that propagate types under applicative functors but notunder true higher-order functors ought to have propagated the types in those cases.

The original criticisms of the MacQueen-Tofte semantics are the lack of support for true

9

signature COMPARABLE =sig

type itemval compare : item ∗ item −> order

endfunctor Set ( Item : COMPARABLE) =struct

type s e t = Item . item l i s tval emptyset : s e t = [ ]fun i n s e r t ( x : Item . item , S : s e t ) : s e t = x : : Sfun member (x : Item . item , S : s e t ) : bool =

. . . Item . compare (x , y ) . . .. . .

end

Figure 7: Set functor example from Dreyer [11]

separate compilation and that the stamp-based operational semantics makes it difficult toextend the module system and to reason about it. Many recent treatments of ML modulesystems abandons true higher-order functors completely due to these issues. The claim isthat the type-theoretic presentations of the module system with applicative functors addressthese problems. This dissertation will consider the question whether an operational semanticsaccount must necessarily be more complicated and if so, why. In contrast to recent work, thisdissertation will take true higher-order module behavior and my revised elaboration-basedsemantics (figs. 18, 20, and 21) as the starting point for developing a formal semantics whileaddressing these criticisms and concerns. My formalism follows the implicit semantics inSML/NJ compiler which enriches the internal representation of functors and functor signa-tures to express the static actions of the functor, thus not having to do a full re-elaborationof the functor body. This approach will also yield some practical benefits. The SML/NJ andMLton implementations have not kept up with the pace of the progress in module systemdesign at least partially due to the fact that most of the research has been a radical departurefrom true higher-order module semantics. Reframing the state-of-the-art in terms of truehigher-order module semantics will bring recent developments closer a practical extension inthese production-quality compilers.

3.2 Full transparency and true separate compilation

The main issue I will study in this dissertation is the the exact nature of the tension betweentrue higher-order modules and separate compilation. Since MacQueen and Tofte introducedtrue higher-order modules, many researchers [15, 30, 48] have studied how to downgradehigher-order functors to regain true separate compilation, by which I mean Modula-2-styleseparate compilation. Although Standard ML enjoys separate typechecking and compilationfor the most part, the MacQueen-Tofte re-elaboration semantics of true higher-order functors

10

necessitates the availability of the functor body source at the point of functor application.To workaround this limitation, SML/NJ uses cut-off incremental recompilation [1, 23] via apowerful compilation manager CM [3]. Incremental recompilation, however, does not solvethe true separate compilation problem.

The separate compilation problem can be reframed as a completeness problem for thesignature language, i.e., can the source-level signature language adequately describe all possi-ble modules including functors. Currently, the SML/NJ compiler elaborates module syntaxinto internal semantic objects. These semantic objects are expressive enough to encodethe functor body relationships that eluded the source signature language. The source andthese semantic objects are then compiled to a predicative System Fω-like calculus [52]. Thissuggests that an Fω-like calculus should be expressive enough to characterize all the staticsemantic actions of functors.

Intuitively, true higher-order modules cannot be fully expressed in the syntactic signaturelanguage because it is limited to definitional specs and type sharing. For example, there isno way to express a functor signature for the apply functor that accounts for all sharing dueto full transparency. Therefore true higher-order modules cannot in general be separatelycompiled. In particular, HO functor applications cannot always be separately compiled fromthe functor definition. In the past, various researchers have approached this problem byincorporating applicative functors into the language to varying degrees [2, 13, 31, 48] some-times limiting the generative functors in the process. In this dissertation, I will argue thatapplicative functors cannot replace true higher-order functors in the general case. Moreover,if fully transparent generativity is the goal, then applicative functors only serve to supporttrue separate compilation in a limited number of cases.

Aside from describing the static semantic actions of HO functors, a signature languagesupporting separate compilation needs a mechanism to ensure coherence of the abstracttypes in imported modules. ML presently solves the coherence problem through a combina-tion of type sharing constraints, definitional type specs, and where type clauses. In simplermodule systems, the problem of coherence is not as pronounced because compilation unitsonly import external units using definite references [54]. The current implementation fortype sharing constraints resolution, called instantiation, is a complicated process with aconsiderable amount of folklore. Simply put, instantiation constructs the free instantiationof a functor formal parameter that imposes the required amount of type sharing but no more.The instantiation phase in SML/NJ imposes a semantics stricter than that of the Definition.Instantiation guarantees inhabitability of signatures where the Definition does not. AlthoughHarper and Pierce [26] claim that type sharing constraints can be superseded by definitionaltype specs in all cases, my study has identified examples (fig. 8) where this is not the case ab-sent a mechanism for signature self-reference, e.g., structure M : S0 where type t = self.N.u.

Both Russo [48] and Swasey et al. [54] have suggested that the separate compilationproblem can be solved by identifying an alternate form of compilation unit, one that is nota module. OCaml, in fact, already implements such a regime. However, in the presence oftrue higher-order functors, these approaches merely punt on the real problem by compellingthe programmer to put otherwise independent modules together in a single compilation unit

11

signature S0 = sig datatype s = A type t endsignature S1 = sig datatype u = B type v end

signature S3 =sig

structure M : S0structure N : S1sharing type M. t = N. u and N. v = M. s

end

Figure 8: Criss-crossing type sharing constraints cannot be reduced to definitional type specifica-tions or uses of where type.

and then abstracting over that unit. This dissertation will either prove definitely that truehigher-order functors and true separate compilation are incompatible or develop a modulesystem integrating these two features.

3.3 Signature calculus

Since the study of the true separate compilation problem points in the direction of the sig-nature calculus, it will be fruitful to take this opportunity to reconsider the design of ML’ssignature language. After Harper-Lillibridge and Leroy, despite the continuing pace of thedevelopment of ML module systems, the signature language generally did not see much at-tention except for Ramsey et al.’s paper [46]. Ramsey et al. [46] describe a signature languagethat includes operations for post hoc manipulation such as adding, removing, rebinding com-ponents, and merging signatures. SML/NJ’s semantics for include is richer than the simplesyntactic inclusion found in the Definition [40]. In particular, certain kinds of compatiblesignatures can be merged. In the SML/NJ 110.68 compiler, two signatures are compatiblewhen their overlapping specifications (i.e., specifications with the same name) have the samearity and follow the rules summarized in table 1. However, the current compatibility rulesare inconsistent and incomplete. For example, merging an eqtype and a type specificationresults in an eqtype in one direction and a type in the other as shown in fig. 9.

Despite its present incomplete state, SML/NJ can do the appropriate consistent mergefor Garcia et al.’s example (fig. 10). In the case of Garcia et al.’s example, the typecheckerneeded to do is to note that the repeated components t and u due to inclusion are identicalspecifications.

The SML/NJ semantics goes further and merges consistent yet unequal specificationssuch as abstract types and datatypes. The merging semantics can be substantially improvedby making the table more symmetrical and adjusting some of the precedences to somethingmore sensible. Both Ramsey [46] and Dreyer and Rossberg [15] offer language supportfor a signature calculus that can safely compose signatures, effectively permitting a kindof multiple signature inheritance. Both accounts only model signature merging for a smalllanguage without support to features such as eqtype and generative datatypes. In particular,

12

signature S0 = sig eqtype t endsignature S1 = sig type t endsignature S2 = sig include S0 include S2 end

S2 : sig eqtype t end

signature S0 = sig type t endsignature S1 = sig eqtype t endsignature S2 = sig include S0 include S2 end

S2 : sig type t end

Figure 9: Unsound behavior of SML/NJ signature merging by include

type eqtype datatype deftypetype 3 eqtype 7 7

eqtype type 3 7 7

datatype 3 datatype 7 7

deftype 7 7 7 7

datatype withtype 3 datatype withtype 7 7

Table 1: SML/NJ 110.68 Signature elaboration consistent signature merging: 3 can be merged,7 cannot be merged, otherwise indicates specs mergable but indicated spec takes precedence

a fine-grain merging of eqtype can be nontrivial. For example, it is safe to merge eqtype tand datatype t = K where K is a data constructor. In contrast, merging eqtype t anddatatype t = K of int −> int is unsafe. Consistent merge rules of this flavor can alreadybe found elsewhere in the compiler, namely in signature matching. This dissertation willdevelop a formal semantics for a safe but flexible consistent signature merging that coversthese features of ML.

The signature merging semantics found in Ramsey et al. is quite aggressive. In oneexample (fig. 11), the merging semantics creates a new definitional type spec type u = t inorder to merge two signatures that disagree on an entangled value specification val x : t listand val x : u list . This kind of aggressiveness likely goes beyond the intention or expectationof the programmer. The programmer may have difficulty deciphering typechecking errorsrelating to S2.u and S2.x after this point because of this aggressive induced type sharing. Itwould be more sensible to have the typechecker complain that S0.x and S1.x are incompatiblevalue specifications because as far as the typechecker and programmer are concerned, S0.tand S1.u are simply flexible type specifications.

Inspired by Ramsey et al., my study of signature calculi will go beyond the semantics ofconsistent signature merging to consider the design implications of adding parameterized sig-natures [27], signature variables, and related features to the ML signature language. The MLsignature language permits type definitions that may refer to general type expressions. Type

13

signature S0 =sig

type teqtype u

end

signature S1 =sig

include S0val x : i n t

end

signature S2 =sig

include S0val y : un i t

end

signature S3 =sig

include S1include S2

end

Figure 10: The naive macro expansion semantics of the Definition rejects S3. SML/NJ acceptsit. This example was derived from Garcia et al.’s GraphSig, IncidenceGraphSig, and VertexList-GraphSig [20].

expressions may involve both primitive type constructors such as → and programmer-definedtype operators. It is the inclusion of type operators that gives the signature language muchof its expressiveness. The semantics of type sharing constraints differs significantly betweenSML90 and SML97. Type sharing constraints could be imposed on two type constructorswithout restriction in SML90. In SML97, the designers partitioned the semantics of typesharing into type definitions which expressed sharing between an abstract type and an ar-bitrary type expression, and regular type sharing constraints which can only be imposedbetween two flexible (or primary) types whose names must be in scope.

A module system that permits both type definitions and type sharing constraints in signa-tures introduces significant new complexity. For example, whereas in Leroy’s [32] TypModllanguage, which only permits SML90-style definitional type sharing constraints and no typedefinitions, type sharing constraints can be “normalized” by pushing them up the signatureand eliminated by turning them into type definitions, type sharing constraints cannot beeliminated in a language that permits both type definitions and type sharing constraints.

In ML modules, structures can be arranged in a hierarchy. This feature enables flexiblenamespace management. In contrast, signatures cannot be arranged in such a hierarchy.Signatures must be defined at the top-level and can never be enclosed in any other signatureor module. For complex hierarchies such the SML/NJ’s Control module that contains layersof submodules, the corresponding signature CONTROL and the signatures of the submodulesPRINT and ELAB are related only incidentally by occurrence in structure specifications inCONTROL. This shortcoming in the signature language unnecessarily pollutes the signaturenamespace and complicates browsing through and working with highly nested hierarchies.It would be desirable to permit (transparent) signature specifications within signatures. Foradded flexibility and perhaps increased expressiveness, it may be useful permit signaturedefinitions within structures and functors. Furthermore, in order for modules to match

14

signature S0 =sig

type ttype uval x : t l i s t

end

signature S1 =sig

type ttype uval x : u l i s t

end

signature S2 =sig

type ttype u = tval x : t l i s t

end

Figure 11: This is an example from Ramsey et al. [46]. S2 is the merge (greatest lower bound) ofS0 and S1 according to their semantics.

structure M =struct

type t = in ttype u = bool ∗ s t r i n gval a : u ∗ t

endsignature S = s ign (M) removing u adding val b : t ∗ t

Figure 12: Accessing and modifying (via Ramsey et al. signature operations) an inferred signature

these signatures enriched with signature specifications, modules must permit correspondingsignature definitions.

Since the semantics of ML already supports the extraction or inference of module signa-tures from the implementation, perhaps it makes sense to permit the programmer to operatedirectly on the inferred signatures of implementations (modules) or generate/synchronizemodule implementation based on the signature. Programmers often skip the step of writingproper signatures for modules because the necessary notation is cumbersome and potentiallyrepetitive with respect to the module implementations of signatures. Some programming en-vironments can help programmers automatically generate interfaces, but changes usually arenot propagated bidirectionally. Synchronization of modules and signatures is ill-defined inthe ML module system which supports a many-to-many relationship between modules andsignatures. However, this is exactly where the existence of a principal signature or of afull signature might be useful. If programmers can leverage the structure of existing modulestructure when writing signatures, this might lower the barrier of entry for programmers with

15

existing non-modularized source thus providing a path to “gradual modularization”. For ex-ample, in fig. 12 an inferred signature can be modified after the fact to serve as a templatefor future structures without having to explicitly write out any signature. This signaturelanguage will enable programmers to quickly integrate modular and non-modular code byfacilitating rapid construction of variations on inferred signatures. Admittedly, this featuremay run against the very spirit of splitting out explicit interfaces from implementations.

4 Related work

Module systems have generally followed either in the extralinguistic style of Modula andMesa or the functional style of ML. Dreyer and Rossberg’s recent module system leanstowards the extralinguistic style with the subtle mixin merge linking construct that servesmany roles including encoding functor application. Cardelli [7] brought some amount offormalization to this ad hoc approach to modularity, but module systems in this style stillvary considerably in terms of semantics. Moreover, the extralinguistic semantics of linkingare typically fairly involved and delicate.

The ML module system has inspired a bevy of formalisms describing its semantics andcommon extensions. These formalisms run the gamut from Leroy’s presentation based onsyntactic mechanisms alone, to Harper-Lillibridge [24] and Dreyer, Crary, and Harper’s [13]type-theoretic approach, to the elaboration semantics approach as represented by the Defi-nition [40]. As I have discussed at length in sec. 3.1, full transparency is a desirable propertyfor module system designs. Designs range from no support for transparency (type prop-agation) such as non-propagating functors and complete support for MacQueen-Tofte fulltransparency. In between are the various gradations,i.e., different combinations of applica-tive and generative functors. Another important quality of the account of module systemdesign is the formalism used for expressing the semantics. In the early days, module sys-tem semantics were quite ad hoc, each relying on their own peculiar notation and collectionof semantic objects [21, 56, 61]. Since then, there has been a strong move toward morestandardized logical frameworks that are more amenable to encoding in a theorem provingsystem such as Isabelle/HOL and Coq [13, 24]. Curiously, the most recent accounts havebeen moving back towards a semantic objects-based approach [12, 15]. Fig. 13 shows veryroughly where the major families of module system designs fall in relationship to each otherin terms of support for full transparency and adherence to some formal logic framework. Inthe figure, it appears that the left-hand side has been well-explored. I will study the right-hand side, and perhaps the upper-right-hand quadrant which represents a fully transparentsemantics in something close to a mechanized metatheory for Coq. This study will ideallyresult in a module system design that reconciles a current understanding of fully transparentfunctors with a simpler and more accessible semantics, a clear evolutionary step from thestate-of-the-art.

Besides the main module system features described earlier in this proposal, there are anumber of other useful properties developed in the different module languages. The phasedistinction [25] plays a fundamental role in ensuring that module systems can be type-

16

target

MacQueen-Tofte

Harper-Lillibridge

MixMLUnits

(Owens-Flatt)

RMC(Dreyer 07)

standard

ad hoc

fullynone transparency

semantics

Definition

Syntacticpaths

(Leroy)

DCH 03

Moscow ML

A semantics is more standard in the sense that it uses more formal logic frameworks. It maypotentially be easier to mechanize the metatheory of such semantics.

Figure 13: The spectrum of major module system families

checked fully at compile-time. This property is desirable, being consistent with our goal ofstatic type safety. It says that a language, in this case a module, can be split into static anddynamic parts such that the static part does not depend on the dynamic. Some accountsof module systems respect phase distinction at the surface language level [31, 48]. Othersrespect the phase distinction in an internal language but not the surface language [38].

The other key property in a module system design is the existence of a principal signa-ture for modules. The term principal signature has been overloaded in meaning. Becauseof the many-to-many relationship between signatures and modules and the signature sub-typing relationship, many signatures may be safely ascribed to a single structure. I will callthe most precise signature for a structure (i.e., one that constrains all components of thestructure with exact, most constraining types) the full signature. A related concept is thefree instantiation of a functor formal parameter. The free instantiation is an instance ofthe functor formal parameter signature S that admits exactly enough type sharing to satisfyS and no more. In particular, it avoids any extraneous type sharing that would constrainthe free instantiation more than is necessary. What Dreyer [11] calls the principal signatureis the full signature in this terminology. Tofte defines principal signature of a signature ex-pression sigexp as one whose flexible components (i.e., abstract types) can be instantiatedto obtain all instantiations of sigexp [40, 56]. I will adopt Tofte’s terminology.

17

4.1 Type-theoretic approach

Beginning with Harper-Lillibridge’s translucent sums module calculus [24], this large, prolificfamily of module systems pushed the state-of-the-art in terms of the type theoretic approachto module system design. Although Harper-Lillibridge originally explored first-class modules,the bulk of the research in this family was directed towards an applicative higher-ordermodule semantics for type-directed compilers TIL/TILT and adding recursion.

Crary et al. [8] and later Dreyer [10, 12] have explored adding support for recursion.Harper-Lillibridge [24] and then Russo [49] studied mechanisms for making modules first-class entities in the core language. With first-class modules, programmers can leveragethe familiar module system to take advantage of the System F-like power of the modulesystem for programming-in-the-small. Unfortunately, as Garcia [20] remarks, the syntacticoverhead of the ML module system makes this undesirable and impractical. When used in asimilar context, Garcia suggests that the type inference used for type classes makes modularprogramming-in-the-small more succinct. This observation holds only in specialized use caseof type classes.

4.2 Syntactic paths approach

One of the first formal accounts of an ML-like module system is Leroy’s manifest typescalculus [30] where manifest types are definitional type specifications. The surface languagefor the manifest types calculus is equivalent to that of the translucent sums calculus describedby Harper-Lillibridge. The key observation in the manifest types calculus is that one cantypecheck manifest types by comparing the rooted syntactic paths to those types whichuniquely determine type identity. Thus, type equivalence is syntactic path equivalence.Leroy introduced the notion of applicative functors which held types in the result of a functorapplication to be equivalent to corresponding types in all other results of applications of thatfunctor to the “same” argument [31]. There is a design space for module equivalence, fromstatic equivalence to full observational equivalence. Several designs [13, 48, 53] have triedto incorporate both applicative and generative functors in a single calculus. Cregut [9]enriched signatures with structure equalities to obtain complete syntactic signatures forseparate compilation.

Leroy introduces a relatively simple approach to module system semantics that precludesshadowing of core and module bindings [32, 33]. The semantics supports type generativityand SML90-style definitional sharing by reducing them to solving path equivalence by wayof A-normalization (for functor applications) and S-normalization (a consolidation of sharingconstraints by reordering). In his module system, Leroy claims that all type sharing can berewritten in his calculus with generative datatypes and manifest types. However, Leroy’ssimplified module system does not include value specifications and datatype constructorsboth of which can constrain the order in which specifications must be written and thereforeresult in situations where sharing constraints cannot be in general reduced to manifest types.

For full transparency, Leroy proved that there is a type-preserving encoding of a stratifiedcalculus with strong sums without generativity using applicative functors [31], claiming that

18

f is a fresh higher-order dependency variable

Γ,W ` type t ⇒ ((t 7→ f(W), ∅), {f})

Figure 14: Biswas’s elaboration rule for abstract type specifications: Γ is the type environmentmapping program variables to types. W is a list of formal parameters variables the specificationmay depend on.

the existence of such an encoding is a strong hint that applicative functors support fulltransparency. My HO apply functor example in fig. 4 casts some doubt to this claim. AsLeroy pointed out, under the strong sums model, first-class modules is at odds with phasedistinction because of the typechecker would have to do arbitrary reductions [30]. In contrast,because the weak sum model of the manifest types calculus does not require any reductionsat typecheck time regardless of the presence of first-class modules, it does not violate thephase distinction [30]. In the most recent paper in the manifest types series [33], Leroyabstracts away most of the core language details from the manifest calculus to obtain amostly core language independent module system.

Shao [53] offers a signature language based on gathering (and internally factoring out) allflexible components (i.e., abstract type components unconstrained by sharing) in a higher-order type constructor that can be applied to obtain a signature that expresses functor bodysemantic actions at a later point. The resultant signature language superficially resemblesapplicative functors. However, type constructor applications in the signature language mustbe on paths. Consequently, it does not support full transparency in the general case.

Although the syntactic paths approach may very well provide the simplest account ofmodule systems, this is at the cost of some very fundamental shortcomings such as theinability to support shadowing and full transparency. Because the account’s support fortype sharing is incomplete, there may also be limits to how the semantics deals with thecoherency issue. The proposed dissertation will address these issues which are fundamentalto the power of the ML module system.

4.3 Moscow ML

Biswas gave a static semantics for a simpler form of higher-order modules [2]. The accountrelies on semantic objects and a stamp-based semantics similar to the Definition. The typepropagation in higher-order functors is captured by a “higher-order” dependency variablethat abstracted possible dependencies on the argument. These variables are only presentin the internal language produced during elaboration. Consequently, Biswas’s semanticsdoes not support true separate compilation and neither does it enrich the surface syntax forsignatures. Biswas’s elaboration rule in fig. 14 maps an abstract type name t to the freshhigher-order abstract dependency variable f applied to the list of all abstract dependencyvariables W it could possibly depend upon. For example, the functor parameter of the Applyfunctor, functor F(X:sig type t end): sig type t end, is given the semantic representation∀f0({t 7→ f0} ⇒ {t 7→ f1(f0)}).

Biswas’s formal account was extended in a somewhat more type-theoretic style to the full

19

SML language and implemented by Russo in the Moscow ML compiler. Moscow ML alsoadds support for first-class modules [49] and a form of recursion [50]. This family of semanticsattempts to incorporate both non-propagating generative functors and applicative functors.The main limitation, as pointed out of Dreyer [11], is that Moscow ML’s combination ofgenerative and applicative functors is unsound. In particular, any generative functor canbe η-expanded into an applicative functor thereby circumventing the generative functorabstraction.

4.4 Units and other extralinguistic linking system

Flatt-Felleisen [19] and Owens-Flatt [43] develop a module system semantics based on acalculus with stratified distinct hierarchically composable modules and recursively linkableunits. Because both accounts appeal to extralinguistic linking semantics, they fall underthe Module/Mesa line of module systems. As pointed out of Dreyer [15], the fundamentallimitation in this semantics is that the stratification of units and modules makes precludesusing unit linking and hierarchical composition together. One strength of their modulesystem design is that it is one of the few accounts that includes an operational dynamicsemantics, unlike all the other accounts discussed in this section. All other accounts ofmodule systems merely give a static semantics and perhaps a typechecking algorithm whichthey prove is sound with respect to the static semantics. Owens and Flatt prove typesoundness of their semantics.

Swasey et al. [54] described a calculus SMLSC that is modeled after Cardelli’s linksetapproach to separate compilation. SMLSC introduces a compilation unit that sits on top ofthe module system that can be separately compiled from unimplemented dependencies bymeans of a handoff units whose role resembles that of header files in C.

4.5 The Definition and MacQueen-Tofte

The Definition of Standard ML [39,40] semantics for the module system evolved throughoutthe late 1980s and early 1990s. Early on, Harper et al. gave a fairly complete account ofthe static semantics of the first-order ML module system in terms of an operational stamp-based semantics [22]. Tofte proves that signature expressions in the first-order, generativesemantics have principal signatures in his thesis [55]. He also extended this proof to covernon-propagating higher-order generative functors [57]. Then MacQueen and Tofte introducedtrue higher-order functor semantics [38].

Apart from type propagation transparency issues, the evolution of the Definition alsoaddressed other key issues type sharing issues. The role of type sharing constraints hasevolved through the development of the Standard ML semantics and SML/NJ implementa-tion. Type sharing constraints solve two problems in ML, type specification refinement andcoherence. Originally, type specifications only declared the name of an expected type. It isquite useful to be able to refine type specifications to restrict it to particular definite types.The coherence problem is the challenge of constraining type components of two structureswhich may or may not be identical to be equivalent regardless of the actual identity of that

20

type. In SML90 [39], explicit sharing equations among visible abstract types and generativestructure sharing served as the sole means for constraining type specifications. Under gener-ative structure sharing semantics, each structure had a unique identity. Thus, two structuresshared only when they were identical in the sense that they were defined at the same pointin the program and are merely aliases (fig. 15). Structures that shared in this sense wereequivalent both statically and dynamically. This kind of rich sharing semantics turned outto be quite complicated and was soon abandoned in favor of a structure sharing that simplyreduced to type sharing constraints on the type specifications inside the signature.

SML93 introduced definitional type specifications, giving programmers two ways for con-straining types to definite ones. SML97 added where type and definitional type specifica-tions completely replaced the definitional type sharing found in SML90. Definitional typespecifications and type sharing were finally disentangled. Generative structure sharing waseliminated in favor of the simpler semantics of a structural sharing that amounted to a typesharing equation for each common type specification, no matter how deeply nested. Thesemantics of type sharing and related mechanisms such as where type are still somewhatproblematical and unsettled [41, 46]. Type sharing as it stands gives rise to a delicate andnon-obvious resolution algorithm, instantiation. Ramsey et al. has argued that the scopingof where type definitions should be more symmetric thereby permitting more flexible typespecification refinements.

Shao [52] (in a paper unrelated to the one on applicative functor-like module system [53])extends MacQueen-Tofte fully transparent modules with support for type definitions, typesharing (normalized into type definitions), and hidden module components. This treatmentof higher-order modules is a more recent form of what is currently in the SML/NJ compiler.Elsman presents a module system compilation technique used in the ML Kit compiler [17].The semantics follows the style of the Definition. The compilation technique is comparableto Shao’s FLINT compilation scheme.

The SML/NJ compiler implements a version of the module system that departs fromthe Definition in a number of aspects. Some of these extensions have not been formalizedas of yet. In particular, the compiler has a richer semantics for include and the elaboratornow compiles a functor body to a static lambda calculus which is what is used in place ofthe actual functor body during the re-elaboration at application. The scoping of sharingconstraints has also changed. In the current implementation, SML/NJ no longer permitsnonlocal forms of sharing of the flavor illustrated in the example in MT (fig. 16). Instead,structure definitions express the same kind of sharing. The module system also has somesignificant limitations such as recursion, the tension between separate compilation and fullytransparent generative functors, and the limited signature language.

4.6 Alice ML

Alice ML provides a number of the features mentioned in this proposal especially in thearea of signature language enrichments. The language supports nested signatures of bothvarieties: those that must be repeated in the signature for the enveloping structure and thosethat are abstract. The abstract signatures do not, however, appear to be complemented with

21

signature S0 =sig

type tval f : t −> tval s t a t e : t r e f

end

functor F(X: sigstructure A : S0structure B : S0sharing A = B

end) = . . .

functor G() =struct

type t = uni tval f = . . .val s t a t e = r e f 0

end

structure M0 = G()structure M1 = G()structure M2 = M0

structure M3 = F( structstructure A=M0structure B=M2

end)

structure M4 = F( structstructure A=M0structure B=M1

end)

Figure 15: Under SML90 identity-based structure sharing, A and B have to be aliases, so thefunctor application at M4 fails to typecheck. Under SML97, the sharing constraint merely rewritesto sharing type A.t = B.t, thus both functor applications typecheck.

22

structure S = struct end ;signature SIG =sig

structure A : sig endstructure B : sig endstructure C : sig endsharing A = Ssharing B = C

end

Figure 16: In the current implementation of SML/NJ, the first kind of sharing constraint is nolonger permitted. Both sides of the constraint must be in local scope as is in the second sharingconstraint.

any bounded polymorphism features.

4.7 MixML

Dreyer and Rossberg [15] show how to encode ML signatures, structures, and functors in amixin module calculus that appeals to something similar to Bracha’s merge [4] as its onlylinking mechanism. When linking a module A and B, the semantics tries to satisfy theimports of A using the exports of B and vice versa. The mixin merging syntax is link x=M0with M1 in the surface language. It binds the name x to a module M0 and concatenatesit with M1 merging components where appropriate. The scope of x is the body of M1.This mixin merging semantics supports recursion and separate compilation. Modules in thislanguage consist of atomic modules that only contain values, types, type constructors, etc.,labeled modules ({` = M}), and the merging form link ... with ....

The peculiarity in this language is that modules and indeed anything that can be encodedin these modules are stateful. For example, signatures in MixML are fundamentally stateful.Linking against a signature S mutates it. Consequently, the typechecker rejects the followingprogram.

module U = l i n k x={module S = {type t }}with {module A=( l i n k x=x . S with {type t = in t } ) ,

module b=( l i n k x=x . S with {type t = bool } )} ;

Signatures must be suspended and then new’ed in order to be matched multiple times. Thissuspension is called a unit in Dreyer and Rossberg’s terminology. Functors are also repre-sented by suspending modules with unsatisfied imports. Although units have full supportfor hierarchical composition, MixML’s design still retains the problem of stratifying modulesand units, a problem inherited from the Flatt-Felleisen units that inspired it.

As it stands, the part of the MixML language that encodes the ML module system isbut a small fraction of the whole. The question remains what implications the rest of thelanguage has. Part of the language is obviously semantically meaningless such as link X =

23

[int] with [3], which is a well-formed program. The approach that the proposed dissertation’ssemantics will take is to extricate the part of MixML that encodes the ML module systemand hopefully simplify the semantics by omitting the rest of features. One feature I think isworth pursuing is the fact that in MixML signatures can be composed together. In ML, onecan only project on modules. MixML [15] loosens this restriction by conflating signaturesand modules. Signatures can be projected out of an enclosing signature.

4.8 First-class polymorphism inference and type classes

Jones [28] motivated first-class polymorphism (FCP) by appealing to the constructive logictautologies for existentials, universals, and implication [28]. I observed that the constructivelogic rule 〈w, τw〉 → τ ′ ↔ w ⇒ τw → τ ′ corresponds to the FLINT transformation (currying)in the forward direction and an uncurrying operation, perhaps a kind of module inference.At any rate, I would like to investigate the use cases for FCP where modules would besufficient. More recent first-class polymorphic calculi such MLF [29] and FPH [58] add somelimited type inference. Although inference in general may be undecidable, these limitedinferencers still go a long way to make programming in these first-class polymorphic calculimore practical. If some of the ideas for inference for FCP calculi can be transferred over toa module calculus, one might also address the syntactic overhead of ML module systems.Adding FCP to the core introduces a certain amount of redundancy with respect to the FCPafforded by the module system. It would be useful to consider what exactly is redundantand whether that can be minimized.

Another language construct related to module systems that enjoys type inference is thetype class. Type classes are a special case of modular programming where a kind of automaticdeduction would be useful. Unfortunately, the scope of class instances are global. In ModularType Classes, Dreyer et al. [14] develop semantics and a translation from type classes to astylized use of the ML module system. I hypothesize that the type inference features of typeclasses might be able to be “back-ported” to module systems.

4.9 Summary

Not all module system designs enjoy the principal signature and phase distinction properties.Fig. 17 summarizes the key features of the main ML-like module system families. Note thatmany module systems have various combinations of these features, but none are complete.Ideally, a module system would have true higher-order semantics and all the other featuresexcept for applicative functors which would be redundant.

5 Methodology

Informed by MixML, MacQueen-Tofte semantics, and FLINT semantics as the main sourcesof inspiration, my dissertation research will define a new formal semantics (including a dy-namic semantics) and type system for true higher-order module system based on the current

24

System higher-order first-class sep comp rec app gen phaseHL [24] 7 3 3 7 7 7 3

Leroy [31] 7 7 3 7 3 7 3

Russo [50] 7 3 3 3 3 3 3

DCH [13] 7 3 7 7 3 3 3

RMC [12] 7 7 7 3 7 3 3

MT [38] 3 7 7 7 7 3 3

MixML [15] 7 3 3 3 7 3 7

Ideal 3 3 3 3 7 3 3higher-order = true higher-orderrec = recursive modulesapp = applicative functorsgen = non-propagating generative functorsphase = respects the phase distinction

Figure 17: A comparison of major ML-like module systems

module system design in the SML/NJ compiler. The semantics will clarify and extend theimplicit compiler semantics. Part of this study will include experimental prototypes eval-uated according to the design criteria outlined above. This prototype module system willvalidate the practicality of the formal design. The prototype will include a module languageelaborator including typechecker and basic compilation into a suitable typed intermediatelanguage.

Through the course of formalizing the module system, the dissertation will establish typesoundness of the module language for the dynamic semantics and type system in the styleof Owens-Flatt but for the more powerful ML module system. Moreover, it will preciselydefine full transparency, separate compilation, and the relationship between the two. Thehypothesis is that these two features are mutually exclusive because I conjecture that type-checking a signature language powerful enough to encode all possible relationships betweenfunctor parameter and body would be undecidable. If the hypothesis turns out to be false,then the dissertation should develop a signature calculus powerful enough to represent allpossible static semantic actions of higher-order functor application. The final component ofthe dissertation will be the decidability and soundness of the signature calculus typechecking.

5.1 Primary and secondary components

SML/NJ’s implementation of elaboration and translation into the FLINT language offerssome unique insight into the semantics of the module system. These insights are not merelyimplementation choices or details. They reflect novel fundamental issues in the design andunderstanding of module systems. In studying translation, it has become clear that notall abstract type components are equal. The form of a functor argument is constrainedby the functor parameter signature possibly modified by a where type definition. In theparameter signature, there can be structure specifications, formal functor specifications,

25

structure/type sharing constraints, and two classes of type specifications. Type specificationsmay be abstract or definitional. Abstract type specifications that remain abstract after theelaborator resolves all sharing and where type constraints are called flexible or primarycomponents. These primary type components are those essential components that must bekept to maintain the semantics of functor application (i.e., the type application associatedwith the functor application). The specific function of primary type components is to capturea canonical representative of an equivalence class of abstract types induced by type sharingconstraints. Each equivalence class has exactly one primary type component that serves asa representative element. References to all other members of the equivalence class shouldbe redirected to the associated primary type component. The remaining type componentsare secondary and therefore should be fully derivable from the primary components andexternally defined types. Secondary types do not have to be explicitly represented in theparameter signature because all occurrences of these secondary types can be expanded outaccording to their definitions.

functor F( type stype ttype u = s ∗ tsharing type t = s ) = . . .

In the above example, s can be primary, representative for the equivalence class containingboth s and t, and u is secondary.

6 Conclusion

ML module systems have evolved throughout the last decade by avoiding the implications oftrue higher-order functors and adopting a combination of applicative and opaque generativefunctors. At the cost of the complexity of two coexisting kinds of functors and the lossof some abstraction power, recent module systems gained true separate compilation. Thisdissertation will revisit true higher-order functors, motivated by the search for the exactnature of the relationship between true higher-order functors and true separate compilation.Because the problem of true separate compilation concerns the expressiveness of the signaturecalculus, I will also take this opportunity to revisit the ML signature calculus and extend itto support more flexible modular software composition.

References

[1] Andrew W. Appel and David B. MacQueen. Separate compilation for Standard ML.In PLDI ’94: Proceedings of the ACM SIGPLAN 1994 conference on Programminglanguage design and implementation, pages 13–23, New York, NY, USA, 1994. ACM.

26

[2] Sandip K. Biswas. Higher-order functors with transparent signatures. In POPL ’95:Proceedings of the 22nd ACM SIGPLAN-SIGACT symposium on Principles of program-ming languages, pages 154–163, New York, NY, USA, 1995. ACM.

[3] Matthias Blume. Standard ML of New Jersey compilation manager. Manual accompa-nying SML/NJ software, 1995.

[4] Gilad Bracha. The programming language Jigsaw: mixins, modularity and multipleinheritance. PhD thesis, University of Utah, Salt Lake City, UT, USA, 1992.

[5] Gilad Bracha and William Cook. Mixin-based inheritance. In OOPSLA/ECOOP’90: Proceedings of the European conference on object-oriented programming on Object-oriented programming systems, languages, and applications, pages 303–311, New York,NY, USA, 1990. ACM.

[6] Gary Bray. Implementation implications of Ada generics. Ada Lett., III(2):62–71, 1983.

[7] Luca Cardelli. Program fragments, linking, and modularization. In POPL ’97: Pro-ceedings of the 24th ACM SIGPLAN-SIGACT symposium on Principles of programminglanguages, pages 266–277, New York, NY, USA, 1997. ACM.

[8] Karl Crary, Robert Harper, and Sidd Puri. What is a recursive module? In PLDI ’99:Proceedings of the ACM SIGPLAN 1999 conference on Programming language designand implementation, pages 50–63, New York, NY, USA, 1999. ACM.

[9] P. Cregut and D. MacQueen. An implementation of higher-order functors. In ACMSIGPLAN Workshop on Standard ML and its Applications, June 1994.

[10] Derek Dreyer. A type system for well-founded recursion. In POPL ’04: Proceedings ofthe 31st ACM SIGPLAN-SIGACT symposium on Principles of programming languages,pages 293–305, New York, NY, USA, 2004. ACM.

[11] Derek Dreyer. Understanding and evolving the ML module system. Technical report,School of Computer Science, Carnegie Mellon University, 2005.

[12] Derek Dreyer. A type system for recursive modules. In ICFP ’07: Proceedings ofthe 2007 ACM SIGPLAN international conference on Functional programming, pages289–302, New York, NY, USA, 2007. ACM.

[13] Derek Dreyer, Karl Crary, and Robert Harper. A type system for higher-order mod-ules. In POPL ’03: Proceedings of the 30th ACM SIGPLAN-SIGACT symposium onPrinciples of programming languages, pages 236–249, New York, NY, USA, 2003. ACM.

[14] Derek Dreyer, Robert Harper, Manuel M. T. Chakravarty, and Gabriele Keller. Modulartype classes. In POPL ’07: Proceedings of the 34th annual ACM SIGPLAN-SIGACTsymposium on Principles of programming languages, pages 63–70, New York, NY, USA,2007. ACM.

27

[15] Derek Dreyer and Andreas Rossberg. Mixin’ up the ML module system. In ICFP’08: Proceeding of the 13th ACM SIGPLAN international conference on Functionalprogramming, pages 307–320, New York, NY, USA, 2008. ACM.

[16] Dominic Duggan and Constantinos Sourelis. Mixin modules. In ICFP ’96: Proceedingsof the first ACM SIGPLAN international conference on Functional programming, pages262–273, New York, NY, USA, 1996. ACM.

[17] Martin Elsman. Static interpretation of modules. In ICFP ’99: Proceedings of the fourthACM SIGPLAN international conference on Functional programming, pages 208–219,New York, NY, USA, 1999. ACM.

[18] Kathleen Fisher and John Reppy. Statically typed traits. Technical Report TR-2003-13,Department of Computer Science, University of Chicago, Chicago, IL, December 2003.

[19] Matthew Flatt and Matthias Felleisen. Units: cool modules for HOT languages. In PLDI’98: Proceedings of the ACM SIGPLAN 1998 conference on Programming languagedesign and implementation, pages 236–248, New York, NY, USA, 1998. ACM.

[20] Ronald Garcia, Jaakko Jarvi, Andrew Lumsdaine, Jeremy Siek, and Jeremiah Willcock.An extended comparative study of language support for generic programming. Journalof Functional Programming, 17(2):145–205, March 2007.

[21] Charles M. Geschke, Jr. James H. Morris, and Edwin H. Satterthwaite. Early experiencewith Mesa. Commun. ACM, 20(8):540–553, 1977.

[22] R. Harper, R. Milner, and M. Tofte. A type discipline for program modules. In 2ndColloquium on Functional and Logic Programming and Specifications (CFLP) on TAP-SOFT ’87: Advanced Seminar on Foundations of Innovative Software Development,pages 308–319, New York, NY, USA, 1987. Springer-Verlag New York, Inc.

[23] Robert Harper, Peter Lee, Frank Pfenning, and Eugene Rollins. Incremental recompi-lation for Standard ML of New Jersey. Technical report, Carnegie Mellon University,Pittsburgh, PA, USA, 1994.

[24] Robert Harper and Mark Lillibridge. A type-theoretic approach to higher-order mod-ules with sharing. In POPL ’94: Proceedings of the 21st ACM SIGPLAN-SIGACTsymposium on Principles of programming languages, pages 123–137, New York, NY,USA, 1994. ACM.

[25] Robert Harper, John C. Mitchell, and Eugenio Moggi. Higher-order modules and thephase distinction. In POPL ’90: Proceedings of the 17th ACM SIGPLAN-SIGACTsymposium on Principles of programming languages, pages 341–354, New York, NY,USA, 1990. ACM.

28

[26] Robert Harper and Benjamin C. Pierce. Advanced Topics in Types and ProgrammingLanguages, chapter Design Considerations for ML-Style Module Systems. MIT Press,2005.

[27] Mark P. Jones. Using parameterized signatures to express modular structure. In POPL’96: Proceedings of the 23rd ACM SIGPLAN-SIGACT symposium on Principles ofprogramming languages, pages 68–78, New York, NY, USA, 1996. ACM.

[28] Mark P. Jones. First-class polymorphism with type inference. In POPL ’97: Proceed-ings of the 24th ACM SIGPLAN-SIGACT symposium on Principles of programminglanguages, pages 483–496, New York, NY, USA, 1997. ACM.

[29] Didier Le Botlan and Didier Remy. MLF: Raising ML to the power of System F.In Proceedings of the Eighth ACM SIGPLAN International Conference on FunctionalProgramming, pages 27–38, August 2003.

[30] Xavier Leroy. Manifest types, modules, and separate compilation. In POPL ’94: Pro-ceedings of the 21st ACM SIGPLAN-SIGACT symposium on Principles of programminglanguages, pages 109–122, New York, NY, USA, 1994. ACM.

[31] Xavier Leroy. Applicative functors and fully transparent higher-order modules. InPOPL ’95: Proceedings of the 22nd ACM SIGPLAN-SIGACT symposium on Principlesof programming languages, pages 142–153, New York, NY, USA, 1995. ACM.

[32] Xavier Leroy. A syntactic theory of type generativity and sharing. Journal of FunctionalProgramming, 6(5):667–698, 1996.

[33] Xavier Leroy. A modular module system. J. Funct. Program., 10(3):269–303, 2000.

[34] Mark Lillibridge. Translucent Sums: A Foundation for Higher-Order Module Systems.PhD thesis, School of Computer Science, Carnegie Mellon University, May 1997. Avail-able as Technical Report CMU-CS-97-122.

[35] David MacQueen. Modules for Standard ML. In LFP ’84: Proceedings of the 1984ACM Symposium on LISP and functional programming, pages 198–207, New York, NY,USA, 1984. ACM.

[36] David MacQueen. An implementation of Standard ML modules. In LFP ’88: Proceed-ings of the 1988 ACM conference on LISP and functional programming, pages 212–223,New York, NY, USA, 1988. ACM.

[37] David B. MacQueen. Using dependent types to express modular structure. In POPL’86: Proceedings of the 13th ACM SIGACT-SIGPLAN symposium on Principles ofprogramming languages, pages 277–286, New York, NY, USA, 1986. ACM.

29

[38] David B. MacQueen and Mads Tofte. A semantics for higher-order functors. In ESOP’94: Proceedings of the 5th European Symposium on Programming, pages 409–423, Lon-don, UK, 1994. Springer-Verlag.

[39] Robin Milner, Mads Tofte, and Robert Harper. The Definition of Standard ML. MITPress, Cambridge, MA, USA, 1990.

[40] Robin Milner, Mads Tofte, Robert Harper, and David MacQueen. The Definition ofStandard ML - Revised. The MIT Press, May 1997.

[41] Philippe Narbel. Type sharing constraints and undecidability. J. Funct. Program.,17(2):207–214, 2007.

[42] Martin Odersky, Vincent Cremet, Christine Rockl, and Matthias Zenger. A nominaltheory of objects with dependent types. In Proc. ECOOP’03, Springer LNCS, July2003.

[43] Scott Owens and Matthew Flatt. From structures and functors to modules and units.In ICFP ’06: Proceedings of the eleventh ACM SIGPLAN international conference onFunctional programming, pages 87–98, New York, NY, USA, 2006. ACM.

[44] D. L. Parnas. On the criteria to be used in decomposing systems into modules. Commun.ACM, 15(12):1053–1058, 1972.

[45] Benjamin C. Pierce. Types and Programming Languages. MIT Press, 2002.

[46] Norman Ramsey, Kathleen Fisher, and Paul Govereau. An expressive language of signa-tures. In ICFP ’05: Proceedings of the tenth ACM SIGPLAN international conferenceon Functional programming, pages 27–40, New York, NY, USA, 2005. ACM.

[47] D. Remy. Type checking records and variants in a natural extension of ML. In POPL’89: Proceedings of the 16th ACM SIGPLAN-SIGACT symposium on Principles ofprogramming languages, pages 77–88, New York, NY, USA, 1989. ACM.

[48] Claudio V. Russo. Types for Modules. PhD thesis, Edinburgh University, 1998.

[49] Claudio V. Russo. First-class structures for Standard ML. Nordic J. of Computing,7(4):348–374, 2000.

[50] Claudio V. Russo. Recursive structures for Standard ML. In ICFP ’01: Proceedings ofthe sixth ACM SIGPLAN international conference on Functional programming, pages50–61, New York, NY, USA, 2001. ACM.

[51] Nathanael Scharli, Stephane Ducasse, Oscar Nierstrasz, and Andrew P. Black. Traits:Composable units of behaviour. In In Proc. European Conference on Object-OrientedProgramming, pages 248–274. Springer, 2003.

30

[52] Zhong Shao. Typed cross-module compilation. In ICFP ’98: Proceedings of the thirdACM SIGPLAN international conference on Functional programming, pages 141–152,New York, NY, USA, 1998. ACM.

[53] Zhong Shao. Transparent modules with fully syntactic signatures. In ICFP ’99: Proceed-ings of the fourth ACM SIGPLAN international conference on Functional programming,pages 220–232, New York, NY, USA, 1999. ACM.

[54] David Swasey, VII Tom Murphy, Karl Crary, and Robert Harper. A separate compila-tion extension to Standard ML. In ML ’06: Proceedings of the 2006 Workshop on ML,pages 32–42, New York, NY, USA, 2006. ACM.

[55] Mads Tofte. Operational Semantics and Polymorphic Type Inference. PhD thesis, De-partment of Computer Science, Edinburgh University, Mayfield Rd., EH9 3JZ Edin-burgh, May 1988.

[56] Mads Tofte. Principal signatures for higher-order program modules. In POPL ’92: Pro-ceedings of the 19th ACM SIGPLAN-SIGACT symposium on Principles of programminglanguages, pages 189–199, New York, NY, USA, 1992. ACM.

[57] Mads Tofte. Principal signatures for higher-order program modules. Journal of Func-tional Programming, 4(03):285–335, 1994.

[58] Dimitrios Vytiniotis, Stephanie Weirich, and Simon Peyton Jones. FPH: first-classpolymorphism for Haskell. In ICFP ’08: Proceeding of the 13th ACM SIGPLAN inter-national conference on Functional programming, pages 295–306, New York, NY, USA,2008. ACM.

[59] P. Wadler and S. Blott. How to make ad-hoc polymorphism less ad hoc. In POPL’89: Proceedings of the 16th ACM SIGPLAN-SIGACT symposium on Principles ofprogramming languages, pages 60–76, New York, NY, USA, 1989. ACM.

[60] Phil Wadler. The expression problem. Email to the Java Genericity mailing list, De-cember 1998.

[61] Niklaus Wirth. The module: A system structuring facility in high-level program-ming languages. In Proceedings of a Symposium on Language Design and ProgrammingMethodology, pages 1–24, London, UK, 1980. Springer-Verlag.

A Initial progress on formal semantics

The following is a summary of an initial progress on the formalization of the true higher-ordermodule semantics found in SML/NJ 110.60+. It is a work-in-progress. Fig. 18 summarizesthe surface module calculus. It supports higher-order functors, sharing constraints, eqtype,and exception specifications. Figs. 19, 22, and 23 give the first few rules of the elaboration

31

d ::= signature s = sig{spec}| local d in d end| ld

ld ::= structure X = m

| functor F (X : sigexp) cnstr = m

| val x = e | type α t = τ

| local ld in ld end| open q

m ::= q | struct{ld} | q arg | let d in m end| m : sigexp | m :> sigexp

arg ::= (d) arg | (m) arg | (m) | (d)cnstr ::= : sigexp | :> sigexp

sigexp ::= s | sig{spec} | sigexp where type α q = τ

spec ::= structure q : sigexp[= q]| functor q (X : sigexp) : sigexp| type α t[= τ ] | val x : τ | sharing type p = p

| sharing p = p | exception exn

| eqtype α t[= τ ]τ ::= τ → τ | int | t

Figure 18: Module surface language: Design caveat – In SML/NJ the AST permits signaturedeclarations within structures. The parser, however, does not support this. The surface languagefollows the parser. The implementation AST has an Abstract Structure form, but the parser doesnot seem to ever produce it.

semantics. More importantly, fig. 20 gives the internal module representation including forhigher-order functors and the static lambda calculus of entities. Fig. 21 describe how entitiesare evaluated.

32

Γ ` d1 ⇒ (d1, entdec1,Γ1,∆1)Γ⊕ Γ1 ` d2 ⇒ (d2, entdec2,Γ2,∆2)

Γ ` local d1 in d2 end⇒ (local d1 in d2 end, entdec,Γ2,∆)

(local)

Γ ` type α t = τ ⇒ (, entdec,Γ,∆)(type)

Γ ` sig{spec} ⇒ (sig{spec}, •,Γ′, ∅)(sig)

Γ ` struct{ld} ⇒ struct{ld}(structure)

Figure 19: Static Semantics

33

Realization expressionϕ ::= ρ entity path

| χ structure entity| (µ, η) stamping| θ(ϕ) functor application| ϕ ↓ Σ abstraction matching| let η in ϕ local definition| Ξ formal functor body| ρ.ϕ . ϕ constraint

η ::= ρ, t | ρ, strid = ϕ | ρ, θ entity declarationsθ ::= λρ.ϕ | ρ | let η in θ | ψ functor expressionΣ ::= 〈m,x, x = spec, typs, strs〉

spec ::= ρ : Σ str no def| ρ : Ξ functor| τ semantic tycon| datacon

| ρ : Σ =ρ Σ str const def| ρ : Σ = Σ;ϕ str relative def

υ ::= ψ | χ entityψ ::= Jλρ.ϕ; ΥK functor entityχ ::= JΥK structure entity

| JΣ; ρK strsigΞ ::= 〈Σp, ρ, x,Σr〉 functor signatureΘ ::= • | (Ψ, ρ, ρ,Θ) entity path contextt ::= ρ | Const τ | Formal τ tyc expressionµ ::= new | get ϕ stamp expression

Θ⊕ ρ , (Ψ, ρ1ρ, ρ2,Θ) Ψ : → ρ

σ : sp→ µ Entity Env Υ : ρ→ υ

Figure 20: Module Representation

34

Υ;Θ ` ϕ ⇓ (ϕ,∆Υ)ψ = Jm,λρ.ϕ,Υ, rpK Υ[ρ : χ]; Θ `• ϕ ⇓ (ϕ′,∆Υ)

Θ ` ψ(χ) ⇓ ϕ′ (evalapp)

Υ;Θ ` ρ ⇓ (Υ(ρ),Υ)(fctvar)

Υ;Θ ` ψ ⇓ (ψ,Υ)(fctconst)

(m new)Υ;Θ ` λρ.ϕ ⇓ (Jm,λρ.ϕ, rpK,Υ)

(fctλ)

Υ;Θ ` η ⇓ Υ′ Υ′; Θ ` θ ⇓ (ψ,Υ′′)Υ;Θ ` let η in θ ⇓ (ψ,Υ′′)

(fctlet)

Υ;Θ `ρ ϕ ⇓ (χ,Υ′)Υ;Θ ` ρ, strid = ϕ ⇓ Υ′[ρ : χ]

(decstr)

Υ;Θ `tyc ρ, t ⇓ τΥ;Θ ` ρ, t ⇓ Υ[ρ : τ ]

(dectyc)Υ;Θ ` θ ⇓ (ψ,Υ′)

Υ;Θ ` ρ, θ ⇓ Υ′[ρ : ψ](decfct)

Υ;Θ `ι ρ ⇓ (Υ(ρ),Υ))(strvar)

Υ;Θ `ι χ ⇓ (χ,Υ)(strconst)

Θ′ = Θ⊕ ι Υ;Θ′ ` µ ⇓ m Υ;Θ′ ` η ⇓ Υ′

Υ;Θ `ι (µ, η) ⇓ (Jm,Υ′, rpK,Υ)(strstp)

Υ;Θ ` θ ⇓ (ψ,Υ′)Υ′; Θ `ι ϕ ⇓ (χ,Υ′′) Θ′ ` ψ(χ) ⇓ ϕ′

Υ;Θ `ι θ(ϕ) ⇓ ϕ′ (strapp)

Υ;Θ ` η ⇓ Υ′ Υ′; Θ `ι ϕ ⇓ (χ,Υ′′)Υ;Θ `ι let η in ϕ ⇓ (χ,Υ′′)

(strlet)

Υ;Θ `ι ϕ ⇓ (χ,Υ′) . . .

Υ;Θ `ι ϕ ↓ Σ ⇓ (χ′,Υ′)(strabs)

Υ;Θ `ρ ϕ ⇓ (χ,Υ′) Υ;Θ′ ⊕ ρ : χ ` ϕ′ ⇓ (χ′,Υ′′)Υ;Θ `ι ρ.ϕ . ϕ′ ⇓ (χ,Υ′′)

(strcstr)

ι ::= • | ρ

Figure 21: Entity semantics

35

ρ freshΓ;Θ `spc X : Σ ⇒ (Γ[X : Σ; ρ], ρ : Σ)

(strspec)

ρ fresh

Γ;Θ `spc X : Σ = spath

⇒ (Γ[X : Σ; ρ], ρ : Σ = Θ(Γ(spath)))

(strspec-def)

Γ;Θ `sig sigexpp ⇒ Σp ρp freshΓ[X : Σp; [ρp]]; Θ `sig sigexpr ⇒ Σr

Γ;Θ `spc (X : sigexpp) : sigexpr(fctspec)

Γ;Θ `spc spec⇒ Σ X = SpecNames(spec)

Γ;Θ `sig sig{spec} ⇒ 〈X : Σ, typs, strs〉(sig)

Figure 22: Signature elaboration

Θ′ = Θ⊕ ι Γ;Υ; Θ′ `dcl ld⇒ (td, entdcl,Γ′,Υ′)Σ = 〈m, elems, [], []〉

Γ;Υ; Θ `ιstr struct{ld}

⇒ (Σ = let Σ;ϕ in locs,Σ, (new, η))

(strdec)

ρ fresh ψ = Γ(sp)Γ;Υ; Θ `ρ strexp⇒ (argDec, argStr, argExp,∆Υ)

Γ;Υ; Θ `ι sp(strexp)(strapp)

Figure 23: Module elaboration

36