A package system for maintaining large model distributions in vle software

A Package System for Maintaining Large ModelDistributions in VLE Software

WETICE/Comets 2012

Gauthier Quesnel, Ronan Trepos

INRA - French National Institute for Agricultural ResearchMIA - The Division of Applied Mathematics and Computer Sciences

BIA - Biometrics and Artificial Intelligence Unit

Quesnel et al. (INRA) Package system for VLE 1 / 32

1 Context

2 Problem

3 Package system

4 Results

5 Conclusions and perspectives


VLEVirtual Laboratory EnvironmentA multi-modeling and simulation environment:

A DEVS simulation engineA command line interface : VLEA modeling graphical user interface: GVLEA python package to develop web services: PyVLEA R package to analyze and visualize simulations results: RVLE

VLE is developed with the support of:I INRA, FranceI Computer Science Laboratory, Calais, FranceI CIRAD, Montpellier, FranceI Computer Science Laboratory, University of Mons-Hainaut, Belgium

Since 2007, VLE has more and more users and developers


RECORD, an INRA projectAn open platform for modeling and simulation of agro-ecosystemsProject initiated in 2006 which relies on VLE software

That makesmulti-disciplinary work easierCapacity to take intoaccount all the differentelements of croppingsystemsEach element can beexpressed in the most suitedmathematical formalismTarget: all researchers andengineers at INRA that usedynamic systems


Need to improve the collaboration

Needs for a collaboration system due to the heterogeneity of domainsinvolved: agronomy, economists, statistician, computer scientists, . . .Needs in sharing models; eg. integration of models at the plot scaleto little territories scale (thousands of plots)Needs in distributing existing models; eg. Stics (more than 25000lines of fortran code)Needs in helping the collaboration between modelers. More and morepeople trained to the platform through 3 days practice sessions (2 peryear, about 200 persons trained since 2007) and access to anE-learning web site


1 Context

2 Problem

3 Package system

4 Results



1 Context

2 Problem

3 Package system

4 Results



How to develop models ?

DEVSInitiated in 1976 by B. Zeigler to model and simulate dynamic systemsA discrete event formalism: the events drive the simulationModular and hierarchical approach: two entities atomic and coupledmodelsFormal approach for model coupling:

I The closed under coupling property: a coupled model is equivalent toan atomic

Abstract simulators (algorithms) to develop simulation tools:I One simulator for driving one atomic modelI One coordinator for driving one coupled model

Extensible:I DS-DEVS: dynamic-structure DEVS, to add or remove, models or

connections at run-timeI Cell-DEVS, Fuzzy-DEVS, RT-DEVS etc.


Atomic model definition (DEVS)

M = 〈X ,Y ,S, δext , δint , δcon, ta, λ〉

Where:δext , δint , δcon, ta, λ are functions, S the set of states

Coupled model definition (DEVS)

N = 〈X ,Y ,D,EIC ,EOC , IC〉

Executive model definition (DS-DEVS)

Mχ = 〈Xχ,Yχ,Sχ, γ,Σ∗, δintχ , δextχ , δconχ , λχ, τχ〉

Where:γ and Σ∗ are structure change functions


How to develop models ?VLE

VLE provides C++ shared library which embeds the DEVS simulationkernel, thus:

I users have to develop C++ class to develop atomic (DEVS) orexecutive models (DS-DEVS), or sub-formalisms

class Dynamics{public :

virtual Time timeAdvance () const ;

virtual void internalTransition ( const Time& time );

virtual void externalTransition ( const Time& time ,const ExternalEventList & lst );

virtual void output ( const Time& time ,ExternalEventList & out) const ;

virtual void confluentTransition ( const Time& time ,const ExternalEventList & lst );

virtual Value * observation ( const ObservationEvent & event ) const ;}



Sub-formalismsDEVS allows to develop sub-formalism of DEVS. With VLE, we provide:

dynamics which encapsulate mathematical formalisms such asdifferential equations, difference equations, Petri nets, etc.

structurals which manipulates the structure of the model: cellularautomaton, graph of communications, etc.

Dynamic sub-formalisms are atomic models with a specifiedbehavior: we provide transition functionsStructural sub-formalisms are atomic models and executive models ofDS-DEVS



For sub-formalismsWe provide simplified API (object oriented development)We provide C++ generators

For example, with the difference equation sub-formalism (which inheritsthe previous Dynamics class), only the compute function is available:class MyModel : public vle :: extension :: DifferenceEquation{

[...]

virtual double compute ( const Time& time){

x = y( -1) + z( -1);y = z( -1);

}};


How to develop models ?GVLE: GUI of VLE, an IDETo develop source code, the structure of the model, to assign initialconditions, observations, to define experimental frames, etc.:


The problem

ContextModelers develop models in C++ and can reuse models from othermodelersModels are stored into shared library. VLE loads and extracts atomicmodel from shared library

ProblemsWe clearly identify a problem of sharing and reusing code:

VLE C++ API / ABI versionSub-formalism C++ API / ABI versionModel change version (for example: the input or output DEVS ports)

SolutionTo improve collaborative work under VLE we develop a package system


1 Context

2 Problem

3 Package system

4 Results



Package system

We identify the following parts to developDefine the format of VLE’s packageDevelop and upgrade the tools and libraries of VLE (VLE, GVLE,PyVLE and RVLE) to take into account packages:

I Open, install, read, compile, check, remove packagesDevelop internet access from VLE to download and install packagesfrom distant distributions of packagesAdd a checking consistency onto distributions and user’s installation

This work is inspired from the FOSS (Free Open Source Software)community


What’s a package in VLE

We need to share with the package system:Data: eg. climate data files etc.External shared library: eg. scientific librariesSub-formalism libraries: structurals and dynamicsModels: classes inherit from atomic, executive or sub-formalismsclassesGUI plug-ins: to improve GVLE with new featuresOutput plug-ins: specific output simulation, database etc.Documentation

Package must have a version number, an unique name and a list ofrun-time and build dependencies



Finite State automatonThe vle.extension.fsa package provides:

Harel’s state-chart, Mealy, Moore and FDDevs sub-formalismsA graphical user interface to draw automaton an generate C++

For the user of the package:Shared library which stores binary code of sub-formalismsC++ Headers of the sub-formalismsGUI plug-ins

The vle.extension.fsa package, only depends of VLE 1.1


Example: FSA packageGUI to develop automaton


Example: FSA packageGenerated C++ source code



The content and the structure of a package

package /data/ # data ( climate data file)exp/ # experiment file ( structure of DEVS model )src/ # source codetest/ # unit testlib/ # shared sub - formalism librariesplugins / # simulators , GUI plug -ins shared librariesAuthors .txtDescription .txt # the most important fileLicense .txt

A source package does not have the lib and plugins directoriesA binary package does not have the src and test directories



The description file

Source : weather -genVersion : 1.0.0 -0Section : agronomicMaintainer : quesnel@users . sourceforge .netHomepage : http :// www.vle - project .orgBuild - Depends : vle (>= 1.1) ,

differential - equation (>= 1.0)Depends : weather (= 1.2.3) , output -file (>= 1.0)Conflicts :Description : A weather generator .

The build-dependencies are required to build and compile the packageThe dependencies are required at run-time of the simulationThe conflicts package must be removed before the compilation of thepackage or at run-time of the simulation



An example of package dependencies with an agronomic model

DifferenceEquation,FSA, Agent, PRNGare sub-formalisms:indirect build-dependspackages formaize-optimOptim issub-formalism: directbuild-depend packagefor maize-optimOther packages arerun-time depends

Depends and build-depends ofmaize-optim package


Distribution

DefinitionThe model developers build models or build sub-formalisms and makeupstream packagesThe package maintainers get, check, fix upstream packages and pullit into distributionThe distribution stores all packages. A consistency check isautomatically done before adding or removing a package.Users of the distribution user community can download and installdistribution’s packages

Users can be connected to several distributionsDistributions are accessible via http or ftp protocols


Distribution: Workflow








How to check package consistency ?

A distribution allows users to install and uninstall packages from a distantrepository

A distribution is a finite set of packages rulesEach package rule is a tuple of the form (p,B,D,C): where:

I p is a package,I B is a set a dependency build clauses for p,I D is a set of dependency clauses for p,I C is a set of conflict clauses for p

The dependency build clauses and dependency clauses must be present tocompile and use the package p

The dependency build packages can be removed after the installation



Each dependency clause or dependency build clause is a disjunction ofpackages p1|...|pk

A dependency clause requires that some packages from the set{p1, ..., pk} to be presentConflict clause requires that package p′ is not present in order toinstall the package p

Thus, a valid installation is a subset of the packages in thedistributionAn installation profile is valid if dependency clauses and conflictclauses are satisfiedA valid installation profile for a distribution should satisfy thedependency and conflict clauses of each package rule of thedistribution



The package system has two critical problems:To install the new package p given a distribution R and an installationprofile P, the package system has to check if there is a profile P ′containing p such that P ∪ P ′ is a valid installation profile for RIt may happen that a new package p cannot be installed because it isin conflict with some packages already installed on the machine. Inthis case, we must first uninstall some packages before attempting toinstall p. The task of the package system is then to identify a set ofpackages P ′ such that p can be installed on P − P ′

To address these two problems, we can use a satisfiability problem solver(SAT) or a constraint satisfaction problem (CSP)


1 Context

2 Problem

3 Package system

4 Results



Results

The major parts of the package system has been developedThe major packages of VLE are converted into packages (near 20packages)RECORD project is converting its packages in new VLE packages(near 100 packages)Package system is working to download and install packages from aserverAll tools of VLE are converted to package systems: VLE, GVLE,RVLE and PyVLEHowever,

I The package system does not embed a complete solver yetI The distribution package does not provide consistency checking

The package system of VLE is available. VLE and RECORD users use it !


1 Context

2 Problem

3 Package system

4 Results



Conclusions and perspectives

A success:Users are happy to use the package system to share and reuse model,sub-formalism and dataThe use of package seems to simplify the construction of complexmodels by aggregating atomic models and data from other packagesUsers intensively use generic models in Weather or Glue packages:these packages are now robustPackage system improves the global quality of the models andsimplify the maintenance of models

However,The package system needs to have computer engineers:

To integrate and maintain packages of the communityTo perform release of the distribution


Technology

A package system for maintaining large model distributions in vle software