1

Towards a Generic Deployment Framework for J2EE, OSGi, Web Services(or Everything)

Tuesday, February 28th, 2006

Philippe Merle

Email: Philippe.Merle@inria.frJacquard Project - INRIA Futurs,Laboratoire d‘Informatique Fondamentale de Lille (LIFL),Université des Sciences et Technologies de Lille (USTL),Villeneuve d’Ascq, France

ITEA S4ALL Task 3.5 Meeting, INT, Evry, France

2

Un brin d’histoire

• OpenCCM• Plate-forme de composants répartis CORBA

• Automatisation du déploiement

• Tout est spécifique au CCM

• Identifier des abstractions logicielles pour construire des plates-formes de déploiement

• Qu’est ce qui peut être factoriser en concepts/codes entre le déploiement OpenCCM et le déploiement Fractal ?

• Voir articles [DECOR 2004], [LMO 2005]

• Le travail présenté aujourd’hui est dans la poursuite de cette recherche du Graal

3

Agenda

• Context• Deployment of distributed services and systems

• Motivation• No generic deployment approach/tool/framework exists!

• Objectives• Build a generic deployment framework

• Principles• A fine grain component-based framework with Fractal

• Illustrations• Some of the Fractal-based components

• Work plan

• Conclusion and perspectives

4

Context - Multiple Services and Providers

• The user-centric approach considers the users’ services as gathering of the many services provided by the all-around world.

• Within each involved platform, deploying these services is made easy.

Why not everything deployed in once?

OSGi Gateway J2EE Server

Web Server <insert yours>

Directories

Banking Facilities

Extra Stuff

Monitoring

5

We Would Like To Have Something Which…

• Starts various processes on remote hosts

• Enables complex configurations for the launched processes

• Deploys applications’ services inside the hosting platforms

• Adds some test / control over this deployment

• Relays on nothing more than existing facilities• Operating systems, remote shell accesses, etc.

Via an easy to use, automation, extensible, and efficient approach!

6

Dependencies for remote deployment/launching

ServiceTowards

Users

Towards Systems

But Heterogeneity At Every Layer

Operating System Linux Windows Mac OS

SSH Telnet OSGiRemote Access ProActive

JOnAS Oscar JOnASApplication Server JOnAS

Requires a unified approach!

7

Motivation

• Deployment of distributed applications is a complex task, e.g.:

• Connect to remote hosts

• Download application (and middleware) binaries

• Launch middleware services and application servers– Set up environment variables– Start processes

• Install application binaries into application servers/gateways

• Configure and publish application services

• Many other specific tasks!

8

Motivation - Related Work

• Scripting is just a tool instead of a paradigm, e.g.:• How to reuse scripts?

• Many work done to automatically deploy applications• Into application servers/gateways, e.g.:

– J2EE, OSGi, Fractal, etc.

• On distributed systems, e.g.:– OLAN, OpenCCM, ProActive, JADE, etc.

• Via generic frameworks, e.g.– [DECOR 2004], etc.

• Via model driven approaches, e.g.– [LMO 2005], ORYA, OMG DnC, etc.

9

Motivation - Related Work

• No work cover the whole deployment process in a generic and technological independent way, e.g.:

• J2EE/OSGi do not address the launching of J2EE/OSGi gateways

• OSGi does not address deployment of distributed applications

• ProActive only deploys ProActive-based servers and applications

• OpenCCM only deploys CCM-based applications but not OpenCCM servers

• JADE deploys J2EE systems but not JADE servers

• Etc.

10

Motivation - Requirements

• Providing a unified approach• Heterogeneity of technologies

• Independent against granularity

• Automation instead of manual operations

• Distributed synchronizations, e.g.:• Connect to remote hosts before do anything

• Set up environment variables before launch processes

• Launch a transaction service before launch application servers

• Launch a registry service before publish application services

• Etc.

• Describing instead of programming

11

Objectives

• Build a generic deployment framework• Unified approach• Covering any task required to deploy

• Independent against granularity• Distributed systems, applications, services, components, objects, etc.

• Independent against technology• Usable with J2EE, OSGi, Web Services, Fractal, OpenCCM, etc.

• Easy to use by end-users• Describing what must be deployed instead of how to do

• Highly customizable and extensible by developers• Adapting the framework to specific purposes• Adding new concepts/mechanisms easily

12

Principles for a Unified Approach

• A framework instead of a single tool• Build tools from the framework

• Everything is reified as components• From end-user configurations to technical entities

• Fine grain component-based approach• Micro components and interfaces

• Intensive use of composition and sharing

13

Everything is Reified as Components

• A hosting system is a component containing a remote access

• This remote shell provides the interfaces to set variables and execute processes on the hosting system

• An execution platform is a component managing applications and services, with many interfaces such as deployment, monitoring, configuration, and so on. hosting system

remote shell

execution platform

service

service

14

Everything is Reified as Components

Network

hosting system

remote shell

execution platform

serviceservice

Reify and Control

15

Everything Reified as Components• Remote access protocols such as SSH, TELNET, etc.

• File transfer protocols such as SCP, FTP, HTTP, etc.

• Shells such as SH, CSH, WINBAT, etc.• Bound to a protocol

• Shell variables and commands• Bound to a shared shell

• Servers such as registry services and transaction managers (RMI, CORBA, WS), etc.

• Composed of shell variables and commands

• Application servers as JOnAS, OSGi, etc.• Composed of shell variables and commands

• Application beans as J2EE jars, OSGi bundles, etc.• Bound to a shared application server

• Etc.

16

Everything Reified as Components

protocol: SSH SSHd

SHshell: SH

host

Reified System Real System

JOnAS

EJB

server: JOnAS

bean: EJB

17

Technical Approach

This generic deployment framework is

• Designed with the Fractal component model• Each deployment concept/mechanism is a Fractal component

• Implemented with the Fraclet annotation framework• Leverage Component-Based Software Engineering

• Configured with Fractal ADL• End-users’ configurations

• Developers’ components

• Managed with Fractal Explorer

• Distributed with Fractal RMI

• Could benefit from any other Fractal-based work• AOKell, FAC, etc.

18

Technical Architecture

Network

End-UserConfiguration

Fractal ADL

Fractal ADL

ComponentsComponentsDeploymentComponents

hosting system

remote shell

execution platform

serviceservice

Fractal Explorer

Deploy

Translation Manage

Insta

ntiat

e

Fraclet

19

A Simple End-User ConfigurationThe End-User Description

MySystem = {

MyBean = BEAN {

location = URL(http://www.lifl.fr/bean.jar) ;

server = JONAS {

home = JONAS_HOME(/usr/jonas) ;

host = HOST {

hostname = HOSTNAME(host.lifl.fr) ;

protocol = SSH ;

shell = SH ;

}; }; }; };

An easy to use end-users’ configuration languageDescribing what the configuration is instead of how to deploy it

20

A Simple End-User ConfigurationThe Fractal Components

MySystem

MyBean: BEAN

location: URL server: JONAS

home: JONAS_HOME

host: HOST

hostname: HOSTNAME

protocol: SSH

shell: SH

21

A Simple End-User ConfigurationThe Fractal ADL Definition

<?xml version="1.0" encoding="ISO-8859-1" ?><!DOCTYPE definition PUBLIC "-//objectweb.org//DTD Fractal ADL 2.0//EN” "classpath://org/objectweb/fractal/adl/xml/standard.dtd">

<definition name="MySystem">

<component name=“MyBean" definition="BEAN">

<component name="location" definition="URL(http://www.lifl.fr/bean.jar)"/>

<component name="server" definition="JONAS">

<component name="home" definition="JONAS_HOME(/usr/jonas)"/>

<component name="host" definition="HOST">

<component name="hostname" definition="HOSTNAME(host.lifl.fr)"/>

<component name="protocol" definition="SSH"/>

<component name="shell" definition="SH"/>

</component>

</component>

</component>

</definition>

22

A Simple End-User ConfigurationThe Expected Runtime Behavior

Deploying MyBean implies to automatically

1. Connect to host.lifl.fr via the SSH protocol

2. Setup the JONAS_HOME variable to /usr/jonas

3. Add $JONAS_HOME/bin/unix to the PATH variable

4. Launch the command jonas start

5. Download http://www.lifl.fr/bean.jar

6. Launch the command jonas admin –a bean.jar

23

A More Complex End-User Configuration

MySystem = ITEA_S4ALL {

hosts = {

...

};

servers = {

...

};

beans = AUTO_INSTALL {

...

};

};

24

A More Complex End-User Configuration

MySystem = ITEA_S4ALL {

hosts = {

M = HOST {

hostname = HOSTNAME(host.lifl.fr) ;

protocol = SSH ;

shell = SH ;

};

… other host definitions ...

}; };

25

A More Complex End-User Configuration

MySystem = ITEA_S4ALL {

servers = {

jonas_on_M = JONAS {

home = JONAS_HOME(/usr/jonas) ;

host = hosts/M ;

};

oscar_on_M = OSCAR {

home = OSCAR_HOME(/usr/oscar) ;

host = hosts/M ;

};

... Other server definitions …

}; };

Sharing:- The M host- The SSH channel- The SH shell

26

A More Complex End-User ConfigurationMySystem = ITEA_S4ALL {

beans = AUTO_INSTALL {

bean1 = BEAN { location = LOCAL_FILE(bean1.jar) ; server = servers/jonas_on_M ;

};

bean2 = BEAN { location = LOCAL_FILE(bean2.jar) ; server = servers/jonas_on_M ; };

bean3 = BEAN { location = LOCAL_FILE(bean3.jar) ; server = servers/oscar_on_M ; };

bean4 = BEAN { location = LOCAL_FILE(bean4.jar) ; server = servers/oscar_on_M ; };

... Other bean definitions …

}; };

JOnAS on host Mshared by beans

OSCAR on host Mshared by beans

Ask for auto installation

27

A More Complex End-User ConfigurationThe Expected Runtime Behavior

1. Connect to host.lifl.fr via the SSH protocol

2. Setup the JONAS_HOME variable to /usr/jonas

3. Add $JONAS_HOME/bin/unix to the PATH variable

4. Launch the command jonas start

5. Download http://www.lifl.fr/bean1.jar

6. Launch the command jonas admin –a bean1.jar

7. Download http://www.lifl.fr/bean2.jar

8. Launch the command jonas admin –a bean2.jar

9. Setup system variables related to OSCAR

10. Launch OSCAR

11. Execute OSCAR commands to install bean3.jar and bean4.jar

• Other execution sequences and/or parallelization are possible

28

Everything is Reified as Components

Network

hosting system

remote shell

execution platform

serviceservice

Control

29

Everything is a Service(Not a Web Service)

• Each component has a Service interface to control the life cycle of the real entity reified

• interface Service { void start(); void stop();}

• Semantics depends on components/implementations• Protocol.start ~ initializes the network connection

• Protocol.stop ~ shutdowns the network connection

• Server.start ~ launches the server process

• Server.stop ~ stops the server process

• Bean.start ~ launches the bean into the application server

• Bean.stop ~ uninstalls the bean from its server

30

Composition

Composition of Services

Host.inria.fr

ShellSH

ProtocolSSH

JOnAS

OSCAR

Bean

Bean

Bean

Bean

31

Orchestration of Services

• Goal• Address distributed synchronizations of composition of services

• Orchestration• Implements the Service interface

• Schedules execution of services automatically

• Implementation• Introspects the composition and bindings between components to

plan the execution of services

• Various execution strategies– Sequential– Parallel

32

Orchestration

Orchestration of Services

Host.inria.fr

ShellSH

ProtocolSSH

JOnAS

OSCAR

Bean

Bean

Bean

Bean

Service

33

Orchestration

Orchestration of Services

Host.inria.fr

ShellSH

ProtocolSSH

JOnAS

OSCAR

Bean

Bean

Bean

Bean

(1)start (2)

start(3)start

(4)start

(5b)start

(6) start

(5a)start

(6) start

(6) start

(6) start

34

Orchestration

Orchestration of Services

Host.inria.fr

ShellSH

ProtocolSSH

JOnAS

OSCAR

Bean

Bean

Bean

Bean

(1)stop (6)

stop(5)stop

(4)stop

(3)stop

(2) stop

(3)stop

(2) stop

(2) stop

(2) stop

35

Protocol Components

• Reify and encapsulate protocols to access the real system

• interface Protocol { void send(String command); }

• Possible implementations• LOCAL by wrapping an external local shell• SSH by wrapping

– an external local ssh process– the BSD open source JSCH library

– http://www.jcraft.com/jsch/index.html• TELNET by wrapping

– an external local telnet process– the open source JTelnet library

– http:// Source Forge– the Apache open source Jakarta Commons Net library

– http://jakarta.apache.org/commons/net

Protocol

Service

36

Internet Protocol Components

Internet Protocol

Basic Protocolas

SSH, TELNET

IP HostName

IP Port

37

Implementation of the IP Port Component with Fraclet

/** @fractal.itf name=“port” */interface IpPort { int getIpPort();}

class PORT implements IpPort { /** @fractal.ac */ protected int port;

public int getIpPort() { return port; }}

• Fractal AttributeController interface & implementation, and Fractal ADL definitions generated automatically by Fraclet :-)

38

/** @fractal.itf name=“hostname” */interface IpHostName { String getIpHostName();}

class HOST_NAME implements IpHostName { /** @fractal.ac */ protected String hostname;

public String getIpHostName() { return hostname; }}

• Fractal AttributeController interface & implementation, and Fractal ADL definitions generated automatically by Fraclet :-)

Implementation of the IP Host Name Component with Fraclet

39

Implementation of the Basic Internet Protocol Components with Fraclet

abstract class InternetProtocol implements Service, Protocol { /** @fractal.bc */ protected IpHostName hostname; /** @fractal.bc */ protected IpPort port;}

class Telnet extends InternetProtocol { public void start() { . . . hostname.getIpHostName() . . . . . . port.getIpPort() . . . } public void stop() { . . . } public void send(String command) { . . . }}

• Ditto for all possible implementations, e.g.:• TELNET, SSH, etc.• Via an external process or an embedded Java library

40

Shell Components

• Reify shells executed in the real system

• interface Shell { void set_variable(String name, String value); void unset_variable(String name); void execute(String command); }

• Each shell is bound to a single Protocol component

• Behavior• Construct commands according to the shell syntax• Format path separator, i.e. / \ : ;

• Possible implementations• SH for Borne Unix shells• CSH for C Unix shells• WINBAT for Windows command shells

ProtocolShell

41

The HOST Component

• Provide an abstract composition of (Shell, InternetProtocol, Hostname) <definition name=“HOST”> <component name=“hostname” definition=“IpHostName”/> <component name=“protocol” definition=“InternetProtocol”/> <component name=“shell” definition=“ShellComponent”/> <binding client=“shell.protocol” server=“protocol.protocol”/> <binding client=“protocol.hostname” server=“hostname.hostname”/> </definition>

• Simplify the configuration for end-users <component name=“MyHost” definition=“HOST”> <component name=“hostname” definition=“HOSTNAME(host.lifl.fr)”/> <component name=“protocol” definition=“SSH”/> <component name=“shell” definition=“SH”/> </component>

shell protocol hostname

42

The VARIABLE Component

• Reify shell variables

• Each variable is bound to a shared Shell component

• Possible implementation class VARIABLE implements Service {

/** @fractal.bc */protected Shell shell;/** @fractal.ac */protected String name;/** @fractal.ac */protected String value;public void start() { shell.set_variable(name, value); }public void stop() { shell.unset_variable(name); }

}

• Possible instantiations• JONAS_HOME(path) = VARIABLE(JONAS_HOME, path)• OSCAR_HOME(path) = VARIABLE(OSCAR_HOME, path)

VARIABLE

ShellService

43

The Command Component• Reify shell commands on the real system• interface Command { void execute(Map arguments); }

• Each command is bound to a shared Shell component

• Possible implementation class COMMAND implements Command {

/** @fractal.bc */protected Shell shell;/** @fractal.ac */protected String command;private String computeCommand(Map arguments) { . . . }public void execute(Map arguments) { shell.execute(computeCommand(arguments));

}}

• Possible instantiations• COMMAND(echo hello world!)• COMMAND(jonas start)• COMMAND(jonas admin –a %url%)

COMMAND

Shell

Command

44

Server Components

• Reify a server/daemon executed in the real system

• Examples• Middleware servers

– Registry

– Transaction Manager

• Apache servers

• Various implementations• Wrapping a protocol to interact with the remote server

• Bound to a shell

• Etc.

SERVERService

45

Implementation of the SERVER Component by Composition

VARIABLE

shell

COMMANDstart

ServerImpl

SERVER

Service

*

COMMANDstop

SERVER*

depend

46

Implementation of the SERVER Component by Composition (Java code)

class ServerImpl implements Service {

/** @fractal.bc signature=“Service” */ protected Map required-servers;

/** @fractal.bc signature=“Service” */ protected Map variables;

/** @fractal.bc */ protected Command start-command;

/** @fractal.bc */ protected Command stop-command;

public void start() { start-command.execute(null);

}

public void stop() { stop-command.execute(null);} }

48

Other Concerns to Address• Display output from remote hosts

• Various GUI • Also Fractal components• Configurable by end-users

• Take care of system resource management• Threads and Sockets are limited• Study work done on DREAM• Use AOKell to build new controllers for system resources• Quid for deployment of large scale systems

• Distribute Fractal components into several JVM/host• To address system resource limitation• Using Fractal RMI

• Transactional deployment• GoTM framework

• Dynamic reconfiguration• Fractal Explorer to manage components, as done by JADE• Rules for autonomous computing

49

Work Plan

• Proof of concept already done• Areski Flissi and Philippe Merle involved

• Engineer recruitment from March 1st 2006

• Define Fractal interfaces and components

• Develop micro-components with Fraclet

• Compose components with Fractal ADL

• Implement Fractal ADL translator

• Implement management with Fractal Explorer

• Evaluate on various ITEA S4ALL scenarios

50

Conclusion

• A deployment framework• Generic

• Independent against technology and granularity

• Easy to use

• Highly customizable and extensible

• Fine grain component-based approach• Each deployment mechanism is reified as a component

• Intensive use of the Fractal component model and tools• Composition and sharing

• Julia, AOKell, Fraclet, Fractal ADL, Fractal Explorer, etc.

• Will be developed soon• Prototyping already done

• Implementation details could change but not the overall approach

51

Perspectives

• Deployment on Grid’5000 Specific components for OAR and KaDeploy services

• Deployment of OpenCCM Specific components for OpenCCM daemons Automation on PDA prototyped

• Deployment of Fractal Specific components for Fractal RMI Registry and Server

• Autonomous computing• Rules for dynamic reconfiguration – Ph.D. Jérémy Dubus

52

Thank you for your attention…

If you have any questions?

53

TODO

The JRE HOST Component

Counter Start/stop + synchronization sur start

ApplicationServer

JONAS

OSGi

BEAN

File transfer

54

class COMPUTED_HOSTNAMEimplements IpHostName {

/** @fractal.bc */ protected IpHostName computer;

protected String hostname = null;

public String getIpHostName() { if(hostname==null) hostname = computer.getIpHostName(); return hostname; }}

• Fractal BindingController code generated automatically by Fraclet :-)

• Hostname computation could be done by a reservation system like OAR on Grid’5000 :-)

Another Implementation of the IP Host Name Component with Fraclet

COMPUTED_HOSTNAME

COMPUTED_HOSTNAME

HOSTNAME_COMPUTER

55

Automation Protocol Components

PROTOCOL

Do start()at the 1stsend()

AnyProtocol

56

Implementation of Automation Protocol Component with Fraclet

class AutomationProtocol implements Protocol { /** @fractal.bc */ protected Protocol next;

private boolean isInit = false;

public synchronized void init() { if(isInit == false) { next.init(); isInit = true; } }

public void send(String command) { this.init(); next.send(command); }

public void shutdown() { next.shutdown(); }}

57

The JRE Component

• Provides an abstract composition for JRE

VARIABLEJAVA_HOME shell

VARIABLEPATH

JREImpl

JRE

58

Java Implementation of the JRE Component

class JREImpl implements Environment { /** @fractal.bc */ protected Environment home;

/** @fractal.bc */ protected Environment path;

public void set(){ home.set(); path.set(); }

public void unset() { home.unset(); path.unset(); }

}

59

Fractal ADL for the JRE Component• Fractal ADL definition

<definition name=“JRE” extends=“Environment”> <component name=“jre” definition=“JREImpl”/> <component name=“home” definition=“JAVA_HOME()”/> <component name=“path” definition=“VARIABLE(PATH,$JAVA_HOME)”/> <component name=“shell” definition=“Shell”/>

<binding client=“this.env” server=“jre.env”/> <binding client=“jre.java_home” server=“home.env”/> <binding client=“jre.path” server=“path.env”/>

<binding client=“home.shell” server=“shell.shell”/><binding client=“path.shell” server=“shell.shell”/>

</definition>

• Fractal ADL usage

<component name=“MyJre” definition=“JRE”> <component name=“home” definition=“JAVA_HOME(/usr/java)”/> <component name=“shell” definition=“hosts/M/shell”/> </component>