Implementing Simple Replication Protocols using CORBA Portable Interceptors

and Java Serialization

T. Bennani, L. Blain, L. Courtes, J.-C. Fabre, M.-O. Killijian, E. Marsden, F. Taïani

DSN-2004, Florence, Italy, July 1st

Toulouse, France

2DSN’04

Outline1. Motivation

2. Background information• Reflection• FT-CORBA

3. Architecture and protocol• Primary-backup replication mechanism

4. Performance evaluation

5. Conclusion• Limits observed

3DSN’04

Motivation Reflection as a means to implement FT

Transparent for application Separation of concerns

Reflection has been introduced into middleware CORBA Portable Interceptors Java Serialization

COTS based systems are cheaper Increasing COTS software for critical systems

Experiment using COTS reflective mechanisms for implementing FT

4DSN’04

Principles of Reflection

separating fault-tolerance from functional concerns

"the ability of a system to think and act about itself"

original system

fault-tolerance

meta-interfaces

meta-level

base-level

observation control

5DSN’04

From FRIENDS To DAISY

Base Object

Meta Object

• Interception : EVENTS

• Introspection : STATE

• Intercession : ACTION

Source-to-source trans. C++ classes Add reflective features

MetaObject Protocol Object life cycle Requests life cycle State handling

Limits Language dependent Access to source code External state Determinism

6DSN’04

From FRIENDS To DAISY

Java CORBA Object

FT Portable Interceptor

• Interception : EVENTS

• Introspection : STATE

• Intercession : ACTION

Source-to-source trans. C++ classes Add reflective features

MetaObject Protocol Object life cycle Requests life cycle State handling

Limits Language dependent Access to source code External state Determinism

DAISY

COTS

COTS

COTS

7DSN’04

FT-CORBA Augment CORBA with FT capabilities

Object Group Addressing (IOGRs)Transparent reference to group of servers

Extensions to failover semanticUnique request’s Ids and Retries upon request failures

Replication ManagementCreation, modification of groups

Fault ManagementFaults detection, report

Recovery ManagementState handling and checkpointing

Very few implementations yet

not a COTS

8DSN’04

The DAISY PlatformJava based Object Request Broker

ServerClient

ORB

JVMORB

JVM

IIOP Requests

9DSN’04

ServerClient

ORB

JVMORB

JVM

IIOP RequestsPIC PIS

REQUEST

REQUEST

The DAISY PlatformCORBA Portable Interceptors

Observe, delay and retarget out/in requests/exceptions

Cannot modify requests/replies (bad for SWIFI)

10DSN’04

The DAISY PlatformJava Serialization

ServerClient

ORB

JVMORB

JVM

IIOP RequestsPIC PIS

State of Server

Serialization

Save and restore state of Java objects

11DSN’04

The DAISY PlatformDependable Adaptive Interceptors & Serialization-based sYstem

Primary

Server

Client

ORB

JVM

ORB

JVMPIC

PIS

Backup

Server

ORB

JVM

PIS

12DSN’04

The DAISY PlatformDependable Adaptive Interceptors & Serialization-based sYstem

Primary

Server

Client

ORB

JVM

ORB

JVMPIC

PIS

Backup

Server

ORB

JVM

PIS

FT Algorith

m

13DSN’04

PrimaryServer

Client

ORB

JVM

ORB

JVMPIC

PIS

BackupServer

ORB

JVM

PIS

Algorithm Overview

Primary-backup strategy

PIC responsible for ID requests Managing faults

PIS responsible for Requests handling State Management Replica Management

14DSN’04

PrimaryServer

Client

ORB

JVM

ORB

JVMPIC

PIS

BackupServer

ORB

JVM

PIS

Client Side Fault Handling

Main role: « detecting faults »

Simple detection scheme

Transient communication faults

Upon exception ForwardRequest trick N retries Switch to backup

Exceptions

15DSN’04

PrimaryServer

ORB

JVM

PIS

BackupServer

ORB

JVM

PIS

Primary Side

Main role: « handling requests »

Upon request Invoke the request Obtain server’s state Forwards to backup

Request Info Reply message State

16DSN’04

PrimaryServer

ORB

JVM

PIS

BackupServer

ORB

JVM

PIS

Backup Side

Main role: « recover primary failures »

Buffer and manage « Primary packets »

Request Info Reply message State

Apply previous one

Upon request reception Ping primary Enter recovery mode

?

17DSN’04

Recovery

PrimaryServer

ORB

JVM

PIS

BackupServer

ORB

JVM

PIS

Crash occures when1. Primary idle2. Handling request3. Primary packet delivered

but not the reply

Case 1 & 2 Apply buffered state Handle request

Case 3 Discard buffered state Re-execute request

Multi-client more complex

18DSN’04

024681012141618

ApplicationApp with InterceptionApp with FT

Time (ms)MinimumAverageMaximum

Performance Evaluation Simple banking application

Account management Withdrawal, deposit, etc.

Testbed I686 1Ghz Linux 2.4 100 Mb/s Ethernet

1000 experiments

1000 operations FT Algorithms48%

Serialization15%

De-serialization15%

Interception7%

Application15%

19DSN’04

PI Drawbacks Can’t modify input params

Prohibit mechanisms E.g. ciphering

Can’t modify output params Cannot forge replies Complexifies

implementation

Must invoke every requests Cannot prevent invocation Must raise exceptions PB for some mechanisms E.g. leader-follower

Not CORBA objects Cannot implement easily

non-functional interface Not transparent for the

application

Don’t have a thread No “I am alive” messages

Cannot reorder requests Limit complexity of

strategies

20DSN’04

Conclusion Middleware standards embbed

Simple reflective mechanisms CORBA Portable Interceptors Java Serialization

Useful for implementing simple FT mechanisms Simple wrapping techniques

– IIOP level CRC32

– Synchronization interface (libc)

But new generation of Portable Interceptors More complex mechanisms (active replication) Better implementation Without being too intrusive

Implementing Simple Replication Protocols using CORBA Portable Interceptors

and Java Serialization

T. Bennani, L. Blain, L. Courtes, J.-C. Fabre, M.-O. Killijian, E. Marsden, F. Taïani

DSN-2004, Florence, Italy, July 1st

Toulouse, France