Fault-tolerance for Component-based Systems – An Automated Middleware Specialization Approach

Fault-tolerance for Component-based Systems – An Automated Middleware Specialization Approach

Sumant Tambe*

Akshay Dabholkar

Aniruddha Gokhale

Abhishek Dubey (Presenter)

Institute of Software Integrated Systems (ISIS),

Vanderbilt University, Nashville, TN, USA

*Contact : [email protected]

12th IEEE International Symposium on Object/component/service-oriented Real-time distributed Computing (ISORC 2009)

2

Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009

Motivation

Contemporary General-purpose middleware (CORBA, J2EE) Generic: Well designed for broad applicability Feature-rich: Supports non-functional

properties, such as security, real-time, FT etc. However, Does not support domain-specific variations of

non-functional semantics out-of-the-box E.g., Domain-specific fault-tolerance

(coming next) Cost of developing proprietary middleware is

high

3


Motivational Case Study (Material Handling System)

Representative Examples FedEx, UPS, DHL Airport Baggage Handling Food Processing/Bottling

High availability and safety is critical

Communicating software components Material Flow Control (MFC) Hardware Interface Layer (HIL)

E.g. Flipper and Motor Controllers

4


Material Handling System (Fault model and domain-specific recovery)

Hardware and software faults E.g., Jamming of the flipper Crash faults of software components

Detected by software components Communicated using software

exceptions E.g., FlipperJamException Communication failure

CORBA::COMM_FAILURE Domain-specific group recovery semantics

Shut down entire primary assembly (F, MC1, MC2)

Start the replica assembly (F’, MC1’, MC2’)

5


Backup Distributed Processing Unit (DPU)

Primary Distributed Processing Unit (DPU)

System Design and Implementation Challenges Lack of middleware abstractions

Failure of one component means failure of all in a DPU

Recover a collection of components simultaneously (even those who don’t have direct exposure to the fault)

Application-level solution is undesirable Technical concerns is (ideally) no

business of applications Failure handling behavior crosscuts

every component in a DPU Lot of manual programming

A B C

A’ B’ C’

Problem Statements1.How to add new semantics in COTS middleware retroactively?2.How to automate it to improve productivity and reduce cost?

Problem Statements1.How to add new semantics in COTS middleware retroactively?2.How to automate it to improve productivity and reduce cost?

6


Solution Approach

Promising approach for retroactive behavior augmentation Aspect-oriented Programming (AOP) Modularizes domain-specific functionality, potentially cross-cutting Aspect compiler can weave in the domain-specific behavior into

the original fabric of the COTS middleware

Promising approach for automation Domain-specific modeling (DSM) Inherently supports tool-driven generation of programming artifacts Simplifies specification of domain-specific requirements

Our SolutionGRAFT

(GeneRative Aspects for Fault-Tolerance)

Our SolutionGRAFT

(GeneRative Aspects for Fault-Tolerance)

7


GRAFT - Overview

GeneRative Aspects for Fault-Tolerance A two step process for specializing middleware

1. Design-time support to specify domain-specific FT requirements Component Availability Modeling Language (CAML) Automate FT modeling using a model transformation

2. Run-time support for middleware specialization Generate application-specific aspect code for group failover Weave in generated code in application-specific stubs.

8


Component Availability Modeling Language (CAML) Annotate component structural

models with fault-tolerance attributes

FT requirements captured using FailOverUnit (FOU)

FOU abstracts away the details of granularity of protection E.g., Component, Assembly

Treats a group of components as a single unit of failover Semantics: If one fails, all fail; Clients failover to replica Configurable degree of replication Captures application-specific failure exceptions

E.g., FlipperJamException

Replica = 2

9


Step 1: Automated Structural FT Modeling

10


Step 2: Automated Aspect Code Generation Two behaviors based on

component role FOU participant’s behavior

Detects the failure, if any Shuts down all other

participants, including itself

FOU client’s behavior Detects the failure, if any Shuts down the FOU Does an automatic failover

to a replica FOU

Generated code:AspectC++ AspectC++ compiler weaves in the generated code in the

respective component stubs

11


Sample Generated Aspect Code (MFC component)aspect FailOverUnit_Client {

// Auto-generated array of names of FailOverUnit participants.

char * FOU_Participants[] = { "FlipperController“, "MotorController1“, "MotorController2“, 0 };

size_t failure_count_; // Initialized to zero.

// Contains remote object reference of the replica.

HIL::IFlipperController_var replica_ref_;

// Weave advice around local stub of the flip() method of MFC.

advice execution ("void HIL::IFlipperController::flip()")

: around ()

// The advice is applied around the flip method.

{

do {

// Use the remote reference of the backup FlipperController component only if the primary component has failed.

if (failure_count_ > 0)

// "_that" is used to change "this" pointer before proceeding.

// Use live object reference of the replica.

tjp->action()._that = replica_ref_.in();

try {

// Continue the flip() function call as usual.

tjp->proceed ();

break;

}

catch(HIL::FlipperJamException & e) {

handle_exception(e);

// deactivates FailOverUnit participants

}

catch(CORBA::COMM_FAILURE & e) {



}

catch(CORBA::TRANSIENT & e) {



}

// Application-specific non-catastrophic exceptions are passed.

} while (replica_ref_.in() != NULL_POINTER);

}

};

12


Run-time Coordination

1. FOU participant detects the failure of another participant

2. Shuts down the primary FOU

3. Deployment infrastructure (DAM) removes the components

4. Clients detect the failure of FOU

5. Clients obtain the replica references from the naming service

6. Successful failover of all the clients

13


Evaluation of Efforts Reduction (Replica = 2)

Component name # of original connections

# of replica components

# of replica connections

Material Flow Control 1 / 1 0 / 0 2 / 0

Flipper Controller 2 / 2 2 / 0 4 / 0

Motor Controller 1 1 / 1 2 / 0 2 / 0


Fault-tolerance Modeling Efforts Without/With GRAFT

Component name # of try blocks # of catch blocks Total # of lines

Material Flow Control 1 / 0 3 / 0 45 / 0

Flipper Controller 2 / 0 6 / 0 90 / 0



Fault-tolerance Programming Efforts Without/With GRAFT

14


Concluding Remarks Specializing middleware for non-functional

properties is desirable using aspects Aspect-oriented code can be auto-generated from

higher-level domain/application-specific models Higher to lower-level model transformations, and

code generation improves productivity GRAFT realizes this approach

for fault-tolerant component-based systems built using Component Integrated ACE ORB (CIAO)

using Component Availability Modeling Language (CAML) using C-SAW and ECL for model transformation using AspectC++ for aspect-oriented programming

www.dre.vanderbilt.edu/cosmicwww.cis.uab.edu/gray/Research/C-SAW

www.dre.vanderbilt.edu/cosmicwww.cis.uab.edu/gray/Research/C-SAW

15


Thank you!

Documents

Fault-tolerance for Component-based Systems – An Automated Middleware Specialization Approach