Upload
lucy-gray
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
KARMA with ProActive Parallel Suite12/01/2009
Air France, Sophia Antipolis
Solutions and Services for Accelerating your Applications
ProActive Parallel Suite
Speed-up applications by distributing them over a set of resources
The result of 9 years of R&D at INRIA
KARMA performances constraints
• KARMA processes computation needs refer to : - HEAVY PROCESS (Batch long time)- LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features:- Triggering mode (start and kill)- Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative)- Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization )- Fault error management
• Open system- Interoperability (upper and lower stream)- API (language, completeness, reliability)- Tools (friendliness, eclipse integration ability,..)
• Parallelization- Multiple processes capability (functional)- Parallelization programming capability (technical).
ProActive Technology Integration
Simple and Seamless integration
Installation : one directory Deployment : one XML file (GCM Deployment)
Possible automated agent-based deployment
Support for common standards Connection/Acquisition: Ssh, Rsh, LSF, PBS, SGE,… Communication: RMI, RMI/SSH, HTTP, Ibis,… Export any activity as a WebService J2EE (in and out)
Validated on common JDKs 1.5, 1.6 and 1.7
Activities are monitored through standard JMX Compliant with common monitoring tools (TIBCO Hawk)
KARMA performances constraints
• KARMA processes computation needs refer to : - HEAVY PROCESS (Batch long time)- LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features:- Triggering mode (start and kill)- Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative)- Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization )- Fault error management
• Open system- Interoperability (upper and lower stream)- API (language, completeness, reliability)- Tools (friendliness, eclipse integration ability,..)
• Parallelization- Multiple processes capability (functional)- Parallelization programming capability (technical).
KARMA performances constraints
• KARMA processes computation needs refer to : - HEAVY PROCESS (Batch long time)- LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features:- Triggering mode (start and kill)- Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) - Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization )- Fault error management
• Open system Interoperability (upper and lower stream)- API (language, completeness, reliability)- Tools (friendliness, eclipse integration ability,..)
• Parallelization- Multiple processes capability (functional)- Parallelization programming capability (technical).
Scheduler / Resource Manager Overview
• Multi-platform Graphical Client (RCP)
• File-based or LDAP authentication
• Static Workflow Job Scheduling, Native and Java tasks, Retry on Error, Priority Policy, Configuration Scripts,…
• Dynamic and Static node sources, Resource Selection by script, Monitoring and Control GUI,…
• ProActive Deployment capabilities : Desktops, Clusters, ProActive P2P,…
KARMA performances constraints
• KARMA processes computation needs refer to : - HEAVY PROCESS (Batch long time)- LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features:- Triggering mode (start and kill)- Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) - Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization )- Fault error management
• Open system Interoperability (upper and lower stream)- API (language, completeness, reliability)- Tools (friendliness, eclipse integration ability,..)
• Parallelization- Multiple processes capability (functional)- Parallelization programming capability (technical).
KARMA performances constraints
• KARMA processes computation needs refer to : - HEAVY PROCESS (Batch long time)- LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features:- Triggering mode (start and kill)- Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) - Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization )- Fault error management
• Open system Interoperability (upper and lower stream)- API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization- Multiple processes capability (functional)- Parallelization programming capability (technical).
Parallelism with ProActive Scheduling
Task 1(input 1) Task 2(input 2)
Task 3(res1,res2)
res1 res2
Task 1(input 1) Task 2(input 2) Task N(input N)…
Task 1(input 1)
Fu
nction
al // Tech
nica
l //
Computations are defined as Jobs made of Tasks
Bag of Tasks
Tasks Flow
ProActive application
KARMA performances constraints
• KARMA processes computation needs refer to : - HEAVY PROCESS (Batch long time)- LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features:- Triggering mode (start and kill)- Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) - Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization )- Fault error management
• Open system Interoperability (upper and lower stream)- API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization- Multiple processes capability (functional)- Parallelization programming capability (technical).
KARMA performances constraints
• KARMA processes computation needs refer to : - HEAVY PROCESS (Batch long time)- LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features:- Triggering mode (start and kill) Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) - Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization )- Fault error management
• Open system Interoperability (upper and lower stream)- API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).
Tasks can be defined in Java or as Native Application
Dynamic Task creation through Java APIJavaTask aTask = new JavaTask();aTask.setName("task 1");aTask.setRerunnable(2);aTask.setRestartOnError(RestartMode.ELSEWHERE);aTask.addArgument("foo",new Boolean(true));aTask.setExecClassName("com.activeeon.Compute");aTask.setPreScript("/path/to/script_file");
Can attach pre/post script to a task Any JSR-223 supported language (JS, Python, Ruby,…) Used for configuration and data transfer
Data are handled separately
Defining Tasks
Jobs can be defined in Java or XML
Dynamic job creation through Java APITaskFlowJob job = new TaskFlowJob();
job.setName("job_name");job.setPriority(JobPriority.NORMAL);job.setCancelOnError(true);job.addTask(new NativeTask(cmd));
Can easily develop tools that generate and submit jobs Specialized Automated
Static description in XML descriptor Can be defined by any non specialized user
Defining Jobs
KARMA performances constraints
• KARMA processes computation needs refer to : - HEAVY PROCESS (Batch long time)- LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features:- Triggering mode (start and kill) Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) - Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization )- Fault error management
• Open system Interoperability (upper and lower stream)- API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).
KARMA performances constraints
• KARMA processes computation needs refer to : HEAVY PROCESS (Batch long time) LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features:- Triggering mode (start and kill) Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization )- Fault error management
• Open system Interoperability (upper and lower stream) API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).
Executing Jobs
Submission through… Java API Graphical interface Command line
Recovery is fully configurable with User/System level distinction
1. Host failure : automatically re-dispatched with needed input Number of retries bounded by the administrator
2. Task failure : can be either… Automatically re-dispatched with needed input
Number of retries bounded by the user Aborted (Task) : exception is transmitted as task result Aborted (Job) : job is fully canceled
Scheduler itself is recoverable from underlying database
KARMA performances constraints
• KARMA processes computation needs refer to : HEAVY PROCESS (Batch long time) LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features:- Triggering mode (start and kill) Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization )- Fault error management
• Open system Interoperability (upper and lower stream) API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).
KARMA performances constraints
• KARMA processes computation needs refer to : HEAVY PROCESS (Batch long time) LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features: Triggering mode (start and kill) Parallelization capability (functional / technical) Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization ) Fault error management
• Open system Interoperability (upper and lower stream) API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).
Monitoring Jobs
System and ProActive level monitoring Through standard JMX Provide generic tool for monitoring and reporting Compliant with common monitoring tool
Monitoring Jobs
System and ProActive level monitoring Through standard JMX Provide generic tool for monitoring and reporting Compliant with common monitoring tool
Scheduler level monitoring On-going porting from MVC to JMX
KARMA performances constraints
• KARMA processes computation needs refer to : HEAVY PROCESS (Batch long time) LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features: Triggering mode (start and kill) Parallelization capability (functional / technical) Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization ) Fault error management
• Open system Interoperability (upper and lower stream) API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).
KARMA performances constraints
• KARMA processes computation needs refer to : HEAVY PROCESS (Batch long time) LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features: Triggering mode (start and kill) Parallelization capability (functional / technical) Re-run ability (fault tolerance / recovery) Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization ) Fault error management
• Open system Interoperability (upper and lower stream) API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).
KARMA performances constraints
• KARMA processes computation needs refer to : HEAVY PROCESS (Batch long time) LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features: Triggering mode (start and kill) Parallelization capability (functional / technical) Re-run ability (fault tolerance / recovery) Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization ) Fault error management
• Open system Interoperability (upper and lower stream) API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).
KARMA performances constraints
• KARMA processes computation needs refer to : HEAVY PROCESS (Batch long time) LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features: Triggering mode (start and kill) Parallelization capability (functional / technical) Re-run ability (fault tolerance / recovery) Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory) Strategy / Policy (Arbitration resources and tasks request prioritization ) Fault error management
• Open system Interoperability (upper and lower stream) API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).
Specializing the Scheduler
Business Level specialization and automation
ActiveEon provides business specific interface for the Scheduler Bio-technology (Gold, MapRead) Finance MatLab/Scilab
Specializing the Scheduler
Business Level specialization and automation
ActiveEon provides business specific interface for the Scheduler Bio-technology (Gold, MapRead) Finance MatLab/Scilab
User-level hooks to (very) easily specialize the Scheduler
Plug-in for RCP client e.g. ActiveEon’s Gold module
Result preview mechanism Can preview result in the scheduler GUI (textual/graphical) Can attach any external application to a task for result display
Specializing the Scheduler
Business Level specialization and automation
ActiveEon provides business specific interface for the Scheduler Bio-technology (Gold, MapRead) Finance MatLab/Scilab
User-level hooks to (very) easily specialize the Scheduler
Plugin for RCP client e.g. ActiveEon’s Gold module
Result preview mechanism Can preview result in the scheduler GUI (textual/graphical) Can attach any external application to a task for result display
KARMA performances constraints
• KARMA processes computation needs refer to : HEAVY PROCESS (Batch long time) LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features: Triggering mode (start and kill) Parallelization capability (functional / technical) Re-run ability (fault tolerance / recovery) Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory) Strategy / Policy (Arbitration resources and tasks request prioritization ) Fault error management
• Open system Interoperability (upper and lower stream) API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).
KARMA performances constraints
• KARMA processes computation needs refer to : HEAVY PROCESS (Batch long time) LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features: Triggering mode (start and kill) Parallelization capability (functional / technical) Re-run ability (fault tolerance / recovery) Performance (elapse time) Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory) Strategy / Policy (Arbitration resources and tasks request prioritization ) Fault error management
• Open system Interoperability (upper and lower stream) API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).
ProActive Resource Manager
Rely on ProActive Programming features GCM deployment standard Agent based deployment Support for many LAN / WAN / Grid / Cloud standards…
MultiCPU/Core Cluster
Desktop Grid
Clusters / Grille Clouds / RAS
Straightforward Upgrade
ProActive Resource Manager
Rely on ProActive Programming features GCM deployment standard Agent based deployment Support for many LAN / WAN / Grid / Cloud standards…
Deal with resources heterogeneity with Selection Scripts User defined : no need for system reconfiguration Any JSR-223 supported language (JS, Python, Ruby,…)
Can rely on dynamic decision or on static host description Optimized (probabilistic selection of candidates)
KARMA performances constraints
• KARMA processes computation needs refer to : HEAVY PROCESS (Batch long time) LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features: Triggering mode (start and kill) Parallelization capability (functional / technical) Re-run ability (fault tolerance / recovery) Performance (elapse time) Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)
• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory) Strategy / Policy (Arbitration resources and tasks request prioritization ) Fault error management
• Open system Interoperability (upper and lower stream) API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).
KARMA performances constraints
• KARMA processes computation needs refer to : HEAVY PROCESS (Batch long time) LIGHT PROCESS (Interactive short time)
• Processes are expressed according to the following features: Triggering mode (start and kill) Parallelization capability (functional / technical) Re-run ability (fault tolerance / recovery) Performance (elapse time) Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)
• Manage resources consumption Availability Resources optimization (CPU, Core, Memory) Strategy / Policy (Arbitration resources and tasks request prioritization ) Fault error management
• Open system Interoperability (upper and lower stream) API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)
• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).