Embed Size (px)
Citation preview
Managed by
A Lightweight Platform for Integration of Resource Limited Devices into Pervasive
Grids
Stavros Isaiadis and Vladimir Getov
University of Westminsterhttp://www.coregrid.net
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 2
• Current Grid Systems• Motivation• Project Overview• Research Issues• Operational Overview• Conclusion
Outline
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 3
Current Grid Systems
• Grids were initially developed to support large (mainly) scientific data-centric applications
• Resources were in abundance and Grids orchestrated sharing in large scale and/or harnessing of idle resources
• Current Grid systems assume:– Plenty of resources– Relatively reliable and static networks
• Traditionally, Grid systems are not concerned with the platform’s footprint: platform is “heavy” and cannot be deployed in resource limited devices
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 4
Grids and “Small” Devices
• But networks have evolved. The “edge” of networks increasingly consists of resource limited and/or mobile devices
• The Grid, however, hasn’t yet allowed the integration of these devices
• While access to the Grid and job submission is generally supported (e.g. through portals), contribution of resources is very difficult
• Resource constraints make installation of Grid middleware prohibiting
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 5
• Current Grid Systems
• Motivation• Project Overview• Research Issues• Operational Overview• Conclusion
Outline
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 6
But Why Integrate Resource Limited Devices?
• Future Grid systems should be truly pervasive and ubiquitous
• Mobile and pervasive computing attract a big part of current research and industry funds and hold a big share of the market pie
• Small and/or mobile devices increasingly offer functionality not available in traditional Grid nodes –multimedia equipment, sensors, global positioning systems etc.
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 7
• Current Grid Systems• Motivation
• Project Overview• Research Issues• Operational Overview• Conclusion
Outline
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 8
A Lightweight Grid Platform
A lightweight platform will provide the foundations for our project:
• Should be based on the CoreGRID component model
• Will consist of a small well defined set of core components
• Extra functionality through pluggable components
• Small footprint and communication overhead so that it won’t drain resources
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 9
• Current Grid Systems• Motivation• Project Overview
• Research Issues• Operational Overview• Conclusion
Outline
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 10
Dealing with Device Dynamicity
• Increased dynamicity is not acceptable in complex and heavily loaded Grid systems
• Solution: – Hide devices behind a proxy and delegate
finer control (scheduling, monitoring, recovery etc) to a local community system
– Devices now form a virtual “cluster” and are presented as a single entity to the Grid.
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 11
Virtual Aggregator Services
• Similar services (that implement the same interface) are aggregated and published through a single interface at the proxy
• The aggregator service implementation orchestrates access to the services it represents (by coordinating the various components like schedulers, indexes etc.)
• A single consistent interface to all aggregated resources is presented regardless of the internal state of the “cluster”
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 12
Making an Unstable Environment Virtually Stable
• When aggregating services that provide controlled access to raw resources (like cpu cycles, memory, storage etc.), we don’t present the total aggregate but only a fraction
• This way we can mask internal failures by reallocating resources from within the cluster without notifying (and thus placing extra burden on) higher level Grid components
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 13
Dealing with Failures
• Even behind a cluster, small devices still suffer from a high number of failures and we must provide the best possible quality of service
• Solution:– A monitoring and failure recovery framework
to predict, detect, diagnose and recover from failures
– Forecasting components to support the community scheduling system providing recommendations on the resource usage plans
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 14
The Monitoring Framework
• A set of small monitoring agents installed in the devices. They collect state information and forward them to the proxy
• A set of monitoring components at the proxy that collect and analyze the information– A monitoring “gateway” makes all decisions
and orchestrates the other components– A forecasting component provides predictions
on the best possible usage plan (according to usage and failure history)
– Failure recovery components decide on the recovery plans based on predefined scenarios
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 15
• Current Grid Systems• Motivation• Project Overview• Research Issues
• Operational Overview• Conclusion
Outline
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 16
Component Prototypes
• Currently using Globus Toolkit 3.2 middleware
• Will port to GT4 in the very near future
• Prototyping components using Java
• Prototypes are quite restrictive at this stage but they will:Help us get a better insightHelp us identify possible obstaclesProvide the skeleton for the functional final
platform
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 17
Operational Overview: Registration of Services
AGGREGATED SERVICE INTERFACES
RESOURCE META-DATA REPOSITORY
Add/Update Interface
Update Index
Discovery and Interface
IdentificationRegistration
LIVE RESOURCE INDEX
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 18
Operational Overview: Job SubmissionJob Submission
Forward
ConsultApplication Tasks
App/ Code
Job Requirements
Distribution
AGGREGATED SERVICE INTERFACES
SERVICE IMPLEMENTATION
LIVE RESOURCE INDEX
TASK EXTRACTION ENGINE
COMMUNITY SCHEDULING SYSTEM
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 19
Operational Overview: Failure Recovery
JOB META-DATA REPOSITORY
Event notification + status info
diagnosisforward
update
Event notification + status info
FORECASTING COMPONENT
LIVE RESOURCE INDEX
MONITORING GATEWAY
FAILURE DIAGNOSIS
FAILURE RECOVERY
plan
PLAN DEPLOYMENT
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 20
• Current Grid Systems• Motivation• Project Overview• Research Issues• Operational Overview
• Conclusion
Outline
European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 21
Conclusion
• The lightweight platform could provide the foundations for truly ubiquitous Grids
• Coupled with a lightweight monitoring framework, it is a big first step towards full and efficient integration of resource limited devices into the Grid
• The early prototyping phase will help us gain a better insight on the issues that might arise and will provide the skeleton for the final deliverable platform
