Grids, Clouds and the Community.Cloud Technology and the NGS

Steve ThornEdinburgh University

Matteo Turilli, Oxford UniversityPresented by David Fergusson

NeSC, University of Edinburgh

• National eScience Centre– Support and develop IT supported advanced

research

• University of Edinburgh– College of Science and Engineering– Coordination and cooperation to make best use of

leading research across research.

NGS

• JISC supported national service to:– Provide a framework for institutions to cooperate

and share services.– Help promote access to JISC services and synthesis

for researchers.

Views of the Clouds• Infrastructure as a Service• Platform as a Service • Service as a Service• Software as a Service

• Private Clouds• Public Clouds• Hybrid Clouds

• Cloud vs Web 2.0 (?)

Views of the Clouds• Infrastructure as a Service• Platform as a Service √• Service as a Service• Software as a Service

• Private Clouds• Public Clouds• Hybrid Clouds

• Cloud vs Web 2.0 (?)

Cloud computing for NGS/NeSC (incomplete list)

• Dynamic provision of services– Individualisation– Peaky demand

• Extending virtualisation• Packaging

– Service configuration– Provenance for research– New publishing models

Head in the clouds?• Dynamic (service) provisioning• How is it applicable to the NGS?• Training

– Rapidly deploy NGS services for training– Isolate training from production

• Other– Specialised research environments– Rapid deployment

• Identify use cases and gather requirements

NGS 3 EWP2

• “NGS Agile Deployment Environments”• EPSRC funded, 2 years• People

– Matteo Turilli (OeRC, Oxford) [0.75 FTE]– Steve Thorn (NeSC, Edinburgh) [0.5 FTE]– David Fergussion (NeSC, Edinburgh) [WP Leader]

Overview

• Agile service deployment• Virtualization vs. Cloud?• Use cases and requirements gathering

– Training– Identify other (scientific) communities

• Create images – NGS Services. Which ones?

Overview (cont.)• Realistic usage

– Training event on virtualized infrastructure• Hosting infrastructure?

– Amazon EC2 compatible• De facto standard currently, with open source

implementation– Ease of deployment– Eucalyptus, Nimbus and others

• Hardware– Edinburgh: 8 cores ⇒ 16+ dual cores => 64 cores– Oxford: 64 cores (older), new cluster => 64 cores

Eucalyptus• “Elastic Utility Computing Architecture Linking

Your Programs To Useful Systems”• Open source and Commercial• Amazon Web Services API compatible

– EC2, storage - S3, Elastic Block Store (EBS)

• Easy to install• Xen and KVM hypervisors

– Commercial version supports others (inc. VMWare)

In the past

• We have worked with Xen in the past to have Live CDs

• Virtualisation• Works, but

– Issues with security setups– networking

Eucalyptus networking challenges

• Eucalyptus 'Managed' networking– Different networking modes– 'Managed' is the most flexible and feature rich – more complex– Allows elastic IP pool and image isolation.

• VMs have private and public IPs– Introspection issues

• Elastic IP – User assignable (may be somewhat different from Amazon)

• X509 Service certificates (NGS Host)• Switch configurations

cont.....

• Security Groups (EC2) – Implemented in Eucalyptus– isolate VMs

• VM public traffic routed through Cluster controller– Instance doesn't have knowledge of its public IP– Bit like a NAT

• Implications for GSI: $GLOBUS_HOSTNAME

Eucalyptus architecture

• Cloud controller– Entry point– Gathers information

• Cluster controller– Schedules VM execution – Manages virtual network

• Node controller– Controls VM execution

• (Xen running on node)

Storage controller (Walrus)

implements Amazon’s S3 interface

cont.....

• Security Groups (EC2) – Implemented in Eucalyptus– isolate VMs

• VM public traffic routed through Cluster controller– Instance doesn't have knowledge of its public IP– Bit like a NAT

• Implications for GSI: $GLOBUS_HOSTNAME

Clouds vs Virtualisation

• Similar security and networking issues in Clouds and Virtualised instances– Virtualisation – virtualise instance– Clouds – virtualise the network (and other things)

too

• All arise from the requirements for rapid, automated, dynamic, reliable, reproducible, robust, provisioning

Progress• Started with Eucalypus 1.4.2• Eucalyptus 1.6.2 deployed at Oxford and

Edinburgh• Existing Images:

– GSI-OpenSSH server– 'Single node cluster': torque/maui + Globus GRAM

& GridFTP

• Next step – some real world testing of phase1 images.

• Image snapshots – not straightforward

Further work• Re-evaluate hosting infrastructure• Develop more images

– Distributed torque/maui cluster + GRAM & GridFTP– Condor & GRAM?– 'Core site'

• Training event in near future• Identify pilot community & gather requirements• Deploy fledgling cloud infrastructure

Further work 2

• Can snap shotting in Clouds side step packaging issues (configuration) in middleware? – By automating the copying and re-deployment of

successful server installs.– Not just having a machine but a set of services

which can be copied and deployed directly

Statement 1• NRENs are still able to provide services that are

generally better or more economic than those from commercial services providers.

• Commercial offerings are currently shaped to particular modes of usage (due to their “by-product” heritage. This may limit utility for some users. EBI have reported issues due to virtual network sharing & unpredictability.

• Data out costs are high• Commercial providers can offer “infinite” resource rapidly – not easy for NRENs• Should NRENs own large computing centres?• Payment models in institutions & research – FEC in the UK

Statement 2• NRENs generally operate as a network for a closed

group of users who have advanced requirements to support their research and education users.

• There are certainly major issues with current commercial provision in regards to data confidentiality, isolation of users, service/data guarantees, internal/external access.– Recent data losses from commercial services

• Commercial providers are almost certainly at an advantage in providing simple generic computing

• NGS/UoE sees need to provide more complex research services.

Statement 3• The NRENs do not compete with commercial ISPs, but

offer a different level of service in parallel with them.

• This is necessarily so. NRENs and other academic providers should find “blue water” for services.– Trust networks and security for public clouds ?– Service discovery ?– Data services ?– Accounting, monitoring ?– ............. >

• Commercial providers will have no interest in some research services, cf. current research computing software provision.