Red Hat Technologies for HPC on Wall St Overvie 5-Red Hat 4.9... · 2013-04-11 · Red Hat Technologies for HPC on Wall St Overview Jan Mark Holzer [email protected] Office of CTO Red

Red HatTechnologies

for HPC on Wall St Overview

Jan Mark [email protected] Office of CTORed Hat Inc.

William [email protected] Office of CTORed Hat Inc.

mailto:[email protected]

2 HPC on Wall St 4/2013

Agenda

● Red Hat Technology Overview

● Linux at the core of Enterprise IT● Enterprise Virtualization ● Emerging Technologies

● HPC & Low Latency Messaging


Open Source Communities as the Foundation


COMPLETE RED HAT PRODUCT PORTFOLIOIT

A

DM

IN S

DE

VE

LO

PE

RS

PaaS

Open Hybrid Cloud

Management

App Platforms / Messaging / Cache / Grid / SOA

PhysicalServers

Network & Storage Infrastructure

VirtualServers

CloudServers

Scale-Out, High-Performance Storage Software

RED HATENTERPRISE VIRTUALIZATION

RED HAT NETWORKSATELLITE

JBoss Operations Network

SY

SA

DM

INS



Red Hat Enterprise Linux Roadmap*

CY2010 CY2011 CY2012 CY2013 CY2014

RHEL 6

RHEL 5

RHEL 4

RHEL 3

Extended Life PhaseProduction 3Production 2Production 1

.4

.10

RHEL 7

.11

*All dates are approximate and subject to change


Red Hat Enterprise Linux 7:Performance Management

● Simplified instrumentation and tuning capabilities

● Ability to dynamically tune processor workloads through TUNA

● Faster tuning profile setup with predefined templates that can be customized

● Support for kernel parameter tuning

● Superior out-of-the-box performance leveraging advances in hardware


● Pool of resources (compute, storage, network)

● Nodes can run native or virtualized

● (Re)deployment/growth as needed and required by business


Red Hat Enterprise Linux 7:Manageability

● Easily configure, monitor, and manage high impact aspects of Red Hat Enterprise Linux through tools and interfaces

● Manage and automate RHEL instances through scripting and APIs

● Increase efficiency via a standards-based common management interface that abstracts complexity of underlying systems

● Support for Red Hat, third-party and custom management frameworks


● Scale out and scale up

● Consistent platform with RHEL

● Virtualized mission critical workloads with RHEV


● Load balancing

● Seamless workload migration


New Storage Infrastructure

RHEV 4.0 – Target H2 - 2013

- Remove need for SPM- Mixed storage types in same pool – iSCSI, FC, NFS

- Based on RHEL7 HypervisorRHEL 7.0

- Cluster-wide Service Level / QOS ManagementSLA Manager

- Third party plugin framework for RHEV-HRHEV-H Plugins

- Network Management ServiceQuantum

- Multilayer Virtual Switch Open vSwitch

RED HAT ENTERPRISE VIRTUALIZATION

RHEV 3.2 – Target H1-2013

- Based on RHEL 6.4 HypervisorRHEL 6.4

- SLA / QoS for CPU, memory and networkHost Level SLA

- Extension framework for RHEV pluginsUI Plugins

- Offload basic storage operations to array – clone, delete, etcArray Offload


Red Hat OpenStack Contribution Highlights

● nova-core, glance-core, quantum-core

● Stable tree maintenance

● Release and vulnerability management teams

● Packaging for Fedora / EPEL

● Upstream trunk chasing (SmokeStack)

● LDAP integration for Keystone

● Puppet modules for Fedora

● openstack-common


Red Hat will include the following in it's Red Hat OpenStack distribution

● All core OpenStack Foslom packages, including Quantum and Cinders

● Support for OpenvSwitch via userspace tools in Red Hat OpenStack + kernel support in RHEL 6.4

● Puppet modules for installing all services for OpenStack● A multi-node installer for small deployments● Reference architectures for large scale deployments● Bug-fixes and features selectively back-ported from Grizzly

Red Hat's OpenStack Offering


Red Hat OpenStack Preview

● Preview Program

● openstack.redhat.com


● Gluster as platform for big data

● Integration with Hadoop

● Build on demand big data storage clusters


● Take advantage of resource wherever they are

● Cater for security, SLA and location requirements

● Create seamless pool of resources


● Drives the need for management tooling

● Consistent tools independent of deployment and location


● Cloudforms/ManageIQ provides a common and open framework

● Execute administrative tasks across clouds

● Define “once” , deploy “multiple”

● Describe security, configuration and dependencies once


Red Hat Open Hybrid Cloud


Advanced Messaging


APIs and Protocols

Messaging API Messaging API

Application Application

Protocol


APIs and Protocols

REST Web Server/CGI


HTTP


APIs and Protocols

JMS JMS


<proprietary>


Sender

Receivermytopic

Topic - mytopic

Receivermytopic

Sender

Receiverworkq1

Queue - workq1

Receiverworkq1

msg1,msg2,msg3,msg4

msg1,msg2,msg3,msg4

msg1,msg2,msg3,msg4

msg1,msg2,msg3,msg4

msg1,msg3

msg2,msg4


Broker

Publisher

PublisherW/ Reply Q

Subscriber

Subscriber

Subscriber

Traditional MOM

Message Queues

PersistentStore


APIs and Protocols

JMS JMS


AMQP


APIs and Protocols

JMS Qpid .NET API


AMQP


Router or Broker

ClientPublisher

ClientSubscriber

Message To Address//hostname/topic


ClientPublisher

ClientSubscriber


AMQP 1.0 Supports peer-to-peer addressing


ClientPublisher


ClientSubscriber

AMQP Network of Smart Routers


AMQP Routing

AMQP Network Architecture

DNS / Geo Load Balancing

Server Load Balancer / VIP

Proxy/Gateway Proxy/Gateway

Broker Broker

ManagementMonitoring

ProvisioningMulti-tennancy

Clients / Services

Back Office


Proton: An AMQP Protocol Engine

AMQP

Application

TransportProton

Application

Transport

Driver

API

Designed to promote adoption of AMQP

Integrates below the API

Not specific to any execution/threading model

Not specific to any transport API (sockets, etc.)

Implementations: C, Java, JavaScript (planned)


AMQP and Vlans across the Clouds

Qpid Dispatch Router

Red HatLab

Machine

Mylaptop LOGICAL

Public Cloud



ClientPublisher

NYC

ClientSubscriber

London

AMQP Network of Smart Routers in a Cloud

ClientSubscriberHong Kong

Private or Public Cloud


MOMBroker

Back Office Apps

JEE Container

JEEApps

JEE Container

JEEAppsWS

Qpid/Proton Lib

ClientPublisher

ClientSubscriber

Intranet/Internet

The AMQP Network


Challenges For AMQP

Finalize addressing – for truly interoperable and portable end points and intermediaries

Some standardized configuration for broker artifacts. ● AMQP 0-10 -> AMQP 1.0+ based assets

Shifting the focus from MOM based


Red Hat Enterprise Linux 7: Networking

Team Device● Mechanism for bonding multiple network devices into a

simple logical interface at the data link (L2)● Alternative to the existing Linux Bonding driver

Ethernet Data Center Bridging (DCB) support● Priority-based Flow Control (PFC) [802.Qbb], Enhanced

Transmission Selection (ETS) [802.1Qaz], and Congestion Notifications (QCN) [802.1Qau]

Network Manager expanded to support more network configurations

Additional support for Virtual Ethernet Port Addressing (VEPA), Network Namespaces (netns), IEEE 1588 Precision Time Protocol (PTP), and Stream Control Transmission Protocol (SCTP)


Red Hat Enterprise Linux 7: Linux Containers and RHEL guest Easily partitions a system into multiple secure compartments

● Simple, low overhead method of deploying multi-tenant applications

Simplified installation process for RHEL as a guest on supported hypervisors

Ability to handle 3D graphics and audio for virtual desktop applications

IP Virtual Server (IPVS)● Support for Linux Containers (LXC) allowing process

virtualization with network namespace transport-layer load balancing


Red Hat Enterprise Linux 7: Virtualization and cloud NUMA capabilities in KVM for better virtualization performance

Strengthening security isolation and fine-grained capabilities in the KVM stack

Smaller virtualization overhead such that SSD-based I/O device and SATA drives overhead is less than 10%


Red Hat Technology Differentiation

OpenStack is an exciting and fast moving project with a very vibrant community. We expect it will be for IaaS what RHEL is for Linux

Red Hat is a founding member of the OpenStack foundation

Committed significant financial and development resources

Numerous Red Hat engineers part of key OpenStack projects

Red Hat is now the #2 contributor to OpenStack

Working with partners and customers to make OpenStack a stable, robust and scalable enterprise solution

Plans in place to offer a Red Hat based OpenStack distribution


Why is Red Hat involved with OpenStack● OpenStack is a growing, exciting and vibrant open-source project. Red Hat is a

leader in open source, cloud and virtualization technologies. OpenStack is a natural fit and strategic to our efforts in advancing open source cloud technologies

Red Hat's achievements to date● Established ourselves as leaders of OpenStack community

● Red Hat engineers began quietly contributing to OpenStack mid-2011

● Red Hat has grown into a significant OpenStack engineering team● Red Hat was the #2 overall code contributor to the Folsom release

and the #3 contributor to Essex● Red Hat engineers are highly influential in the project's technical

meritocracy● Established a long-term governance structure for OpenStack

● Red Hat was a key driver in establishing the OpenStack Foundation to “protect, promote and empower” the project

● Red Hat is a Platinum member of the OpenStack Foundation

Red Hat's OpenStack History


Red Hat's achievements to date (continued)● Released preview versions of Red Hat’s commercial OpenStack distribution

– Essex and Folsom versions available, both based RHEL 6.3– Freely available for download from Red Hat’s Customer Portal– Preview is a non-production, unsupported version aimed at developers and

integrators for testing OpenStack on RHEL● Introduced an invitation-only high-touch, supported preview of Red Hat

OpenStack● Created a Lighthouse Program to enable paid engagement supported by Red

Hat’s OpenStack engineering team

What is Red Hat’s current focus● Creating a commercial, fully-supported OpenStack product

– Paid subscription– Based on OpenStack Folsom with RHEL 6.4– Available early 2013

Red Hat's OpenStack History


Developers with the most patches:

termie 238 (6.8%)Gabriel Hurley 207 (5.9%)Brian Waldon 195 (5.6%)Johannes Erdfelt 146 (4.2%)Vishvananda Ishaya 116 (3.3%)Dolph Mathews 98 (2.8%)Dan Prince 84 (2.4%)Ziad Sawalha 80 (2.3%)Jason Kölker 77 (2.2%)Mark McLoughlin 73 (2.1%)Jake Dahn 73 (2.1%)Rick Harris 71 (2.0%)Alex Meade 70 (2.0%)Trey Morris 62 (1.8%)Joe Heck 58 (1.7%)Chris Behrens 52 (1.5%)Russell Bryant 50 (1.4%)Eoghan Glynn 50 (1.4%)Joe Gordon 47 (1.4%)Jesse Andrews 46 (1.3%)

Covers 54.380925% of changesets

Top patch contributors by employer:

Rackspace 1921 (55.2%)Nebula 348 (10.0%)Red Hat 275 (7.9%)HP 101 (2.9%)Canonical 92 (2.6%)Citrix 83 (2.4%)Nicira 78 (2.2%)Cloudscaling 47 (1.4%)Delta Electronics 44 (1.3%)eNovance 41 (1.2%)SINA 38 (1.1%)Cisco Systems 25 (0.7%)Nimbis Services 22 (0.6%)[email protected] 20 (0.6%)FathomDB 20 (0.6%)Everbread 19 (0.5%)Midokura 19 (0.5%)Wikimedia Foundation 18 (0.5%)[email protected] 14 (0.4%)[email protected] 14 (0.4%)

Covers 93.047975% of changesets


Red Hat's product offering beyond what's available upstream

● RHOS will be to OpenStack what RHEL is to Linux● In many ways, the RHOS and RHEL value propositions are identical● Products. Solutions. Support. Training● Stable, Secure, Scalable, Certified● Product evolution will follow a similar path to RHEL● Customers will influence that evolution as they do for RHEL● Customers benefit from our insight into upstream roadmap● Customers have the ability to leverage Red Hat's upstream

leadership● “Upstream first” development mentality● Killer combination of our Linux, KVM and OpenStack leadership

Red Hat OpenStack and the Community Version


Red Hat leadership in the upstream community● Two members on the Technical Committee● Project Technical Lead for Oslo● Members of Nova, Quantum, Glance, Keystone, Oslo, Heat and

Ceilometer “core” teams● Leading the stable tree maintenance effort● Members vulnerability management and release teams● Upstream continuous integration via Smokestack● Overall, Red Hat Engineers were #3 contributor to Essex and #2 to

Folsom

Red Hat's Upstream Leadership Commitment


Examples of Red Hat's upstream engineering work● Packaging for Fedora and RHEL● LDAP integration● PKI Tokens based authentication● Configuration rationalization● Generalization of messaging infrastructure to support Apache Qpid● Improved KVM performance● Improved file injection security via libguestfs integration● Groundwork for supporting rolling upgrades● Improved Quantum/Nova integration● Enterprise networking support in Quantum● Development of the Heat orchestration service

Red Hat's Upstream Areas of Focus


Focus areas for Grizzly and beyond● Rolling upgrades● Provisioning, management and monitoring tools● Orchestration (ala CloudFormations)● Monitoring and auto-scaling (ala CloudWatch)● Metering and billing● Improved PostgreSQL support● Paying down technical debt (e.g. Oslo)● GlusterFS integration● Moving from Nova networking to Quantum● Quantum support for complex enterprise environments● Cells work for scaling● Better integration of guest image creation tools like Oz● Minimal image based compute node deployments (ala ovirt-node)● Ongoing OpenShift, CloudForms and RHEV integration improvements

Red Hat's Future Focus Areas

Documents

Red Hat Technologies for HPC on Wall St Overvie 5-Red Hat 4.9... · 2013-04-11 · Red Hat Technologies for HPC on Wall St Overview Jan Mark Holzer [email protected] Office of CTO Red