An evening with - files.meetup.comfiles.meetup.com/10751222/Big Data Manchester.pdfResultant Hadoop pain points? 1. Struggle with the storage and movement of data throughout a big

An evening with

Chris Reid

Partner

[email protected]

May 2016

Pointgreen

PointGreen Business 2

Based in Grenoble

14 folks around US and EMEA

Experience is Technology Start-ups

Focus on Cloud, Virtualisation and IoT

We build the References and the Eco-system to ramp-up the Business

https://tuleap.net/3

Open Source Software Forge


S3 Cloud storage solutionsObject Storage


Apache Cloudstack

Iaas Model


Compute

XenServer VMwareKVM

Hyper-VOracle VM Bare metal

Hypervisor

Storage

Local Disk iSCSI NFSFiber Channel Swift

Block & Object

Network

Network Type Isolation Load balancerFirewall VPN

Network & Network Services

Primary Storage Secondary Storage


VM or LxC

SD WAN

Application Lifecyling

Virtual Visualisation

Application Defined Networks

Community of Interest

Code Migration and Micro Services

Messaging Center

HLR Objects DBMS

ObjectPresence

Location & Status

Rating & Billing

SMS USSD IP

BSS

Alerts

Objects Provisioning

Core Engine

Cache &Buffer

IMS as a Service

IoT Orchestration

VoIP Fraud/BillingGeneric Self Service Portal

Eco-System exampleHeathcare

Plexxi

Application Defined Networking


Virtualised vision (IaaS)

10

Compute

XenServer VMwareKVM

Hyper-VOracle VM Bare metal

Hypervisor

Storage

Local Disk iSCSI NFSFiber

ChannelSwift

Block & Object

NetworkNetwork

TypeIsolation

Load balancer

Firewall VPN

Network & Network Services

Primary Storage Secondary Storage

Physical Reality

Router

L3 Core Switch

Access Layer 2Switches

………… …

Availability Zone

Servers

CloudStack MgmtServer Cluster

Secondary Storage

Pod 1 Pod 2 Pod 3 Pod N

MySQL

Load Balancer

Operations Admin and Cloud API

Users

Resultant Hadoop pain points?

1. Struggle with the storage and movement of data throughout a big data cluster Networks

Cause workload placement restrictions

Avoid Cross rack communications

2. Struggle with sequencing, need to avoid spikes

Throw bandwidth at it?

3. Struggle with the Application Variable ‘fest’

Too many variables for a single network to handle?

4. Struggle with Agile IP storage

Separate networks?12

View, dynamically manage and automate from the Application stand point not the Network.

Dedicate Bandwidth, Latency and a back-up route to run the service efficiently.

Sub-priorities around HDFS, YARN, Ambari and ZooKeeper.

Integrate with existing Security and Ticketing systems

Visualise, per flow, who, what and where


What if you could…

Multiple Application service guarantees

Accommodate Vmware, OpenStack, Nutanix , VDI etc

SDK for bespoke offerings

Service chain on a per application basis.


What if you could…

Plexxi model and the Integration ecosystem

Ple

xx

i S

wit

chFa

bri

cs

Ple

xx

i C

on

tro

l

Data Plane

Algorithmic Topology Computation (Fitting)

Event Exchange Policy Automation

Infrastructure Workload

Ple

xx

i C

on

ne

ct

Commodity Switching(Brite Box)

Commodity Switching + Integrated Optics + L1 +

WDM

vSwitching / Specialized HW Add-ons

Traffic Control Policies (Flow / Workload)Workload Data Model (Affinities)

Open Extensible Platform

DevOps & ChatOps

VD

I

Control Plane

Integration Plane

1

5

PLEXXI Switch

16

Agile, Scale-Out Application Network Fabrics for Cloud, Big Data and Data Centers delivered as an integrated product

Plexxi Switch

Single tier, scale-out networking solution for

distributed application environments

• Dynamic, application-centric traffic mgt.

• Broadcom Ethernet Switching

• Innovation

• LightRail® Photonic Interconnect

• Layer 1 cross-connect

Controller architecture for automation and

orchestration of network fabrics

• Application-centric network visualization• Application-based Fitting and Rendering

• Innovation:

• Dynamic Fitting Engine (DFE)

• Single point of control

Plexxi Control

Scale-out application, data source and

infrastructure integration

• Group-based automation and system level control

• Unified application and network workflows

• Innovation:

• Network-level API abstraction

• Plexxi Services Engine

Plexxi Connect

Migration Path – Hyperconverged (Customer Example)

17

WAN EdgeLayer

Hyperconverged Core/Access

Layer

L2

L3

BeforeAfter

(Actual Customer Example)

Building data center networks simply better…

18

High-density Bare Metal Access

• Bare Metal (disaggregated) model

• Easy to support new models based on

customer connectivity needs and density

• High value integration to compute/storage

cloud and rack management software

(automation, operations)

18

Logical Fabric Topology

Comparing Fabric Latency

19

Hierarchical network

Fewer fabric hops, lower end-to-end latency

1 2

Plexxi LightRail™

1

+Forwarding delay+Serialization delay

*

+Forwarding delay*+Serialization delay+Congestion/queuing delay




*Note: forwarding delay in spine layer switches up to 2-3x when using “spline” or chassis switches

Plexxi LightRail™ photonic fabric allows for greater 1-hop reach and for custom-engineered L1 paths

Allocate bandwidth and Latency

20

Fabric bandwidth allocation

Deliver SLAs with bandwidth and latency control

Lower latency through direct paths

Direct optical paths

single-hop, direct path

AVAILABILITYCONGESTIONCONSISTENCYSCALABILITYMULTI-TENANT

Distribute traffic to avoid congestion

21

Automatic fabric load-balancing

…add more unequal-cost paths as bandwidth is needed

…algorithmically optimized to reduce congestion points

Plexxi Topology Controller computes paths

based on fabric topography and traffic models

Traffic is load-balanced over all available pathsController-based fitting distributes traffic across all paths

(distributed forwarding model reduces internal congestion)

AVAILABILITYCONGESTIONCONSISTENCYSCALABILITYMULTI-TENANT

Automatic Fabric Load Balancing

PLEXXI Control

22


Plexxi Switch





• Innovation






• Innovation:



Plexxi Control





• Innovation:



Plexxi Connect

Plexxi Control – Distributed Architecture

23

Designed for Continuous Operation

Designed for scale and self-healing‒ Fabric interconnects are self-discovered

‒ Continuous monitoring of all fabric interconnect links

‒ 25ns Latency

Architected for resiliency‒ Central controller never active in the data plane

‒ Fast, local repair driven by switches

‒ No Controller / Switch fate sharing

Plexxi fabric keeps running when disconnected from the Plexxi Controller

SSL-protected IP control protocol between controller and switches

Internal fabric control protocol between switches

Out-of-band Ethernet management network

Co-Controller

Co-Controller

Co-Controller

Co-Controller

Plexxi Control Engine: unprecedented network control

2

• Application-centric network visualization and

management

Matches Hadoop workloads to available resources

• Policy-based network fitting and dynamic topology

rendering

Meets priority application workload SLAs and efficiently

distributes traffic

• Application workload policy model and APIs

Makes network resources relevant to business priorities

• Stock Linux Distribution for Switch OS

Integrates 3rd party agents easily on Linux network OS

24

PLEXXI Connect

25


Plexxi Switch





• Innovation






• Innovation:



Plexxi Control





• Innovation:



Plexxi Connect

Api driven networking through open interfaces

2

6

Plexxi Connect Open Integration Platform

VD

IInfrastructure Workload DevOps & ChatOpsIntegration Packs & Super Packs

OpenStack

Simplivity

Nutanix

vCenter vRealize

vSphere Web Client

VSAN

Informatica

MapR

Cloudera

Cloud & Converged Infrastructure

Big Data & Analytics

Nimble

VSAN

ScaleIO

Storage

Embotics

VMTurbo

Elastic

Ubersmith

SFDC

Egenera

Puppet

Ansible

Jira

Visibility Ops & Security

Portal & NOC

DevOps & ChatOps

Fabric Control & Management, per Workload or per Flow

Workflow Fabric (Logic and Interfaces)

Control

Plane

Workload Responsive, Multi-purpose Switching Fabric

Compute/Storage HostsLegacy Networks External Networks

Data

Plane

Plexxi

Control

Plexxi

Switches

Integration

Plane

Plexxi

Connect

PLEXXI CONTROL

Flows

From Plexxi Connect - Find out what nodes HDFS services are running

Find out which switches the nodes are connected to; For this Plexxi Connect probes Ambari to get the endpoint node ip address

PLEXXI CONNECT

PlexxiUI

Dynamically push down UDAT information (path and qualifiers) via Plexxi view webapp to Plexxi Control. UDAT (Fitting) info includes src/dst ip & tcp port for qualifiers. Path information is provided to Control via Connect

PLEXXI CONTROL AND AMBARI INTEGRATION

Flowstats information received via Plexxi View

AMBARI SERVER

PROVISION | MANAGE| MONITOR

AMBARI

REST

API

Compute &

storage. . .

.

.

.

.

.

.

.

.

. . .

. . .

. .Compute

& storage

AMBARI STACKS

OPERATE EXTEND

Services

HDFS

MAP REDUCEAMBARI MANAGER

YARN

VIEW

PLEXXI CORE

Job-SpecificAffinity Group

PLEXXI FABRIC

Plexxi Solution – Hadoop Affinities

Smart Optics giving an Aggregated view of your network Reachability Capacity Latency Workflow

Application Requirements and Orchestration Simple integration with existing Systems Workflow specific network requirements Sensors and Effectors Open and agile

Fit the Application directly onto the Optics Zero Touch Traffic Control Guaranteed Capacity/Latency/Backup between Logical groups and Application phases. Dynamic feedback loop Time-of-day networking

28

PLEXXI Big Data Fabric

29

CapEx OpEx

Dynamic Scale

• Flatter network / fewer devices• Simplified interconnect cabling and

included optics• Controller-based fitting maximizes use

of intra-DC bandwidth

• Easier to manage: fewer devices and automated setup/discovery

• Lower power/space/cooling costs• Easier to integrate into external

systems

• Easier policy provisioning via Affinity network services abstraction model

• Responsive, programmable fabric interconnect

• Cross-infrastructure integration• DevOps-ready

• High-capacity, low-latency interconnect fabric

• Controller-based fitting dynamically matches capacity to application requirements

• Scale out without adding additional network tiers

http://www.rivermeadow.com/ - transitional aid for application migration

http://www.corenttech.com/ - suitability of apps for Cloud / SaaS migration


Cloud Migration

http://www.rivermeadow.com/

http://www.corenttech.com/

Apache BrooklynApplication Life cycling

(thanks to Cloudsoft)


Agility comes from Policy Management

Page: 32

Business & Operational policies

Ap

plicatio

ns

Infr

astr

uct

ure

s Brooklyn

Cost Balancing

Disaster Recover / Elasticity / Scalability

The real value of Agility

Page: 33

Use-Cases

07/07/2016 Page: 34

Telecom Migration towards NFV

Brooklyn for getting the best from NFV

Core: Use dynamic policing

for elasticity

Demand/ Response

framework

Edge: Allow multiple

deployment scenarios &

flexibility

vCPE case

Cloud Services to Enterprises

Best use of your existing

Applications

Migration through hybrid architectures

Compose as you want & need

Improve your Apps on 1+ axis

(responsiveness, business

continuity, availability)

Apps unleashed

Enterprise IT reshaping

Brooklyn for “blueprinting” your

new Applications

Address immediate issues

& simplify the App’s control

layer

From scripts towards ABC (Application

BluePrint Control)

Increase performance and

capacities for business critical

Applications

Mainframe migration


Application Blueprint

Ready to deploy application framework in a scalable topology through

a lifecycle enforced bypolicies

+ +

Hosted Environment

Version Control Public Private Hybrid

Foundation


What Blueprints are: a combination of

Topology + Policies

Where

Topology

Entity wiring, groupings and management hierarchy

Policy

Governs an application’s behaviour along time using sensors & actuators

Leveraging

Management elements

Introducing Blueprints (1/3)



Autonomic Management is achieved througha hierarchical wiring of elements


Policies illustrations

Failure Detection

Service Restarter

Service Replacer

Elasticity

General

Various specialized for NoSQL:

Couchbase, Cassandra, MongoDB, Riak.

Load Balancing

Wide Area

FollowtheSun

FollowtheMoon



Blueprint Composition


https://brooklyn.apache.org/

http://www.cloudsoft.io/

https://jclouds.apache.org/ - Multicloud

https://www.oasis-open.org/ - Security, IoT

http://www.opengamma.com/ - Risk Analysis


Information

https://brooklyn.apache.org/

http://www.cloudsoft.io/

https://jclouds.apache.org/

https://www.oasis-open.org/

http://www.opengamma.com/

Don’t Mention Clocker!(Docker in the Cloud)



https://www.jfrog.com/open-source/

The world’s most advanced repository manager

https://www.projectcalico.org/

Which morphed into

https://www.tigera.io/ the home of Canal utilise Calico and Flannel

https://www.jfrog.com/open-source/

https://www.projectcalico.org/

https://www.tigera.io/

Version 4.1 Beta - Basic Functions User Guide

www.hyperglance.com [email protected] Follow us @hyperglance

MonitorMap Manage

http://www.hyperglance.com/

mailto:[email protected]?subject=Hypergance Interest - Tell Me More

https://twitter.com/hyperglance

The Problem

New Agile deployments mean:

(growing) Virtualisation

Service Migration

Use of Public, Private & Hybrid Cloud architectures

Evolution towards Elastic, dynamic resource allocation

Produces:

Many Virtual Machines & Docker containers

Distributed Application and Network resources

Lack of real time visibility, traceability

The Solution

Hyperglance

Visualisation of all Virtual resources (nodes, networks, applications)

Across private and public clouds

Real time mapping, monitoring, management and control

Produces

A simple 3D model of your virtual assets

Interconnected across your Hybrid Cloud

A compliance snapshot of your virtual World

Relationship visualisation

4

2

3

5

8

1

10

6 7

119

To Expand / Contract Groups

2 3

Expanded Group View

1

3a

2

4

3b

Documents

An evening with - files.meetup.comfiles.meetup.com/10751222/Big Data Manchester.pdfResultant Hadoop pain points? 1. Struggle with the storage and movement of data throughout a big