PayPal’s Private Cloud @ Scale · strive to increase our relevance for consumers, merchants, friends and family to access and move their money anywhere in the world, anytime, on

©2015 PayPal Inc. Confidential and proprietary.

PayPal’s Private Cloud @ ScaleJigar Desai• @jigardesai • March 17 2016


Table of contents

2

Introduction to PayPal

Current Cloud Stack

SDN Journey

Future Cloud Stack

Q&A


Introduction to PayPalA Technology Perspective

3


About PayPal

4

PayPal is a leading technology platform company that enables digital and mobile payments on behalf of consumers and merchants worldwide. We put our customers at the center of everything we do. We strive to increase our relevance for consumers, merchants, friends and family to access and move their money anywhere in the world, anytime, on any platform and through any device.


Architecture @ PayPal

5

Robust Infrastructure, Reusable Platforms, Payments Operating System, Delightful ExperiencesC

usto

mer

Fac

ing

Infra

stru

ctur

e

Analytics Platform

HadoopTeradata FPTIBI

Technology Platform

Framework Data Access Tools MediaMessagingSOA CrawlerCache

Infrastructure & Operations Layer

Power Data Center Hardware Network Database OperationsToolsCloudStorage Monitoring

Experience

Payments Operating System

LoginIdentity EmailCart CouponsPayment Ship SupportRisk InvoiceWallet POS Notify Merchant


Cloud@PayPal

6

© 2015 PayPal Inc. All rights reserved. Confidential and proprietary.

x 100K Cores

x 10 PB Storage

OpenStack Cloud

> 10K Physical Servers

x 10K VMs

x 1000engineers

x 10M Lines

x 1000 Releases/year

> 1000 Services

x 1000 Builds/day

x 1000Deploy/day

Key StatisticsDeveloper Statistics

9 Availability Zones

3 Regions

© 2015 PayPal Inc. All rights reserved. Confidential and proprietary.

Steps to Production 25 Manual Steps Automated

Time to Market

QA Deployment

Build Time

Prod Deployment

Infrastructure

2 years back

Now

8 Weeks

5 Hours

2.5 Hours

4 Hours

1 Month

1 Day

30 Minutes

5 Minutes

< 30 Minutes

30 Minutes

Impact On Life Of A Developer


The Current Cloud Stack

9

Datacenters (Geographic regions, Availability Zones)

OpenStack Cloud (Infrastructure as a Service)

Platform as a Service

Web/Mid Tier Applications

Common Infrastructure

Common Platforms and Services

• 100% of web and mid tier applications on OpenStack

• Proprietary PaaS on top of OpenStack to automate E2E application development life cycle

• Support for polyglot applications

• Test and production environments available on-demand


Cloud SDN Journey

10


Why SDN at PayPal?

11

•Ability to logically isolate cloud resources (compute, storage, network) for different business use cases needing different security policies while co-existing on shared infrastructure•Solve compute capacity islands by moving computes between security zones as needed•Programmatic APIs to enable both cloud users and operator reducing operational overhead•Use cases:• External zone hosting beta apps reachable from internet but separated from

other zones•Developer zone hosting all developer tools with no direct access from internet

but available from corp

Global Platform & Infrastructure

Neutron Core

Neutron API

SDN PluginLBaaSPlugin

LB SpecificDriver(s)

LB Agent1 … n

LB Pair-1LB Pair-2

LB Pair - ..LB Pair-n

LB Device APIs

SDN Controller-1SDN Controller-2

SDN Controller-3SDN Controller-4

SDN Controller-n

LB for Controllers

SDN APIs

Overlay Hypervisors 1….n

VM1 VM2

Open vSwitch

OVS DB

SDN Gateway

OVS DB

Overlay Networkper VPC

VM1 VM2

Open vSwitch

OVS DB

Bridge Hypervisors 1….n

Bridged Network per VPC

•PayPal deploys multiple VPCs (Virtual Private Cloud)s in every AZ •Every overlay VPC is running on a

separate routed network and firewalled off

•Bridged VPCs are deployed using VRF

(Virtual Routing and Forwarding)

•SDN controllers are horizontally scaled

Our SDN Architecture


SDN Challenges

13

• Off the shelf solutions are not battle-tested for scale well• Example: Control plane doesn’t scale well beyond 2500 hypervisors

• Bit early in software maturity curve• Example: Latency issues with early OVS implementation• Security groups performance issues with multiple rules. Later solved with

“megaflows” feature• Strong architectural principles don’t always mean strong implementation• Initial SDN implementation didn’t have full separation between control plane

and data plane. Control plane outages resulted in site outages• Security concerns specially around centralized control plane


The Future

14


The Future Cloud Stack

15

Datacenters (Geographic regions, Availability Zones)

OpenStack Cloud (Infrastructure as a Service)

Web and Mid Tier Applications

Common Infrastructure

Common Platforms and Services

• New PaaS is being built using Mesos and Docker on top of OpenStack

• Goal is to achieve industry leading datacenter efficiency and utilization

• High application resiliency against infrastructure failures

• Hybrid cloud POC • PaaS support for stateful

applications

Public Clouds


Container Networking

16


Container Networking Needs

17

Needs:• Containers as first class citizens of the network• IP per container or per pod• Container IP must be a private address routable within the AZ

• Low latency container to container communication• Support up to 100 containers per host • Distributed SLB• IP Mobility within the AZ for stateful apps• Distributed firewall

Exploratory Options:Non-overlay: Ipvlan, MacvlanOverlay: Vxlan, MPLS

Questions?Jdesai at paypal dot com

Documents

PayPal’s Private Cloud @ Scale · strive to increase our relevance for consumers, merchants, friends and family to access and move their money anywhere in the world, anytime, on