View
1
Download
0
Category
Preview:
Citation preview
©2015 PayPal Inc. Confidential and proprietary.
PayPal’s Private Cloud @ ScaleJigar Desai• @jigardesai • March 17 2016
©2015 PayPal Inc. Confidential and proprietary.
Table of contents
2
Introduction to PayPal
Current Cloud Stack
SDN Journey
Future Cloud Stack
Q&A
©2015 PayPal Inc. Confidential and proprietary.
Introduction to PayPalA Technology Perspective
3
©2015 PayPal Inc. Confidential and proprietary.
About PayPal
4
PayPal is a leading technology platform company that enables digital and mobile payments on behalf of consumers and merchants worldwide. We put our customers at the center of everything we do. We strive to increase our relevance for consumers, merchants, friends and family to access and move their money anywhere in the world, anytime, on any platform and through any device.
©2015 PayPal Inc. Confidential and proprietary.
Architecture @ PayPal
5
Robust Infrastructure, Reusable Platforms, Payments Operating System, Delightful ExperiencesC
usto
mer
Fac
ing
Infra
stru
ctur
e
Analytics Platform
HadoopTeradata FPTIBI
Technology Platform
Framework Data Access Tools MediaMessagingSOA CrawlerCache
Infrastructure & Operations Layer
Power Data Center Hardware Network Database OperationsToolsCloudStorage Monitoring
Experience
Payments Operating System
LoginIdentity EmailCart CouponsPayment Ship SupportRisk InvoiceWallet POS Notify Merchant
©2015 PayPal Inc. Confidential and proprietary.
Cloud@PayPal
6
© 2015 PayPal Inc. All rights reserved. Confidential and proprietary.
x 100K Cores
x 10 PB Storage
OpenStack Cloud
> 10K Physical Servers
x 10K VMs
x 1000engineers
x 10M Lines
x 1000 Releases/year
> 1000 Services
x 1000 Builds/day
x 1000Deploy/day
Key StatisticsDeveloper Statistics
9 Availability Zones
3 Regions
© 2015 PayPal Inc. All rights reserved. Confidential and proprietary.
Steps to Production 25 Manual Steps Automated
Time to Market
QA Deployment
Build Time
Prod Deployment
Infrastructure
2 years back
Now
8 Weeks
5 Hours
2.5 Hours
4 Hours
1 Month
1 Day
30 Minutes
5 Minutes
< 30 Minutes
30 Minutes
Impact On Life Of A Developer
©2015 PayPal Inc. Confidential and proprietary.
The Current Cloud Stack
9
Datacenters (Geographic regions, Availability Zones)
OpenStack Cloud (Infrastructure as a Service)
Platform as a Service
Web/Mid Tier Applications
Common Infrastructure
Common Platforms and Services
• 100% of web and mid tier applications on OpenStack
• Proprietary PaaS on top of OpenStack to automate E2E application development life cycle
• Support for polyglot applications
• Test and production environments available on-demand
©2015 PayPal Inc. Confidential and proprietary.
Cloud SDN Journey
10
©2015 PayPal Inc. Confidential and proprietary.
Why SDN at PayPal?
11
•Ability to logically isolate cloud resources (compute, storage, network) for different business use cases needing different security policies while co-existing on shared infrastructure•Solve compute capacity islands by moving computes between security zones as needed•Programmatic APIs to enable both cloud users and operator reducing operational overhead•Use cases:• External zone hosting beta apps reachable from internet but separated from
other zones•Developer zone hosting all developer tools with no direct access from internet
but available from corp
Global Platform & Infrastructure
Neutron Core
Neutron API
SDN PluginLBaaSPlugin
LB SpecificDriver(s)
LB Agent1 … n
LB Pair-1LB Pair-2
LB Pair - ..LB Pair-n
LB Device APIs
SDN Controller-1SDN Controller-2
SDN Controller-3SDN Controller-4
SDN Controller-n
LB for Controllers
SDN APIs
Overlay Hypervisors 1….n
VM1 VM2
Open vSwitch
OVS DB
SDN Gateway
OVS DB
Overlay Networkper VPC
VM1 VM2
Open vSwitch
OVS DB
Bridge Hypervisors 1….n
Bridged Network per VPC
•PayPal deploys multiple VPCs (Virtual Private Cloud)s in every AZ •Every overlay VPC is running on a
separate routed network and firewalled off
•Bridged VPCs are deployed using VRF
(Virtual Routing and Forwarding)
•SDN controllers are horizontally scaled
Our SDN Architecture
©2015 PayPal Inc. Confidential and proprietary.
SDN Challenges
13
• Off the shelf solutions are not battle-tested for scale well• Example: Control plane doesn’t scale well beyond 2500 hypervisors
• Bit early in software maturity curve• Example: Latency issues with early OVS implementation• Security groups performance issues with multiple rules. Later solved with
“megaflows” feature• Strong architectural principles don’t always mean strong implementation• Initial SDN implementation didn’t have full separation between control plane
and data plane. Control plane outages resulted in site outages• Security concerns specially around centralized control plane
©2015 PayPal Inc. Confidential and proprietary.
The Future
14
©2015 PayPal Inc. Confidential and proprietary.
The Future Cloud Stack
15
Datacenters (Geographic regions, Availability Zones)
OpenStack Cloud (Infrastructure as a Service)
Web and Mid Tier Applications
Common Infrastructure
Common Platforms and Services
• New PaaS is being built using Mesos and Docker on top of OpenStack
• Goal is to achieve industry leading datacenter efficiency and utilization
• High application resiliency against infrastructure failures
• Hybrid cloud POC • PaaS support for stateful
applications
Public Clouds
©2015 PayPal Inc. Confidential and proprietary.
Container Networking
16
©2015 PayPal Inc. Confidential and proprietary.
Container Networking Needs
17
Needs:• Containers as first class citizens of the network• IP per container or per pod• Container IP must be a private address routable within the AZ
• Low latency container to container communication• Support up to 100 containers per host • Distributed SLB• IP Mobility within the AZ for stateful apps• Distributed firewall
Exploratory Options:Non-overlay: Ipvlan, MacvlanOverlay: Vxlan, MPLS
Questions?Jdesai at paypal dot com
Recommended