Greenplum on Pivotal Container Service · Greenplum for Containers Greenplum on Pivotal Container...

© Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0

gpcloud@pivotal.ioPostgres Conf, April 2018

Speaker: Goutam Tadi

Greenplum for ContainersGreenplum on Pivotal Container Service

1

Cover w/ Image

Agenda

■ Greenplum Architecture

■ Greenplum Data Platform

■ Kubernetes on PCF

■ Greenplum on Kubernetes

■ Demo

■ Q+A

2

MASSIVELY PARALLEL DATA WAREHOUSE

WHAT IS GREENPLUM?

3

Greenplum = Massively Parallel Postgres for Analytics

StandbyMaster

…

MasterHost

SQL

Interconnect

Segment Host

Node1

Segment Host

Node2

Segment Host

Node3

Segment Host

NodeN

LocalStorage

OtherRDBMSes

SparkGemFireCloudObject

StorageHDFS KafkaETL

SpringCloud

Data Flow

Master ServersQuery planning and dispatch

Segment ServersQuery processing and data storage

Interconnect

External Sources & PipelinesParallel loading and streaming

Greenplum Data Platform

ANALYTICALAPPLICATIONS

NATIVE INTERFACES

PIVOTALGREENPLUMPLATFORM

MULTI-STRUCTURED DATA

SOURCES &PIPELINES

Structured Data

JDBC, ODBC

SQL

ANSI SQL

FLEXIBLE DEPLOYMENT

LocalStorage

OtherRDBMSes

SparkGemFireCloudObject

StorageHDFS

JSON, Apache AVRO, Apache Parquet and XML

Teradata SQL

Other DB SQL

Apache MADlib

ML/Statistics/Graph

Python. R,Java, Perl, C

Programmatic

Apache SOLR

Text

PostGIS

GeoSpatial

Custom Apps BI / Reporting Machine Learning AI

On-Premises

NEXT GENERATION

DATA PLATFORM

KafkaETLSpringCloud

Data Flow

MassivelyParallel(MPP)

PostgreSQLKernel

PetabyteScale

Loading

QueryOptimizer(GPORCA)

WorkloadManager

PolymorphicStorage

Command Center

SQL Compatibility

(Hyper-Q)

DS AnalystsIT Dev

PublicClouds

PrivateClouds

FullyManaged

Clouds

5

Faster Deployments… How?

6

Have youHave you...● Ran out of disk space ?

● Been able to provision more than 100 postgres instances in few minutes?

● Faced issues in recovering failures ?

● Faced issues in expanding the database?

7

experienced these before with any database?

RELIABLY DEPLOY AND RUN

CONTAINERIZED WORKLOADS.

WHAT IS PKS?

8

Kubernetes on Pivotal Cloud Foundry

Continuously deliver any app to every major private and public cloud with a single platform.

9

10

Faster Deployments… How?

Greenplum Data Platform + PKS

11

Kubernetes 101

Kubernetes Master

12

Kubernetes Master

kubelet kube-proxy docker

Node

kubelet kube-proxy docker

Node

13

Kubernetes 101

Kubernetes Master

Pod

kubelet kube-proxy docker

Node

Pod

kubelet kube-proxy docker

Node

14

Kubernetes 101

Kubernetes Master

Pod

kubelet kube-proxy docker

Node

Pod

kubelet kube-proxy docker

Node

15

Kubernetes 101

Kubectl

Kubernetes Master

Pod

kubelet kube-proxy docker

Node

Pod

kubelet kube-proxy docker

Node

Storage volumes

16

Kubernetes 101

Kubernetes Master

Pod

kubelet kube-proxy docker

Node

Pod

kubelet kube-proxy docker

Node

Storage volumes

17

Kubernetes 101

Kubernetes MasterLoad Balancer Service

Node Node

Storage volumes

18

Pod

kubelet kube-proxy docker

Node

Pod

Node

Kubernetes 101

Greenplum on Kubernetes

Node

Pod

mirror

kubelet kube-proxy docker

19

Greenplum Service

Pod

kubelet kube-proxy docker

standby

Pod

kubelet kube-proxy docker

Node

Storage volumes

primary

Pod

kubelet kube-proxy docker

master

Benefits

Greenplum on PKS

20

1. On Demand Cluster Provisioning

PKS

Alana

21

1. On Demand Cluster Provisioning

PKS

Alana

Give me a Greenplum

Cluster

22

1. On Demand Cluster Provisioning

PKS

Alana

Give me a Greenplum

Cluster

Cluster Alana

23

1. On Demand Cluster Provisioning

PKS

Alana

Give me a Greenplum

Cluster

Cluster Alana

gpdb-alana:5432

24

1. On Demand Cluster Provisioning

PKS

Alana

Cluster Alanapsql gpdb-alana:5432

25

1. On Demand Cluster Provisioning

PKS

Alana

Cluster Alanapsql gpdb-alana:5432

Dev Team26

1. On Demand Cluster Provisioning

PKS

Alana

Cluster Alanapsql gpdb-alana:5432

Dev Team27

1. On Demand Cluster Provisioning

PKS

Alana

Cluster Alanapsql gpdb-alana:5432

Dev Team

Cluster Dev

28

1. On Demand Cluster Provisioning

PKS

Alana

Cluster Alanapsql gpdb-alana:5432

Dev Team

Cluster Dev

29

1. On Demand Cluster Provisioning

PKS

Alana

Dev Team

Cluster Alana

Cluster Dev

psql gpdb-alana:5432

psql gpdb-dev:5432

30

2. Service Discovery

master

ContainerPod

We can always discover a container by DNS.

For example, DNS address for different roles:master.greenplum.svc.cluster.localstandby.greenplum.svc.cluster.localsegment-0a.greenplum.svc.cluster.localsegment-0b.greenplum.svc.cluster.local

standby

ContainerPod

Mirrorsegment-0b

ContainerPod

Primarysegment-0a

ContainerPod

31

3. HA without Rebalancing

seg-0a

ContainerPod

seg-0b

ContainerPod

seg-1a

ContainerPod

seg-1b

ContainerPod

Primary

Mirror

32

3. HA without Rebalancing

seg-0a

ContainerPod

seg-0b

ContainerPod

seg-1a

ContainerPod

seg-1b

ContainerPod

Mirror becomes primaryAnd no need for rebalancing.

Primary

Mirror

33

4. Kubernetes Plugins Support : Container Storage Interface

34

seg-1b

ContainerPod

Cloud Storage

4. Kubernetes Plugins Support : Logging

35

seg-0b

ContainerPod

seg-1b

ContainerPod

syslog / stderr syslog / stderr

LoggingAgent Log Store

HEY PKS! GIVE ME A GREENPLUM CLUSTER OF “N”

SEGMENTS

GREENPLUM ON PKS DEMO

36

Deploy Greenplum on PKS

Demo

37

38

Expand Greenplum on PKS

Demo

39

40

Greenplum Segment Failover

Demo

41

42

More Components,StatefulSets,

Custom Resource Definitions

Future Work

43

gpcloud@pivotal.io

QUESTIONS?

44

Transforming How The World Builds Software

© Copyright 2017 Pivotal Software, Inc. All rights Reserved.45