Distributed Database Architecture for GDPR · 10/15/2018 · ü Apache HBase committers and early...

Distributed Database Architecture for GDPR

Karthik RanganathanPostgresConf Silicon Valley

Oct 15, 2018

About Us

Kannan Muthukkaruppan, CEONutanix ♦ Facebook ♦ Oracle

IIT-Madras, University of California-Berkeley

Karthik Ranganathan, CTONutanix ♦ Facebook ♦Microsoft

IIT-Madras, University of Texas-Austin

Mikhail Bautin, Software ArchitectClearStory Data ♦ Facebook ♦ D.E.Shaw

Nizhny Novgorod State University, Stony Brook

ü Founded Feb 2016

ü Apache HBase committers and early engineers on Apache Cassandra

ü Built Facebook’s NoSQL platform powered by Apache HBase

ü Scaled the platform to serve many mission-critical use cases• Facebook Messages (Messenger)• Operational Data Store (Time series Data)

ü Reassembled the same Facebook team at YugaByte along with engineers from Oracle, Google, Nutanix and LinkedIn

Founders

WHAT ISYUGABYTE DB?

A transactional, planet-scale database

for building high-performance cloud services.

NoSQL + SQL Cloud Native

TRANSACTIONAL PLANET-SCALEHIGH PERFORMANCE

Single Shard & Distributed ACID Txns

Document-Based, Strongly Consistent Storage

Low Latency, Tunable Reads

High Throughput

OPEN SOURCE

Apache 2.0

Popular APIs ExtendedApache Cassandra, Redis and PostgreSQL (BETA)

Auto Sharding & Rebalancing

Global Data Distribution

Design Principles

CLOUD NATIVE

Built For The Container Era

Self-Healing, Fault-Tolerant

WHAT IS GDPR?

GDPR : General Data Protection Regulation

Citizens of EU can control sharing and protection

of their personal data by businesses.

Personal Data, also called

PII (Personally Identifiable Information)

• User name

• Email address

• Date of birth

• Bank details

• Location details

• Computer IP address

Control over personal data

• Consent & data location

• Data privacy and safety

• Right to be forgotten

• Data access on demand

• Notify on data breach

• Data portability

• Ability to fix errors in data

• Restrict processing

Database concerns Application concerns

#1 USER CONSENTAND DATA LOCATION

Data must be stored in EU by default. Businesses

need explicit user consent to move it outside.

Why is this hard?

• EU user data lives in that region

• Other countries have compliance regulation – more geo’s

• Public clouds may not have coverage – hybrid deployments

• Architecture depends on data – multiple per service

Think Global Deployments first!

Example – online ecommerce site

• Products table needs globally replication – not PII data

Read Replicas

Global Replication

Non-PII Data

Global Replication with YugaByte DB

Example – online ecommerce site

• Users, orders and shipments needs locality – PII data

• Product locations table needs scale – may be PII

Primary Data in EU

PII Data

Non-EU Data

Non-EU DataGeo-Partitioning

with YugaByte DB

Replicate data on demand to other geo’s

• User may be ok with replicating data

• Read replicas on demand (for remote, low-latency reads)

• Change data capture (for analytics)

Read Replicas

Primary Data in EU

PII Data with YugaByte DB

Read Replicas with YugaByte DB

#2 DATA PRIVACYAND SAFETY

Data must be secured by using best practices by

default. Users need to be notified on breach.

Implement end-to-end encryption on day #1

• Use TLS Encryption

• Between client and server for app interaction

• Between database servers for replication

Encrypt All Network Communication

TLS Encryption

Database Cluster

Server to server communication

• Encryption at rest

• Integrate with external Key Management Systems

• Ability to rotate keys on demand

Encryption All Storage

Have a key-value table with id to cipher key. Encrypt PII data with

the cipher key for fine-grained control. More in the next section.

Encryption at Rest

Database Cluster

Encryption on disk

Key Management Service

#3 RIGHT TO BE FORGOTTEN

Data must be erased if on explicit request or when

data is no longer relevant to original intent.

• Have a key-value table with id to cipher key

• Encrypt PII data with the cipher key on write

• Decrypt PII data on access

• Delete cipher key to forget PII data

Use Encryption of Data Attributes

SET email=foo@bar.com FOR USER ID=XXX

Example - Storing User Profile Data

SET email=ENCRYPTED FOR USER ID=XXX

Get encryption key for user

Encryption PII DataStore encrypted data

• Reads require decryption• Data not accessible without key

• Many cases where value not needed

• Anonymize PII data with one way hash functions

• Use hashed ids for in data warehouse

• There is no PII data if hashed ids are used!

Use Anonymization of Data Attributes

USER=foo@bar.com CHECKED OUT PRODUCT=X, CATEGORY=Gadget

Example – Website Analytics

USER=HASHED_VAL CHECKED OUT PRODUCT=X, CATEGORY=Gadget

One-way hash user id

Analytics

Example – Website Analytics

• User no longer identifiable• Hashed data still useful!

#4 DATA ACCESSON DEMAND

Ability to inform a user about what data is being

used, for what purpose and where it is stored.

• Store in a separate information architecture table

• Make tagging a part of the process

• Easy to find what PII data is stored on demand

Tag Tables and Columns with PII

• Ensure PII are encrypted

• Ensure non-PII columns do not have sensitive data

• Use Spark/Presto to perform scan periodically

• Run scan on a read replica to not impact production

Run Continuous Compliance Checks

Ensure PII columns are encrypted

Ensure no PII data in other columns

Tag PII Columns

PUTTING IT ALL TOGETHER

GDPR Reference Architecture

Primary Cluster(in EU)

Read Replica Clusters(Anywhere in the World)

Encrypted Encrypted

App clients

Encrypted Async Replication

Reads & Writes, Encrypted

Analytics clients

Read only, Encrypted

At-Rest Encryption for All Nodes At-Rest Encryption for All Nodes

PII Columns Encrypted w/ Cipher Key

Tag PII Columns

Ensure PII columns are encrypted

Ensure no PII data in other columns

Questions?Try it at docs.yugabyte.com/quick-start

Distributed Database Architecture for GDPR · 10/15/2018 · ü Apache HBase committers and early...

Documents

Early Literacy StationTM - AWE Learning · Merriam Webster Dictionary / Diccionario ü ü Mi Increíble Cuerpo Humano ü Mia Maths ü Mia Reading ü Mia Sciences ü Mia Sodoku ü

with Apache Beam (incubating) Fundamentals of Stream ... · Apache Beam Committers & Google Engineers Fundamentals of Stream Processing with Apache Beam (incubating) QCon San Francisco

40 mm systems - Amazon S3s3.amazonaws.com/hk-manuals/files/Catalogs/Military/HK...40 mm systems ü ü − ag36/glm 40 mm x 46 ü ü − ü ü ü ü ü ü ü ü hckler & e koch offers

ù ù ÿ ù ü þ ÿ ù ù ù + + ü ù ù...ù ! $ . . ü . ! . . 2 . (Ancient Greek Maxims) m 3 ÿ ù ù ü ÿ ü ü (TABLE OF CONTENTS) ù - ÿ ü ü ÿ (DEDICATIONS) 5 þ ü ÿ

G2 >Ý>à>Ø>Ü>Ü>Ü Ç >Ü Ç G2 >Þ>Ý>Ø>Ü>Ü>Ü Ç

learning.acm.org...Apache Arrow Project Overview 2020 Development Status 17 major releases Over 500 unique contributors Over 50M package installs in 2019 ASF roster: 52 committers,

INDUSTRIAL LED LIGHTING · 2019. 4. 6. · Pracovná teplota -30 – 40°C Krytie IP 65 ü ü ü ü ü ü ü GP Technické údaje Parametre Uvedené hodnoty príkonu a svetelného

Woodland spotter sheet - Fastly · Oak tree Squirrel Blue Tit Woodpecker Badger Red Kite Red Admiral Black bird Slug ü ü ü ü ü ü ü ü ü Woodland spotter sheet Woodlands are

Committers (Section 1)

Plan Comparisons - IDShield · Reduce Pre-Approved Credit Card Offers/Junk Mail ü ü ü ü Credit Inquiry Alerts ü ü ü ü ü ü Lost Wallet Assistance ü ü ü ü ü ü ü Data

So what’s included in your upgraded member program? · Thyroid Profile Total Triiodothyronine (T3) ü ü ü ü Total Thyroxine (T4) ü ü ü ü Thyroid-Stimulating Hormone (TSH)

· ÷ øvù#ú ûü ý.þ$ÿ #ûü|û #ú û #ü|û #ù#ü ü ü^û ü #û"! ú û # ü #$# ø %'& ( øvü #% ü^û ü^û

Silver Gold Platinum Platinum+ ü ü ü ü ü ü ü ü ü ü ü€¦ · Red Cell Distribution Width (RDW-CV) ü ü ü ü Platelet Count ü ü ü ü Platelet Distribution Width ü

Apache Flink Hands-On - JCConf.tw · Apache Flink an open-source platform for distributed stream and batch data processing ~230 contributors, 23 Committers / PMCs (growing) User adoption:

Managing Containers with Helix - events.static.linuxfound.org · Apache Helix Committers @ LinkedIn helix.apache.org @apachehelix. Intersection of Job Types OracleDB OracleDB. Intersection

TUNGSTEN SMART-HEAT™ ELECTRIC · 2019. 11. 6. · 240V/16.6A 208V/19.23A 277V/14.44A 240V/25A ... ü ü ü ü – ü ü ü ü ü ü ü ü ü * Control up to six zones of heaters

White Clay Creek State Park Hunting Map - Possum Hill 2019-20 · ü ü ü ü ü ü ü ü ü ü ü G # # # # # # # # # # # # # # # # # # # G!!! G " " " " "! " " " " G!!!! " " #!!!!!

Live Panel: Appium Core Committers Answer Your Questions

SLAZ052E– May 2009– Revised January 2011 …carlson/download/datasheets/errata.pdf · wdg4 cc430f5133 e ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü

Annual Report 2014-15G/F, Ajanta Appartment, Near Lakdi Pul, 2439132 Dt. 26-11-2014 Dandiya Bazar, Baroda-390 001 ü ü üüü üüü üüü ü ü ü ü ü ü ü ü ü ü ü ü ü