Introdução ao data warehouse Amazon Redshift

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Eric Ferreira, Principal Database Engineer, AWS

2 de Junho de 2016

Introdução ao data warehouse

Amazon Redshift

Agenda

• Introdução

• Benefícios

• Testemunho

• Começando

• Criando um cluster

• Modelo de dados

• Carregando dados

• Consultando

• Mais Recursos

AnalyzeStore

Amazon

Glacier

Amazon S3

Amazon

DynamoDB

Amazon RDS,

Amazon Aurora

AWS big data portfolio

AWS Data Pipeline

Amazon

CloudSearch

Amazon EMR Amazon EC2

Amazon

Redshift

Amazon

Machine

Learning

Amazon

Elasticsearch

Service

AWS Database

Migration Service

Amazon

QuickSight

Amazon

Kinesis

Firehose

AWS Import/Export

AWS Direct

Connect

Collect

Amazon Kinesis

Streams

Relational data warehouse

Massively parallel; petabyte scale

Fully managed

HDD and SSD platforms

$1,000/TB/year; starts at $0.25/hour

Amazon

Redshift

a lot faster

a lot simpler

a lot cheaper

The Amazon Redshift view of data warehousing

10x cheaper

Easy to provision

Higher DBA productivity

10x faster

No programming

Easily leverage BI tools,

Hadoop, machine learning,

streaming

Analysis inline with process

flows

Pay as you go, grow as you

need

Managed availability and

disaster recovery

Enterprise Big Data SaaS

The Forrester Wave™ is copyrighted by Forrester Research, Inc. Forrester and Forrester Wave™ are trademarks of Forrester Research, Inc. The Forrester Wave™ is a graphical

representation of Forrester's call on a market and is plotted using a detailed spreadsheet with exposed scores, weightings, and comments. Forrester does not endorse any

vendor, product, or service depicted in the Forrester Wave. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change.

Forrester Wave™ Enterprise Data Warehouse

Q4‘15

Selected Amazon Redshift customers

Amazon Redshift architecture

Leader Node

Simple SQL endpoint

Stores metadata

Optimizes query plan

Coordinates query execution

Compute Nodes

Local columnar storage

Parallel/distributed execution of all queries, loads,

backups, restores, resizes

Start at just $0.25/hour, grow to 2 PB (compressed)

DC1: SSD; scale from 160 GB to 326 TB

DS2: HDD; scale from 2 TB to 2 PB

Ingestion/Backup

Backup

Restore

JDBC/ODBC

10 GigE

(HPC)

Benefit #1: Amazon Redshift is fast

Dramatically less I/O

Column storage

Data compression

Zone maps

Direct-attached storage

Large data block sizes

analyze compression listing;

Table | Column | Encoding

---------+----------------+----------

listing | listid | delta

listing | sellerid | delta32k

listing | eventid | delta32k

listing | dateid | bytedict

listing | numtickets | bytedict

listing | priceperticket | delta32k

listing | totalprice | mostly32

listing | listtime | raw

10 | 13 | 14 | 26 |…

… | 100 | 245 | 324

375 | 393 | 417…

… 512 | 549 | 623

637 | 712 | 809 …

… | 834 | 921 | 959

10

324

375

623

637

959


Parallel and distributed

Query

Load

Export

Backup

Restore

Resize


Hardware optimized for I/O intensive workloads, 4 GB/sec/node

Enhanced networking, over 1 million packets/sec/node

Choice of storage type, instance size

Regular cadence of autopatched improvements


New Dense Storage (HDD) instance type

Improved memory 2x, compute 2x, disk throughput 1.5x

Cost: Same as our prior generation!

Performance improvement: 50%

Enhanced I/O and commit improvements (Jan ‘16)

Reduce amount of time to commit data

Performance improvement: 35%

Benefit #2: Amazon Redshift is inexpensive

DS2 (HDD)Price Per Hour for

DW1.XL Single Node

Effective Annual

Price per TB Compressed

On-Demand $ 0.850 $ 3,725

1 Year Reservation $ 0.500 $ 2,190

3 Year Reservation $ 0.228 $ 999

DC1 (SSD)Price Per Hour for

DW2.L Single Node

Effective Annual

Price per TB Compressed

On-Demand $ 0.250 $ 13,690



Pricing is simple

Number of nodes x price/hour

No charge for leader node

No upfront costs

Pay as you go

Benefit #3: Amazon Redshift is fully managed

Continuous/incremental backups

Multiple copies within cluster

Continuous and incremental backups

to S3

Continuous and incremental backups

across regions

Streaming restore

Amazon S3

Amazon S3

Region 1

Region 2

Benefit #3: Amazon Redshift is fully managed

Amazon S3

Amazon S3

Region 1

Region 2

Fault tolerance

Disk failures

Node failures

Network failures

Availability Zone/region level disasters

Benefit #4: Security is built-in

• Load encrypted from S3

• SSL to secure data in transit

• ECDHE perfect forward security

• Amazon VPC for network isolation

• Encryption to secure data at rest

• All blocks on disks and in S3 encrypted

• Block key, cluster key, master key (AES-256)

• On-premises HSM & AWS CloudHSM support

• Audit logging and AWS CloudTrail integration

• SOC 1/2/3, PCI-DSS, FedRAMP, BAA

10 GigE

(HPC)

Ingestion

Backup

Restore

Customer VPC

Internal

VPC

JDBC/ODBC

Benefit #5: We innovate quickly

Well over 100 new features added since launch

Release every two weeks

Automatic patching

Service Launch (2/14)

PDX (4/2)

Temp Credentials (4/11)

DUB (4/25)

SOC1/2/3 (5/8)

Unload Encrypted Files

NRT (6/5)

JDBC Fetch Size (6/27)

Unload logs (7/5)

SHA1 Builtin (7/15)

4 byte UTF-8 (7/18)

Sharing snapshots (7/18)

Statement Timeout (7/22)

Timezone, Epoch, Autoformat (7/25)

WLM Timeout/Wildcards (8/1)

CRC32 Builtin, CSV, Restore Progress (8/9)

Resource Level IAM (8/9)

PCI (8/22)

UTF-8 Substitution (8/29)

JSON, Regex, Cursors (9/10)

Split_part, Audit tables (10/3)

SIN/SYD (10/8)

HSM Support (11/11)

Kinesis EMR/HDFS/SSH copy, Distributed Tables, Audit

Logging/CloudTrail, Concurrency, Resize Perf., Approximate Count Distinct, SNS

Alerts, Cross Region Backup (11/13)

Distributed Tables, Single Node Cursor Support, Maximum Connections to 500

(12/13)

EIP Support for VPC Clusters (12/28)

New query monitoring system tables and diststyle all (1/13)

Redshift on DW2 (SSD) Nodes (1/23)

Compression for COPY from SSH, Fetch size support for single node clusters, new

system tables with commit stats, row_number(), strotol() and query

termination (2/13)

Resize progress indicator & Cluster Version (3/21)

Regex_Substr, COPY from JSON (3/25)

50 slots, COPY from EMR, ECDHE ciphers (4/22)

3 new regex features, Unload to single file, FedRAMP(5/6)

Rename Cluster (6/2)

Copy from multiple regions, percentile_cont, percentile_disc (6/30)

Free Trial (7/1)

pg_last_unload_count (9/15)

AES-128 S3 encryption (9/29)

UTF-16 support (9/29)

Benefit #6: Redshift is powerful

• Approximate functions

• User defined functions

• Machine Learning

• Data Science

Benefit #7: Amazon Redshift has a large ecosystem

Data Integration Systems IntegratorsBusiness Intelligence

Benefit #8: Service oriented architecture

DynamoDB

EMR

S3

EC2/SSH

RDS/Aurora

Amazon

Redshift

Amazon Kinesis

Amazon ML

Data Pipeline

CloudSearch

Amazon

Mobile

Analytics

Testemunho

“O Redshift permitiu que integrássemos todos

dados da empresa de forma coesa e acessível

para todas áreas de negócio”

Global Fashion Group ecommerce

presente 4 países com três marcas

(Dafiti, Kanui e Tricae)

Maior e-commerce de moda e

lifestyle da América Latina

“Diferente dos

modelos mainstream,

na AWS conseguimos

cadenciar nosso

investimento em DWH

de acordo com o uso

que fazemos de

nossos dados”

- Caio Motta – CIO

O Desafio

Ampliação do uso para novas áreasdo negócio

Plataforma instável

Manutenção Lenta e Constante

Performance e Escalabilidade• Storage trabalhando no limite

• Indisponibilidade de paralelismo

• Features limitadas

Solução

Começando…

Criando um cluster

Enter Cluster Details

Select Node Configuration

Select Security Settings and Provision

Point and click resize

Resize

• Resize while remaining online

• Provision a new cluster in the

background

• Copy data in parallel from node to

node

• Only charged for source cluster

Modelo de dados

SELECT COUNT(*) FROM LOGS WHERE DATE = ‘09-JUNE-2013’

MIN: 01-JUNE-2013

MAX: 20-JUNE-2013

MIN: 08-JUNE-2013

MAX: 30-JUNE-2013

MIN: 12-JUNE-2013

MAX: 20-JUNE-2013

MIN: 02-JUNE-2013

MAX: 25-JUNE-2013

Unsorted Table

MIN: 01-JUNE-2013

MAX: 06-JUNE-2013

MIN: 07-JUNE-2013

MAX: 12-JUNE-2013

MIN: 13-JUNE-2013

MAX: 18-JUNE-2013

MIN: 19-JUNE-2013

MAX: 24-JUNE-2013

Sorted By Date

Zone Maps

• Single Column

• Compound

• Interleaved

Single Column• Table is sorted by 1 column

Date Region Country

2-JUN-2015 Oceania New Zealand

2-JUN-2015 Asia Singapore

2-JUN-2015 Africa Zaire

2-JUN-2015 Asia Hong Kong

3-JUN-2015 Europe Germany

3-JUN-2015 Asia Korea

[ SORTKEY ( date ) ]

• Best for:

• Queries that use 1st column (i.e. date) as primary filter

• Can speed up joins and group bys

• Quickest to VACUUM

Compound

• Table is sorted by 1st column , then 2nd column etc.

Date Region Country







[ SORTKEY COMPOUND ( date, region, country) ]

• Best for:

• Queries that use 1st column as primary filter, then other cols

• Can speed up joins and group bys

• Slower to VACUUM

Interleaved

• Equal weight is given to each column.

Date Region Country







[ SORTKEY INTERLEAVED ( date, region, country) ]

• Best for:

• Queries that use different columns in filter

• Queries get faster the more columns used in the filter

• Slowest to VACUUM

• KEY

• ALL

• EVEN

ID Gender Name

101 M John Smith

292 F Jane Jones

139 M Peter Black

446 M Pat Partridge

658 F Sarah Cyan

164 M Brian Snail

209 M James White

306 F Lisa Green

2

3

4

ID Gender Name

101 M John Smith

306 F Lisa Green

ID Gender Name

292 F Jane Jones

209 M James White

ID Gender Name

139 M Peter Black

164 M Brian Snail

ID Gender Name

446 M Pat Partridge

658 F Sarah Cyan

RoundRobin

DISTSTYLE EVEN

ID Gender Name

101 M John Smith

292 F Jane Jones

139 M Peter Black

446 M Pat Partridge

658 F Sarah Cyan

164 M Brian Snail

209 M James White

306 F Lisa Green

HashFunction

ID Gender Name

101 M John Smith

306 F Lisa Green

ID Gender Name

292 F Jane Jones

209 M James White

ID Gender Name

139 M Peter Black

164 M Brian Snail

ID Gender Name

446 M Pat Partridge

658 F Sarah Cyan

DISTSTYLE KEY

ID Gender Name

101 M John Smith

292 F Jane Jones

139 M Peter Black

446 M Pat Partridge

658 F Sarah Cyan

164 M Brian Snail

209 M James White

306 F Lisa Green

HashFunction

ID Gender Name

101 M John Smith

139 M Peter Black

446 M Pat Partridge

164 M Brian Snail

209 M James White

ID Gender Name

292 F Jane Jones

658 F Sarah Cyan

306 F Lisa Green

DISTSTYLE KEY

ID Gender Name

101 M John Smith

292 F Jane Jones

139 M Peter Black

446 M Pat Partridge

658 F Sarah Cyan

164 M Brian Snail

209 M James White

306 F Lisa Green

101 M John Smith

292 F Jane Jones

139 M Peter Black

446 M Pat Partridge

658 F Sarah Cyan

164 M Brian Snail

209 M Lisa Green

306 F James White

101 M John Smith

292 F Jane Jones

139 M Peter Black

446 M Pat Partridge

658 F Sarah Cyan

164 M Brian Snail

209 M Lisa Green

306 F James White

101 M John Smith

292 F Jane Jones

139 M Peter Black

446 M Pat Partridge

658 F Sarah Cyan

164 M Brian Snail

209 M Lisa Green

306 F James White

101 M John Smith

292 F Jane Jones

139 M Peter Black

446 M Pat Partridge

658 F Sarah Cyan

164 M Brian Snail

209 M Lisa Green

306 F James White

ALL

DISTSTYLE ALL

• KEY

• Large Fact tables

• Large dimension tables

• ALL

• Medium dimension tables (1K – 2M)

• EVEN

• Tables with no joins or group by

• Small dimension tables (<1000)

Carregando dados

AWS CloudCorporate Data center

Amazon S3Amazon

Redshift

Flat files

Data loading options

AWS CloudCorporate Data center

ETL

Source DBs

Amazon

Redshift

Amazon

Redshift


AWS Cloud

Amazon

RedshiftAmazon

Kinesis


Use the COPY command

Each slice can load one file at a

time

A single input file means only one

slice is ingesting data

Instead of 100MB/s, you’re only

getting 6.25MB/s

Use multiple input files to maximize

throughput

Use the COPY command

You need at least as many input files as you have slices

With 16 input files, all slices are working so you maximize throughput

Get 100MB/s per node; scale linearly as you add nodes

Use multiple input files to maximize

throughput

Consultando

JDBC/ODBC

Amazon Redshift

Amazon Redshift works with your existing

analysis tools

ODBC/JDBC

BI ClientsRedshift

ODBC/JDBC

BI Server Redshift

Clients

Monitor query performance

View explain plans

Resources

Eric Ferreira | [email protected] |

Detail Pages

• http://aws.amazon.com/redshift

• https://aws.amazon.com/marketplace/redshift/

Best Practices

• http://docs.aws.amazon.com/redshift/latest/dg/c_loading-data-best-practices.html

• http://docs.aws.amazon.com/redshift/latest/dg/c_designing-tables-best-practices.html

• http://docs.aws.amazon.com/redshift/latest/dg/c-optimizing-query-performance.html

Open Source Tools

• https://github.com/awslabs/amazon-redshift-utils

mailto:[email protected]

http://aws.amazon.com/redshift

https://aws.amazon.com/marketplace/redshift/

http://docs.aws.amazon.com/redshift/latest/dg/c_loading-data-best-practices.html

http://docs.aws.amazon.com/redshift/latest/dg/c_designing-tables-best-practices.html

http://docs.aws.amazon.com/redshift/latest/dg/c-optimizing-query-performance.html

https://github.com/awslabs/amazon-redshift-utils

Obrigado !

Technology

Introdução ao data warehouse Amazon Redshift