Upload
others
View
10
Download
0
Embed Size (px)
Citation preview
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Junpei Ozono, Solutions Architect, Amazon Web Services K.K.September 26, 2019
Amazon Redshift Deep Dive & Update
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AgendaAmazon Redshift recap
Redshift Deep Dive -SETDW-
���
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Redshift recap
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Redshift : "����������
��������
BI
OLTP ERP CRM LOB �������&�����!$
������0*#$� 0*-5,3
���%
Amazon Redshift
(2����"�$�/+)1.�(���������������%
'4���������
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Redshift�������$
JDBC/ODBC
Amazon Redshift������
Amazon S3���(��
���������&��)�"�)�
���������Redshift �� ',-��(���Redshift �� '����&�+*�)�)
Amazon Redshift SpectrumS3 ���'����&�+*�)�)(���(��)�%��&���)
�!)���#�����',-��(��(Parquet, ORC, JSON, CSV etc)
���������������� ����
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Redshift '"Redshift ����� 1 ���� 2 �������3������.#
2���/8�106 �2
2� 200(� !$,�40$,�*1
%,�$, -+7� ��)5�&���
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
������������ (2018 10�)
��
���������
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
(1 0 (2019 7 )
0
/ 8 2 )
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Redshift Deep Dive
-SETDW-
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
������ ���� 5 ��
SETDW
RedshiftPerformance
Sortkey
Encoding
Table Maintenance
Distribution
WLM
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sortkey (�����)������������������� ����� �
CREATE TABLE deep_dive (aid INT --audience_id
,loc CHAR(3) --location,dt DATE --date
);
CREATE TABLE deep_dive (aid INT --audience_id
,loc CHAR(3) --location,dt DATE --date
) SORTKEY (dt, loc);
deep_diveaid loc dt1 SFO 2017-10-202 JFK 2017-10-203 SFO 2017-04-014 JFK 2017-05-14
deep_dive (sorted)aid loc dt3 SFO 2017-04-014 JFK 2017-05-142 JFK 2017-10-201 SFO 2017-10-20
deep_dive (sorted)aid loc dt3 SFO 2017-04-01
deep_dive (sorted)aid loc dt3 SFO 2017-04-014 JFK 2017-05-14
deep_dive (sorted)aid loc dt3 SFO 2017-04-014 JFK 2017-05-142 JFK 2017-10-20
deep_dive (sorted)aid loc dt3 SFO 2017-04-014 JFK 2017-05-142 JFK 2017-10-201 SFO 2017-10-20
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
������ '6�Redshift �������8A�����<�%�� (1MB) +�?;��1->�1'>��&"#$(�8A��,)4953*�7/• ��$=.�0:2 �%���B�@�� (��&!� )
MIN: 01-JUNE-2017MAX: 06-JUNE-2017
MIN: 07-JUNE-2017MAX: 12-JUNE-2017
MIN: 21-JUNE-2017MAX: 30-JUNE-2017
MIN: 01-JUNE-2017MAX: 20-JUNE-2017
MIN: 08-JUNE-2017MAX: 30-JUNE-2017
MIN: 02-JUNE-2017MAX: 25-JUNE-2017
SELECT count(*) FROM deep_dive WHERE dt = '06-09-2017';
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
%�)#��<�6� ���2�$1�*�� COMPOUND %�)#��<S���
XU�F����MD��TP��V�B�HC (WHERE W) #��?8�AL• 4O9��3G� ID
R=#��?8�� �IE���IE�J� 4 "/.>:�Q��1. HC�?8�X;�5�N2. "�( +0' ��K�N�?8• @��"/.�?8� �1�*, Vacuum 7�!�,�-&*�
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
������� ��������SVL_QUERY_SUMMARY ��� is_rrscan�� t �������������� (= ��� �������� ��������)
# select … from svl_query_summary where query = xxx order by stm, seg, ste; maxtime | avgtime | rows | bytes | label | is_diskbased | workmem | is_rrscan | is_delayed_scan | rows_pre_filter--------+---------+---------+----------+---------------------------------------------+--------------+---------+-----------+-----------------+----------------103579 | 102174 | 2338770 | 28065240 | scan tbl=587136 name=test_sort_key_order | f | 0 | t | f | 16501910 103579 | 102174 | 2338770 | 0 | project | f | 0 | f | f | 0 103579 | 102174 | 2338770 | 0 | project | f | 0 | f | f | 0 103579 | 102174 | 2 | 16 | aggr tbl=296 | f | 0 | f | f | 0 65 | 58 | 2 | 16 | scan tbl=296 name=Internal Worktable | f | 0 | f | f | 0 65 | 58 | 2 | 0 | return | f | 0 | f | f | 0 354 | 354 | 2 | 16 | scan tbl=52828 name=Internal Worktable | f | 0 | f | f | 0 354 | 354 | 1 | 16 | aggr tbl=300 | f | 0 | f | f | 0 74 | 74 | 1 | 16 | scan tbl=300 name=Internal Worktable | f | 0 | f | f | 0 74 | 74 | 1 | 0 | project | f | 0 | f | f | 0 74 | 74 | 1 | 0 | project | f | 0 | f | f | 0 74 | 74 | 1 | 13 | return | f | 0 | f | f | 0
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Encoding (�('<=)
�('<=�.�����-0�"����� ��?�7��"�!;�6� ��$%��&+��5,�/7��
<=��+��# (�*�)�') �@423�����CREATE TABLE�9>�1A����:8
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
N���������COPY
(COMPUPDATE = PRESET ; )
• - RAW (• BOOLEAN / REAL / DOUBLE PRECISION - RAW ( )• - LZO
S E T D W
!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
� �45�"����"�(%$&���������ANALYZE COMPRESSION ��"���� ��� �"����"�����26(%�.*��,(%����1-• ANALYZE COMPRESSION <���!)>;
�"�������/3���������!,+' & INSERT-SELECT�#0
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
:;�$����$��+�)�Redshift �� �)0��
1. 5%*�� COPY �!$����'.:;�8�
2. �� #����21�<���"��$�37��6/�(�4��ANALYZE COMPRESSION �!$��,& 9-�37
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
\^ 6'�,�6%�@�<�7Z:D�
SY�O���ZSTD� � LZO ������@���QU
• ZSTD �\^P�;IV & IO [F�LW���&�(�G���$ 4A9=�\^X?�!�/�1*. (CPUT`) �;�
• LZO �]�\^P� ZSTD �b���BKJ��M����&�(�
Ra>HC�N��+�05 (~10,000H) �"32�E8)�-#��RAW �@_��
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
������56����/*• ������4��" �.7��56�2���������#��)�1���+��3�(-� ,��" ��#���%0���!$����������'&���
0000
100K
0000
100K
5K
COL1 COL2
row_id
row_id
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
�� ���78����3-• .'�� ���78�25 (RAW) ������04
• �����������COL1 �1* $��!�+,��COL2 �"%�����#&�6/���)(���
����
���
0000
100K
5K
COL1 COL2
10K
row_id
row_id
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
<;4����"SVL_QUERY_SUMMARY � ��rows_pre_filter= (���"��!���:���1.) rows = ()0�5*&81.) �%7���9�(�����$#�!��'�+3,�����/�:�����62-��
# select … from svl_query_summary where query = xxx order by stm, seg, ste; maxtime | avgtime | rows | bytes | label | is_diskbased | workmem | is_rrscan | is_delayed_scan | rows_pre_filter--------+---------+---------+----------+---------------------------------------------+--------------+---------+-----------+-----------------+----------------103579 | 102174 | 2338770 | 28065240 | scan tbl=587136 name=test_sort_key_order | f | 0 | t | f | 16501910 103579 | 102174 | 2338770 | 0 | project | f | 0 | f | f | 0 103579 | 102174 | 2338770 | 0 | project | f | 0 | f | f | 0 103579 | 102174 | 2 | 16 | aggr tbl=296 | f | 0 | f | f | 0 65 | 58 | 2 | 16 | scan tbl=296 name=Internal Worktable | f | 0 | f | f | 0 65 | 58 | 2 | 0 | return | f | 0 | f | f | 0 354 | 354 | 2 | 16 | scan tbl=52828 name=Internal Worktable | f | 0 | f | f | 0 354 | 354 | 1 | 16 | aggr tbl=300 | f | 0 | f | f | 0 74 | 74 | 1 | 16 | scan tbl=300 name=Internal Worktable | f | 0 | f | f | 0 74 | 74 | 1 | 0 | project | f | 0 | f | f | 0 74 | 74 | 1 | 0 | project | f | 0 | f | f | 0 74 | 74 | 1 | 13 | return | f | 0 | f | f | 0
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Table Maintenance
Vacuum
Sort
Analyze Vacuum
Delete
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Analyze �%*(Analyze ��+>���!'$/52-��� 6,�����"�</1.�?&��8)
���"����7(���!�;@:= 0���>4����#�%*'$���
� ��� auto_analyze�7?�����%*'$� OFF �����93
Amazon RedshiftQuery planner
Table statistics Amazon Redshift Worker Thread Auto Analyze
Anal
yze
this
tabl
e
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Vacuum Delete Vacuum Delete Delete Updatet o 2
t cmr 1
u 9
V 99 a 0
S
Col1 Col2 Col3 Col1 Col2 Col3
Vacuum Delete
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Distribution ($8���#)�� #��������� 20����'����
ALL
Node 1
Slice 1
Slice 2
Node 2
Slice 3
Slice 4
KEY
Node 1
Slice 1
Slice 2
Node 2
Slice 3
Slice 4
keyA
keyB
keyC
keyD
Node 1
Slice 1
Slice 2
Node 2
Slice 3
Slice 4
EVEN
*%����4����!+/�1�"�� $820
1�"��9 -)�65 ����
$820
1���&�,3��"�� ����(-$����.7
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
!6�����%�"�Redshift �����"*�Redshift ����
1. ����/�!6�����(4 �������-)(CREATE TABLE #�!6�����+$��)
2. & ������.,�0'�2��$3�15���
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AUTO );� �&Redshift �);� �&�1*�!�%&�4/����"$�&#�AUTO );� �&�8-
����,����� ALL);�����3��� EVEN );�(.69
'+ EVEN �69�����"� 57�2 ��(.�� ALL �:���
����30
AUTOALL
Node 1
Slice 1
Slice 2
Node 2
Slice 3
Slice 4
Node 1
Slice 1
Slice 2
Node 2
Slice 3
Slice 4
EVEN
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Advisor DISTKEY Recommendation
Redshift
Redshift Advisor
N :ALTER TABLE
S E T D W
!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
+?���'�.�-�)=*4��ALL, EVEN, KEY �8�+��
1. JOIN � �"'����63�:�� �"' ($��� �"') �ALL+?�63�5� �"' (Fact �"') �KEY+?� �EVEN+?�.@
2. BA� JOIN �63�5� �"',9��; �"'� JOIN �80��&%�DISTKEY��71
3. ���� JOIN ������� �"'�>2</�EVEN+?�71
#�(!� “JOIN”
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Redshift JOIN Deep Dive��#���� ��� JOIN "%���!������� (Fact-Fact� ) � JOIN ���
Redshift�� JOIN ���3���• Nested Loop JOIN• Hash JOIN• Merge JOIN
Redshift��$� JOIN �Merge JOIN
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Nested Loop JOIN• 4?��"�('�Nested Loop JOIN �-G��
• Nested Loop JOIN �E=�@9�B�>A• :5�;��F7�1'82�<�������"�6�D��• ��" !$% 0,�Warning �&+������������
----- Nested Loop Join in the query plan - review the join predicates to avoid Cartesian products -----
id
1
1
3
5
9
10
id
1
5
9
10
/*.
)*.
%��1#�����C3
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hash JOIN• :+.�8��-,���Hash JOIN �1@���
• Hash JOIN �$%(��*�*�" =3• ���'��")�$%(�<�� �-,�=3������>� !��#*��A2���?5
id
5
1
10
9
3
1
id
9
5
10
1
col1
9
5
10
1
C���'��");7(on $%()
B��&* D��&*�9,
604
/04
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Merge JOIN• JOIN����"*5������ '?���*%�(&��
Merge JOIN �-@���• A�=�>�����*5�8B�� JOIN �������:!����);�6.�73
• 24 �� Concurrency ���� #��1$��<9
id
1
1
3
5
9
10
id
1
5
9
10
0,/
+,/
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
WLM (8�%7�-SB)S E T D W
WLM��%"5�N�$3���' 9��A;��$3���� ��A;��%"5�ICPL�125KU�=>A;G�A;) 0�!,�@R��
���������� ���(SQA)
E_D[����%"5�A;>?�J]�A;>?�\��J]������YC�<X$3��6�+�9&����
Concurrency Scaling
%"5�=>A;G�:Q�W�OBT�����>��.*%&4!9-�M�%4()��VH�%"5OB�#/7�-�Z^FL������
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto WLMWLM( )
WLM (Auto WLM )
Auto WLM N
Redshift�����
…
Concurrency Query Slots = Auto
S E T D W!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto WLM – Query Priorities•• Highest• High• Normal ( )• Low• Lowest
• Redshift N
• QMR • CHANGE_QUERY_PRIORITY• CHANGE_SESSION_PRIORITY• CHANGE_USER_PRIORITY
! S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto WLM – Query Priorities
•
ETL N
Highest• N 1
Normal
• BI N
Normal
ETL / Priority : Highest
BI / Priority : Normal
S E T D W!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
�'� ��)���(+��',(SQA)
���� 9H6F ����)3-/0�<G��
1
�'� ��)�=B�����)�E5.D�%�*���,����
2
)�����'� ��)��72A4���
3
SQA ��������BI / �� �����
�,#%� "�!
�,#%� "�!
�,#%� "�!
Amazon Redshift
��� �
9>��$�* �@C(&���?�;1�8:)
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Concurrency Scaling Deep Dive
Redshift S3
1
2
N1
2
e
Concurrency Scaling
S E T D W!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Concurrency Scaling ����������������������������WLM ����� !�� �Concurrency Scaling mode” �� auto ���
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
������������������ �max_concurrency_scaling_clusters” �������������� 0-10 ������ ( ��� : 1)
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Concurrency Scaling -%*(��"����� ���• �"��"�����dc2.8xlarge, ds2.8xlarge, dc2.large, or ds2.xlarge• ���'�2�32�
�� ���• Amazon Redshift local ���!��� Spectrum ���!�#��
Read-only�� (SELECT, Unload)
,&• ���!�Interleaved Sort Key�)������• ���!�����$+ Temporary Table�)������
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PC$.&'�� ')('!+1%Q�max_concurrency_scaling_clusters = 5 �65
1.,!1$.&'��$"/�#-�O��;L��8�?D�PC$.&'�� ')(
2. ')(���PC$.&'�� ���$"/&0)*�TK���65�2 I<�PC$.&'�� ')(�BS�F:���?2A 5 ��7R�>G�V�N
PC$.&'���$"/=4�MW ���H�����$"/�J;�����3:@9�$.&'��UE �� �PC$.&'��//�& �������
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Concurrency ScalingR=?• Concurrency Scaling �)�. +"#�� �,��*�L�� B�3;• �*�L����H�� �,�)�. +"#��TO��• 5C4�MA +"#��F9�EJ����<G
• MA +"#��80�� �,�)�. +"#���80�>D�Q��GD>���• K�� �,�80N6�N���21����S���21�����• ����*�L������ ��!"%(:7�"-�'$&�@/�PI
S E T D W
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
���
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
���
RedshiftPerformance
Sortkey
Encoding
Table Maintenance
Distribution
WLM
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!Junpei OzonoTwitter: @jostandard