142
Masterclass [email protected] @IanMmmm Ian Massingham — Technical Evangelist Amazon S3

Amazon S3 Masterclass

Embed Size (px)

Citation preview

Page 1: Amazon S3 Masterclass

Masterclass

[email protected]@IanMmmm

Ian Massingham — Technical Evangelist

Amazon S3

Page 2: Amazon S3 Masterclass

Masterclass

Intended to educate you on how to get the best from AWS services

Show you how things work and how to get things done

A technical deep dive that goes beyond the basics123

Page 3: Amazon S3 Masterclass

Amazon S3

Secure, durable, highly-scalable object storageAccessible via a simple web services interface

Store & retrieve any amount of data Use alone or together with other AWS services

Page 4: Amazon S3 Masterclass

Amazon S3

Durable Available

Low Cost

Scalable High Performance

Secure

Integrated

Easy to Use

Page 5: Amazon S3 Masterclass

Amazon S3

Backup & Archiving Big Data Analytics

Static Website HostingDisaster Recovery

Content Storage & Distribution

Cloud-native Application Data

Page 6: Amazon S3 Masterclass

Agenda

Amazon S3 Concepts & FundamentalsNamespaces

Access Controls Storage Classes

Encryption & Other Security Features Versioning & Cross-Region Replication

Lifecycle Rules Website Hosting

Page 7: Amazon S3 Masterclass

AMAZON S3 CONCEPTS

Page 8: Amazon S3 Masterclass

BUCKETS

Containers for objects stored in S3 Serve several purposes:

Organise the Amazon S3 namespace at the highest level Identify the account responsible for charges Play a role in access control Serve as the unit of aggregation for usage reporting

Page 9: Amazon S3 Masterclass

Fundamental entities stored in Amazon S3 Consist of data & metadata

Data portion is opaque to Amazon S3 Metadata is a set of name-value pairs that describe the object Object is uniquely identified within a bucket by a key (name) and a version ID

OBJECTS

Page 10: Amazon S3 Masterclass

Unique identifier for an object within a bucket. Every object in a bucket has exactly one key Combination of a bucket, key & version ID uniquely identify each object

KEYS

http://doc.s3.amazonaws.com/2006-03-01/AmazonS3.wsdl

Page 11: Amazon S3 Masterclass

The geographical region where Amazon S3 will store the buckets that you create Choose a region to optimise latency, minimise costs, or address regulatory requirements.

REGIONS

Page 12: Amazon S3 Masterclass

AMAZON S3 FUNDAMENTALS

A web store, not a file system

Highly scalable data storage

Fast Economical

Access via APIs

Highly available & durable

Page 13: Amazon S3 Masterclass

Access via APIshttp://aws.amazon.com/documentation/s3/

Page 14: Amazon S3 Masterclass

Access via SDKshttp://aws.amazon.com/tools/

Page 15: Amazon S3 Masterclass

Access via SDKs

https://github.com/awslabs/aws-sdk-go

Page 16: Amazon S3 Masterclass

Access via AWS CLI

List buckets

List buckets contents

Copy a file to an object

Stream the contents of an object to STDOUT

Delete an object

Sync a directory with a bucket

List buckets contents

Delete buckets contents

AWS S3 CLI help

AWS CLI

Page 17: Amazon S3 Masterclass

Access via AWS CLI

Detailed help on a specific command

AWS CLI

Page 18: Amazon S3 Masterclass

AMAZON S3 FUNDAMENTALS

A web store, not a file system

Highly scalable data storage

Fast Economical

Access via APIs

Highly available & durable

Page 19: Amazon S3 Masterclass

A web store, not a file system

Eventually consistent

Write once, Read many

Page 20: Amazon S3 Masterclass

A web store, not a file system

Eventually consistent

Write once, Read many

Page 21: Amazon S3 Masterclass

A web store, not a file system

Eventually consistent

Write once, Read many

Region

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

Page 22: Amazon S3 Masterclass

A web store, not a file system

Eventually consistent

Write once, Read many

Region

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

Page 23: Amazon S3 Masterclass

A web store, not a file system

Eventually consistent

Write once, Read many

Region

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

Page 24: Amazon S3 Masterclass

A web store, not a file system

Eventually consistent

Write once, Read many

Region

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

Page 25: Amazon S3 Masterclass

Region

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

A web store, not a file system

Eventually consistent

Write once, Read many

Page 26: Amazon S3 Masterclass

Region

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

A web store, not a file system

Eventually consistent

Write once, Read many

Page 27: Amazon S3 Masterclass

A web store, not a file system

Eventually consistent

Write once, Read many

Region

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

Availability.Zone

Indexing Storage

Load.balancers

Web.servers

Page 28: Amazon S3 Masterclass

A web store, not a file system

Eventually consistent

Write once, Read many

Page 29: Amazon S3 Masterclass

A web store, not a file system

Eventually consistent

Write once, Read many

New Objects

Updates

Deletes

Synchronously stores your data across multiple facilities before returning SUCCESS Read-after-write consistency,

except US-STANDARD region

Write then read: could report key does not exist Write then list: might not include key in list Overwrite then read: old data could be returned

Delete then read: could still get old data Delete then list: deleted key could be included in list

Find out more here: docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html

Page 30: Amazon S3 Masterclass

NAMESPACES

Page 31: Amazon S3 Masterclass

Bucket Name + Object Name (key)

Globally Unique

Page 32: Amazon S3 Masterclass

Bucket Name + Object Name (key)

Globally Unique

Amazon S3

bucket

object object

bucket

object object

bucket

object object

Page 33: Amazon S3 Masterclass

Bucket Name + Object Name (key)

Globally Unique

Amazon S3

ianm-aws-bootstrap

wp/bootstrap.sh wp/credentials.txt

ianm-aws-docs

s3-webinar.pptx vid/s3-webinar.mp4

aws-exampl.es

index.html logo.png

Page 34: Amazon S3 Masterclass

Object key

Unique within a bucket

Page 35: Amazon S3 Masterclass

Object key

Unique within a bucketMax 1024 bytes UTF-8 Including ‘path’ prefixes

Page 36: Amazon S3 Masterclass

Object key

Unique within a bucketMax 1024 bytes UTF-8 Including ‘path’ prefixes

assets/js/jquery/plugins/jtables.js  

an example object key

Page 37: Amazon S3 Masterclass

Throughput Optimisation

2134857/gamedata/start.png  2134857/gamedata/resource.rsrc  2134857/gamedata/results.txt  2134858/gamedata/start.png  2134858/gamedata/resource.rsrc  2134858/gamedata/results.txt  2134859/gamedata/start.png  2134859/gamedata/resource.rsrc  2134859/gamedata/results.txt

mynewgame

Bucket Object keys

S3 automatically partitions based upon key prefix:

Page 38: Amazon S3 Masterclass

Throughput Optimisation

2134857/gamedata/start.png  2134857/gamedata/resource.rsrc  2134857/gamedata/results.txt  2134858/gamedata/start.png  2134858/gamedata/resource.rsrc  2134858/gamedata/results.txt  2134859/gamedata/start.png  2134859/gamedata/resource.rsrc  2134859/gamedata/results.txt

mynewgame

Bucket Object keys

S3 automatically partitions based upon key prefix:

Incrementing game IDs

Page 39: Amazon S3 Masterclass

Throughput Optimisation

2134857/gamedata/start.png  2134857/gamedata/resource.rsrc  2134857/gamedata/results.txt  2134858/gamedata/start.png  2134858/gamedata/resource.rsrc  2134858/gamedata/results.txt  2134859/gamedata/start.png  2134859/gamedata/resource.rsrc  2134859/gamedata/results.txt

mynewgame

Bucket Object keys

S3 automatically partitions based upon key prefix:

Partition: mynewgame/2

Page 40: Amazon S3 Masterclass

Throughput Optimisation

7584312/gamedata/start.png  7584312/gamedata/resource.rsrc  7584312/gamedata/results.txt  8584312/gamedata/start.png  8584312/gamedata/resource.rsrc  8584312/gamedata/results.txt  9584312/gamedata/start.png  9584312/gamedata/resource.rsrc  9584312/gamedata/results.txt

mynewgame

Bucket Object keys

S3 automatically partitions based upon key prefix:

Reversed game ID

Page 41: Amazon S3 Masterclass

Throughput Optimisation

7584312/gamedata/start.png  7584312/gamedata/resource.rsrc  7584312/gamedata/results.txt  8584312/gamedata/start.png  8584312/gamedata/resource.rsrc  8584312/gamedata/results.txt  9584312/gamedata/start.png  9584312/gamedata/resource.rsrc  9584312/gamedata/results.txt

mynewgame

Bucket Object keys

S3 automatically partitions based upon key prefix:

Partitions: mynewgame/7 mynewgame/8 mynewgame/9

Page 42: Amazon S3 Masterclass

ACCESS CONTROLS

Page 43: Amazon S3 Masterclass

SECURE BY DEFAULT

You decide what to share Apply policies to buckets and objects

Policies, ACLs & IAM Use S3 policies, ACLs or IAM to define rules

Page 44: Amazon S3 Masterclass

Fine grained Administer as part of role

based access Apply policies to S3 at role, user & group level

Allow  

Actions  PutObject  

Resource  arn:aws:s3:::mybucket/*

Bob Jane

IAM Policies

Find out more here: aws.amazon.com/iam

Page 45: Amazon S3 Masterclass

Fine grained Administer as part of role

based access Apply policies to S3 at role, user & group level

Allow  

Actions  PutObject  

Resource  arn:aws:s3:::mybucket/*

Bob JaneFind out more here: aws.amazon.com/iam

IAM Policies

Page 46: Amazon S3 Masterclass

Fine grained Administer as part of role

based access Apply policies to S3 at role, user & group level

Allow  

Actions  PutObject  

Resource  arn:aws:s3:::mybucket/*

Bob JaneFind out more here: aws.amazon.com/iam

IAM Policies

Page 47: Amazon S3 Masterclass

Fine grained Administer as part of role

based access Apply policies to S3 at role, user & group level

Allow  

Actions  PutObject  

Resource  arn:aws:s3:::mybucket/*

Bob Jane

Bucket PoliciesFine grained

Apply policies at the bucket level in S3

Incorporate user restrictions without using IAM

Allow  Bob,  Jane  Actions  PutObject  

Resource  arn:aws:s3:::mybucket/*

mybucket

IAM Policies

Page 48: Amazon S3 Masterclass

Bucket PoliciesFine grained

Apply policies at the bucket level in S3

Incorporate user restrictions without using IAM

Allow  Bob,  Jane  Actions  PutObject  

Resource  arn:aws:s3:::mybucket/*

mybucket

{      "Version":"2012-­‐10-­‐17",      "Statement":[          {              "Sid":"AddPerm",              "Effect":"Allow",              "Principal":  "*",              "Action":["s3:GetObject"],              "Resource":["arn:aws:s3:::examplebucket/*"]          }      ]  }

Granting Read-Only Permission to an Anonymous User

Page 49: Amazon S3 Masterclass

{  

   "Version":"2012-­‐10-­‐17",  

   "Statement":[  

       {  

           “Sid":"AddPerm",  

           "Effect":"Allow",  

           "Principal":  "*",  

           "Action":["s3:GetObject"],  

           "Resource":["arn:aws:s3:::examplebucket/*"]  

       }  

   ]  

}

Granting Read-Only Permission to an Anonymous User

Who the policy applies to. * means everyoneThe actions allowed/denied by this policy

The AWS resource that this policy applies to. In this case all objects in example bucket

The effect of the policy, allow or deny

Page 50: Amazon S3 Masterclass

Bucket PoliciesFine grained

Apply policies at the bucket level in S3

Incorporate user restrictions without using IAM

Allow  Bob,  Jane  Actions  PutObject  

Resource  arn:aws:s3:::mybucket/*

mybucket

{      "Version":  "2012-­‐10-­‐17",      "Id":  "S3PolicyId1",      "Statement":  [          {              "Sid":  "IPAllow",              "Effect":  "Allow",              "Principal":  "*",              "Action":  "s3:*",              "Resource":  "arn:aws:s3:::examplebucket/*",              "Condition":  {                    "IpAddress":  {"aws:SourceIp":  "54.240.143.0/24"},                    "NotIpAddress":  {"aws:SourceIp":  "54.240.143.188/32"}                }            }        ]  }

Restricting Access to Specific IP Addresses

Page 51: Amazon S3 Masterclass

Bucket PoliciesFine grained

Apply policies at the bucket level in S3

Incorporate user restrictions without using IAM

Allow  Bob,  Jane  Actions  PutObject  

Resource  arn:aws:s3:::mybucket/*

mybucket

Other Example Use-Cases for Bucket Policies

• Granting Permissions to Multiple Accounts with Added Conditions

• Restricting Access to a Specific HTTP Referrer • Granting Permission to an Amazon CloudFront

Origin Identity • Adding a Policy to Require MFA Authentication • Granting Cross-Account Permissions to Upload

Objects While Ensuring the Bucket Owner Has Full Control

Find out more here: docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html

Page 52: Amazon S3 Masterclass

Fine grained Administer as part of role

based access Apply policies to S3 at role, user & group level

Allow  

Actions  PutObject  

Resource  arn:aws:s3:::mybucket/*

Bob Jane

Bucket Policies ACLsFine grained

Apply policies at the bucket level in S3

Incorporate user restrictions without using IAM

Coarse grained Apply access control rules at the bucket and/or object

level in S3

Allow  Bob,  Jane  Actions  PutObject  

Resource  arn:aws:s3:::mybucket/*

mybucket

Allow  Everyone,  Bob,  Jane  Actions  Read  

mybucket myobject

IAM Policies

Page 53: Amazon S3 Masterclass

ACLsCoarse grained

Apply access control rules at the bucket and/or object

level in S3

Allow  Everyone,  Bob,  Jane  Actions  Read  

mybucket myobject

You can use ACLs to grant basic read/write permissions to other AWS accounts. There are limits to managing permissions using ACLs. For example, you can grant permissions only to other AWS accounts, you cannot grant permissions to users in your account.

Page 54: Amazon S3 Masterclass

DEMO: GETTING STARTED

Page 55: Amazon S3 Masterclass

STORAGE CLASSES

Page 56: Amazon S3 Masterclass

Designed to provide 99.999999999% durability and 99.99% availability of objects over a given year Designed to sustain the concurrent loss of data in two facilities

Reduces costs by storing data at lower levels of redundancy than the Standard storage Designed to provide 99.99% durability and 99.99% availability of objects over a given year

Suitable for archiving data, where data access is infrequent and a retrieval time of several hours is acceptable Uses the very low-cost Amazon Glacier storage service, but managed through Amazon S3

Standard

Reduced Redundancy

Storage

Glacier

Page 57: Amazon S3 Masterclass
Page 58: Amazon S3 Masterclass

Moving Objects between S3 storage classes

You can specify the storage class of an object when uploading or creating it

$  aws  s3  cp  aws_uki.txt  s3://aws-­‐ianm-­‐s3-­‐masterclass/  -­‐-­‐storage-­‐class  REDUCED_REDUNDANCY

AWS CLI

Page 59: Amazon S3 Masterclass

Moving Objects between S3 storage classes

You can change the storage class of an object that is already stored in Amazon S3 by copying it to the same key name in the same bucket

$  aws  s3  cp  s3://aws-­‐ianm-­‐s3-­‐masterclass/aws_uki.txt  s3://aws-­‐ianm-­‐s3-­‐masterclass/aws_uki.txt  -­‐-­‐storage-­‐class  STANDARD

AWS CLI

Page 60: Amazon S3 Masterclass

Moving Objects between storage classes

AWS Console

Page 61: Amazon S3 Masterclass

Moving Objects between storage classes$  python  

>>>  import  boto  

>>>  conn  =  boto.connect_s3()  

>>>  mybucket  =  conn.get_bucket(‘aws-­‐ianm-­‐s3-­‐masterclass’)  

>>>  mybucket.copy_key('aws_uki.txt','aws-­‐ianm-­‐s3-­‐masterclass','aws_uki.txt',storage_class='REDUCED_REDUNDANCY')

Python

Page 62: Amazon S3 Masterclass

What about Amazon Glacier?

We will come to this when we talk about Lifecycle Management

Page 63: Amazon S3 Masterclass

ENCRYPTION

Page 64: Amazon S3 Masterclass

Securing Data in Transit

Securely upload or download your data via SSL-encrypted endpoints using HTTPS Alternatively, use a client encryption library such as the Amazon S3 Encryption Client to encrypt your data before uploading to Amazon S3

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3EncryptionClient.html

Page 65: Amazon S3 Masterclass

SECURING DATA AT REST

Page 66: Amazon S3 Masterclass

Amazon S3 Server Side Encryption (SSE)

Amazon S3 will automatically encrypt your data on write and decrypt your data on retrieval Uses Advanced Encryption Standard (AES) 256-bit symmetric keys There are three different ways to mange encryption keys:

Page 67: Amazon S3 Masterclass

SSE with Amazon S3 Key Management (SSE-SE)

With SSE-S3, Amazon S3 will encrypt your data at rest and manage the encryption keys for you

Data bucket Encrypted data

Encrypted per-object key

Key management (monthly rotation)

Per-object key

Master key

Page 68: Amazon S3 Masterclass

SSE with Customer-Provided Keys (SSE-C)

With SSE-C, Amazon S3 will encrypt your data at rest using the custom encryption keys that you provide Amazon S3 doesn’t store your encryption key anywhere; the key is immediately discarded after Amazon S3 completes your requests

Data bucket Encrypted data

Customer Provided Key

Page 69: Amazon S3 Masterclass

SSE with AWS KMS (SSE-KMS)

With SSE-KMS, Amazon S3 will encrypt your data at rest using keys that you manage in the AWS Key Management Service (KMS)

AWS KMS provides an audit trail so you can see who used your key to access which object and when

Page 70: Amazon S3 Masterclass

ADDITIONAL SECURITY FEATURES

Page 71: Amazon S3 Masterclass

AUDIT LOGS

Page 72: Amazon S3 Masterclass

MULTI-FACTOR AUTHENTICATION DELETE

Page 73: Amazon S3 Masterclass

TIME-LIMTED ACCESSTO OBJECTS

Page 74: Amazon S3 Masterclass

Signed URLs Provide time-limited access to

specific objects that expires after a set period

Access Permissions Use on objects in non-public

buckets to prevent access once the signed URL has expired

https://ianmas-­‐aws.testbucket.s3.amazonaws.com/testfile.txt  ?Signature=JHCa39GV1fKRKkEnAWzI88lH7f8%3D  &Expires=1391425438  &AWSAccessKeyId=AKIAIRBKBJ3ZAYAXFC2Q

Page 75: Amazon S3 Masterclass

>>>  import  boto  >>>  conn  =  boto.connect_s3()  >>>  conn.generate_url(3600,  'GET',  bucket='aws-­‐ianm-­‐s3-­‐masterclass',  key=‘aws_uki.txt')  'https://aws-­‐ianm-­‐s3-­‐masterclass.s3.amazonaws.com/aws_uki.txt?Signature=hEBUPczy8DXCyqTz1JHgEaihvMo%3D&Expires=1431697820&AWSAccessKeyId=AKIAI65L23YDGKGQTRFA'  

>>>  import  boto  >>>  conn  =  boto.connect_s3()  >>>  conn.generate_url(30,  'GET',  bucket='aws-­‐ianm-­‐s3-­‐masterclass',  key=‘aws_uki.txt’,  force_http=True)  'http://aws-­‐ianm-­‐s3-­‐masterclass.s3.amazonaws.com/aws_uki.txt?Signature=yIYPyn0DMXk2cOcZkWPRuSHoKPA%3D&Expires=1431694649&AWSAccessKeyId=AKIAI65L23YDGKGQTRFA'

PythonGenerating time-limited signed links

Page 76: Amazon S3 Masterclass

>>>  import  boto  >>>  conn  =  boto.connect_s3()  >>>  conn.generate_url(3600,  'GET',  bucket='aws-­‐ianm-­‐s3-­‐masterclass',  key=‘aws_uki.txt')  'https://aws-­‐ianm-­‐s3-­‐masterclass.s3.amazonaws.com/aws_uki.txt?Signature=hEBUPczy8DXCyqTz1JHgEaihvMo%3D&Expires=1431697820&AWSAccessKeyId=AKIAI65L23YDGKGQTRFA'  

>>>  import  boto  >>>  conn  =  boto.connect_s3()  >>>  conn.generate_url(30,  'GET',  bucket='aws-­‐ianm-­‐s3-­‐masterclass',  key=‘aws_uki.txt’,  force_http=True)  'http://aws-­‐ianm-­‐s3-­‐masterclass.s3.amazonaws.com/aws_uki.txt?Signature=yIYPyn0DMXk2cOcZkWPRuSHoKPA%3D&Expires=1431694649&AWSAccessKeyId=AKIAI65L23YDGKGQTRFA'

PythonGenerating time-limited signed links1st parameter is link lifetime in seconds

Force a non-SSL link

Page 77: Amazon S3 Masterclass

Generating time-limited signed linksError response: link expired

Page 78: Amazon S3 Masterclass

VERSIONING &CROSS REGION REPLICATION

Page 79: Amazon S3 Masterclass

VERSIONING

Bucket level Automatically preserves

all copies of objects

Persistent Even deleted object

history is held

Page 80: Amazon S3 Masterclass

AWS Console

Page 81: Amazon S3 Masterclass

AWS Console

Page 82: Amazon S3 Masterclass

AWS CLI

uses the ‘aws s3api’ CLI command, which has additional functionality over ‘aws s3’

$  aws  s3api  list-­‐object-­‐versions  -­‐-­‐bucket  aws-­‐ianm-­‐s3-­‐masterclass

Working with versioned objects via the CLI

http://docs.aws.amazon.com/cli/latest/reference/s3api/index.html#cli-aws-s3api

Page 83: Amazon S3 Masterclass

AWS CLI

$  aws  s3api  list-­‐object-­‐versions  -­‐-­‐bucket  aws-­‐ianm-­‐s3-­‐masterclass  

None   None  VERSIONS   "36bc67941830bb388c9bf201440683a4"   True   s3-­‐masterclass-­‐logo.txt   2015-­‐05-­‐18T15:47:38.000Z  337   STANDARD   q2FcbXFAgU7P82Hh6y81hYkrbGYtOCCc  OWNER   ianm   4ee381d180ee58aa815e7d4a3a5f739b20bb8980a568947384e59c8d0ff8379b  VERSIONS   "e0253c9354f61097cbf6ce239afd0464"   False   s3-­‐masterclass-­‐logo.txt   2015-­‐05-­‐18T15:47:30.000Z  337   STANDARD   p_4oF4eG7Be.0aNXqUYFlZL7Q9OuK9nU  OWNER   ianm   4ee381d180ee58aa815e7d4a3a5f739b20bb8980a568947384e59c8d0ff8379b  VERSIONS   "84defb05031845e8b0616a9b70b2ae93"   False   s3-­‐masterclass-­‐logo.txt   2015-­‐05-­‐18T15:47:10.000Z  328   STANDARD   BykgMQ6bRY02Y6krxvMaMvwrL2Ep2e6X  OWNER   ianm   4ee381d180ee58aa815e7d4a3a5f739b20bb8980a568947384e59c8d0ff8379b

Page 84: Amazon S3 Masterclass

AWS CLI

$  aws  s3api  get-­‐object  -­‐-­‐bucket  aws-­‐ianm-­‐s3-­‐masterclass  -­‐-­‐key  s3-­‐masterclass-­‐logo.txt  -­‐-­‐version-­‐id  q2FcbXFAgU7P82Hh6y81hYkrbGYtOCCc  version.txt  

bytes   337   text/plain   "36bc67941830bb388c9bf201440683a4"   Mon,  18  May  2015  15:47:38  GMT  q2FcbXFAgU7P82Hh6y81hYkrbGYtOCCc  

$  more  version.txt    ____  _____      __    __                      _                                _  /  ___|___  /    |    \/    |  __  _  ___|  |_  ___  _  __  ___|  |  __  _  ___  ___  \___  \  |_  \    |  |\/|  |/  _`  /  __|  __/  _  \  '__/  __|  |/  _`  /  __/  __|    ___)  |__)  |  |  |    |  |  (_|  \__  \  ||    __/  |  |  (__|  |  (_|  \__  \__  \  |____/____/    |_|    |_|\__,_|___/\__\___|_|    \___|_|\__,_|___/___/  

Version  3

http://docs.aws.amazon.com/cli/latest/reference/s3api/index.html#cli-aws-s3api

Page 85: Amazon S3 Masterclass

>>>  import  boto  >>>  conn  =  boto.connect_s3()  >>>  bucket=conn.get_bucket('aws-­‐ianm-­‐s3-­‐masterclass')  >>>  versions  =  bucket.list_versions()  >>>  for  version  in  versions:  ...      print  version.name  +  '  '  +  version.version_id  ...  s3-­‐masterclass-­‐logo.txt  q2FcbXFAgU7P82Hh6y81hYkrbGYtOCCc  s3-­‐masterclass-­‐logo.txt  p_4oF4eG7Be.0aNXqUYFlZL7Q9OuK9nU  s3-­‐masterclass-­‐logo.txt  BykgMQ6bRY02Y6krxvMaMvwrL2Ep2e6X

PythonListing object versions

Page 86: Amazon S3 Masterclass

>>>  key  =  bucket.get_key('s3-­‐masterclass-­‐logo.txt',  version_id='p_4oF4eG7Be.0aNXqUYFlZL7Q9OuK9nU')  >>>  key.get_contents_as_string()  "\n  ____  _____      __    __                      _                                _                              \n/  ___|___  /    |    \\/    |  __  _  ___|  |_  ___  _  __  ___|  |  __  _  ___  ___  \n\\___  \\  |_  \\    |  |\\/|  |/  _`  /  __|  __/  _  \\  '__/  __|  |/  _`  /  __/  __|\n  ___)  |__)  |  |  |    |  |  (_|  \\__  \\  ||    __/  |  |  (__|  |  (_|  \\__  \\__  \\\n|____/____/    |_|    |_|\\__,_|___/\\__\\___|_|    \\___|_|\\__,_|___/___/\n\nVersion  2\n”  >>>  key.generate_url(300)  'https://aws-­‐ianm-­‐s3-­‐masterclass.s3.amazonaws.com/s3-­‐masterclass-­‐logo.txt?Signature=c%2BjgGY5EZ4tDuI0xcKg572qL%2B9Y%3D&Expires=1431965853&AWSAccessKeyId=AKIAI65L23YDGKGQTRFA&versionId=p_4oF4eG7Be.0aNXqUYFlZL7Q9OuK9nU'  >>>

PythonGetting a specific object version

Page 87: Amazon S3 Masterclass

CROSS REGION REPLICATION

Page 88: Amazon S3 Masterclass

AWS Console

Page 89: Amazon S3 Masterclass

AWS Console

Page 90: Amazon S3 Masterclass

AWS Console

Page 91: Amazon S3 Masterclass

AWS Console

Page 92: Amazon S3 Masterclass

AWS Console

Page 93: Amazon S3 Masterclass

AWS Console

Page 94: Amazon S3 Masterclass

AWS CLI

$  aws  s3  cp  s3-­‐masterclass-­‐logo.txt  s3://aws-­‐ianm-­‐s3-­‐masterclass  upload:  ./s3-­‐masterclass-­‐logo.txt  to  s3://aws-­‐ianm-­‐s3-­‐masterclass/s3-­‐masterclass-­‐logo.txt  

$  aws  s3  ls  s3://aws-­‐ianm-­‐s3-­‐masterclass-­‐fra  -­‐-­‐region=eu-­‐central-­‐1  2015-­‐05-­‐18  17:27:47                337  s3-­‐masterclass-­‐logo.txt

Page 95: Amazon S3 Masterclass

LIFECYCLE RULES

Page 96: Amazon S3 Masterclass

LIFECYCLE RULES

Object Deletion Permanently delete objects from S3

Object Archiving Move objects from S3 to Glacier

Page 97: Amazon S3 Masterclass

Amazon Glacier

Page 98: Amazon S3 Masterclass

Amazon Glacier

Durable Designed for 99.999999999%

durability of archives

Cost Effective Write-once, read-never. Cost effective for long

term storage. Pay for accessing data

Page 99: Amazon S3 Masterclass

Logsaccessible from S3

logs

Expi

ry

time

Page 100: Amazon S3 Masterclass

logs✗Objects expire and are deleted

Logsaccessible from S3Ex

piry

time

Page 101: Amazon S3 Masterclass

Txns

Object transition to Glacier invoked

Logs logs✗Objects expire and are deleted

accessible from S3

accessible from S3

Expi

ryTr

ansit

ion

time

Page 102: Amazon S3 Masterclass

Restoration of object requested for x hrs

Logs logs✗Objects expire and are deleted

accessible from S3

accessible from S3Txns

Expi

ryTr

ansit

ion

Object transition to Glacier invoked

time

Page 103: Amazon S3 Masterclass

time

3-5hrs

Object held in S3 RRS for x hrs

Expi

ryTr

ansit

ion

Logs logs✗Objects expire and are deleted

accessible from S3

accessible from S3Txns

Object transition to Glacier invoked

Restoration of object requested for x hrs

Page 104: Amazon S3 Masterclass

Configuring Lifecycle Rules AWS Console

Page 105: Amazon S3 Masterclass

Configuring Lifecycle Rules AWS Console

Page 106: Amazon S3 Masterclass

Configuring Lifecycle Rules AWS Console

Page 107: Amazon S3 Masterclass

Configuring Lifecycle Rules AWS Console

Page 108: Amazon S3 Masterclass

Configuring Lifecycle Rules AWS Console

Page 109: Amazon S3 Masterclass

Configuring Lifecycle Rules AWS Console

Page 110: Amazon S3 Masterclass

Configuring Lifecycle Rules AWS Console

Page 111: Amazon S3 Masterclass

using  (client  =  new  AmazonS3Client()){      var  lifeCycleConfiguration  =  new  LifecycleConfiguration()      {          Rules  =  new  List<LifecycleRule>          {                  new  LifecycleRule                  {                            Id  =  "Archive  and  delete  rule",                            Prefix  =  "projectdocs/",                            Status  =  LifecycleRuleStatus.Enabled,                              Transition  =  new  LifecycleTransition()                              {                                        Days  =  365,                                        StorageClass  =  S3StorageClass.Glacier                              },                              Expiration  =  new  LifecycleRuleExpiration()                              {                                        Days  =  3650                              }                  }          }      };

.NetConfiguring Lifecycle Rules

Page 112: Amazon S3 Masterclass

using  (client  =  new  AmazonS3Client()){      var  lifeCycleConfiguration  =  new  LifecycleConfiguration()      {          Rules  =  new  List<LifecycleRule>          {                  new  LifecycleRule                  {                            Id  =  "Archive  and  delete  rule",                            Prefix  =  "projectdocs/",                            Status  =  LifecycleRuleStatus.Enabled,                              Transition  =  new  LifecycleTransition()                              {                                        Days  =  365,                                        StorageClass  =  S3StorageClass.Glacier                              },                              Expiration  =  new  LifecycleRuleExpiration()                              {                                        Days  =  3650                              }                  }          }      };

.NetConfiguring Lifecycle Rules

Transition to Glacier after 1 year

Page 113: Amazon S3 Masterclass

using  (client  =  new  AmazonS3Client()){      var  lifeCycleConfiguration  =  new  LifecycleConfiguration()      {          Rules  =  new  List<LifecycleRule>          {                  new  LifecycleRule                  {                            Id  =  "Archive  and  delete  rule",                            Prefix  =  "projectdocs/",                            Status  =  LifecycleRuleStatus.Enabled,                              Transition  =  new  LifecycleTransition()                              {                                        Days  =  365,                                        StorageClass  =  S3StorageClass.Glacier                              },                              Expiration  =  new  LifecycleRuleExpiration()                              {                                        Days  =  3650                              }                  }          }      };

.NetConfiguring Lifecycle Rules

Delete object after 10 years

Page 114: Amazon S3 Masterclass

Restoring from Amazon Glacier AWS Console

Page 115: Amazon S3 Masterclass

Restoring from Amazon Glacier AWS Console

Page 116: Amazon S3 Masterclass

Restoring from Amazon Glacier AWS Console

Page 117: Amazon S3 Masterclass

Restoring from Amazon Glacier AWS Console

Page 118: Amazon S3 Masterclass

Restoring from Amazon Glacier AWS Console

Page 119: Amazon S3 Masterclass

WEBSITE HOSTING

Page 120: Amazon S3 Masterclass

Static Website Hosting with Amazon S3

You can host your entire static website on Amazon S3 for a low-cost, highly available hosting solution that can scale automatically to meet traffic demands

With Amazon S3, you can reliably serve your traffic and handle unexpected peaks without worrying about scaling your infrastructure

docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html

Page 121: Amazon S3 Masterclass

Static Website Hosting Bucket Properties

Page 122: Amazon S3 Masterclass

Setting Default Documents

Page 123: Amazon S3 Masterclass

Redirecting Requests

Page 124: Amazon S3 Masterclass

{      "Version":"2008-­‐10-­‐17",      "Statement":[{     "Sid":"PublicReadGetObject",                  "Effect":"Allow",        "Principal":  {                          "AWS":  "*"                    },              "Action":["s3:GetObject"],              "Resource":["arn:aws:s3:::example-­‐bucket/*"              ]          }      ]  }  

Bucket Policy

Page 125: Amazon S3 Masterclass

{bucket-­‐name}.s3-­‐website-­‐{region}.amazonaws.com  e.g.  mybucket.s3-­‐website-­‐eu-­‐west-­‐1.amazonaws.com

Website Addressing

s3-­‐{region}.amazonaws.com/{bucket-­‐name}/{object-­‐key}  e.g.  s3-­‐eu-­‐west-­‐1.amazonaws.com/mybucket/img.png  

{bucket-­‐name}.s3-­‐{region}.amazonaws.com/{object-­‐key}  e.g.  mybucket.s3-­‐eu-­‐west-­‐1.amazonaws.com/img.png  

Normal Addressing

Page 126: Amazon S3 Masterclass

DNS Record set for:

aws-­‐exampl.es

Route 53

Page 127: Amazon S3 Masterclass

DNS Record set for:

aws-­‐exampl.es

Route 53

bucket bucket

Website bucket name: Website bucket name:www.aws-­‐exampl.es aws-­‐exampl.es

Error .html

Index .html

Page 128: Amazon S3 Masterclass

DNS Record set for:

aws-­‐exampl.es

Route 53

bucket bucket

Website bucket name: Website bucket name:www.aws-­‐exampl.es aws-­‐exampl.es

Error .html

Index .html

Website redirect to:aws-­‐exampl.es

Page 129: Amazon S3 Masterclass

DNS Record set for:

aws-­‐exampl.es

Route 53

bucket

Website bucket name: Website bucket name:www.aws-­‐exampl.es aws-­‐exampl.es

Error .html

Index .html

Website redirect to:aws-­‐exampl.es

A Record ‘Alias’ to S3 website:aws-­‐exampl.es  @  s3website-­‐eu-­‐west1-­‐amazonaws.com  

bucket

Page 130: Amazon S3 Masterclass

DNS Record set for:

aws-­‐exampl.es

Route 53

bucket

Website bucket name: Website bucket name:www.aws-­‐exampl.es aws-­‐exampl.es

Error .html

Index .html

Website redirect to:aws-­‐exampl.es

A Record ‘Alias’ to S3 website:aws-­‐exampl.es  @  s3website-­‐eu-­‐west1-­‐amazonaws.com  

bucketbucket

CNAME for www. to:www.aws-­‐exampl.es.s3-­‐website-­‐eu-­‐west-­‐1.amazonaws.com  

Page 131: Amazon S3 Masterclass

docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html

Page 132: Amazon S3 Masterclass

SUMMARY

Page 133: Amazon S3 Masterclass

S3 provides developers with secure, durable & highly scalable object storage1

S3 can be used alone with other AWS services or 3rd party tools & services2

Cost effective for a wide variety of use-cases from cloud applications, content distribution, backup, archiving & disaster recovery to analytics

3

Page 134: Amazon S3 Masterclass

THINGS WE DIDN’T COVER

Page 135: Amazon S3 Masterclass

Amazon CloudFrontaws.amazon.com/cloudfront

Page 136: Amazon S3 Masterclass

S3 EVENT NOTIFICATIONS

http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html

Page 137: Amazon S3 Masterclass

AWS Lambda

S3 event notifications

AutomaticCloud Functions

aws.amazon.com/lambda

Page 138: Amazon S3 Masterclass

RESOURCES YOU CAN USETO LEARN MORE

Page 139: Amazon S3 Masterclass

aws.amazon.com/s3

Page 140: Amazon S3 Masterclass

Getting Started with Amazon S3:

docs.aws.amazon.com/AmazonS3/latest/gsg/GetStartedWithS3.html

Amazon S3 Deep Dive & Best Practices session from AWS re:Invent 2014

https://youtu.be/2DpOS0zu8O0

Amazon S3 Documentation:

aws.amazon.com/documentation/s3/

Page 141: Amazon S3 Masterclass

Certification

aws.amazon.com/certification

Self-Paced Labs

aws.amazon.com/training/self-paced-labs

Try products, gain new skills, and get hands-on practice working

with AWS technologies

aws.amazon.com/training

Training

Validate your proven skills and expertise with the AWS platform

Build technical expertise to design and operate scalable, efficient applications on AWS

AWS Training & Certification

Page 142: Amazon S3 Masterclass

Follow us fo

r more

events

& webina

rs

@AWScloud for Global AWS News & Announcements

@AWS_UKI for local AWS events & news

@IanMmmmIan Massingham — Technical Evangelist