Aws introduction tocloudstorage

Amazon Web ServicesIntroduction to Cloud Storage

Kevin WittkopfEnterprise Solutions Architect…

Cloud StorageAWS offers multiple cloud-based storage options.

• Amazon S3 (Simple Storage Service)

• Amazon EBS (Elastic Block Storage)

• Amazon EC2 Local Instance Store (Ephemeral volumes)

• Amazon Cloudfront

Each has a unique combination of performance, durability, cost, and interfaceOften used to form a Storage Hierarchy

Storage by the Numbers

Cloudfront S3 EBS Local Disk

Scale Unlimited Temporary Storage

Unlimited Storage5TB objects100 name spaces (buckets)

1TB per volume~44TB per server

147 – 1690GB1-5 volumes

Durability N/A 99.999999999% (standard)99.99%(RRS)

.1-.5% AFR Ephemeral design requires application-level redundancy

Latency Low High Low Very Low

Management Zero Maintenance

Zero Maintenance Snapshots, RAID, Consistency

Manual Redundancy

Locality Global Region-Centric AZ-Specific In-machine

Interface REST or SOAPWeb Services

REST or SOAPWeb Services

Block Device Block Device

S3 Amazon Simple Storage Service

©2011 Amazon Web Services May not be reused or redistributed without permission

STORAGE

Amazon

EBSAmazon

SimpleDB

Amazon

S3

COMPUTE

Amazon

EC2

Amazon

RDS


Amazon Simple Storage Service

Amazon S3: Simple Storage Service

Highly scalable data storage in-the-cloud

Programmatic access via web services API

Is a Web Store, not a file system Optimized for WORM

Eventually consistent

Fast, highly available and durable

Economical


http://aws.amazon.com/s3

Region

Datacenter

Datacenter

Is Your Data Storage This Resilient?Note: Conceptual drawing only. Actual number of nodes & datacenters may vary

Datacenter

...Node

1Node

n


Amazon S3 Adoption Rate: Billions of Objects Stored


Q4 2006 Q4 2007 Q4 2008 Q4 2009 Q4 2010 Q2 20110

100

200

300

400

500

600

2.914

40

102

262

490

6 months!

Peak Requests:290,000+

per second

Amazon S3 Namespace

Amazon S3

bucket bucket

object object objectobject

bucket

object object


Amazon S3 Namespace

Amazon S3

mculver-images media.mydomain.com

Beach.jpg img1.jpg img2.jpg2009/maui/sun.jpg

public.blueorigin.com

Installer.msi img/pic1.jpg


Reduced Redundancy Storage Saves $$

Designed to provide 99.99% durability and 99.99% availability of objects over a given yearThis durability level corresponds to an average annual expected loss of 0.01% of objects Designed to sustain the loss of data in a single facility


S3 Pricing has Three Dimensions

Storage Data Transfer Requests

Tier Normal

RRS Tier Price Type Price

Frist 1 TB/mo

$0.14/GB

$0.093/GB All transfer in Free PUT, COPY, POST, or LIST

$0.01 per

1,000

Next 49 TB $0.125/GB

$0.083/GB First 1 GB out Free GET and all other requests

$0.01 per

10,000

Next 450 TB

$0.11/GB $0.073/GB <= 10 TB out $0.12/GB

Next 500 TB

$0.095/GB

$0.063/GB Next 40 TB out

$0.09/GB

Next 4000 TB

$0.08/GB

$0.053/GB Next 100 TB out

$0.07/GB

> 5000 TB $0.055/GB

$0.037/GB > 150 TB out $0.05/GB


Access Control Lists• Control access to bucket or to object• No inheritance from bucket to object• Set at create, write, or any time• List of up to 100 grants by:

– Owner– Email– Any AWS account– Anyone

• Permissions:– READ – Object or bucket– WRITE – Create, overwrite, delete objects in

bucket– READ_ACP – Read ACL for bucket or object– WRITE_ACP – Overwrite ACL for bucket or object– FULL_CONTROL – All of the above


Bucket Policy

Allow Jason or Mark to do anything in my S3 bucket when coming from the Office internal network

{"Statement":[{ "Effect":"Allow", "Principal":{"AWS":["4649-6425", "5243-0045"]}, "Action":"*", "Resource":"/mybucket/*", "Condition":{ "IpAddress":{"AWS:SourceIp":"176.13.0.0/12"}}}]}


S3 Multipart Uploads

Improved throughput—upload parts in parallel

Quick recovery from any network issues—Smaller part size minimizes the impact of restarting a failed upload due to a network error.

Pause and resume object uploads— upload object parts over time Once you initiate a multipart upload there is no expiry must explicitly complete or abort the multipart upload

Begin an upload before you know the final object size— can upload as you are create.


Even More Amazon S3 (not so simple, is it?)Website supportVersioningBrowser Upload to S3 LoggingRequestor PaysSigned URLsBittorrent supportIAMMeta-data


AWS Import/Export

Get your data into AWS faster - load it onto a portable storage device and ship it to an Amazon data centerFaster than Internet transfer and more cost effective than upgrading your connectivityUse cases: data migration, offsite backup, direct data interchange, disaster recovery


AWS Import/Export vs. S3 upload costs

100Gb 500Gb 1 Tb 2 Tb 3 Tb 5 Tb * 10 Tb **$0

$200

$400

$600

$800

$1,000

$1,200

Import/ExportS3 upload

AWS Import/Export vs. S3 upload time savings

100 Gb 500 Gb 1 Tb 2 Tb 3 Tb 5 Tb0

10

20

30

40

50

60

Import/ExportS3 uploadD

ays

STORAGE

Amazon

EBSAmazon

SimpleDB

Amazon

S3

COMPUTE

Amazon

EC2

Amazon

RDS


EBS Features

Persistent storageVolume lifetime is independent of any particular EC2 instance.

General purposeRaw, unformatted, block device. Use from Linux, Solaris or Windows.

PerformanceEqual to or better than local EC2 drive.

High reliabilityBuilt-in redundancy within availability zone.AFR (Annual Failure Rate) between 0.1% and 1%.

ScalableVolume sizes ranging from 1 GB to 1 TB.

EasyEasy to create, attach, back up, restore, and delete volumes.


EU West Region

Availability Zone A

Availability Zone B

Availability Zone C

Japan

Availability Zone A

Availability Zone B

Amazon Web Services: Durable & Available

US East Region

Availability Zone A

Availability Zone C

Availability Zone B

US West Region

Availability Zone A

Availability Zone B

Note: Conceptual drawing only. The number of Availability Zones may vary

Singapore

Availability Zone A

Availability Zone B

©2011 Amazon Web Services May not be reused or redistributed without written permission

EBS Volumes are in a Single Availability Zone

Note: Conceptual diagram only

Availability Zone A

EBS Volume 1

Availability Zone B

EBS Volume 2


Internet

AWS Cloud

How EBS Interacts With EC2 and S3

EC2 Availability Zone

EC2

S3 Storage

EBS

EC2 EC2

EBS EBS EBS EBS EBSEBS Snapshot

EBS Snapshot

EBS Snapshot

EBS Snapshot

EBS Snapshot

Create Snapshot

Clone From Snapshot


What’s the Difference BetweenAmazon S3 and Amazon EBS?

Amazon S3 EBS

Paradigm Object store File system

Performance Very fast Very very fast

Redundancy Across data centers Within data center

Security Public Key / Private Key

Visible only to your EC2

Pricing $0.14/GB/Mo. stored $0.10/GB/Mo. allocated

Access from the Net? Yes(1) No(2)

Typical use case Write once, read many It’s a disk drive

(1) Only with proper credentials, unless ACLs are world readable(2) Accessible from Net if mounted to server and set up as FTP, etc.


EBS Pricing

EBS Volumes $0.10 per GB-month

of provisioned storage $0.10 per 1 million I/O requests No charge for mounting/unmounting

volume

EBS Snapshots to Amazon S3 $0.14 per GB-month of data stored $0.01 per 1,000 PUT requests

(when saving a snapshot) $0.01 per 10,000 GET requests

(when loading a snapshot)


EBS Volume Lifecycle

Create

Vast amounts of unused space

Call CreateVolume1 GB to 1 TB

Attach

Deleted

Call AttachVolume to affiliate with one EC2 instance

Attached &

In Use

• Format from EC2 instance OS

• Mount formatted drive

CreateSnapshot

Snapshot to Amazon S3

Detach

Call DetachVolume

Call DeleteVolume


http://images.google.com/imgres?imgurl=http://larvalsubjects.files.wordpress.com/2008/12/universe.jpg&imgrefurl=http://larvalsubjects.wordpress.com/2008/12/&usg=__STC6iVgZ1zvcyoRsfV1uKnbIY0g=&h=307&w=425&sz=28&hl=en&start=1&tbnid=XVmONP9NL94C_M:&tbnh=91&tbnw=126&prev=/images?q=universe&gbv=2&hl=en

How Do Snapshots Work?

Time

Snapshot 1

Snapshot 2 Snapshot 3

EBS Volume

S3Block 1

Block 2

Block 3

Block 4

Chunk 1

Chunk 2

Chunk 3

Chunk 4

X©2011 Amazon Web Services May not be reused or redistributed without permission

Things You Should Know… (Part I)

EBS is a network resource Networks have variable performance

Striping will improve performance, although it’s usually fine to start with 4-8 drives per logical volume is observed sweet spot Great approach for high-performance database workloads

EBS is already redundant behind the scenes. It makes no sense to implement RAID on top of EBS


Things You Should Know… (Part II)

Lazy loads from snapshots When you create a volume, space is allocated immediately Data from snapshot written to disk on 1st request for maximum

performance For maximum IO performance, run dd to touch all blocks

Like all of AWS EBS is constantly innovating.These tips may not be appropriate in a few months


Get your data into AWS faster - load it onto a portable storage device and ship it to an Amazon data center

Faster than Internet transfer and more cost effective than upgrading your connectivity

Use cases: data migration, offsite backup, direct data interchange, disaster recovery

AWS Import/Export

Easy-to-use content delivery networkSame pay-as-you-go pricing as all of AWS– No up-front contracts– No long-term commitments– Self-service sign-up

Uses Amazon S3 as the origin storeWorldwide network of edge locations– Seattle, Palo Alto, Los Angeles, St. Louis, Dallas,

Ashburn, Newark, Miami, Amsterdam, Dublin, Frankfurt, London, Hong Kong, Tokyo

Amazon CloudFront

Technology

Aws introduction tocloudstorage