© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Guy Farber, AWS Business Development
1/27/2016
Cloud Data Migration: 6 Strategies for Getting Data into AWS
When do we need to migrate data?
• Migrate applications to AWS• Build hybrid model• Deploy disaster recovery site
in an AWS region• Backup and archive into
Amazon S3, Glacier• Load data into AWS analytics
services, such as EMR and Redshift.
Where do we migrate to?
Amazon S3Durable object
storage for all types of data
Amazon EBSBlock storage for use
with Amazon EC2
Amazon GlacierArchival storage for infrequently accessed data
Economics Easy to Use Reduce risk Agility, Scale
Pay as you go
No upfront investmentNo commitment
No risky capacity planning
Self service administration
SDKs for simple integration
Durable and Secure
Avoid risks of physical media handling
Reduce time to market
Focus on your business, not your infrastructure
Amazon EFSFile storage for use with Amazon EC2
Roadmap to this Webinar: Transport and Ingest Services
AWS Direct Connect
What is AWS Direct Connect…
Dedicated, 1 or 10 GE private pipes into AWS
Create private (VPC) or public virtual interfaces to AWS
Reduced data-out rates (data-in still free)
Consistent network performance
At least 1 location to each AWS region
Option for redundant connections
Uses BGP to exchange routing information over a VLAN
Physical Connection
• Cross Connect at the location
• Single Mode Fiber - 1000Base-LX or 10GBASE-LR
• Potential onward Delivery via Direct Connect Partner
• Customer Router
At the Direct Connect Location
CORP
AWS DirectConnect Routers
Customer Router
Colocation
DX Location
CustomerNetwork`
AWS BackboneNetwork
Cross Connect
Customer Router
Customers Network
Demarcation
Dedicated Port via Direct Connect Partner
CORP
AWS DirectConnect Routers
Colocation
DX Location
Partner Network
AWS BackboneNetwork
Cross Connect
Customer Router
PartnerNetwork
AccessCircuit
Demarcation
PartnerEquipment
Direct Connect - LocationsAWS Region AWS Direct Connect LocationAsia Pacific (Singapore) Equinix SG2, GPX, MumbaiAsia Pacific (Seoul) KINX, SeoulAsia Pacific (Sydney) Equinix SY3, Global SwitchAsia Pacific (Tokyo) Equinix OS1, Equinix TY2China (Beijing) Sinnet JiuXianqiao IDC, CIDS Jiachuang IDCEU (Frankfurt) Equinix FR5, Interxion FrankfurtEU (Ireland) TelecityGroup, London Docklands’, Eircom Clonshaugh
Equinix LD4 - LD6, LondonSouth America (Sao Paulo) Terremark NAP do Brasil, TivitUS East (Virginia) CoreSite NY1 & NY2, Equinix DC1 - DC6 & DC10US West (Northern California)
CoreSite One Wilshire & 900 North Alameda, CA, Equinix SV1 & SV5
US West (Oregon) Equinix SE2 & SE3, Switch SUPERNAP, Las VegasAWS GovCloud (US) Equinix SV1 & SV5
Deep Dive – AWS Direct Connect
Re:Invent 2015 Session - Deep Dive: AWS Direct Connect and VPNs (NET406)
Youtube - https://www.youtube.com/watch?v=SMvom9QjkPk
SlideShare - http://www.slideshare.net/AmazonWebServices/net406-deep-dive-aws-direct-connect-and-vpns
Service details and pricing - https://aws.amazon.com/directconnect/details/
AWS Import/Export Services
AWS Import/Export Disk
• Accelerates moving large amounts of data into and out of Amazon S3, Glacier and EBS
• Transfers your data directly onto and off of customer owned storage devices
• Uses Amazon high-speed internal network to complete the transfer
• Supports up to eSATA and USB 2,3 attached drives up to 6 TBs or 16 TB arrays
AWS Import/Export
What is Snowball? Petabyte scale data transport
E-ink shipping label
Ruggedizedcase
“8.5G Impact”
All data encrypted end-to-end
Rain & dust resistant
Tamper-resistant case & electronics
50 TB10GE network
How it works
How fast is Snowball?• Less than 1 day to transfer 250TB via 5x10G connections with 5
Snowballs, less than 1 week including shipping• Number of days to transfer 250TB via the Internet at typical utilizations
Internet Connection SpeedUtilization 1Gbps 500Mbps 300Mbps 150Mbps
25% 95 190 316 63250% 47 95 158 31675% 32 63 105 211
When to use AWS Import/Export Snowball
Cloud Migration
Disaster Recovery
DatacenterDecommission
ContentDistribution
AWS Snowball AWS Import/Export Disk
When to use Disk vs Snowball?
• Import only, Export coming soon
• Currently available in US East and US West 2
• Import to S3 only• Supports large data transfers,
from TBs to PBs
Supports import and export for S3 buckets and EBS snapshot import in:
• US East (N. Virginia)• US West (Oregon)• US West (Northern California)• EU (Ireland)• Asia Pacific (Singapore)
Supports import into Glacier in:• US East (N. Virginia)• US West (Oregon)• US West (Northern California)• EU (Ireland) regions.
Deep Dive – AWS Import/Export
Re:Invent 2015 Session - AWS Import/Export Snowball: Large-Scale Data Ingest into AWS (STG202)
Youtube - https://www.youtube.com/watch?v=86ogJHFSJRo
SlideShare - http://www.slideshare.net/AmazonWebServices/stg202-aws-importexport-snowball-largescale-data-ingest-into-aws
Service details and pricing - https://aws.amazon.com/importexport/
AWS Storage Gateway
What is AWS Storage Gateway?
Works with your existing applications
Secure and durable storage in AWS
Low-latency for frequently used data
Scalable and cost-effective on-premises storage - $125 per gateway per month + S3/Glacier storage fees
Service connecting an on-premises software appliance with cloud-based storage
Common uses for AWS Storage Gateway
Backup and archive
Disaster recovery
Data migration
How does AWS Storage Gateway work?
Amazon EBS snapshots
Amazon S3Amazon Glacier
AWSStorage Gateway
appliance
Applicationserver
AWSStorage Gateway
backend
AWSDirect
Connect
Internet
Customer premises
AWS Storage Gateway configurations
iSCSI block storage
Gateway-stored volumes
iSCSI virtual tape storage
Low-latency for all your data with point-in-time backups to AWS
Replacement for on-premises physical tape infrastructure for backup and archive
Gateway-cached volumes
Gateway-virtual tape library (VTL)
Low-latency for frequently used data with all data stored in AWS
Gateway-virtual tape library (VTL)• Replace or augment your aging tape infrastructure with durable object
storage• Virtual tapes stored in AWS. Frequently accessed data cached on-premises• Up to 1,500 tapes, up to 2.5 TB each, for up to 150 TB per gateway-VTL• Unlimited number of tapes in virtual tape shelf (VTS)
Customer data center
VTS storage backed by Amazon Glacier
AWS Storage Gateway VM
BackupServer IN
ITIA
TOR
AWSStorage Gateway
service
MED
IA
CHAN
GER
UploadBuffer
CacheStorage
Gateway-VTLstorage backedby Amazon S3
VTS
TAPE
DR
IVE
Deep Dive – AWS Storage Gateway
Re:Invent 2015 Session - AWS Storage Gateway Deep Dive (STG311)
Youtube - https://www.youtube.com/watch?v=VmjDfz-MIZE
SlideShare - http://www.slideshare.net/AmazonWebServices/stg311-aws-storage-gateway-secure-costeffective-backup-archive
Service details and pricing - https://aws.amazon.com/storagegateway/
AWS Technology Partnerships
Amazon Storage Partner Ecosystem
Backup to AWS Approaches
Amazon S3
Amazon GlacierAWS
DirectConnect
InternetAmazon S3-IA
Applicationservers
Cloud Gateway
Local disk
MediaServer
Cloud Gateway
HTTPS/API
Applicationservers
Backup SW cloud connector
Local diskMedia
Server with cloud
connector
HTTPS/API
CommVault Ties Together On Premise and Cloud Data StrategiesCommvault Orchestrates the Enterprise
• Back up in the Cloud: Keep backups of cloud workloads internal to the cloud
• Back up to the Cloud: Allow on premise workloads the ability to leverage AWS
• Disaster Recovery to the Cloud: Automate disaster recovery to the cloud on a scheduled basis
• Workload Portability: Rest assured that virtual servers can be moved from on-premise to the cloud and back, keep your data available wherever you need it
• Archiving to the Cloud: Moving legacy data to tier 2 storage in the cloud for long term archive
Centralized and Simple Management
AWS VPC Data Center
AWS and Commvault together combine to minimize networking, storage and infrastructure
costs, while providing the business a sound data protection and disaster recovery strategy.
Amazon S3Amazon Glacier
Backup to AWS Approaches
Amazon S3
Amazon GlacierAWS
DirectConnect
InternetAmazon S3-IA
Applicationservers
Cloud Gateway
Local disk
MediaServer
Cloud Gateway
HTTPS/API
Applicationservers
Backup SW cloud connector
Local diskMedia
Server with cloud
connector
HTTPS/API
NetApp AltaVault Backup from On-premises to S3/Glacier
Common backup applications integrated with AltaVaultSolve backup & archive headaches with cloud-integrated storage 90% reduction in time, cost, and data volumes Shrink recovery times from days to minutes 85% of backup & software providers supported
Glacier
On Premises
AWS
Cloud-integrated storage appliance
NetApp AltaVault
FAS
E-SeriesNon-NetApp
Storage
Seamlessly integrates into existing storage and backup
software environment
Deduplicates, compresses, and encrypts
Caches recent backups locally, vaults older copies to
the cloud
Store data in the public or private cloud of choice
NetApp SnapProtect Arcserve CommVault Simpana EMC NetWorker HP Data Protector IBM Tivoli
Storage Mgr
Symantec Backup Exec
Symantec NetBackup
Veeam Microsoft SQL
Server Oracle RMAN
S3
AltaVault also available on marketplace to protect cloud-native workloads
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
unified file services that extend from endpoints, to
remote offices, to the cloud.
snapshots, file versioning and file sync runs across all access points via the
cloud
data is secured and optimized at
the source
all stored in your AWS VPC, data is stored on AWS S3-
IA
Integrated with trusted enterprise security and
management tools
ROBO NAS Gateways
Endpoint Apps
Cloud Server Agents
Data ProtectionEngine
File SyncEngine
with centralized automation, management and multi-tenancy
Identitymanagement
datagovernance
cloudorchestration
S3 Infrequent Access
CTERA GlobalDeduplication
Ctera: Enterprise File Services Platform
AWS Kinesis Firehose
Amazon Kinesis PlatformAmazon Kinesis streaming data on the AWS cloud• Amazon Kinesis Streams• Amazon Kinesis Firehose • Amazon Kinesis Analytics
Amazon Kinesis FirehoseLoad massive volumes of streaming data into Amazon S3 and Amazon Redshift
Zero administration: Capture and deliver streaming data into S3, Redshift, and other destinations without writing an application or managing infrastructure.
Direct-to-data store integration: Batch, compress, and encrypt streaming data for delivery into data destinations in as little as 60 secs using simple configurations.
Seamless elasticity: Seamlessly scales to match data throughput w/o intervention
Capture and submit streaming data to Firehose
Firehose loads streaming data continuously into S3 and Redshift
Analyze streaming data using your favorite BI tools
Vertical/Use Case Accelerated Ingest-Load to final destination for Analytics
Ad Tech/ Marketing Analytics
Advertising data aggregation
Consumer Online/Gaming
Online customer engagement data aggregation
Financial Services Market/ Financial Transaction order data collection
IoT / Sensor Data Fitness device , vehicle Sensor, telemetry data ingestion
Amazon Kinesis FirehoseAmazon Kinesis Firehose Use Cases
Deep Dive – AWS Kinesis Firehose
Re:Invent 2015 Session - Streaming Data Flows with Amazon Kinesis Firehose (BDT320)
Youtube - https://www.youtube.com/watch?v=lkRoQlhWDXA
SlideShare - http://www.slideshare.net/AmazonWebServices/bdt320-new-streaming-data-flows-with-amazon-kinesis-firehose
Service details and pricing - https://aws.amazon.com/kinesis/firehose/
Summary – When to Use each ServiceIF YOU NEED: CONSIDER:
An optimized or replacement Internet connection to:
connect directly into an AWS regional datacenter Direct Connect
migrate TB or PB of data to the cloud Import/Export Snowball
migrate GB of data over a <10Mbps network Import/Export Disk
A friendly interface into S3 to: cache data locally in a hybrid model (for performance reasons)
Gateways (AWS or Partner)
redirect backups or archives with minimal disruption
Technology Partnerships
aggregate data streams from multiple devices Kinesis Firehose
Thank you!
Recommended
