Cloud Computing With AWS

© 2008 Wipro Ltd - Confidential

Cloud Computing

With Amazon Web Service

Munish K Gupta


Cloud ComputingWhat

2April 10, 2023 2© 2008 Wipro Ltd - Confidential


What is Cloud Computing ?

• Cloud computing refers to the use of Internet ("cloud") based computer technology for a variety of services. It is a style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet. Users need not have knowledge of, expertise in, or control over the technology infrastructure "in the cloud" that supports them - Wikipedia

CLOUD CLOUD


Cloud Computing Comparisons

• Cloud computing is often confused with grid computing, utility computing and autonomic computing.


What is Grid Computing ?

• Multiple independent computing clusters which act like a “grid” because they are composed of resource nodes not located within a single administrative domain

• Grid computing depends on software to divide and apportion pieces of a program among several computers, sometimes up to many thousands.

• Grid computing can also be thought of as distributed and large-scale cluster computing, as well as a form of network-distributed parallel processing.

* Source - Wikipedia


What is Utility Computing ?

• Utility computing is the packaging of computing resources, such as computation and storage, as a metered service similar to a traditional public utility (such as electricity, water, natural gas, or telephone network).

• “Utility computing” usually envisions some form of virtualization so that the amount of storage or computing power available is considerably larger than that of a single time-sharing computer. Multiple servers are used on the “back end” to make this possible. These might be a dedicated computer cluster specifically built for the purpose of being rented out, or even an under-utilized supercomputer. The technique of running a single calculation on multiple computers is known as distributed computing.



What is autonomic computing ?

• Autonomic Computing is an initiative started by IBM in 2001. Its ultimate aim is to develop computer systems capable of self-management, to overcome the rapidly growing complexity of computing systems management, and to reduce the barrier that complexity poses to further growth.

• An autonomic system makes decisions on its own, using high-level policies; it will constantly check and optimize its status and automatically adapt itself to changing conditions.



Cloud Computing as of 2009

• The majority of cloud computing infrastructure consists of reliable services delivered through data centers and built on servers with different levels of virtualization technologies. The services are accessible anywhere in the world, with The Cloud appearing as a single point of access for all the computing needs of consumers.

• Many cloud computing deployments depend on grids, have autonomic characteristics and bill like utilities but cloud computing can be seen as a natural next step from the grid-utility model



Cloud ComputingWhy

April 10, 2023 9© 2009 Wipro Ltd - Confidential


Web Application Pains

• 70% of Web development work is all about– Data Centers– Bandwidth/Power/Cooling– Operations– Staffing

• Application Scaling is difficult and expensive– Large up front

investment– Invest ahead of time– Load is unpredictable

(seasonal spikes)


Why Cloud Computing ?

• Almost zero upfront infrastructure investment

• Just-in-time Infrastructure

• More efficient resource utilization

• Usage-based costing• Potential for shrinking

the processing time• Device and location

independence• ilities (Reliability,

Scalability, Sustainability, Maintainability)


Cloud ComputingWhere



Where to use Cloud Computing ?

Examples of applications that could utilize the power of Cloud

• Processing Pipelines– Document processing pipelines – convert documents from Microsoft Word to PDF,

OCR– Image processing pipelines - create thumbnails of an image, resize images– Video transcoding pipelines - transcode AVI to– MPEG movies– Indexing – create an index of web crawl data– Data mining – perform search over millions of records

• Batch Processing Systems– Back-office applications (in financial, insurance or retail sectors)– Log analysis – analyze and generate daily/weekly reports– Nightly builds – perform nightly automated builds of source code repository every

night in parallel– Automated Unit Testing and Deployment Testing – Test and deploy and perform

automated unit testing (functional, load, quality) on different deployment configurations every night

• Websites– Websites that “sleep” at night and auto-scale during the day– Instant Websites – websites for conferences or events (Super Bowl, sports

tournaments)– Promotion websites– “Seasonal Websites” - websites that only run during the tax season or the holiday

season(“Black Friday” or Christmas)


Cloud ComputingAmazon Web Service (AWS) > Infrastructure Services



Amazon Web Service (AWS)

• What is Amazon Web Service – A set of APIs and business models which give

developer-level access to Amazon’s infrastructure and content


AWS Products

• AWS provides the following

We will look at the Infrastructure Service as these form the core of the Cloud Computing infrastructure provided by AWS

Payments and Billing ServicesAmazon Flexible Payments Service

Amazon DevPay

Alexa Web ServicesAlexa Web Information Service

Alexa Top Sites

Alexa Site Thumbnail

Infrastructure ServicesAmazon Elastic Compute Cloud

Amazon Simple DB

Amazon Simple Storage Service

Amazon CloudFront

Amazon Simple Queue Service On Demand WorkforceAmazon Mechanical Turk

Fulfillment & AssociatesAmazon Fulfillment Service

Amazon Associate Web Service


AWS - Infrastructure Services

• Amazon Elastic Compute Cloud (Amazon EC2) – A web service that provides resizable compute capacity in the cloud. One can configure an Amazon Machine Instance (AMI) and load it into the Amazon EC2 service. Allows to quickly scale capacity, both up and down, as your computing requirements change.

• Amazon SimpleDB – A web service for running queries on structured data in real time. This service works in close conjunction with Amazon S3 and Amazon EC2, collectively providing the ability to store, process and query data sets in the cloud

• Amazon Simple Storage Service (Amazon S3) – A simple web services interface that can be used to store and retrieve large amounts of data, at any time, from anywhere on the web.

• Amazon CloudFront – A web service for content delivery. It integrates with other Amazon Web Services to provide an easy way to distribute content to end users with low latency and high data transfer speeds.

• Amazon Simple Queue Service (Amazon SQS) – A reliable, highly scalable, hosted queue for storing messages as they travel between computers. By using Amazon SQS, developers can simply move data between distributed components of their applications that perform different tasks, without losing messages or requiring each component to be always available.


Amazon Elastic Compute Cloud (Amazon EC2)

What > Concepts>Setup >API



Amazon Elastic Compute Cloud (Amazon EC2)

• Amazon EC2 is a web service that enables you to launch and manage server instances in Amazon's data centers using APIs or available tools and utilities

• Instances are available in different sizes and configurations• For example, one can use an m1.small instance (one

Amazon EC2 Compute Unit) as a web server, an m1.xlarge instance (eight Amazon EC2 Compute Units) as a database server, or an extra large High-CPU instance (twenty Amazon EC2 Compute Units) for processor intensive applications

• All the instances can be managed using the Web Service APIs

This is akin to virtualization concept (albeit as a Web Service)


Concepts Amazon EC2

• Amazon Machine Image (AMI):– Bootable root disk– Pre-defined or user-built– Catalog of user-built AMIs– OS: Fedora, Centos, Gentoo, Debian,

Ubuntu, Windows Server– App Stack: LAMP, mpiBLAST, Hadoop

• Instance:– Running copy of an AMI– Launch in less than 2 minutes– Start/stop programmatically

• Network Security Model:– Explicit access control– Security groups

• Inter-service bandwidth is free


Setting up Amazon EC2

1. One can create an AMI (Amazon Machine Images) from scratch or use an existing AMI

2. Next, you bundle the AMI and obtain an AMI ID so one can launch as many instances of the AMI

3. Launch one or more instances of your AMI 4. One can administer and use your instances and you would

with any servers but making use of Web Services


Amazon EC2 SOAP API

• AllocateAddress• AssociateAddress• AttachVolume• AuthorizeSecurityGroupIngress• BundleInstance• CancelBundleTask• ConfirmProductInstance• CreateKeyPair• CreateSecurityGroup• CreateSnapshot• CreateVolume• DeleteKeyPair• DeleteSecurityGroup• DeleteSnapshot• DeleteVolume• DeregisterImage• DescribeAddresses• DescribeAvailabilityZones• DescribeBundleTasks• DescribeImageAttribute

• DescribeImages• DescribeInstances• DescribeKeyPairs• DescribeRegions• DescribeSecurityGroups• DescribeSnapshots• DescribeVolumes• DetachVolume• DisassociateAddress• GetConsoleOutput• ModifyImageAttribute• RebootInstances• RegisterImage• ReleaseAddress• ResetImageAttribute• RevokeSecurityGroupIngress• RunInstances• TerminateInstances

The Amazon EC2 API consists of web service operations for every task the service can perform


Amazon EC2 At Work

• Startups– Cruxy – Media transcoding– GigaVox Media – Podcast Management

• Fortune 500 clients:– High-Impact, S hort-Term Projects– Development Host

• Science / Research:– Hadoop / MapReduce– mpiBLAST

• Load-Management and Load Balancing Tools:– Pound – Weogeo– Rightscale


Amazon SimpleDBWhat > Setup >API



Amazon SimpleDB

• Amazon SimpleDB is a web service for running queries on structured data in real time

• Provides the core functionality of a database - real-time lookup and simple querying of structured data - without the operational complexity

• Amazon SimpleDB requires no schema, automatically indexes your data and provides a simple API for storage and access. This eliminates the administrative burden of data modeling, index maintenance, and performance tuning


Setting up Amazon SimpleDB

• Using Amazon SimpleDB, one can organize the structured data in domains within which one can put data, get data, or run queries

• Domains consist of items which are described by attribute name-value pairs

• Regardless of how the data is stored, Amazon SimpleDB automatically indexes your data for quick and accurate retrieval


Amazon SimpleDB API

Amazon SimpleDB service consists of a small group of API calls that provide the core functionality one needs to build the application

• Create Domain—Create domains to contain your data; you can create up to 100 domains

• Delete Domain—Delete any of your domains• List Domains—List all domains within your account• Put—Add, modify, or remove data within your Amazon SimpleDB

domains• Delete—Remove items, attributes, or attribute values from your

domain• Get—Retrieve the attributes and values of any item ID that you

specify• Select—Query the specified domain using a SQL SELECT

expression• Query—Query the specified domain using an expression that

performs value tests on one or more attributes• QueryWithAttributes—Query the specified domain and retrieve

the results for matching items


Amazon Simple Storage Service (Amazon

S3) What > Concepts >API



Amazon Simple Storage Service

• Amazon S3 has a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web

• Amazon S3 provides a minimal feature set that focus on simplicity and robustness. Following are some of features of the Amazon S3 service:– Enables you to write, read, and delete objects from 1 byte

to 5 gigabytes in size with accompanying metadata. There is no fixed limit on the number of objects you can store.

– Provides a straightforward flat object store model, where each object is stored and retrieved using a unique developer-assigned key.

– Provides authentication mechanisms to ensure that data is kept secure from unauthorized access. Objects can be made private or public and rights can be granted to specific users.

– Uses standards-based REST and SOAP interfaces designed to work with any Internet-development toolkit.


Concepts Amazon S3

• Objects:– Opaque data to be stored (1 byte … 5 Gigabytes)– Authentication and access controls

• Buckets:– Object container – any number of objects– 100 buckets per account / buckets are “owned”

• Keys:– Unique object identifier within bucket– Up to 1024 bytes long– Flat object storage model

• Standards-Based Interfaces:– REST and SOAP– URL-Addressability – every object has a URL


Amazon S3 SOAP API

• Service:– ListAllMyBuckets

• Buckets:– CreateBucket– DeleteBucket– ListBucket– GetBucketAccessControlPolicy– SetBucketAccessControlPolicy– GetBucketLoggingStatus– SetBucketLoggingStatus

• Objects:– PutObject– PutObjectInline– GetObject– GetObjectExtended– DeleteObject– GetObjectAccessControlPolicy– SetObjectAccessControlPolicy


Amazon CloudFrontWhat > Setup >API



Amazon CloudFront

• Amazon CloudFront is a web service for content delivery

• CloudFront delivers your content through a worldwide network of edge locations. End users are routed to the nearest edge location, so content is delivered with the best possible performance.


Setting up Amazon CloudFront

• Objects - Objects are the files you want CloudFront to deliver. This typically includes web pages, images, and digital media files

• Origin Server - An origin server is the location where you store the original, definitive version of your objects. One can place any objects that are required to be delivered through CloudFront in the bucket.

• Distributions - After the objects are stored in the origin server, there is a link create between an Amazon S3 bucket (the origin server) and a domain name (which CloudFront automatically assigns)

• Edge Locations - An edge location is a geographical site where CloudFront caches copies of your objects.


Amazon CloudFront API

1. You create the Amazon S3 bucket that will be your origin server2. You place your objects in the bucket and make them publicly readable3. You create your CloudFront distribution and get the domain name that

CloudFront assigns.

Example distribution ID: PDFDVBD632BHDS5Example domain name: e604721fxaaqy9.cloudfront.net

The distribution ID will not necessarily match the domain name.4. You create the URLs that your end users will use to get your objects

and include them as needed in your web application or web site.Example URL: http://e604721fxaaqy9.cloudfront.net/images/image.jpg


Amazon Simple Queue ServiceWhat >Concept> Setup >API



Amazon Simple Queue Service (SQS)

• A queue is a temporary repository for messages that are awaiting processing

• Amazon SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component

• The queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing. This means the queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer are only intermittently connected to the network

• SQS ensures delivery of each message at least once, and supports multiple readers and writers interacting with the same queue.

• A single queue can be used simultaneously by many distributed application components, with no need for those components to coordinate with each other to share the queue


Concept Amazon SQS

• Redundant infrastructure—Guarantees delivery of the messages at least once, highly concurrent access to messages, and high availability for sending and retrieving messages

• Multiple writers and readers—Multiple parts of the system can send or receive messages at the same time. SQS locks the message during processing, keeping other parts of the system from processing the message simultaneously.

• Configurable settings per queue—All of the queues don't have to be exactly alikeFor example, one queue can be optimized for messages that require a longer processing time than others.

• Variable message size— the messages can be up to 8 KB in sizeUnlimited queues and messages—One can have as many queues and messages in the Amazon SQS system as you want


Setting up Amazon SQS

• There are three main actors in the overall system:– The components of your distributed system– Queues– Messages in the queues

• In the following diagram, your system has several components that send messages to the queue and receive messages from the queue. The diagram shows that a single queue, which has its messages (labeled A-E), is redundantly saved across multiple SQS servers.


Amazon SQS SOAP API

• Queues:– ListQueues– DeleteQueue– SetVisibilityTimeout– GetVisibilityTimeout

• Messages:– SendMessage– ReceiveMessage– DeleteMessage– PeekMessage

• Security:– AddGrant– ListGrants– RemoveGrant


Case StudyGIGAVOX


GIGAVOX

• Implemented Amazon S3, Amazon EC2 and Amazon SQS in November 2006

• Created an infinitely scalable infrastructure for less than $100 - building the same infrastructure themselves would have cost thousands of dollars

• Reduced staffing requirements - far less responsibility for 24x7 operations

© 2008 Wipro Ltd - Confidential© 2009 Wipro Ltd - Confidential

Munish K GuptaSenior Consultant

[email protected]

Thank You


Technology

Cloud Computing With AWS