67

Serverless · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

  • Upload
    lamdien

  • View
    222

  • Download
    2

Embed Size (px)

Citation preview

Page 1: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 2: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Roger S. Barga, Ph.D.

General Manager

Amazon Web Services

Serverless ComputingRedefining the Cloud

Page 3: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Technology Triggers

Page 4: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 5: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Highly Recommended http://a16z.com/2016/12/16/the-end-of-cloud-computing/

Page 6: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Serverless

Page 7: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Amazon Kinesis FirehoseLoad massive volumes of streaming data into S3, Redshift, Elasticsearch,…

• Zero administration: Capture and deliver streaming data into Amazon S3, Amazon Redshift, and

other destinations without writing an application or managing infrastructure.

• Direct-to-data store integration: Batch, compress, and encrypt streaming data for delivery into

data destinations in as little as 60 secs using simple configurations.

• Elastic: Scales to match data throughput w/o intervention

• Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform

incoming source data.

Capture and submit

streaming data

Analyze streaming data using your

favorite BI tools Firehose loads streaming data

continuously into Amazon S3, Redshift

and Elasticsearch

Page 8: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Amazon Kinesis Analytics

• Apply SQL on streams: Easily connect to a Kinesis Stream or Firehose Delivery Stream and apply SQL skills.

• Build real-time applications: Perform continual processing on streaming data with sub-second processing latencies using ANSI SQL

• Automatics Scalability : Serverless, elastically scales to match data throughput.

Connect to Kinesis streams,

Firehose delivery streamsRun standard SQL queries

against data streams

Kinesis Analytics can send processed data to

analytics tools so you can create alerts and

respond in real-time

Page 9: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 10: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 11: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 12: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 13: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 14: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 15: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 16: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 17: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Provisioning and Utilization

Availability and Fault Tolerance

Scaling

Operations and Management

Page 18: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 19: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 20: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 21: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 22: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 23: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 24: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Provisioning and Utilization

Availability and Fault Tolerance

Scaling

Operations and Management

Page 25: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Provisioning and Utilization

Availability and Fault Tolerance

Scaling

Operations and Management

Page 26: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Provisioning and Utilization

Availability and Fault Tolerance

Scaling

Operations and Management

Page 27: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 28: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 29: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 30: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 31: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 32: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 33: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Common Use Cases

Web Applications

• Static websites

• Complex web apps

• Packages for Flask and Express

Big Data

• Real time

• MapReduce

• Batch

Chatbots

• Powering chatbot logic

Backends

• Apps & services

• Mobile

• IoT

</></>

Amazon Alexa

• Powering voice-enabled apps

• Alexa Skills Kit

IT Automation

• Policy engines

• Extending AWS services

• Infrastructure management

Page 34: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Common Use Cases

Web Applications and Backends

Internet

Mobile Apps

Websites

Services

AWS Lambda

function

AWS

API Gateway

Cache

All publicly

accessible endpoints

Amazon

CloudWatch Monitoring

Amazon

CloudFront

AmazonS3AWS Lambda

function

AWS Lambda

functionAmazon DynamoDB

Web Applications

• Static websites

• Complex web apps

• Packages for Flask and Express

Backends

• Apps & services

• Mobile

• IoT

</></>

Page 35: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Task

Choice

Fail

Parallel

Lambda + Step FunctionsImage Recognition and Processing Backend

Common Use Cases

Web Applications

• Static websites

• Complex web apps

• Packages for Flask and Express

Backends

• Apps & services

• Mobile

• IoT

</></>

Page 36: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Serverless IoT with AWS Lambda at iRobot

Page 37: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

AWSLambda

AmazonCloudFront

AmazonS3

AmazonDynamoDB

Amazon Redshift

AmazonRoute 53 Amazon VPC

AmazonCloudWatch

AWSCloudFormation

AWSCloudTrail

AWSConfig AWS IAMAWS KMSAWS WAF

Amazon Cognito AmazonSNS

Amazon APIGateway

AmazonSQS

AmazonElasticsearch Service

AmazonKinesis

AmazonQuickSight

AWS IoT

How iRobot leverages AWS

Page 38: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Serverless is Distributed by Nature

• Component graph becomes call graph

• Distributed systems thinking is required from the start

• Event-based architecture

Page 39: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Common Use Cases

Web Applications

• Static websites

• Complex web apps

• Packages for Flask and Express

Data Processing

• Real time

• MapReduce

• Batch

Chatbots

• Powering chatbot logic

Backends

• Apps & services

• Mobile

• IoT

</></>

Amazon Alexa

• Powering voice-enabled apps

• Alexa Skills Kit

IT Automation

• Policy engines

• Extending AWS services

• Infrastructure management

Page 40: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Common Use Cases

Web Applications

• Static websites

• Complex web apps

• Packages for Flask and Express

Page 41: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 42: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 43: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Common Use Cases

Web Applications

• Static websites

• Complex web apps

• Packages for Flask and Express

Data Processing

• Real time

• MapReduce

• Batch

Chatbots

• Powering chatbot logic

Backends

• Apps & services

• Mobile

• IoT

</></>

Amazon Alexa

• Powering voice-enabled apps

• Alexa Skills Kit

IT Automation

• Policy engines

• Extending AWS services

• Infrastructure management

Lambda + S3

Page 44: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Common Use Cases

Data Processing

• Real time

• MapReduce

• Batch

Lambda + Kinesis + DynamoDB

Page 45: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Common Use Cases

Data Processing

• Real time

• MapReduce

• Batch

Lambda + DynamoDB + Redshift

Page 46: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

What to expect from the session

15-20 minutes of processing now in seconds2x order of magnitude for cost savings

https://www.youtube.com/watch?v=TXmkj2a0fRE

Nordstrom Recommendations

Page 47: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Common Use Cases

Web Applications

• Static websites

• Complex web apps

• Packages for Flask and Express

Data Processing

• Real time

• MapReduce

• Batch

Chatbots

• Powering chatbot logic

Backends

• Apps & services

• Mobile

• IoT

</></>

Amazon Alexa

• Powering voice-enabled apps

• Alexa Skills Kit

IT Automation

• Policy engines

• Extending AWS services

• Infrastructure management

Page 48: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Common Use Cases

Web Applications

• Static websites

• Complex web apps

• Packages for Flask and Express

Data Processing

• Real time

• MapReduce

• Batch

Chatbots

• Powering chatbot logic

Backends

• Apps & services

• Mobile

• IoT

</></>

Amazon Alexa

• Powering voice-enabled apps

• Alexa Skills Kit

IT Automation

• Policy engines

• Extending AWS services

• Infrastructure management

Page 49: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 50: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 51: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 52: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 53: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 54: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 55: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 56: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 57: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 58: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 59: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 60: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 61: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 62: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 63: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 64: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 65: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 66: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture
Page 67: Serverless  · PDF file•Serverless ETL using AWS Lambda - Firehose can invoke your Lambda function to transform ... and Elasticsearch. ... •Event-based architecture

Takeaways

Serverless is a Fundamental Component of Modern Applications

• Many enterprise applications can go serverless

• Move to event driven computing

The ecosystem continues to grow

• Tooling, languages, and application capabilities

• But we still have a long ways to go…

Serverless and Edge are technology triggers with the potential to

reshape distributed computing and the role of cloud computing