SUMMITBerlin
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Making S3 more resilientusing Lambda@Edge
Júlia Biró, Yann HamonReliability TeamContentful
SessionID
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Agenda
Introduction
Proof of Concept
Going live
Improving our Lambda@Edge software platform
Review
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Our file delivery infrastructure
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Our goal: multi-region active-active
https://www.youtube.com/watch?v=2e29I3dA8o4
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Current state-of-the-art (?)
“Highly available multi region S3 website
Cloudfront distributions” - Derek Higgins (2017)
Could work but:
Failover solution
No guaranteed propagation time for configuration changes in Cloudfront
Manual reset
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
A possible solution... that doesn’t work
Use an origin with DNS Round-Robin?
Does not work:
The request's Host needs to match the name of the S3 bucket
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
There must be a way...
"Dynamically Route Viewer Requests
to Any Origin Using Lambda@Edge"
Jake Wells, AWS Blog (Nov. 2017)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
A/B Testing with Lambda@Edge
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Load-balancing with Lambda@Edge
dns.resolveCname('files-origin.contentful.com', (function(err, result) {
if (result[0].includes('us-east-1')) {
bucketName = 'cf-files.s3.us-east-1.amazonaws.com';
region = 'us-east-1';
else {
bucketName = 'cf-files.s3.us-west-2.amazonaws.com';
region = 'us-west-2';
}
request.origin.s3.region = region;
request.origin.s3.domain = bucketName;
request.headers['host'] = [{key: 'host', value: bucketName}];
}
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Technology validation
We build a small proof-of-concept. Only 20 lines of code!
Proof: different image with the same path in both regions.
Learnings:Our Javascript is not great - we don't do this every day!
We do a DNS resolution on every cache miss
us-east-1 us-west-2
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
DNS Caching
static async ResolveCname(fqdn) {
let now = Date.now();
let cachedEntry = this.cache[fqdn];
if (cachedEntry && now - cachedEntry.updatedAt < this.defaultTTL) {
return cachedEntry.answers;
}
[...]
DNS lookups add latency to our requests...
But we can cache the results [1]
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
It works!
Thank you for your attention.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Leap of faith
Our team is not used to writing Javascript
Lambda@Edge is a new technology to the company
High cost of failure
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
High cost of failure
contractual SLAs
100s of requests/second
no graceful degradation
We need safety gear.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Why do we trust our current software platform?
version control
dash-boards
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Production-readiness list
1. Solution-agnostic criteria2. Translated for the specific solution 3. Gap analysis
To add an image, select Click to insert image, and find the
image you want to use.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Production-readiness list is an ideal
No need to meet all requirements but..It needs to be a conscious and
documented decision
Our existing software platforms did
not meet all requirements
The goal is to reduce uncertainty and risk
RUNNIN' RHINO
Design by Allan Faustino
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Risk-aversion
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Going live… gradually
1. Attach the Lambda - new feature turned off completely 2. Feature-flag / dark release3. Whitelist for some internal test customers4. Gradually roll out to all traffic 5. Monitor at all steps
Big red button: ability to quickly revertat all time
Expect to meet unknowns.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Our new file delivery infrastructure
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Migrating a larger service to Lambda@Edge
Apply what we learned to our first Lambda project
Rewrite in Typescript
Backport all the tooling
Lambda@Edge becoming a first class citizen in our software platform
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Working with lambda@Edge
Near-immediate scale-up
Marginal costs
Highly available
But..Challenging development environment (esp. integration testing)
Logs saved in each region
Our deployment workflow still manual
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Working with lambda@Edge
TypeScript helped us write safer codewe also contributed to the Cloudfront specific package
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Future improvements
Use Geo-routing to forward requests to the closest S3 bucket
Run the DNS resolution outside of the main event loop
Automate deployments
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
A few months later...
Similar implementations emerged:Using Amazon CloudFront with Multi-Region Amazon S3 Origins (Sept. 30th, Seldam)
Amazon S3 Region Failover — Part 2: CloudFront S3 origin failover (Oct. 30th, Frias)
Cloudfront origin failover was introduced:Amazon CloudFront announces support for Origin Failover (Nov. 20th, AWS)
But…Current solution has proven cheap, fast and stable
Has laid the groundworks for other multi-region projects
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Takeaways
Lambda@Edge is a cheap, scalable & highly reliable platform to build stateless APIs
When the cost of failure is high, use production-readiness lists
Feature-flagging, canarying, gradual rollouts are easy to use with Lambda@Edge to reduce risk of large-scale changes
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT
Thank you!
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Júlia Biró, Yann [email protected], [email protected]