Moving to DynamoDB: Practical Lessons Learned

Ryan ParkInfrastructure Engineer

@StanfordRyan

Build Better APIs, Togetherwith Runscope API Monitoring & Testing

API TRAFFIC DEBUGGING

Quickly diagnose integration headaches.

Verify that your APIs are returning the right data.

DATA VALIDATION

UPTIME MONITORING

Be the first to know when an API is down.

Solve API problems as a team.

COLLABORATION

50 MM

100 MM

Jan 2014 Mar May Jul Sep Nov Jan 2015 Mar May Jul Sep Nov

50 MM

100 MM

Jan 2014 Mar May Jul Sep Nov Jan 2015 Mar May Jul Sep Nov

Monthly API Test Runs

100 MM

200 MM

300 MM

400 MM

500 MM

Jan 2014 Mar May Jul Sep Nov Jan 2015 Mar May Jul Sep Nov

100 MM

200 MM

300 MM

400 MM

500 MM

Jan 2014 Mar May Jul Sep Nov Jan 2015 Mar May Jul Sep Nov

Cumulative Test Runs(database rows)

What now?

Some results are huge

PROBLEM

SOLUTION

DynamoDB (record size < 10 KB)

Amazon S3 (record size > 10KB)

+

Hash Key: Result ID Global Secondary Index: Test ID

REVERSE INDEX IN DYNAMODB

s3_index

Selecting a Partition Key and Sort Key

Partition Key: Test ID

TEST RESULTS: THE OBVIOUS CHOICE

Sort Key: Result ID

HOW MANY PARTITIONS WILL WE HAVE?

EXAMPLE: SMALL, BUSY TABLE

Table Size 6 GB

Read Capacity Units 15,000

Write Capacity Units 2,000

Partitions:

EXAMPLE: LARGE TABLE

Table Size 380 GB

Read Capacity Units 2,100

Write Capacity Units 400

Partitions:

EXAMPLE: BULK IMPORT

Table Size 0 GB

Read Capacity Units 30

Write Capacity Units 10,000

Partitions:

EXAMPLE: BULK EXPORT & BACKUP

Table Size 4 GB

Read Capacity Units 30,600

Write Capacity Units 200

Partitions:

🦄 Partition

🐱 Partition 🐝 Partition

data hits 10 GB or

provisioned capacity hits limits

🐙 Partition 🐒 Partition

data hits 10 GB or

provisioned capacity hits limits

🐱 Partition 🐝 Partition

🐝 Partition

data hits 10 GB or

provisioned capacity hits limits

🐔 Partition 🐳 Partition

🐙 Partition 🐒 Partition

🐙 Partition 🐒 Partition 🐔 Partition 🐳 Partition

HOW MANY PARTITIONS?

Usually a power of 2.

TABLE PARTITIONS

test_results

Partition Reads Writes Hash Keys

( 200 100 α, δ, η, κ…

🕵 200 100 β, ε, θ, λ…

* 200 100 γ, ζ, ι, μ…

TABLE PARTITIONS

test_results

(α

δ

η

κ

🕵β

ε

θ

λ

*γ

ζ

ι

μ

200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU

TABLE PARTITIONS

test_results

(α 1 α 2

δ 1 δ 2

🕵β 1 β 2

ε 1 ε 2

*γ 1 γ 2

ζ 1 ζ 2

200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU

TABLE PARTITIONS

test_results

(α 1 α 2 α 3 α 4

δ 1 δ 2

🕵 *γ 1 γ 2

ζ 1 ζ 2 ζ 3 ζ 4 ζ 5 ζ 6 ζ 7

β 1

ε 1

200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU

200 RCU, 100 WCU

TABLE PARTITIONS

test_results

Test 1, Result 1 Test 1, Result 2 Test 1, Result 3 Test 1, Result 4

Test 5, Result 1 Test 5, Result 2

Test 4, Result 1

Test 7, Result 1

Test 3, Result 1 Test 3, Result 2

Test 12, Result 1 Test 12, Result 2 Test 12, Result 3 Test 12, Result 4 Test 12, Result 5 Test 12, Result 6 Test 12, Result 7 Test 12, Result 8200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU

200 RCU, 100 WCU

Test 1, Result 1 Test 1, Result 2 Test 1, Result 3 Test 1, Result 4

Test 5, Result 1 Test 5, Result 2

Test 4, Result 1

Test 7, Result 1

Test 3, Result 1 Test 3, Result 2

Test 12, Result 1 Test 12, Result 2 Test 12, Result 3 Test 12, Result 4 Test 12, Result 5 Test 12, Result 6 Test 12, Result 7 Test 12, Result 8

ProvisionedThroughputExceededException

200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU

200 RCU, 100 WCU

Test 1, Result 1 Test 1, Result 2 Test 1, Result 3 Test 1, Result 4

Test 5, Result 1 Test 5, Result 2

Test 4, Result 1

Test 7, Result 1

Test 3, Result 1 Test 3, Result 2

Test 12, Result 1 Test 12, Result 2 Test 12, Result 3 Test 12, Result 4 Test 12, Result 5 Test 12, Result 6 Test 12, Result 7 Test 12, Result 8

ProvisionedThroughputExceededException

200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU

200 RCU, 100 WCU

Test 1, Result 1 Test 1, Result 2 Test 1, Result 3 Test 1, Result 4

Test 5, Result 1 Test 5, Result 2

Test 4, Result 1

Test 7, Result 1

Test 3, Result 1 Test 3, Result 2

Test 12, Result 1 Test 12, Result 2 Test 12, Result 3 Test 12, Result 4 Test 12, Result 5 Test 12, Result 6 Test 12, Result 7 Test 12, Result 8200 RCU, 600 WCU 200 RCU, 600 WCU 200 RCU, 600 WCU

SHORT TERM FIX: OVER-PROVISION ALL PARTITIONS

test_results

200 RCU, 100 WCU

Test 1, Result 1 Test 1, Result 2 Test 1, Result 3 Test 1, Result 4

Test 5, Result 1 Test 5, Result 2

Test 4, Result 1

Test 7, Result 1

Test 3, Result 1 Test 3, Result 2

Test 12, Result 1 Test 12, Result 2 Test 12, Result 3 Test 12, Result 4 Test 12, Result 5 Test 12, Result 6 Test 12, Result 7 Test 12, Result 8

SHORT TERM FIX: OVER-PROVISION ALL PARTITIONS

test_results

200 RCU, 600 WCU 200 RCU, 600 WCU 200 RCU, 600 WCU

Result 58 (Test 7)

Result 55 (Test 4)

LONG TERM FIX: REPLACE THE HASH KEY

Result 41 (Test 7)

Result 8 (Test 3)

Result 3 (Test 9)

Result 37 (Test 7)

Result 73 (Test 7)

Result 87 (Test 6)

Result 27 (Test 3)

Result 63 (Test 7)

Result 85 (Test 2)

Result 78 (Test 7)

test_results_v2

200 RCU, 100 WCU 200 RCU, 100 WCU 200 RCU, 100 WCU

DEBUGGING EXCEPTIONS

AWS APIs return useful debugging data:

"ConsumedCapacity": { "CapacityUnits": 3.0, "TableName": "test_results", "Table": { "CapacityUnits": 3.0 } }

DEBUGGING EXCEPTIONS

AWS APIs return useful debugging data:

Content-Type: application/x-amz-json-1.0 Date: Wed, 27 Jan 2016 02:31:20 GMT X-Amzn-Requestid: S0JI33OC2OPW73KHD028LJWY9W5...

DEBUGGING EXCEPTIONS

Libraries hide the details.

DEBUGGING EXCEPTIONS

RUNSCOPE TRAFFIC INSPECTOR?

“Quickly diagnose integration headaches.”

USING BOTOCORE HOOKS

100 MM

200 MM

300 MM

400 MM

500 MM

Jan 2014 Mar May Jul Sep Nov Jan 2015 Mar May Jul Sep Nov

100 MM

200 MM

300 MM

400 MM

500 MM

Jan 2014 Mar May Jul Sep Nov Jan 2015 Mar May Jul Sep Nov

Cumulative Test Runs(database rows)

Everything is going to be

200 OK