ABOUT NETFLIX!
NETFLIX
ACTIVE - ACTIVE!
WHAT IS ACTIVE ACTIVE?
Also called dual active, it is a phrase used to describe a network of independent processing nodes where each node has access to replicated database. Traffic intended for a failed node is either passed onto an existing node or load balanced across the remaining nodes.
WHY ACTIVE-ACTIVE ?!
ENTERPISE IT SOLUTIONS
WEB SCALE CLOUD
SOLUTIONS
RAPID SCALING
HIGH AVAILABILITY
DOES AN INSTANCE FAIL?!• It can, plan for it!• Bad code / configuration pushes!• Latent issues!• Hardware failure!• Test with Chaos Monkey!
DOES A ZONE FAIL?!• Rarely, but happened before!• Routing issues!• DC-specific issues!• App-specific issues within a zone!• Test with Chaos Gorilla!
DOES A REGION FAIL?!
• Full region – unlikely, very rare!• Individual Services can fail region-wide!• Most likely, a region-wide configuration issues!• Test with Chaos Kong!
EVERYTHING FAILS… EVENTUALLY!
• Keep your services running by embracing isolation and redundancy!
• Construct a highly agile and highly available service from ephemeral and assumed broken components!
ISOLATION!
• Changes in one region should not affect others!• Regional outage should not affect others!• Network partitioning between regions should not affect
functionality / operations!
REDUNDANCY!
• Make more than one (of pretty much everything)!• Specifically, distribute services across Availability
Zones and regions!
HISTORY: X-MAS EVE 2012!
• Netflix multi-hour outage!• US-East1 regional Elastic Load Balancing issue!!• “...data was deleted by a maintenance process
that was inadvertently run against the production ELB state data”!
ACTIVE-ACTIVE ARCHITECTURE!
THE PROCESS!
IDENTIFYING CLUSTERS FOR AA !
SNITCH CHANGES!
EC2Snitch! EC2MultiRegionSnitch!
Uses Private IPs! Uses Public IPs!
PRIAM.MULTIREGION.ENABLE =TRUE!
storage_port : Using Private IPs!
ssl_storage_port : Using Public IPs!
SPIN UP NODES IN NEW REGION!
us-east-1! us-west-2!
APP
UPDATE KEYSPACE!
Update keyspace <keyspace> with placement_strategy = 'NetworkTopologyStrategy'! and strategy_options = {us-east : 3, us-west-2 : 3};!
Existing region and replication factor ! New region and replication factor!
REBUILD NEW REGION
Run – nodetool rebuild us-east-1 on all us-west-2 nodes
RUN NODETOOL REPAIR
VALIDATION!
BENCHMARKING GLOBAL CASSANDRA WRITE INTENSIVE TEST OF CROSS-REGION REPLICATION
CAPACITY 16 X HI1.4XLARGE SSD NODES PER ZONE = 96 TOTAL 192 TB OF SSD IN SIX LOCATIONS UP AND RUNNING
CASSANDRA IN 20 MINUTES!
Cassandra Replicas
Zone A
Cassandra Replicas
Zone B
Cassandra Replicas
Zone C
US-West-2 Region - Oregon
Cassandra Replicas
Zone A
Cassandra Replicas
Zone B
Cassandra Replicas
Zone C
US-East-1 Region - Virginia
Test Load
Test Load
Validation Load
Interzone Traffic 18TB Backup Restored from S3 using Priam
1 Million Writes!CL.ONE (Wait for One Replica to ack)!
1 Million Reads!after 500 ms!CL.ONE with No!Data Loss!
Interregional Traffic!Up to 9Gbits/s, 83ms! 18 TB backups
from S3
TEST FOR THUNDERING HERD!
TEST FOR RETRIES!
FAILURE RETRY
KEY METRICS USED!
• 99 /95 th Read Latency (Client & C*)!• Dropped Metrics on C*!• Exceptions on C*!• Heap Usage on C*!• Threads Pending on C*!
CONFIGURATION FOR TEST!
• 24 Node C* SSDs!• 220 Client instances!• 70+ Jmeter Instances!
C* IOPS
TOTAL READ IOPS
TOTAL WRITE IOPS
95th LATENCY
99th LATENCY
CHECK FOR CEILING!
NETWORK PARTITION!
us-east-1 us-west-2
TAKEAWAYS!
REPAIRS AFTER EXTENSION ARE PAINFUL !!!
TIME TO REPAIR DEPENDS ON!
• Number of regions!• Number of replicas!• Data size!• Amount of entropy!!
ADJUST GC_GRACE AFTER EXTENSION!• Column Family Setting!• Defined in seconds!• Default 10 days!• Tweak gc_grace settings to
accommodate time taken to repair!• BEWARE of deleted columns!
RUNBOOK!
PLAN FOR CAPACITY!
CONSISTENCY LEVEL !
• Check the client for consistency level setting!• In a Multiregional cluster QUORUM <>
LOCAL_QUORUM!• Recommended consistency levels
LOCAL_ONE (CASSANDRA-6202) for reads and LOCAL_QUORUM for writes!
• For region resiliency avoid – ALL or QUORUM calls!
HOW DO WE KNOW IT WORKS? CREATE CHAOS!!!
Benchmark …!!Time Consuming !!But worth it!!
Recommended
