1. v Elephants and Mice vSwitch Elephant Detection and Underlay Elephant Handling Justin Pettit, Kanna Rajagopal, JR Rivers and Martin Casado 20140515

2. Of Elephants and Mice Large data transfers elephants can consume disproportionate network resources (buffer/queues) which causes packet drops in other traffic, greatly increasing the mean-time-to-completion (mttc) of smaller data transfers mice. mice mttc disproportionately affect application mttc cumulusnetworks.com 2 3. Project Goal Evaluate the effects of predictive signaling and distributed reaction on the elephant/mice problem Detect and signal elephants via Open vSwitch React to elephants in HW via Cumulus Linux cumulusnetworks.com 3 4. Elephant Detection and Signaling Detect via per flow statistics (current) explicitly measure flow rate over time TCP Segmentation Offload buffer size (pending) indicates socket is sending large amount of data Signal via IP Differentiated Services Code Point (DSCP) carried through entire IP fabric cumulusnetworks.com 4 vswitch elephants mice nic vm vm vm vnic vnic vnic 5. Elephant Reaction - Escalating Affects Lower drop/wred threshold in a single queue diffserv AF behavior Alternative queue on same physical port Alternative network path cumulusnetworks.com 5 ? drop-threshold queue-select alternate-path 6. Test Topology Sources VMs connected via Open vSwitch 10G connection to network Network Paths 1G normal link easy to congest with VM traffic sources 10G alternative link Sink bare metal server 10G connection from network cumulusnetworks.com 6 1G Open vSwitch n vm vni c 10G 10G 10G vmvni c vm vni c 7. Traffic Generation and Result Measurement Generators elephants nuttcp fixed time transfers, 4M window mice small (10ms) interval pings mimics tcp-acks, lock release, small db transations Results elephants realized bandwidth, drops mice mean-time-to-completion, drops cumulusnetworks.com 7 8. Results flow statistic detection & alternate queue reaction test case (120 sec period) elephant mouse Mbps drops Latency (ms) drops elephant only 941 63 N/A N/A mouse only N/A N/A 0.444 0 mouse vs elephant no detection 941 61 3.055 0 mouse vs elephant w/detection 937 1223 0.401 0 cumulusnetworks.com 8 9. Results flow statistic detection & alternate queue reaction cumulusnetworks.com 9 0 1 2 3 4 5 6 7 8 9 10 500 550 600 650 700 750 800 850 900 950 1000 1 11 21 31 41 51 61 71 81 91 101 111 121 131 Latencyms) Bandwidth(Mbps) Time (Secs) Mice vs Elephants (Detection off) Elephant Mice 10. Results flow statistic detection & alternate queue reaction cumulusnetworks.com 10 0 1 2 3 4 5 6 7 8 9 10 500 550 600 650 700 750 800 850 900 950 1000 1 11 21 31 41 51 61 71 81 91 101 111 121 131 Latency(ms) Bandwidth(Mbps) Time (Secs) Mice vs Elephants (Detection on, OVS timer off) Elephant Mice 11. Next Steps TSO based detection pre-test with Linux kernel patches retest with updated Open vSwitch from vmware Various reactions alternate drop threshold alternate path cumulusnetworks.com 11 12. Thank You! cumulusnetworks.com 12