2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 1
Optimized Storage and Network layer for Next Gen Industry workloadsAjinkya NakavePrajakta ZagadeSumit DigheVeritas
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 2
Agenda
§ Brief overview§ Challenges involved § Solutions§ Performance study § Conclusion
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 3
Hyper-Converged Architecture
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 4
Dis-aggregated Architecture
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 52020 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Possible Optimizations
§ Multithreading§ Batching§ Reduced context switching§ Use of multiple Q pairs/connections§ Load balancing across ports§ Is strict flow control at network layer needed ?
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 6
Deep dive
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 7
Storage Layer
I/O
Storage Layer
Offset 0
Network Network
How IO Flows
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 8
Ordering of I/OsI/O 2 I/O 3I/O 1
Storage Layer
Offset 0
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 92020 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Packet Handling
§ Network Layer§ Ordering of packets§ Flow control§ Guaranteed Delivery (Data Resends-Duplicate detection)
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 10
Ordering of packets
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 112020 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Ordering-Sequencing§ Ordering requires
§ Complex protocols: consumes CPU and increased latency§ Out of order delivery may cause
§ corruption to data§ Some protocols cause retransmissions§ No of acks on network increases due to smaller window
§ Found duplication in handling IO Ordering:§ File System layer: Using range locking§ Network layer: Using Sliding Window Protocol
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 122020 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Flow control§ To cope up with fast Sender§ Found duplication in handling Flow control:
§ Application IO:§ Most of the applications issue finite IOs.§ Next set of IOs wait till previous comes back
§ Network layer: Using Sliding Window Flow control Protocol§ Receiver window field can be adapted to higher values to avoid
unnecessary flow control
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 132020 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Guaranteed Delivery§ Packets may get dropped§ Guaranteed delivery has to ensure data to be delivered§ Sequence numbers allow receivers to discard duplicates§ Network layer protocol is best place to handle this
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 14
Storage
I/O 2 I/O 3I/O 1
Storage
Offset 0
Network Network
Ordering
Sequencing
Flow control
Guaranteed delivery
How IO Flows
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 152020 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
How to optimize I/O Paths
§ Sequencing-Ordering§ Flow control§ Guaranteed Delivery (Data Resends-Duplicate
detection)§ Resiliency / Storage Redundancy (RAID levels,
Erasure coding)
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 16
Dirty region logging processing
APPAPPAPP APP APP
Mirroring Overhead: Dirty region tracking
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 17
APPAPPAPP APP APP
Batch Processing Engine 1
Batch Processing Engine 2
Batch Processing Engine M
Flush Engine 1
Flush Engine 2
Flush Engine 3
Flush Engine N
Mirroring Overhead: Optimizations
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 182020 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Additional considerations
§ Receive Side Scaling (RSS)§ Zero copy§ blk-mq for SSD devices§ TCP – UDP offloading§ Multiple socket / queue-pair (RDMA)§ NUMA aware handling for CPU load balancing
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 19
Performance Study
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 20
Performance Study: Results with no sequencing at N/W layer
§ Default: Sequencing enabled at N/W layer§ Improved: No sequencing at N/W layer
Workload Details:§ Fio – libaio ioengine, iodepth=8,
numjobs=8§ Storage: loopback device (to avoid
storage bottleneck)
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 21
Performance Study: Results with all optimizations
§ Default: No optimizations§ Improved: Consists all optimizations
discussed so far
Workload Details:§ Fio – libaio ioengine, iodepth=8, numjobs=8§ Storage: loopback device (avoid storage
bottleneck)§ Same workload ran from 4 nodes
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 22
Performance Study: Mirror layout
1 4 8 16Default 5687 24541 47118 100871Optimized 11544 42751 78865 148590
020000400006000080000100000120000140000160000
IOPS
Threads
IOPS comparision (4K IO size)*Higher the better*
Default Optimized
1 4 8 16Default 5351 24290 50704 103285Optimized 12548 42245 79816 122134
020000400006000080000100000120000140000
IOPS
Threads
IOPS comparision (8k IO sz)*Higher the better*
Default Optimized
• Default: No optimizations• Improved: All optimizations discussed so farNOTE:• Storage layout is mirror across nodes• Each write updates dirty region bitmap on
disk
Storage: NVME. network link: 2 10GBenchmark: FIO-libaioIO sizes (4k/8k). Iodepth = 8 (Random write)
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 23
Summary
2020 Storage Developer Conference India. © Insert Your Company Name. All Rights Reserved. 242020 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Summary
§ Discussed various techniques to optimize storage and network layer
§ Performance statistics: Up to 1.5X to 2X performance improvement