Scalable Network I/O with C++

Arindam MukherjeePune C++ and Boost Meetup

Content

• Sockets• Boost Asio programming model• Synchronous model• Key challenges to scaling• Asynchronous model• Asio coroutines

Objectives

• Get a feel of the subject• Know where to look, what to look for

Sockets

• Applications bind to an IP address and a numeric port to start communicating.

• Port: an unsigned integer identifying the point of attachment of the app to the protocol.

• Endpoint: The treble of an IP address, port and protocol (TCP, UDP, etc.).

• Socket: an unsigned integer identifying an endpoint in use.

Socket API

• Berkeley Sockets API, part of POSIX standard.• System calls– socket: get a socket– bind: bind to an endpoint (usually passive)– listen: wait for incoming connections (passive)– accept: establish the TCP connection (passive)– connect: connect to a remote endpoint (active)– close: tear down the connection

Boost Asio Programming Model

• asio::io_service as a bridge to the OS, and as a task queue.

• IO objects like asio::ip::tcp::socket, likewise for udp, etc.

• I/O functions both sync and async, like asio::read/write/asycn_read.

• Helper classes for name resolution, IP address manipulation.

Name resolution

• asio::ip::tcp::resolver provides the resolve method.– Takes asio::ip::tcp::resolver::query

object.– Returns iterator of type asio::ip::tcp::resolver::iterator.

– Dereferencing the iterator return asio::ip::tcp::endpoint objects.

Vectored I/O

• Gather data from disparate areas in memory and write it to a connection as a single stream.

• Read data from a single connection and partition the stream across a set of memory areas.

• Provided by readv / writev system calls – requires lesser context switches or processing overheads.

Asio Buffers for vectored I/O

• Wrappers around memory buffers, initialized with buffer base address and length in bytes.

• asio::const_buffer – source, asio::mutable_buffer – sink.

• Sequence of buffers can be vectors, lists, etc.

Sync I/O with TCP (server)

• Create a TCP listener socket.• Bind to a local endpoint.• Listen for and accept incoming connections.• Perform I/O and handle events on each

connection, using a per-connection socket.

Sync I/O with TCP (client)

• Create a TCP socket (and optionally bind to an endpoint).

• Connect to a remote endpoint (server).• Perform communication and handle events on

the connection.

TCP sync I/O interaction model

Key challenges to scaling

• Sync APIs block waiting for completion.• One thread per connection is for toy

programs: scalability anti-pattern.• Use a thread-pool and leverage the io_service

task queue: only limits threads, not waiting.• Go async: wait-free.• Async + thread-pool: wait-free, limited

threads.

io_service as a task queue

• Put a handler with signature void() on the queue, using post or dispatch.

• Handlers associated with tasks, could be ready or pending.

• Call run or run_one from threads to process tasks.• Call poll or poll_one to process ready tasks.• Create object of asio::io_service::work to make

run wait.• Serialize handlers using asio::io_service::strand.

Async I/O

• Submit tasks to the OS via the io_service, and be notified by callbacks.

• Wait-free, but difficult to code.• Maintaining context / state across callback

invocations presents challenges.

Async I/O model

Coroutines

• A coroutine can be called, suspended and resumed.

• Suspension through cooperative yielding.• On suspension, control goes back to counterpart

(caller).• On resumption, coroutine does not restart but

continues from where it left off. Stack preserved.• Boost.Coroutine provides implementation.

Async I/O with coroutines

• Enqueue a coroutine using asio::spawn.• Coroutine runs, but instead of blocking,

registers the yield context as a handler, and then yields.

• Control goes to caller (loop in run function).• The run function handles other I/O requests.• On I/O events, handler is invoked which

resumes coroutine.

Coroutine Sequence diagram

• We’ll make this during the presentation.

Q & A

• Books:Learning Boost C++ Libraries (Packt), Chapter 11.

• References:http://www.gamedev.net/blog/950/entry-2249317-a-guide-to-getting-started-with-boostasio/

• Collaboration diagrams from here:http://think-async.com/Asio/asio-1.4.1/doc/asio/overview/core/basics.html

Howto lambda

• Syntax for generating function objects inline:[] { std::cout << "Hello, world!\n"; }

• Can take args:[](std::string name) { std::cout << "Hello, " + name + "!\n"; }

• Can access vars from surrounding scope:[&var1, var2](…) { std::cout << var2 << '\n'; var1 = 10;}

Howto bind

• Syntax for adapting function arity (signature).void foo(int n, std::string name); // binaryauto f = boost::bind(foo, ::_1, "hello"); // unary

• Binding references:void foo(const std::string& name); // unarystd::string str;auto f = boost::bind(foo, boost::ref(str)); // nullary

• Placeholders for unbound params: ::_1, ::_2.