Improving sanity with structured binary formats and protocols

  • View

  • Download

Embed Size (px)


Did you know that you can turn a JSON API into a binary one using MessagePack with one line of code? There are a lot of myths surrounding working with binary formats, and a lot of things even the "experts" get wrong. Just as important as making a binary format that is high performance, is making one that is easy for developers to work with. This talk describes how not to implement a binary format, and then introduces structured ways to implement them that will maximize language compatibility and make everybody's lives easier. An early version of this talk was introduced at Realtime Conf Europe, and was recorded here:

Text of Improving sanity with structured binary formats and protocols

  • 1.Learning how to let go:Improving sanity with structured binary data-interchange formats and protocolsKyle DrakeNet Brew Ventures

2. If youre doing a lot oftext-based serializedobject passing, andwant a way to improveserialization time andpayload size, try abinary format. 3. Why Smaller size, representing the sameinformation (esp. vs XML) Usually faster for CPU as aconsequence Surprisingly easy to implement (withhelp) Lots of well-tested, streamlinedsolutions 4. Why Not No quick language support (if custom) Hard to understand without good docs Potentially harder to debug low-level ifsomething bad happens (dataentanglement/corruption?) Binary != Performance (Measure!) 5. Two Approaches Custom, hand-rolled format you write Frameworks (Protocol Buers,MessagePack, BERT, others) 6. Custom Binary Format This is often a bad idea. No, seriously. Requires a very compelling reasonIMHO. 7. Lets look at a poorlydesigned custom binaryformat, via: The ApplePush NoticationService. 8. Wait, why notjust use JSONfor the wholething then? 9. Success: no responseError: socket disconnectionInterim: unreported data lossW.T.F. 10. ROFLSCALE TIPS FTW 11. My solution: 12. Meanwhile, at Google: 13. I think this crap is whypeople hate working withbinary so much.But there is a better way. 14. Lets look at somestructured ways to workwith binary! 15. Protocol Buffers Developed by Google 3 to 10 times smaller, 20 to 100times faster than XML Requires a pre-dened .proto le In eect, it has a language agnosticschema. 16. Protocol Buffers 17. Protocol Buffers 18. Protocol BuffersIm doing work with the Bitcoin protocolright now. I really wish it used protocolbuers. Not just for size, but for safetyand ease of use.Bitcoin lead core developer is warm toidea: 19. Lets look at someschemaless structuredbinary formats. 20. JSON Number (50, 2.33) String howdy Boolean (true or false) Array [1,2,3] Object {key: value} null (empty) 21. MessagePackBasically the same thing.Except faster and smaller!Super simple: 22. SOURCE: INDIEGAMR.COMHTTP://INDIEGAMR.COM/CUT-YOUR-DATA-EXCHANGE-TRAFFIC-BY-UP-TO-50-WITH-ONE-LINE-OF-CODE-MSGPACK-VS-JSON/ 23. BINARYJS.COM 24. Painless API Integration /route?format=msgpack Content-Type: application/x-msgpack Accept: application/x-msgpack Plugs right in, if youre using JSON. 25. HAVE A JSON API? TRY ASCHEMALESS STRUCTUREDBINARY FORMAT! ITS EASY. 26. THANKS!