Developing a Redis ModuleItamar Haber

2

Who We Are

The open source home and commercial provider of Redis

Open source. The leading in-memory database platform, supporting any high performance OLTP or OLAP use case.

Chief Developer Advocate at Redis Labsitamar@redislabs.com@itamarhaber

3

Redis Modules Hackathon

http://bit.ly/redishack• Opens today, ends on November 12th• Global & local events (SF and TLV)• Submit your Redis module to participate• Win $10K in cash prizes! Gain world fame!

~10 Things About Redis

5

1. Redis: REmote DIctionary Server2. / rɛdɪs/: “red-iss”3. OSS: http://github.com/antirez/redis4. 3-clause BSD-license: http://redis.io5. In-memory: all in RAM (Flash option)6. A database for: ~9 data structures7. And: 4 (+1) more specialized ones

6

8. Developed & maintained: (mostly) Salvatore Sanfilippo (a.k.a. @antirez) and his OSS team at @RedisLabs

9. History:v1.0 August 9th, 2009… v3.2.4 September 26th, 2016

10.“The Leatherman™ of Databases”: mostly used as a DB, cache & broker

7

11.A couple or so of extra features:(a) atomicity; (b) blocking wait;(c) configurable persistence;(d) data expiration and (e) eviction; as well as transactions, PubSub, Lua scripts, high availability & clustering

12.Next version (v4.0): MODULES!

8

Redis 1011. Redis is “NoSQL”0. No (explicit) schema, access by key1. Key -> structure -> data

9

Redis data stratav1.0 Strings

Lists

Setsv1.2 Sorted Setsv2.0 Hashes

v2.2 Bit arraysv2.8.9HyperLogLogv3.2 Geo Sets

Bit fieldsv4 Neural Network

MODULES!

10

How to Redis in 3 steps:1. 150 OSS clients in 50 languages, e.g:

Java, Node.js, .NET, Python, Ruby…2. You make a request, i.e.:

PING3. The server replies, i.e.g:

PONG

11

~$ redis-cli127.0.0.1:6379> SET counter 1OK127.0.0.1:6379> GET counter"1"127.0.0.1:6379> INCRBY counter 1(integer) 2127.0.0.1:6379> APPEND counter b||!2b(integer) 7127.0.0.1:6379> GETSET counter "\x00Hello\xffWorld""2b||!2b"127.0.0.1:6379>

DEMOTIME!

The Evolution of Versatility

13

Flexibility: model (almost) anything with basic “building blocks” and simple rules (v0.0.1)Composability: transactions (v1.2) and server-embedded Lua scripts (v2.6)Extensibility: modules (v4) for adding custom data structures and commands

MODULES! (a.k.a plugins)

15

First mentioned in release v1.0

https://groups.google.com/forum/#!msg/redis-db/Z0aiVSRAnRU/XezAFFtgyPUJ

“Another interesting idea is to add support for plugins implementing specific commands and associated data types, and the embedding of a scripting language.”

16

Redis before modules:1. Redis is ubiquitous for fast data, fits

lots of cases (Swiss™ Army knife)2. Some use cases need special care3. Open source has its own agenda

So what can you do? FR, PR or fork

17

Redis with modules:1. Core still fits lots of cases2. Module extensions for special cases3. A new community-driven ecosystem4. “Give power to users to go faster”

What to expect? Nothing’s impossible!

18

Redis modules are:1. Dynamically (server-)loaded libraries2. Future-compatible3. (will be mostly) written in C4. (nearly) as fast as the core5. Planned for public release Q3 2016

19

Modules let you:1. Process: where the data is at2. Compose: call core & other modules3. Extend: new structures, commands

20

redis> ECHO "Alpha""Alpha"redis> MODULE LOAD example.soOKredis> EXAMPLE.ECHO "Bravo""Bravo"redis> ^C

~$ wc example.c 13 46 520 example.c~$ gcc -fPIC -std=gnu99 -c -o example.o example.c~$ ld -o example.so example.o -shared -Bsymbolic -lc

core command

module library“new”

command

21

Some exemplary modules:• Neural Redis – a neural network• RediSearch – full text search• countminsketch – streaming counter• redablooms – Bloom filters• redis-tdigest – rank estimator• More at the hub: redismodules.com

Redis Modules API

23

The API1. Where most of the effort was made2. Abstracts & isolates Redis’ internals3. The server’s (C-) binding contract4. Will not be broken once released5. Exposes three conceptual layers

24

Modules API layers1. Operational: admin, memory, disk,

replication, arguments, replies…2. High-level: client-like access to core

and modules’ commands3. Low-level: (almost) native access to

core data structures memory

~$ cat example.c: operational-API-only example

26

#include "redismodule.h"int Echo(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 2) return RedisModule_WrongArity(ctx); return RedisModule_ReplyWithString(ctx,argv[1]); }

int RedisModule_OnLoad(RedisModuleCtx *ctx) { if (RedisModule_Init(ctx, "example", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_CreateCommand(ctx, "example.echo", Echo, "readonly", 1, 1, 1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; }

27

#include "redismodule.h"int Echo(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 2) return RedisModule_WrongArity(ctx); return RedisModule_ReplyWithString(ctx,argv[1]); }

int RedisModule_OnLoad(RedisModuleCtx *ctx) { if (RedisModule_Init(ctx, "example", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_CreateCommand(ctx, "example.echo", Echo, "readonly", 1, 1, 1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; }

#include "redismodule.h"

int RedisModule_OnLoad(RedisModuleCtx *ctx) {

MUST:API definitions

MUST:is called when module is loaded

pointer to context

28

RedisModuleCtx *ctx1. The module’s call execution context2. Used by most calls to the API, just

pass it along3. Exposes the client id if needed

29

#include "redismodule.h"int Echo(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 2) return RedisModule_WrongArity(ctx); return RedisModule_ReplyWithString(ctx,argv[1]); }

int RedisModule_OnLoad(RedisModuleCtx *ctx) { if (RedisModule_Init(ctx, "example", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_CreateCommand(ctx, "example.echo", Echo, "readonly", 1, 1, 1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; }

if (RedisModule_Init(ctx, "example", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_CreateCommand(ctx, "example.echo", Echo, "readonly", 1, 1, 1) == REDISMODULE_ERR) return REDISMODULE_ERR;

register thecommand

register the moduleor die trying

30

#include "redismodule.h"int Echo(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 2) return RedisModule_WrongArity(ctx); return RedisModule_ReplyWithString(ctx,argv[1]); }

int RedisModule_OnLoad(RedisModuleCtx *ctx) { if (RedisModule_Init(ctx, "example", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_CreateCommand(ctx, "example.echo", Echo, "readonly", 1, 1, 1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; }

int Echo(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) if (argc != 2) return RedisModule_WrongArity(ctx); return RedisModule_ReplyWithString(ctx,argv[1]); validate

number of arguments

&err if needed

arguments

&countsend back

the argument

31

RedisModule_ReplyWith• Error – duh• Null – no words• LongLong – integer• String – also Simple and Buffer• Array – Redis array (can be nested)• CallReply – High-Level API reply

High-Level API

33

RedisModule_Call(…)• Does: runs a command • Expects: context, command name,

printf-like format and arguments• Returns: RedisModuleCallReply *

• Not unlike: Redis’ Lua redis.call

35

int Educational_HighLevelAPI_Echo(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 2) return RedisModule_WrongArity(ctx); RedisModule_AutoMemory(ctx); RedisModuleCallReply *rep = RedisModule_Call(ctx, "ECHO", "s", argv[1]); return RedisModule_ReplyWithCallReply(ctx, rep);}

Using the High-Level API to call the Redis core ‘ECHO’ command...

...is impractical but educational :)

36

RedisModule_AutoMemory(…)Automagically manages memory• RedisModuleCallReply *• RedisModuleString *• RedisModuleKey *• RedisModule_Alloc() and family

High-Level Visualization Of The Low-Level API

38

user

app

Redis client

Redis cor

e

data

GET foo

"bar"101010010101101010

39

user

101010010101101010

High

level

API

app

module

40

user

101010010101101010

app

Low

level

API

41

With the low-level API you can:• Manage keys: open, close, type,

length, get/set TTL, delete…• Manipulate core data structures:

e.g. RedisModule_StringSet(…), RedisModule_ListPop(…) and RedisModule_Zset*Range(…)

42

• Directly access String memory: RedisModule_StringDMA(…)

• Register custom data types: RedisModule_CreateDataType(…)

• Asynchronous (blocking) commands: RedisModule_BlockClient(…)

• And much more…

Redis Modules Hackathon

44

Redis Modules Hackathon

http://bit.ly/redishack• Learn new stuff, help the community• Have an amazing idea? Start coding!• Open to individuals and teams globally• Win $10K in cash prizes! Gain world fame!

45

Developing your module• Tools: a text editor and a compiler• Get Redis unstable branch• Docs and code are at

src/modules• Quick jumpstart & SDK: https://

github.com/RedisLabs/RedisModulesSDK

46

Guidelines for a module’s API• Keep it simple• Make it generic (and composable)• Implicitly manage data structures• Commands have predetermined

space/time cost• Inspiration: Redis’ core API

47

A few more pointers• Top performance is key• RAM consumption is also important• Code isn’t everything: docs, tests,

benchmarks and examples of use• Make it useful and awesome!

Thank you