More hacking
• PSYNC2 bug (Alibaba + Amazon + Redis Labs <3)
• RDB modules values
• Modules thread-safe contexts
Sorted sets
• Optimized to *move* elements around
• They are sets: unique elements
• Wasteful if not used for the right task
Log file
• One (complex) entry per line
• Ordered
• One entry one offset
• Repeated entries are OK
• Memory efficient
An abstract log file
• Entries are hashes
• Offsets are logical (entry unique ID)
• O(log N) access to random ranges
• Trow away old data easily
• Memory efficient? Yep!
Streams implementation1;somedata;foo;bar
2;somedata;…
3;somedata;…
4;somedata;…
5;somedata;…
6;somedata;…
…
…
…
…
…
…
…
…
…
1;somedata;foo;bar
2;somedata;…
3;somedata;…
4;somedata;…
5;somedata;…
6;somedata;…
…
…
…
Streams implementation1;somedata;foo;bar
2;somedata;…
3;somedata;…
4;somedata;…
5;somedata;…
6;somedata;…
…
…
…
Streams API
• Blocking API: consume from <ID>
• Ranges
• Deletion of old data
• From Kafka: consumer groups
• ACK and retry