of 40 /40
It’s Not You... Ending a 15 year relationship with RRD OpenNMS User Conference Europe April 11, 2014 Eric Evans [email protected] @jericevans

It's not you, it's me: Ending a 15 year relationship with RRD

Embed Size (px)

DESCRIPTION

OpenNMS User Conference Europe presentation on using Apache Cassandra and Newts for time-series data storage.

Citation preview

Page 1: It's not you, it's me: Ending a 15 year relationship with RRD

It’s Not You...Ending a 15 year relationship with RRD

OpenNMS User Conference EuropeApril 11, 2014

Eric [email protected]

@jericevans

Page 2: It's not you, it's me: Ending a 15 year relationship with RRD

Five Stages of Grief

1. Denial2. Anger3. Bargaining4. Depression5. Acceptance

Page 3: It's not you, it's me: Ending a 15 year relationship with RRD
Page 4: It's not you, it's me: Ending a 15 year relationship with RRD

RRDTool

● Round robin database● First released 1999● Time-series storage● File-based● Constant-size● Automatic, amortized aggregation

Page 5: It's not you, it's me: Ending a 15 year relationship with RRD

Graph All The Things

Page 6: It's not you, it's me: Ending a 15 year relationship with RRD

Consider

● 2 5+ IOPs per update (read-modify-write)!● 1 RRD per data source (storeByGroup=false)● 100,000s of data sources, 1,000s IOPS● 1,000,000s of data sources, 10,000s IOPS● 15,000 RPM SAS drive, ~175-200 IOPS

Page 7: It's not you, it's me: Ending a 15 year relationship with RRD

Also

● Not everything is a graph● Inflexible● Incremental backups impractical● ...

Page 8: It's not you, it's me: Ending a 15 year relationship with RRD
Page 9: It's not you, it's me: Ending a 15 year relationship with RRD

Bottleneck!?

● We need to be collecting even more!● We need to be collecting more frequently!● The Internet of Things is upon us!!

Page 10: It's not you, it's me: Ending a 15 year relationship with RRD
Page 11: It's not you, it's me: Ending a 15 year relationship with RRD

But can’t we … ?

● Serialize RRD writes?● Cache?● Distribute the RRDs?● … ?

Page 12: It's not you, it's me: Ending a 15 year relationship with RRD
Page 13: It's not you, it's me: Ending a 15 year relationship with RRD
Page 14: It's not you, it's me: Ending a 15 year relationship with RRD

How about:● Distributed, decoupled architecture● High throughput● Horizontally scalable● Pluggable, extensible graphing● Facilitate new forms of analytics● More?

Starting Over Can Be Fun!

Page 15: It's not you, it's me: Ending a 15 year relationship with RRD

Observation #1

We collect and write a great deal; We read (graph) relatively little.

We are read-optimized.

Page 16: It's not you, it's me: Ending a 15 year relationship with RRD

Observation #2

Grouping samples that are accessed together is an easy optimization.

Page 17: It's not you, it's me: Ending a 15 year relationship with RRD

Project: NewtsGoals:● Stand-alone time-series data store● High-throughput● Horizontally scalable● Grouped metric storage/retrieval● Late-aggregating

Page 18: It's not you, it's me: Ending a 15 year relationship with RRD

Cassandra

● Apache top-level project● Distributed database● Tunable consistency● NoSQL MoSQL

Page 19: It's not you, it's me: Ending a 15 year relationship with RRD

Partitioning

A

B

C

Key: Apple

...

Page 20: It's not you, it's me: Ending a 15 year relationship with RRD

Placement

A

B

C

Key: Apple

...

Page 21: It's not you, it's me: Ending a 15 year relationship with RRD

Replication

A

B

C

Key: Apple

...

Page 22: It's not you, it's me: Ending a 15 year relationship with RRD

Cap Theorem

Consistency

Availability

Partition tolerance

Page 23: It's not you, it's me: Ending a 15 year relationship with RRD

Consistency

A

?

?

W=1

Page 24: It's not you, it's me: Ending a 15 year relationship with RRD

Consistency

A

B

C

R=3

Page 25: It's not you, it's me: Ending a 15 year relationship with RRD

Consistency

A

B

?

W=2

Page 26: It's not you, it's me: Ending a 15 year relationship with RRD

Consistency

?

B

C

R=2

R+W > N

Page 27: It's not you, it's me: Ending a 15 year relationship with RRD

Properties

● Symmetrical● Linearly scalable● Redundant● Highly available

Page 28: It's not you, it's me: Ending a 15 year relationship with RRD

SSTables

Writes

Commitlog

Memtable

SSTable

DiskMemory

Page 29: It's not you, it's me: Ending a 15 year relationship with RRD

Properties

● Optimized for write throughput● Sorted on disk● Perfect for time series!

Page 30: It's not you, it's me: Ending a 15 year relationship with RRD

Gist

● Samples stored as-is.● Samples can be retrieved as-is.● Measurements are aggregations calculated

from samples (at time of query).

Page 31: It's not you, it's me: Ending a 15 year relationship with RRD

Samples vs. Measurementssam·ple/ˈsampəl/noun

1. a small part or quantity intended to show what the whole is like."investigations involved analyzing samples of handwriting"synonyms: representative, illustrative, selected, specimen, test, trial, typical

meas·ure·ment/ˈmeZHərmənt/noun

1. the action of measuring something."accurate measurement is essential"synonyms: quantification, computation, calculation, mensuration

Page 32: It's not you, it's me: Ending a 15 year relationship with RRD

Sample{ ‘timestamp’: 1395278097, ‘resource’ : ‘localhost.eth0’, ‘name’ : ‘ifInOctets’, ‘type’ : ‘COUNTER’, ‘value’ : 457283782231}

Page 33: It's not you, it's me: Ending a 15 year relationship with RRD

NewtsCREATE TABLE newts.samples ( resource text, collected_at timestamp, metric_name text, metric_type text, value blob, attributes map<text, text>, PRIMARY KEY(resource, collected_at, metric_name));

Page 34: It's not you, it's me: Ending a 15 year relationship with RRD

Behind the scenes...

KSAT (1970-02-10 12:42:00,dewPoint,value): 0xc01a0000

(1970-02-10 12:42:00,maxTemp,value): 0x40280000

...

Ascending Order

Page 35: It's not you, it's me: Ending a 15 year relationship with RRD

Newts

resource | collected_at | metric_name | value

---------+---------------------+--------------+-----------

KSAT | 1970-02-10 12:42:00 | dewPoint | 0xc01a0000

KSAT | 1970-02-10 12:42:00 | maxTemp | 0x40280000

KSAT | 1970-02-10 12:42:00 | maxWindGust | 0x7ff80000

KSAT | 1970-02-10 12:42:00 | maxWindSpeed | 0x40180000

KSAT | 1970-02-10 12:42:00 | meanTemp | 0xbfe00000

Page 36: It's not you, it's me: Ending a 15 year relationship with RRD

POSTPOST /samples HTTP/1.1Host: example.comContent-Type: application/json

[ { ‘timestamp’: 1395278097, ‘resource’ : ‘localhost.eth0’, ‘name’ : ‘ifInOctets’, ‘type’ : ‘COUNTER’, ‘value’ : 457283782231 }, { ... },]

Page 37: It's not you, it's me: Ending a 15 year relationship with RRD

GET samplesGET /samples/localhost.eth0?start=900000000 HTTP/1.1

[ [ {“name”: “ifInOctets”, “timestamp”: 900000000, “type”: “COUNTER”, “value”: 12345678900}, {“name”: “ifOutOctets”, “timestamp”: 900000000, “type”: “COUNTER”, “value”: 87654321000} ], [ {“name”: “ifInOctets”, “timestamp”: 900000300, “type”: “COUNTER”, “value”: 23456789000}, {“name”: “ifOutOctets”, “timestamp”: 900000300, “type”: “COUNTER”, “value”: 98765432100} ]]

Page 38: It's not you, it's me: Ending a 15 year relationship with RRD

GET measurementsGET /measurements/octets/localhost.eth0?resolution=20m HTTP/1.1

[ [ {“name”: “ifInOctets”, “timestamp”: 900000000, “value”: 102400.00}, {“name”: “ifOutOctets”, “timestamp”: 900000000, “value”: 409600.00} ], [ {“name”: “ifInOctets”, “timestamp”: 900001200, “value”: 102400.00}, {“name”: “ifOutOctets”, “timestamp”: 900001200, “value”: 409600.00} ]]

Page 40: It's not you, it's me: Ending a 15 year relationship with RRD