Embed Size (px)
DESCRIPTION
Restats is a highly customized real-time framework that processes data and creates data and geo cubes that later can be queried. It makes use of the data types redis supports, like the sorted-set.
Citation preview
restatsr
Processing real-time data to cubes on data structure servers
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
incement
query interface
response
geoscaling
speedtest
lessonslearned
theProblem
Trackings Calculate
every X hours
Doesn’t Scale!
Warehouse
800 M+ 500 GB+
mySQL
Application
Rails
Statistics
Redis
r
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
incement
query interface
response
geoscaling
speedtest
lessonslearned
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
incement
query interface
response
geoscaling
speedtest
lessonslearned
theSolution
Application Statistics
Trackings Process
Real-Time
RailsRedis
r
restatsr
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
incement
query interface
response
geoscaling
speedtest
lessonslearned
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
incement
query interface
response
geoscaling
speedtest
lessonslearned
rwhat is restats?
Redis Statisticsadvanced key-value store
data structure serverdata cubesgeo-location cubes
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
incement
query interface
response
geoscaling
speedtest
lessonslearned
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
increment
query interface
response
geoscaling
speedtest
lessonslearned
rhow does it work: tracking
Application
Rails
Produces
JSON
{ "created_at": "2012-10-18T21:34:56+02:00", "type": "adrequest","location_id": 1, "campaign_id": 1, "publisher_id": 1, "latitude": "53.36669921875", "longitude": "-1.5", "bannersize": "320x50"
}
At 2012-10-18, there was an adrequest for campaign 1 at 53.366,-1.5 for the bannersize 320x50
rhow does it work: dimensions
Attributetype
location_idcampaign_idpublisher_idbannersize
Geo
latitudelongitude
Time
daymonth
year
rhow does it work: cubingAttribute
typelocation_id
campaign_idpublisher_idbannersize
Geo
latitudelongitude
Time
daymonth
year
Depth
3
1 Attribute Combination
Maps
Redis INCR1 Redis Key
Maps
Redis INCR
INCR day-month-year-type:\18-10-2012-adrequest
day-month-year
Time
type
Attribute
18-10-2012 adrequest
1st Depth
rhow does it work: 1st depthAttribute
typelocation_id
campaign_idpublisher_idbannersize
Geo
latitudelongitude
Time
daymonth
year
Depth
3
day-month-year
Time
type
Attribute
18-10-2012 adrequest
Maps
Redis INCR
INCR day-month-year-type:\18-10-2012-adrequest
month-year
Time
type
Attribute
10-2012 adrequest
Maps
Redis INCR
INCR month-year-type:\10-2012-adrequest
year
Time
type
Attribute
2012 adrequest
Maps
Redis INCR
INCR year-type:\2012-adrequest
rhow does it work: 2nd depthAttribute
typelocation_id
campaign_idpublisher_idbannersize
Geo
latitudelongitude
Time
daymonth
year
Depth
3
INCR day-month-year-type-location_id:\18-10-2012-adrequest-1
Maps
Redis INCR
day-month-year
Time
type
Attribute
18-10-2012 adrequest
location_id
1
INCR day-month-year-type-campaign_id:\18-10-2012-adrequest-1
Maps
Redis INCR
day-month-year
Time
type
Attribute
18-10-2012 adrequest
campaign_id
1
...Combinations
rhow does it work: 3rd depthAttribute
typelocation_id
campaign_idpublisher_idbannersize
Geo
latitudelongitude
Time
daymonth
year
Depth
3
INCR day-month-year-type-location_id\-campaign_id:\18-10-2012-adrequest-1-1
Maps
Redis INCR
day-month-year
Time
type
Attribute
18-10-2012 adrequest
location_id
1
campaign_id
1
INCR day-month-year-type-location_id\-publisher_id:\18-10-2012-adrequest-1-1
Maps
Redis INCR
day-month-year
Time
type
Attribute
18-10-2012 adrequest
location_id
1
publisher_id
1
Combinations...
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
increment
query interface
response
geoscaling
speedtest
lessonslearned
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
increment
query interface
response
geoscaling
speedtest
lessonslearned
rhow does it work: keysAttribute
typelocation_id
campaign_idpublisher_idbannersize
Geo
latitudelongitude
Time
daymonth
year
Depth
3
Total Amount of Keys per Tracking =
n =d =t =
attribute dimensionsmax depthtime dimensions
(5)(3)(3)
(5 + 10 + 10) • 3 = 75 Keys
rhow does it work: increment
Digest SHA1
day-month-year-type:18-10-2012-adrequestmonth-year-type:10-2012-adrequest
year-type:2012-adrequestday-month-year-type-location_id-publisher_id:18-10-2012-adrequest-1-1
...
Computed Keys
INCR 76f374viq4vi7fti374oo76INCR o8qf374fvlqizfvil7v2o8vINCR fqwevlrfli74l3fzlizflzfINCR fbliaguzvf736i4cjsztudz
...
Redis Batch INCR
rhow does it work: geo cubingRedis Sorted Set
set with a score for each member
ZADD key score member # adds a member to keyZINCRBY key 1 member # increases member by 1
Mapping
member = “latitude:longitude”score = # of trackings at that location
Rounded position speci!es precision
ZINCRBY sdvouzg 1 53.3666:-1.5ZRANGE sdvouzg 0 -1 withscores1) 54.3215:0.34562) 53) 53.3666:-1.54) 2763
rhow does it work: geo increment
Round Position ZINCRBY 1
INCR 76f374viq4vi7fti374oo76INCR o8qf374fvlqizfvil7v2o8vINCR fqwevlrfli74l3fzlizflzfINCR fbliaguzvf736i4cjsztudz
...
Computed Keys
INCR 76f374viq4vi7fti374oo76ZINCRBY 76f374viq4vi7fti374oo76g 1 53.3666:-1.5
INCR o8qf374fvlqizfvil7v2o8vZINCRBY o8qf374fvlqizfvil7v2o8vg 1 53.3666:-1.5
INCR fqwevlrfli74l3fzlizflzfZINCRBY fqwevlrfli74l3fzlizflzfg 1 53.3666:-1.5
INCR fbliaguzvf736i4cjsztudzZINCRBY fbliaguzvf736i4cjsztudzg 1 53.3666:-1.5
...
Redis Batch ZINCRBY
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
increment
query interface
response
geoscaling
speedtest
lessonslearned
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
increment
query interface
response
geoscaling
speedtest
lessonslearned
rthe queryinterface
Queryconverts the attributes to keys
multi-fetch to save round-trip time
rthe queryresponse
Query-able Ruby Hashallows to iterate over the responses
merged aggregations
rthe geoqueryresponse
Query-able Ruby Hashlist of positions
scores are trackings at that position
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
increment
query interface
response
geoscaling
speedtest
lessonslearned
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
increment
query interface
response
geoscaling
speedtest
lessonslearned
rthe scalingstrategy
Processing Ratio =
monitor value over a time period, e.g. 10 mins
Trackings In
Trackings Out
PR > 0PR < 0
add workerremove worker
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
increment
query interface
response
geoscaling
speedtest
lessonslearned
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
increment
query interface
response
geoscaling
speedtest
lessonslearned
rthe speedtest
2.4 GHz Intel Core i5 with 8 GB 1067 MHz DDR35 attribute dimensions
5 max depthgeo cube enabled
85 processings / second
(5 + 10 + 10 + 5) • 3 • 2 ==
bottleneck =
180 keys / tracking15,300 keys / secondcpu; calculation of combinations
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
increment
query interface
response
geoscaling
speedtest
lessonslearned
ProblemSolution
What is restats?Tracking
dimensionscubing
1st depth2nd depth3rd depth
keysincrement
geo cubinggeo incrementquery interfacequery response
geo query interfacescaling strategy
speed testlessons learned
scales
restats
problem solution
tracking
dimensions
cubing
1st, 2nd, 3rddepth
keys
increment
query interface
response
geoscaling
speedtest
lessonslearned
rthe lessonslearned
dare to customizeyou’ll be rewarded
use mini-benchmarkswhen you’re unsure during implementation
know your datadynamics kills performance
not NoSQL, newSQL!we’re still querying, right?
process on writeif you need fast read-access
no CRONrecurring tasks don’t scale
denormalizein this !eld, it’s your friend
![ROI in the age of keyword not provided [Mozinar]](https://img.pdfslide.us/doc/110x75/53eabc7a8d7f7289708b51f7/roi-in-the-age-of-keyword-not-provided-mozinar.jpg)
