Presented by Daniel Ortiz Esquivel

R&D Software EngineerYAPC::Europe September 2015

Site: https://developer.williamhill.com

Twitter: https://twitter.com/dortizesquivel

OMNIA

Distributed & Reactive platform for data management

Motivations

2Omnia: Distributed & Reactive platform for data management

Users

Feeds

System

3 Party

In order to be in a position to innovate we need to control and understand our data

SocialNetwork

s

IoT

Omnia

DMP based on the Lambda architecture and the Reactive principles

What is Omnia?

3

Ch

ron

os

Dat

a S

ourc

e NeoCortexSpeed Layer

FatesBatch Layer

Her

mes

Ser

ving

Lay

er

Data Flow

Input Output

Omnia: Distributed & Reactive platform for data management

Resources:• Reactive Extension for Perl:

https://github.com/eilara/Rx.pl• Carl Mäsak YAPC:2013: https

://www.youtube.com/watch?v=EnlYURSKgk0 • Ran Eliam :

https://www.youtube.com/watch?v=ymEGZEi4hO4

Reactive principles

4

Responsive

Resilient

Message Driven

Elastic

The Reactive Manifestohttp://www.reactivemanifesto.org/

Omnia: Distributed & Reactive platform for data management

Chronos is a reliable and scalable component which collect data from different source and organize them into Streams of observable events.

Chronos: Data acquisition

5

Incident: {

type: “bet”,

version: “1.0”,

time: “2015-09-03 06:00:10”,

acquisitionTime: “2015-09-03 06:00:06”,

source: “BetSystem”,

payload: {…. Any valid JSON}

}

Omnia: Distributed & Reactive platform for data management

Ch

ron

os

Dat

a S

ourc

eTCP

HTTP

WS

…

JMS

HTTP Poll

SSE

AdapterStreams

Converter Persistence

Bets

Dep

osits

Pric

es

Stream = Adapter + Converter + Persistence

Chronos: Data acquisition

6Omnia: Distributed & Reactive platform for data management

Chronos 1(SSE, Bets placed)

Chronos 2(JMS, Deposits)

Chronos 3(HTTP, Events)

Chronos N(SSE, Twitter)

….…

Chronos 2(JMS, Deposits)

(SSE, Bet Placed)

Fates represents the long term memory of Omnia. It organizes the incidents that Chronos collected into timelines and also elaborates new information as views by using machine learning, logical reasoning and time series analysis.

Fates: Batch layer

7Omnia: Distributed & Reactive platform for data management

Customer: 123

Login

Deposit

Bet placed

…Logout

Event: 78

Started

Fault

Penalty

…

GoalTimelines & Views

Bets Deposits Events Session

FatesBatch Layer

Fates: Batch layer

8Omnia: Distributed & Reactive platform for data management

Timelines

Views

Jobs

Fates

NeoCortex represents the short term memory of Omnia. It offers a framework to develop micro services on top of Apache Spark. It performs fast and real time data processing with the data acquired by Chronos.

NeoCortex: Speed layer

9Omnia: Distributed & Reactive platform for data management

NeoCortex

Bets

Dep

osits

Eve

nts

Sess

ion

Micro Services

Out

put

Hermes is a scalable and full duplex communication for B2C and B2B.

Hermes: Serving Layer

10

Omnia: Distributed & Reactive platform for data management

B2CBrowser

B2B

Load

bal

ance

r

Push Server

DistributeCache

Push Server

Push Server

…

TCP

WS

HTTP

JS A

PI

WH Apps

Cac

he

Cac

he

Custom advert, bonus, data load prediction, bot detection...

Omnia Inception

11

Ch

ron

os

Dat

a S

ourc

e NeoCortexSpeed Layer

FatesBatch Layer

Her

mes

Ser

ving

Lay

er

Input Output

Omnia: Distributed & Reactive platform for data management

Users become a new data producer

Real time monitoring and elasticity

Docker and Mesos: Scale In&Out based on demand,

Omnia inception +1

12

Ch

ron

os

Dat

a S

ourc

e NeoCortexSpeed Layer

FatesBatch Layer

Her

mes

Ser

ving

Lay

er

Input Output

Omnia: Distributed & Reactive platform for data management

JMX

JMX

JMX

Omnia infrastructure

13

Omnia: Distributed & Reactive platform for data management

Omnia

Docker

Marathon

Mesos

Node Node NodeNodeNode

https://careers.williamhillplc.com/

Questions

14

Omnia: Distributed & Reactive platform for data management

https://omnia.williamhill.com/