Index Activiti data on Elasticsearch

Activiti User Day Paris 2015

Silvio dos Passos NetoCTO at iColabora

@silvioneto

“Don’t  bridge  the  business-­‐IT  divide.  Obliterate  it!”  (2003)

Smith & Fingar

?

?

@mike_dias

The big table problem

ID_ NAME_ VALUE_* …ACT_HI_VARINST

User form

ProcessInstance

ID_ NAME_ VALUE_* …

1 client_name Jonh …

2 client_tel 123456 …

3 due_date 01/06/2015 …4 demand_desc I have a problem… …

ACT_HI_VARINST

User form

ProcessInstance

ProcessInstance

User form

ID_ NAME_ VALUE_* …

1 client_name Jonh …

2 client_tel 123456 …

3 due_date 01/06/2015 …4 demand_desc I have a problem… …5 client_name Bob …6 client_tel 654321 …7 due_date 10/06/2015 …8 demand_desc My internet conn… …

ACT_HI_VARINST

85 fields x

~1000 Process per day =

~85.000 variables per day

~15 million variables

in 9 months

The Tool

Built on top of

Analytics

Distributed

Indexing the data

Historic Data

P

P PP

PP

PP

PP

P P PP

P P

PPP

Process Lake

P

P PP

PP

PP

PP

P P PP

P P

PPP

P

P

PP

P

PP

P

PP

P

P

CPU 1 CPU 2 CPU 3 CPU 4

Process Lake

P

P

P

CPU 1

V V

V V

V V

V V

VariablesT

T

T

Tasks

P

Process

{ }P V V V

V V V V

V T T T

JSON

REST API

Real-Time Data

E E E E

E E E E

E E E E

E E E E

E E E E

E E E E

E E E E

E E E E

Engine Events

E

E

E

E

E

E

E

E

E

E

E

E

E E E E

E E E E

E E E E

E E E E

E E E E

E E E E

E E E E

E E E E

Engine Events

E

E

E

E

E

E

E

E

E

E

E

E

E EE

E EE{ }JSON

REST API

Listeners

Playing with the data

Search

{ "query":{ "path":"variables", "nested":{ "query":{ "match":{ "text":"João Silva" } } } }}

Search results

Compare

SELECT * FROM ACT_HI_VARINSTWHERE NAME_ = 'passport'AND TEXT_ = '1234'

{ "filter":{ "nested":{ "path":"variables", "filter":{ "bool":{ "must":[ { “term": { "name":"passport" }}, { “term": { "text":"1234" }} ] } } } }}

Response Time

0 secs

45 secs

90 secs

135 secs

180 secs

MySQL Elasticsearch

0,08 secs

161 secs

Response Time

0 secs

45 secs

90 secs

135 secs

180 secs

MySQL Elasticsearch

0,08 secs

161 secs

CENSU

RED

Lessons learned

Full text search is a helpful feature

Reduce MySQL workload

ES is great for analytics

Next steps

Apache Spark

Lightning-Fast Cluster Computing

Java EE dependency

Open source

Thank you!@mike_dias @silvioneto

Questions?@mike_dias @silvioneto