Connecting the dots…

Christiaan Douma - Dev Engineer, Stephane Rouault - Dev Engineer

Tracing of events within ING

Amsterdam • 29th October 2015

3

ING

Future

Why

Numbers

Elasticsearch

How

INGIt’s who we are

5

Market leaders Benelux

Growth marketsCommercial Banking

Challengers

World mapOver 40 countries52,000+ employees

6

Market leaders Benelux

Growth marketsCommercial Banking

Challengers

European map

Full-service bankVery strong European baseRanked 7thLargest bank of NL150+ DevOps teams

• Engineer• Software Developer• 7 years of experience in IT• @ING since: 01-03-2011

• Engineer • Jack of all trades• 11 years of experience in

IT• @ING since: 01-01-2011

7

A little bit about usStephane Rouault Christiaan Douma

8

ING

Future

Why

Numbers

ElasticSearch

How

The WhyA little bit of context

10

Happy Flow

A simple illustration of a transaction

Frontend Middleware

Backend

Request------------------------------------------------

Reply------------------------------------------------

11

Error occurs

But what if something goes wrong?

Frontend Middleware

Backend

Request------------------------------------------------

ERROR: Some error at …

What happened ?

Where did it happen ?What has been

affected?

12

What usually happens next ..

Frontend Middleware

Backend

Log Log Log

DevOps teams searching their logs..

• Finding the owner of the problem

• Access to logs is restricted to system owners

• Difficult to link events across systems

• Time Consuming Process13

Issues

14

Complex ING Landscape

Frontend Middleware

Backend

15

ING

Future

Why

Numbers

ElasticSearch

How

The HowSolving some issues

17

Unique Correlation Identifier

Frontend Middleware

Backend

Request:UUID------------------------------------

Link events across systems, extra context info in the request

18

Using Elasticsearch to combine everything

Frontend Middleware

Backend

LogLo

gLogLo

g

LogLo

gLogLo

g

LogLo

gLogLo

g

Elasticsearch

Event:UUIDContextDurationStatus

Event:UUIDContextDurationStatus

Event:UUIDContextDurationStatus

19

Event logging in Elasticsearch : Tracing

Frontend Middleware

Backend

Request:UUID------------------------------------

Elasticsearch

20

Tracing

21

Component Name

Error Description

Event Info

Tracing

22

Tracing in kibana

23

ING

Future

Why

Numbers

ElasticSearch

How

ElasticsearchThe reason why

25

Free text searchShared service within INGData lake for operational events

Scalability

No read/write interference

Flexible data model High Available

How did Elasticsearch help us?

26

How is Elasticsearch set-up?

27

ING

Future

Why

Numbers

ElasticSearch

How

The NumbersWho doesn’t love numbers

29

• Peak number events per second: ~650 events per second

• Avg number events per second: ~300 events per second

• Peak number of events per day: ~24 million events

• Avg number of events per day ~11 million events

• Growth of the number of events: ~40% more than 3 months ago..

• Growth of the number of operations: ~12% more than 3 months ago..

• Longest chain of events: >30

• Number of DevOps teams using Tracing : ~24 teams

• Number of architecture domains in Tracing ~5

Some statistics about Tracing

30

• Peak number events per second ~2500 events per second

• Peak number of events per day ~120 million events

• Index per day takes up to 100-130 Gb

• Retaining 30 days of data (1x replicated) ~4,4 TB

• Number of Elasticsearch queries 30 tps

• Avg responsetime (for most queries) 1-2 secondes

• Number of kibana dashboards: ~740 and growing fast.

• Number of shards: 5

Some statistics about Elasticsearch

31

ING

Future

Why

Numbers

ElasticSearch

How

The FutureWhere we want to go

How it works now (architecture)

33

How it will work (architecture roadmap)

34

35

• Multi-source event correlation; combining tracing with:

• System logs• Alert logs• System metrics• Incidents logs• Deployment logs• Etc..

• Elasticsearch hook up to a graph database for realtime graphical insight

• Forecasting of (business) usage• Business chain alerting with watcher or other tool

Roadmap

How it will work (architecture roadmap)

36

Q&ATime to interface

38

DisclaimerImportant legal informationThis document has been produced by ING solely for use at presentations held externally, and may not be reproduced or redistributed, in whole or in part, to any other persons without the prior written consent of ING. The content of this document is not to be construed as legal, business, investment or tax advice. Each person should consult with its own professional advisors for any such matters and advice. ING comprises a broad spectrum of companies (the "ING companies"), many of them operating under their own brand names. Almost every ING company, business unit or product group, has its own website on the internet where it offers information about its products and services. Reference is made to those websites for further details and hyperlinks to the websites of those ING companies, business units and product groups, if available. However, ING is not responsible for the content of any other websites that are included in this document. A link to such website is meant for convenience only and does not imply that ING endorses the website or any products or services that it describes. Your access to and use of any other websites included in this document is at your own risk and ING assumes no obligation or liability in connection therewith. Certain of the statements contained in this release are statements of future expectations and other forward-looking statements. These expectations are based on management’s current views and assumptions and involve known and unknown risks and uncertainties. Actual results, performance or events may differ materially from those in such statements due to, among other things : (1) changes in general economic conditions, in particular economic conditions in ING’s core markets, (2) changes in performance of financial markets, including developing markets, (3) consequences of a potential (partial) break-up of the euro, (4) ING’s implementation of the restructuring plan as agreed with the European Commission, (5) changes in the availability of, and costs associated with, sources of liquidity such as interbank funding, as well as conditions in the credit markets generally, including changes in borrower and counterparty creditworthiness, (6) the frequency and severity of insured loss events, (7) changes affecting mortality and morbidity levels and trends, (8) changes affecting persistency levels, (9) changes affecting interest rate levels, (10) changes affecting currency exchange rates, (11) changes in investor, customer and policyholder behaviour, (12) changes in general competitive factors, (13) changes in laws and regulations, (14) changes in the policies of governments and/or regulatory authorities, (15) conclusions with regard to purchase accounting assumptions and methodologies, (16) changes in ownership that could affect the future availability to us of net operating loss, net capital and built-in loss carry forwards, (17) changes in credit-ratings, (18) ING’s ability to achieve projected operational synergies and (19) the other risks and uncertainties detailed in the risk factors section contained in the most recent annual report of ING Groep N.V. Any forward-looking statements made by or on behalf of ING speak only as of the date they are made, and, ING assumes no obligation to publicly update or revise any forward-looking statements, whether as a result of new information or for any other reason.  This document does not constitute an offer to sell, or a solicitation of an offer to purchase, any securities in the United States or any other jurisdiction. The securities of NN Group have not been and will not be registered under the U.S. Securities Act of 1933, as amended (the “Securities Act”), and may not be offered or sold within the United States absent registration or an applicable exemption from the registration requirements of the Securities Act.

www.ing.com