- To insert a Zurich picture

click on the "camera"-icon in the Zurich CI toolbar and follow the instructions.

- To insert a picture from your

personal files, click on the "Insert Picture from File" icon here on the right.

Please make sure that this picture follows the Zurich

core elements available on the "book"-icon in the Zurich

CI toolbar.

- To keep this neutral

background, just leave it as it

is.

Note: this message will not be displayed in the presentation mode.

Secured ELK Stack at Zurich Insurance

11.11.2015

Michael Lehmann, Oliver Wulff

Zürich Versicherungs Gesellschaft AG

© Z

uri

ch S

wit

ze

rla

nd

Michael Lehmann

Head of Solution Integration providing shared services for business applications

Working for Zurich for 3 years

Ownership of shared platforms: SOA Integration Platform, SSO Security Platform, Application Development Platform, Java Platform and now ELK Platform

My Ambition:

– Integrate state-of-the-art technologies consistently into our application landscape

– Sell business value of technology internally

Oliver Wulff

Principal Consultant at Talend

Support Zurich Integration, Security, Application Development and ELK

Working in Open Source Projects at Apache

About us

2

© Z

uri

ch S

wit

ze

rla

nd

Motivation to use ELK

3

Application demand for

full-text search capabilities

Shared Platforms and

Business Applications log

gigabytes of data every

day

Basic infrastructure

monitoring in place but

not at application level

Incidents will be handled

reactively only

Insights into production

environment restricted

Challenge

Provide shared ELK

platform service

Feed log and audit data

into one central system:

ELK

Provide business

application and shared

platform dashboards for

specific insights

Monitor application

behaviour

Solution

Increased performance for

full-text search queries

Proactively handle

incidents even before a

business user reports it

Resolve incidents faster

Derive usage patterns and

detect application

problems

Business Value

© Z

uri

ch S

wit

ze

rla

nd

System Architecture

4

© Z

uri

ch S

wit

ze

rla

nd

Shippers:

LSF

FileBeat

Apache Karaf (Talend ESB)

Applications using ElasticSearch REST API

Indexes:

Logstash (logstash-[YYYY-MM-DD])

Apache Karaf (talendesb-[YYYY-MM-DD])

Top Beat (topbeat-[YYYY-MM-DD])

Sources / Document Types:

Access Log (Tomcat, JBoss)

Application Log (JSON, Log4J, Logback)

Legacy Applications (Orbix, Artix)

Shippers / Indexes / Document Types

ELK Architecture

5

© Z

uri

ch S

wit

ze

rla

nd

Input

Heartbeat

Lumberjack

Beats (Planned)

Kafka (Queue Collector/Indexer)

JMX (Kafka Monitoring)

Output

Kafka

ElasticSearch

Filters:

Grok

mutate, useragent, multiline,

Logstash

ELK Architecture

6

© Z

uri

ch S

wit

ze

rla

nd

ElasticSearch

Base monitoring OS Level (Process, Filesystem)

Marvel

Logstash

Base monitoring OS Level (Process, Filesystem)

Dashboard in Kibana to visualize the heartbeats

Kafka:

Base monitoring by OS Level (Process, Filesystem)

JMX input plugin in Logstash (planned)

Dashboard in Kibana for Visualization

Monitoring

ELK Architecture

7

© Z

uri

ch S

wit

ze

rla

nd

Version in use: 1.3

Authentication

Supported Authentication Realms

– File

– LDAP (Planned)

– Active Directory

– PKI (Option for Application based authentication)

Auditing

Role Based Access Control (RBAC)

Features

Shield (1/2)

8

© Z

uri

ch S

wit

ze

rla

nd

Model

Privilege Actions to execute in ES pre-defined list for secured objects

Secured objects ES Cluster Level (Monitor, Admin,) Index

Permission Privileges with a secured object

Role Named set of Permission

Role Based Access Control

Shield (2/2)

9

Examples

user:

cluster: monitor/health

indices:

'logstash-*,topbeat-*,packetbeat-*,

filebeat-*':

- indices:admin/mappings/fields/get

- indices:admin/validate/query

- indices:admin/get

- read

© Z

uri

ch S

wit

ze

rla

nd

Security Architecture

10

© Z

uri

ch S

wit

ze

rla

nd

Field Level Access Control

Protect certain fields without blocking whole index

Security Plugin Customizable

Single Sign On based on WS-Federation/SAML (Apache Fediz)

Features

Shield 2.0

11

© Z

uri

ch S

wit

ze

rla

nd

Integration Platform Services (Web Services based communication)

Security Token Service (Issue/Validate security token)

Service Locator (Find Service Endpoints)

Service Activity Monitoring (Message flow)

Generic Application Dashboards

Access Log

Application Log

Karaf/Talend ESB

Monitoring ELK Infrastructure

Logstash HeartBeat

JMX for Kafka

Dashboards

12

© Z

uri

ch S

wit

ze

rla

nd

Dashboards

13

© Z

uri

ch S

wit

ze

rla

nd

Dashboards

14

© Z

uri

ch S

wit

ze

rla

nd

Dashboards

15

© Z

uri

ch S

wit

ze

rla

nd

Questions

16