AWSSummit Berlin Day1 mohmeim Gain Meaningful Insights ... Mark… · Best analytics and search Log analytics fueling Elasticsearch growth Rank Project Name Overall Project Rating

S U MM I TBe r l i n

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT

Gain Meaningful Insights from your Customer Journey using Amazon Elasticsearch

Tina HowellCloud Practice LeadAND Digital

Imran DawoodSolution ArchitectAWS

Bruce SmithDigital Engineering LeadN Brown Group Plc

SUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.


The explosion of machine-generated data

Transition from IT to DevOps

Increase in IoT and Mobile Devices

Cloud-basedarchitectures

Machine-generated data is growing 10x faster than business data

Source: insideBIGDATA - The Exponential Growth of Data, February 16, 2017

https://insidebigdata.com/2017/02/16/the-exponential-growth-of-data/


Source: TechCrunch survey of popular open source software from April’17

Open source

Fast time to value

Easy ingestion

Easy visualization

High performance and distributed

Best analytics and search

Log analytics fueling Elasticsearch growthRank Project Name Overall Project

Rating

1 Linux 100.00

2 Git 31.10

3 MySQL 25.23

4 Node.js 22.75

5 Docker 22.61

6 Hadoop 16.19

7 Elasticsearch 15.72

8 Spark 14.99

9 MongoDB 14.68

10 Selenium 12.81

11 NPM 12.31

12 Redis 11.61

B E N E F I T S


Amazon Elasticsearch Service is a fully managed service that makes it easy to deploy, manage, and scale Elasticsearch and Kibana

AMAZON ELASTICSEARCH SERVICEA fully managed, scalable, secure Elasticsearch service


Benefits of Amazon Elasticsearch Service

Supports Open-Source APIs and Tools

Drop-in replacement with no need to learn new APIs or

skills

Easy to Use

Deploy a production-ready Elasticsearch cluster in

minutes

Scalable

Resize your cluster with a few clicks or a single API

call

Secure

Deploy into your VPC and restrict access using security

groups and IAM policies

Highly Available

Replicate across Availability Zones, with monitoring and

automated self-healing

Tightly Integrated withOther AWS Services

Seamless data ingestion, security, auditing and

orchestration


Leading use cases

Application Monitoring & Root Cause Analysis

Security Information and Event Management (SIEM)

IoT & Mobile Full-text Search

Provides developers with a high performance, self-service operational

monitoring and analytics platform

Enables security practitioners to centralize and analyze events from across the entire

organization

Gives developers and lines of business users real-time location-aware insights

into their device fleets

Provides applications with a highly performant and feature rich search

experience with support advanced features like faceting and type-ahead


Software & Internet Financial ServicesEducation Technology BioTech and Pharma

Media and Entertainment Social Media Telecommunications Travel & Transportation

Real Estate Logistics & Operations Publishing Other

Amazon Elasticsearch Service customers



Example architecture

Amazon Kinesis Firehose

Amazon CloudWatch Logs

Logstash

AWS IoT

Elasticsearchdata nodes

Kibana

Data Producers Buffer Transform Deliver Amazon Elasticsearch Service

Elasticsearchmaster nodes

Analytics UI


Amazon Lambda architecture

S3 Events

Amazon S3 AWS LambdaFunction

Amazon Elasticsearch Service

Amazon Kinesis

Files

Amazon DynamoDB

Table

DynamoDB streams

AWS LambdaFunction

Data Producers

AWS LambdaFunction



Company Background



Why?• Provide secure, resilient, performant and flexible platforms that enable

a great customer experience

• Reduce our time to market, by improving the flow of changes though the engineering process

• Shorten and strengthen the technical feedback loops, which in turn improves our customer experience

• Give us the data to make the right decisions by greater visibility and transparency into how our digital products behave


Our AWS journey

Development Environments

• EC2 instances used for development and initial testing

• On demand provisioning giving flexibility

Establish Cloud Foundations

• Established best practice account structure

• Implemented initial platform tooling

• Agreed first platform to be built using infrastructure as code

Logging and Monitoring

Platform

• Centralised logs from different platforms

• Built in AWS using ElasticSearch, Filebeat, Logstash and many more tools

Data Platform

• For Financial and Transactional reconciliation from different providers/ platforms

• Built in AWS using Kinesis, Lamda, S3 and AWS Elastic Search

Hybris

• Moving e-commerce platform into the cloud to provide scalability, resilience and flexibility

• Built in EC2, Loadbalancers, Oracle RDS



Logging and Monitoring - High-level architecture

AWS Cloud

Corporate data center

Internet(Whitelisted

Public IP Addresses)

Logstash Aggregator

Traditional servers

(filebeatAgent)

FilebeatAgent



Logging and Monitoring – AWS Architecture

AWS Cloud


MONITORING VPC

TOOLS VPC

Availability zone

Availability zone

Auto Scaling groupInstanceInstance

Availability zone

Availability zone

Auto Scaling group

GRAFANA

KIBANA PROXYFORWARDERS VPC

Availability zone

Availability zone


LOGSTASH FORWARDER

Corporate data center

Logstash Aggregator

Traditional servers

(filebeatAgent)

FilebeatAgent

Amazon Cognito


Logging and Monitoring – AWS Architecture


MONITORING VPC

TOOLS VPC

Availability zone

Availability zone


Availability zone

Availability zone

Auto Scaling group

Corporate data

center

GRAFANA

KIBANA PROXY

FORWARDERS VPC

Availability zone

Availability zone


LOGSTASH FORWARDER

AWS Cloud

Availability zone

Availability zone

Auto Scaling group

INFLUXDB InstanceInstance

InstanceInstance

INFLUXDB RELAY


Here are some of the technical challenges?• Establishing a secure network connection to AWS

• Complex to get out of the managed service datacentre• URL Filtering on firewalls• Outbound DNS needed to be enabled

• Different log types• Syslogs easy to convert• General Application Logs could be transformed using GROK• Custom Logs needed to look at different solutions


What insights have we gained?• Identified errors in our key customers

journeys• Enabled us to prioritize defect fixes against new

functionality development• Enabled us to identify issues in test environments more

quickly (i.e. shift left)

• Application logging needs to be standardized and improved• Developers need consider logging in the development

process• Errors need to have meaningful messages in the logs • Focus logging in the right places

Transition from IT to DevOps



What do we use our Data Platform for?• To reconcile transactions giving a single view of customer order and

payment transactions

• Provide a data stream that other applications can subscribe to

• Identify and correct issues before they impact the customer experience

• Improve the quality of future application and platform releases though enhanced end-to-end test capability


Data Platform – Architecture View

AWS CloudCorporate

data center

Amazon API Gateway

Amazon KinesisLambda function


Amazon Athena


Data Platform – Data Flow

KINESIS STREAM- RAW

KINESIS STREAM- MASTER

KINESIS STREAM- TRANSFORMED

Corporate data

center

INITIAL DOCUMENTXML.CSV

JSONJSON MODIFIED

TRANSFORM TRANSFORM

AWS Cloud

Lambda function Lambda function


Amazon Athena



What have we learnt?

• To achieve a reduced time to market and increased flow we need stronger feedback loops

• Education, education, education ….

• Use of tools to help drive cultural transformation


What’s next?• Focus on transition of Hybris onto AWS which enables further brand

migrations

• Improved logging standards giving a clearer link to customer experience

• Build stronger deployment pipelines with more automation, faster and more predictable releases

• Education and training to embed changes in our ways of working

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM ITSUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

To achieve our technology strategy without utilising AWS would be like walking barefoot along a path of Lego bricks; painful, slow, and whilst entertaining for others to watch, ultimately needless

Adam WarneCIO

N Brown Group Plc


Thank you!


Tina HowellCloud Practice LeadAND Digital

Imran DawoodSolution ArchitectAWS

Bruce SmithDigital Engineering LeadN Brown Group Plc

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM ITSUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Documents

AWSSummit Berlin Day1 mohmeim Gain Meaningful Insights ... Mark… · Best analytics and search Log analytics fueling Elasticsearch growth Rank Project Name Overall Project Rating