S U MM I TBe r l i n
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Gain Meaningful Insights from your Customer Journey using Amazon Elasticsearch
Tina HowellCloud Practice LeadAND Digital
Imran DawoodSolution ArchitectAWS
Bruce SmithDigital Engineering LeadN Brown Group Plc
SUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
The explosion of machine-generated data
Transition from IT to DevOps
Increase in IoT and Mobile Devices
Cloud-basedarchitectures
Machine-generated data is growing 10x faster than business data
Source: insideBIGDATA - The Exponential Growth of Data, February 16, 2017
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Source: TechCrunch survey of popular open source software from April’17
Open source
Fast time to value
Easy ingestion
Easy visualization
High performance and distributed
Best analytics and search
Log analytics fueling Elasticsearch growthRank Project Name Overall Project
Rating
1 Linux 100.00
2 Git 31.10
3 MySQL 25.23
4 Node.js 22.75
5 Docker 22.61
6 Hadoop 16.19
7 Elasticsearch 15.72
8 Spark 14.99
9 MongoDB 14.68
10 Selenium 12.81
11 NPM 12.31
12 Redis 11.61
B E N E F I T S
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Amazon Elasticsearch Service is a fully managed service that makes it easy to deploy, manage, and scale Elasticsearch and Kibana
AMAZON ELASTICSEARCH SERVICEA fully managed, scalable, secure Elasticsearch service
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Benefits of Amazon Elasticsearch Service
Supports Open-Source APIs and Tools
Drop-in replacement with no need to learn new APIs or
skills
Easy to Use
Deploy a production-ready Elasticsearch cluster in
minutes
Scalable
Resize your cluster with a few clicks or a single API
call
Secure
Deploy into your VPC and restrict access using security
groups and IAM policies
Highly Available
Replicate across Availability Zones, with monitoring and
automated self-healing
Tightly Integrated withOther AWS Services
Seamless data ingestion, security, auditing and
orchestration
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Leading use cases
Application Monitoring & Root Cause Analysis
Security Information and Event Management (SIEM)
IoT & Mobile Full-text Search
Provides developers with a high performance, self-service operational
monitoring and analytics platform
Enables security practitioners to centralize and analyze events from across the entire
organization
Gives developers and lines of business users real-time location-aware insights
into their device fleets
Provides applications with a highly performant and feature rich search
experience with support advanced features like faceting and type-ahead
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Software & Internet Financial ServicesEducation Technology BioTech and Pharma
Media and Entertainment Social Media Telecommunications Travel & Transportation
Real Estate Logistics & Operations Publishing Other
Amazon Elasticsearch Service customers
SUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Example architecture
Amazon Kinesis Firehose
Amazon CloudWatch Logs
Logstash
AWS IoT
Elasticsearchdata nodes
Kibana
Data Producers Buffer Transform Deliver Amazon Elasticsearch Service
Elasticsearchmaster nodes
Analytics UI
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Amazon Lambda architecture
S3 Events
Amazon S3 AWS LambdaFunction
Amazon Elasticsearch Service
Amazon Kinesis
Files
Amazon DynamoDB
Table
DynamoDB streams
AWS LambdaFunction
Data Producers
AWS LambdaFunction
SUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Company Background
SUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Why?• Provide secure, resilient, performant and flexible platforms that enable
a great customer experience
• Reduce our time to market, by improving the flow of changes though the engineering process
• Shorten and strengthen the technical feedback loops, which in turn improves our customer experience
• Give us the data to make the right decisions by greater visibility and transparency into how our digital products behave
SUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our AWS journey
Development Environments
• EC2 instances used for development and initial testing
• On demand provisioning giving flexibility
Establish Cloud Foundations
• Established best practice account structure
• Implemented initial platform tooling
• Agreed first platform to be built using infrastructure as code
Logging and Monitoring
Platform
• Centralised logs from different platforms
• Built in AWS using ElasticSearch, Filebeat, Logstash and many more tools
Data Platform
• For Financial and Transactional reconciliation from different providers/ platforms
• Built in AWS using Kinesis, Lamda, S3 and AWS Elastic Search
Hybris
• Moving e-commerce platform into the cloud to provide scalability, resilience and flexibility
• Built in EC2, Loadbalancers, Oracle RDS
SUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Logging and Monitoring - High-level architecture
AWS Cloud
Corporate data center
Internet(Whitelisted
Public IP Addresses)
Logstash Aggregator
Traditional servers
(filebeatAgent)
FilebeatAgent
Amazon Elasticsearch Service
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Logging and Monitoring – AWS Architecture
AWS Cloud
Amazon Elasticsearch Service
MONITORING VPC
TOOLS VPC
Availability zone
Availability zone
Auto Scaling groupInstanceInstance
Availability zone
Availability zone
Auto Scaling group
GRAFANA
KIBANA PROXYFORWARDERS VPC
Availability zone
Availability zone
Auto Scaling groupInstanceInstance
LOGSTASH FORWARDER
Corporate data center
Logstash Aggregator
Traditional servers
(filebeatAgent)
FilebeatAgent
Amazon Cognito
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Logging and Monitoring – AWS Architecture
Amazon Elasticsearch Service
MONITORING VPC
TOOLS VPC
Availability zone
Availability zone
Auto Scaling groupInstanceInstance
Availability zone
Availability zone
Auto Scaling group
Corporate data
center
GRAFANA
KIBANA PROXY
FORWARDERS VPC
Availability zone
Availability zone
Auto Scaling groupInstanceInstance
LOGSTASH FORWARDER
AWS Cloud
Availability zone
Availability zone
Auto Scaling group
INFLUXDB InstanceInstance
InstanceInstance
INFLUXDB RELAY
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Here are some of the technical challenges?• Establishing a secure network connection to AWS
• Complex to get out of the managed service datacentre• URL Filtering on firewalls• Outbound DNS needed to be enabled
• Different log types• Syslogs easy to convert• General Application Logs could be transformed using GROK• Custom Logs needed to look at different solutions
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
What insights have we gained?• Identified errors in our key customers
journeys• Enabled us to prioritize defect fixes against new
functionality development• Enabled us to identify issues in test environments more
quickly (i.e. shift left)
• Application logging needs to be standardized and improved• Developers need consider logging in the development
process• Errors need to have meaningful messages in the logs • Focus logging in the right places
Transition from IT to DevOps
SUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
What do we use our Data Platform for?• To reconcile transactions giving a single view of customer order and
payment transactions
• Provide a data stream that other applications can subscribe to
• Identify and correct issues before they impact the customer experience
• Improve the quality of future application and platform releases though enhanced end-to-end test capability
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Data Platform – Architecture View
AWS CloudCorporate
data center
Amazon API Gateway
Amazon KinesisLambda function
Amazon Elasticsearch Service
Amazon Athena
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Data Platform – Data Flow
KINESIS STREAM- RAW
KINESIS STREAM- MASTER
KINESIS STREAM- TRANSFORMED
Corporate data
center
INITIAL DOCUMENTXML.CSV
JSONJSON MODIFIED
TRANSFORM TRANSFORM
AWS Cloud
Lambda function Lambda function
Amazon Elasticsearch Service
Amazon Athena
SUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
What have we learnt?
• To achieve a reduced time to market and increased flow we need stronger feedback loops
• Education, education, education ….
• Use of tools to help drive cultural transformation
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
What’s next?• Focus on transition of Hybris onto AWS which enables further brand
migrations
• Improved logging standards giving a clearer link to customer experience
• Build stronger deployment pipelines with more automation, faster and more predictable releases
• Education and training to embed changes in our ways of working
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM ITSUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
To achieve our technology strategy without utilising AWS would be like walking barefoot along a path of Lego bricks; painful, slow, and whilst entertaining for others to watch, ultimately needless
Adam WarneCIO
N Brown Group Plc
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM IT
Thank you!
SUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tina HowellCloud Practice LeadAND Digital
Imran DawoodSolution ArchitectAWS
Bruce SmithDigital Engineering LeadN Brown Group Plc
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMM ITSUMM IT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.