Building Your First Big Data Application on AWS

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

David Elliott, Solutions Architecture Manager, AWS

Anvar Karimson, Senior Application Developer, ICAP

7th July 2016

Building your First Big Data App on

AWS

Your First Big Data Application on AWS

PROCESS

STORE

ANALYZE & VISUALIZE

COLLECT

Did NASA’s STS-69 Mission Land …

… on the right homepage?

AccessLog - Common Log Format (CLF)

75.35.230.210 - - [20/Jul/2009:22:22:42 -0700]

"GET /images/pigtrihawk.jpg HTTP/1.1" 200 29236


PROCESS: Amazon EMR with Spark & Hive

STORE

ANALYZE & VISUALIZE: Amazon Redshift and Amazon QuickSight

COLLECT: Amazon Kinesis Firehose

Demo

http://aws.amazon.com/big-data/use-cases/

London, 7 July 2016

Front Office Analytics at Scale

ICAP Front Office Technology

Anvar Karimson

ICAP Global Broking – Front Office Technology

• ICAP, leading markets operator and provider of post trade risk

mitigation and information services

• FTSE 250 company founded in 1986 – 30 years

• Operates in more than 60 locations in 32 countries

• More than 4000 employees

• Revenues over £1.2B

• Active investor in Fintech companies

The Challenge - Analytics prior to AWS

• Bespoke implementation per application

• Difficult to run analytics across applications

• Scalability-issues restricting number of

consumers

• Batch-oriented – No streaming

• On-premise solution restrictive in terms of events

that could be captured

The Solution – The Event Stream

• Amazon Kinesis combined with Amazon Redshift

• Standardised way of collecting, transforming, and analysing data across our estate

• Encrypted by default

• Amazon Kinesis allows for tapping into data as it is being streamed• Empirical evidence suggests ~30 seconds between data being ingested and it being available for

queries in Amazon Redshift

• Fine-grained access/resource controls allow us to grant access to a large number of consumers in a controlled and secure manner

• Much higher fidelity of events• Authentication/Authorization Events

• Connection Quality Events

• Business Events

• Metrics

• Quality-of-Service

• End-to-end view of every action performed by a user across the whole estate over the complete user lifecycle

Using AWS enabled

ICAP to deliver an

industrial-grade

analytics platform in

weeks, dramatically

decreasing costs

compared to an on-

premise solution

Greatest benefit of AWS

What’s next with AWS

• Integrate with Amazon Elastic MapReduce and

Spark Streaming for real-time analytics and

anomaly detection

• Expand the use of the Event Stream throughout

the organization

• Evaluate Amazon QuickSight as a complement

to existing business intelligence tools


PROCESS: Amazon EMR with Spark & Hive

STORE

ANALYZE & VISUALIZE: Amazon Redshift and Amazon QuickSight

COLLECT: Amazon Kinesis Firehose

Demo


Estimated pricing to run this demo is as follows:

Service Est. Cost*

Amazon EC2 $0.00

Amazon Kinesis Firehose $0.04

Amazon S3 $0.51

Amazon EMR $2.28

Amazon Redshift $0.00

Amazon QuickSight $12.00

Est. Total $14.83

Pricing estimate

*Estimated costs assumes: use of free tier where available, lower cost instances, dataset no bigger than 100MB and instances running

for 3 hours. Costs may vary depending on options selected, size of dataset, and usage.

DIYDownload steps: http://bit.ly/29fhcwu


Please remember to rate this

session under My Agenda on

awssummit.london