Why GCP over AWS -...

April, 2018

GCP Workshop

Koivu Solutions Oy

• Porist – already since 2017

• A Solutions Company • Knowledge to help define the digital

roadmap• Capabilities and tools to rapidly develop

digital pilots to showcase benefits • Deployment experience to expand

breakthrough digital technologies to the entire organization

• Extensive Fortune Global 1000 experience• Expertise developed across many digital

assessments in multiple industries• More than 25 years enterprise solutions

development experience• More than 100 enterprise customer projects

delivered across 4 continents

Koivu Founding Team – Born Global

Sami LahtiSenior Technology ArchitectFormer JDA, i2, Innomat

Samu LahtiSenior Software ArchitectFormer JDA, i2, Tieto

Chris MorhardSenior Business ArchitectFormer E2open, MercuryGate, JDA, i2

Janne SalmiSenior Business ArchitectFormer Chainalytics, ROCE Partnerts, McKinsey

Harri RajalaSenior Software ArchitectFormer JDA, i2, Innomat

Our team’s work has been for industry giants

Stop-and-go mentality is Achilles' heel of today’s enterprise software

• Nightly batch processing stops the business. 24 hour day is not 24 hour business day.

• Diminishing nightly time windows for data batch processing are creating pressure for systems to be available earlier on the next morning with data good enough for tomorrow’s business.

• These systems are sequence oriented, not capable to do parallel data processing and hard to scale. Peak load capacity needs to be always on-line.

runStop and

load runStop and

loadStop and

Modern way to build enterprise applications is to connect the dots on cloud platform

IoT and Data Analytics solutions are mean to be real-time.

Message queue is a core component of a modern enterprise application

• When you need messages to be delivered?• People send messages all the time: email, SMS, Chat, Comments, … • These are just fraction of messages systems send to each others.

• Traditionally many message type processes have been managed as batch load, for example nightly. Today real-time operation is a business requirement.

• Message hub consumes messages and sends them for others to operate.• Works as shock absorber for load fluctuation.• Message content can be ‘anything’.

• Message buffering and guaranteed deliver works as enablers for high availability for the complete communication chain.• Publishers and Subscribers work independent and unknowingly of each others. • There can be many topics / channels on which messages are published.• A new consumer can join to listen the messages without changes to publisher(s).

Google Cloud Pub / Sub is a high-available messaging hub for asynchronous communication

• Cloud Pub/Sub is a simple, reliable, scalable foundation for stream analytics and event-driven computing systems.

• As part of Google Cloud’s stream analytics solution, the service ingests event streams and delivers them to Cloud Dataflow for processing and BigQuery for analysis as a data warehousing solution.

Spotify is a music streaming service, and also generates huge amount of messages.

“Whenever a user performs an action in the Spotify client—such as listening to a song or searching for an artist—a small piece of information, an event, is sent to our servers. Event delivery, the process of making sure that all events gets transported safely from clients all over the world to our central processing system, is an interesting problem. ”

https://labs.spotify.com/2016/03/03/spotifys-event-delivery-the-road-to-the-cloud-part-ii/

• Spotify is now on their 3rd generation of messaging solutions (2016).

• How do they handle millions of messages 24/7?

Spotify did a massive Pub / Sub Performance Test

• Spotify used 29 servers to send messages to Pub/Sub to generate continuous test load of 2 million messages.

• “Enabling batching and compression on the Event Service machines resulted in ~1Gbps of network traffic towards Pub/Sub.”

• “Pub/Sub passed the test with flying colours. We published 2M messages [per second] without any service degradation and received almost no server errors from the Pub/Sub backend.”

• With current Google prices (2018) this would generate cost of $17 per hour. • $0,04 per Gb

• 330TB monthly rate.

The second part of the test was to let the Pub/Sub buffer messages for one hour before starting to consume them.

• 7.2 billion messages buffered

• Consumption was just slightly higher rate than publishing.

• It took 8 hours to catch up the backlog.

• No messages were observed to be lost!

• Pub/Sub managed the backlog and hide the ‘blackout’ of the back-end system.

What was required from Spotify to setup, run and maintain Pub/Sub for this size of scalability test?

Absolutely nothing.

Pub / Sub is fully managed, unconfigurable, global service and

operated entirely by Google.

Not even capacity reservations needed, no system configuration, no

administration.

It just runs. Like a telecommunication switch for

phones.

“Based on these tests, we felt confident that Cloud Pub/Sub was the right choice

for us. Latency was low and consistent, and the

only capacity limitations we encountered was the one explicitly set

by the available quota. In short, choosing Cloud Pub/Sub rather

than Kafka 0.8 for our new event delivery platform was an obvious

choice.”

Serverless – NoOps

BigQuery is Google's serverless, highly scalable, low cost enterprise data warehouse designed to make all your data analysts productive.

• Because there is no infrastructure to manage, you can focus on analyzing data to find meaningful insights using familiar SQL and you don't need a database administrator.

• BigQuery enables you to analyze all your data by creating a logical data warehouse over managed, columnar storage as well as data from object storage, and spreadsheets.

• BigQuery makes it easy to securely share insights within your organization and beyond as datasets, queries, spreadsheets and reports.

• BigQuery allows organizations to capture and analyze data in real-time using its powerful streaming ingestion capability so that your insights are always current.

• BigQuery is free for up to 1TB of data analyzed each month and 10GB of data stored.

Public data set used: New York Taxi Trips -2016

Table ID nyc-tlc:yellow.trips

Table Size 130 GB

Long Term Storage Size 130 GB

Number of Rows 1,108,779,463

More than a billion row data set

This dataset includes trip records from all trips completed in yellow taxis in NYC since 2009. Records include fields capturing pick-up and drop-off dates/times, pick-up and drop-off locations, trip distances, itemized fares, rate types, payment types, and driver-reported passenger counts. The data used in the attached datasets were collected and provided to the NYC Taxi and Limousine Commission (TLC) by technology providers authorized under the Taxicab Passenger Enhancement Program (TPEP).

Calculate average price for trips –according to passenger count

• Categorize all 1.1 billion rows by passenger count.

• Calculate average price, distance and time for each category.• So use numbers from every single 1.1

billion rows.

• 41Gb data scanned.

• 6 seconds execution time.

• Theoretical query cost…invisible.

• (Data set cost: Google provides public data sets)

SELECTCASEWHEN passenger_count <= 0 THEN 'Unknown'WHEN passenger_count >= 7 THEN '>= 7'ELSE STRING(passenger_count)

END AS PassengerCount,COUNT(passenger_count) AS TripCount,ROUND(AVG(total_amount),2) AS AverageTotalAmout,ROUND(AVG(trip_distance),2) AS AverageTripDistance,TIME(SEC_TO_TIMESTAMP(AVG((dropoff_datetime-pickup_datetime)/1000/1000))) AS

AverageTripDurationFROM

[nyc-tlc:yellow.trips]GROUP BY

PassengerCountORDER BY

PassengerCount

Demo can be run online using this

Demo Recording

Add more calculation without increase data scanned: no more extra cost

• Find also max trip price.

• The same column used for both average and max calculation.

• No extra costs.

• Less than 6 seconds execution time.

SELECTCASE

WHEN passenger_count <= 0 THEN 'Unknown'WHEN passenger_count >= 7 THEN '>= 7'ELSE STRING(passenger_count)

END AS PassengerCount,COUNT(passenger_count) AS TripCount,ROUND(AVG(total_amount),2) AS AverageTotalAmout,ROUND(MAX(total_amount),2) AS MaxTotalAmout,ROUND(AVG(trip_distance),2) AS AverageTripDistance,TIME(SEC_TO_TIMESTAMP(AVG((dropoff_datetime-

pickup_datetime)/1000/1000))) AS AverageTripDurationFROM

PassengerCount

Adding more results columns means that more data is scanned

• Find also average tip.

• A new column introduced into query.

• More data scanned as BigQuery is columnar store.

• Still Less than 6 seconds execution time – parallel processing.

SELECTCASE

WHEN passenger_count <= 0 THEN 'Unknown'WHEN passenger_count >= 7 THEN '>= 7'ELSE STRING(passenger_count)

END AS PassengerCount,COUNT(passenger_count) AS TripCount,ROUND(AVG(total_amount),2) AS AverageTotalAmout,ROUND(MAX(total_amount),2) AS MaxTotalAmout,ROUND(AVG(tip_amount),2) AS AverageTipAmout,ROUND(AVG(trip_distance),2) AS AverageTripDistance,TIME(SEC_TO_TIMESTAMP(AVG((dropoff_datetime-

pickup_datetime)/1000/1000))) AS AverageTripDurationFROM

PassengerCount

Serverless – NoOps

Google Cloud Dataflow is service for real-time data stream processing

• Cloud Dataflow is a fully-managed service for transforming and enriching data in stream (real time) and batch (historical) modes with equal reliability and expressiveness -- no more complex workarounds or compromises needed.

• And with its serverless approach to resource provisioning and management, you have access to virtually limitless capacity to solve your biggest data processing challenges, while paying only for what you use.

• Hours of nightly runs become distributed around the clock as minute and second size small ‘mini-batches’.

• Windowing for data processing is continuous and data volumes fluctuating up and down requiring dynamic software architecture.

Delay to react to the new data and intelligent decisions becomes minutes instead of days.

Dynamic solution runs continuously and manages changes using small time windows

Dataflow combines batch and stream operations when needed

• The same logic, so code, is used to process both batch and stream data.

• No double maintenance for two different type of operation.

Example Dataflow implementation for real-time metrics and log data processing

• Statics calculation of 300+ metrics coming at 20 messages per second:

• Sums, averages, 10%, 90%, medians• Baseline compute, top contributor compute• Minute, hourly, daily values• Window statistics and cumulative statistics• Bottom up statistcs from instance, environment,

application, customer, company.• Log, exception and error file processing.• Real-time processing of constant data stream costs

about $300 per month.

• Great performance. Cloud Dataflow is 2-3x faster and cheaper than Hadoop when evaluating classic MapReduce based pipelines, such as PageRank and WordCount.

• And with dynamic work rebalancing, Cloud Dataflow effectively optimizes resource utilization which provides additional performance gains without requiring manual intervention.

Customers are saying…

Hands on Workshop

Ingest Pipelines

Storage

Analytics

Application &

Presentation

Standard

Devices

Constrained

Devices

Non-TCP

e.g. BLE

Gateway

Real Time Stream Processing - Internet of Things

Engine

Container

Engine

Storage

Pub/Sub

Studio

Monitoring

Logging

Dataflow

Datastore

Bigtable

BigQuery

Dataproc

Datalab

Compute

Engine

Entry Point

IoT Core

Functions

Ingest Pipelines Analytics

Standard

Devices

Real Time Stream Processing - Internet of Things

Pub/SubBigQuery

Functions

Analytics

Studio

1.4.3. 2.

Demo Data

• Tieliikenne data

• https://www.liikennevirasto.fi/avoindata/tietoaineistot/lam-tiedot

• http://digitraffic.liikennevirasto.fi/

• Pori, Nakkila, Karkkila, Vihti LAM-asemat ( liikenteen automaattisetmittausasema, induktiosilmukat) ajoneuvonopeudetajoneuvotyypeittäin 1.4.2018.

• http://digitraffic.liikennevirasto.fi/tieliikenne/#ajantasaiset-lam-mittaustiedot

GitHub Link

• Guidance to workshop details and source code:

• https://github.com/koivusolutions/koivu-workshops

• Open this to other browser window.

Cloud Console - https://console.cloud.google.com/

Big Query – Create New Dataset

Create Table

Import Data to Table (Create Table)

Preview Table

Query Table

Data Studio - https://datastudio.google.com

Add Data Source

Create Data Connector

Creating Data Connector

Add Table

Add Chart

Add Filter

Extra Report Challenge

Pub/Sub Real-Time Data into Pipeline -https://console.cloud.google.com/cloudpubsub

Create Topic

Manual operations to Pub/Sub

Cloud Functions - Workshop

Cloud Functions -Create

Publish Message from UI

Did Cloud Function Activate?

Do you see data?

What Next?

• Dataprep

• Dataflow

• Datalab

• Machine Learning

• Big Table

• IoT Core

• …

Why GCP over AWS -...

Documents

IBP1086 09 PIPEL INE MAPPING AND STRAIN ASSESSMENT …

The Nightly News - Spring 2011

GCP guidelines

Nightly Sky Guide

ICH GCP guidelines

Report on the Pilot EMA-FDA GCP Initiative on the Pilot EMA-FDA GCP Initiative EXT/INS/GCP/56289/2011 Page 3/30 Report on the Pilot EMA-FDA GCP Initiative EXT/INS/GCP/56289/2011 Page

Differences between indian gcp and ich-gcp

Daily & Nightly Supplications

GCP South Africa

5. Pipel iicr Mf ables · 5. Pipel iicr Mf "ables The PEA (USOI, MMS ivi '} contains e description of the impacts of structure-removal ect ivi cie* or pipelines and cables

GCP INSPECTORATE GCP INSPECTIONS METRICS REPORT - gov.uk · PDF fileGCP INSPECTORATE GCP INSPECTIONS METRICS REPORT ... GCP Compliance IMP Management/ Pharmacy ... Project/Trial Management

Nightly Ghost List

GCP Applied Technologiesinvestor.gcpat.com/~/media/Files/G/Gcp-Applied-Tech-IR-V2/...GCP Applied Technologies First Quarter 2017 ©2017 GCP Applied Technologies Inc

Classification and analysis of the GCP inspection findings of GCP

Nightly System Growth Graphs

Higher Mathematics Nightly Questions

Instek GCP-100 Current Probe - Data Sheet - TestEquity · PDF fileTitle: Instek GCP-100 Current Probe - Data Sheet Author: Instek Subject: GCP-100 Current Probe Keywords: GCP-100,

Nightly Rate: $18,000

Detailed GCP Presentation

GCP Presentation