Streaming analytics on Google Cloud Platform, by Javier Ramirez, teowaki

End-to-end streaming analytics on Google Cloud Platform

From event capture to dashboard to monitoring

Javier Ramirez@supercoco9

Hard problemsand easy problems

And hard problems that look easy

3

Calculate the average of several numbers.

An easy problem

Calculate the average of several numbers. By the way, they might be MANY numbers. They will probably not fit in memory. They might not even fit in one file or on a single hard drive.

An easy big data problem

Calculate the average of several numbers. By the way, they might be MANY numbers. They will probably not fit in memory. They might not even fit in one file or on a single hard drive.

Truth is they will not be in one file, but they will be streamed live from different sensors…

An easy big data and streaming problem

Calculate the average of several numbers. By the way, they might be MANY numbers. They will probably not fit in memory. They might not even fit in one file or on a single hard drive.

Truth is they will not be in one file, but they will be streamed live from different sensors… In different parts of the world

A not so easy streaming data problem

Calculate the average of several numbers. By the way, they might be MANY numbers. They will probably not fit in memory. They might not even fit in one file or on a single hard drive.

Truth is they will not be in one file, but they will be streamed live from different sensors… In different parts of the world

Some sensors might send a few events per hour, some a few thousands per second…

An autoscaling streaming problem

Calculate the average of several numbers. By the way, they might be MANY numbers. They will probably not fit in memory. They might not even fit in one file or on a single hard drive.

Truth is they will not be in one file, but they will be streamed live from different sensors… In different parts of the world

Some sensors might send a few events per hour, some a few thousands per second… We want not just the total average of all the points, but the moving average every 30 seconds, for every sensor. And the hourly, daily, and monthly averages

A hard streaming analytics problem

Calculate the average of several numbers. By the way, they might be MANY numbers. They will probably not fit in memory. They might not even fit in one file or on a single hard drive.

Truth is they will not be in one file, but they will be streamed live from different sensors… In different parts of the world

Some sensors might send a few events per hour, some a few thousands per second… We want not just the total average of all the points, but the moving average every 30 seconds, for every sensor. And the hourly, daily, and monthly averages

Sometimes the sensors will have connectivity issues and will not send their data until later, but of course I want the calculations to still be correct

A real life analytics problem

All of the above, plus monitoring, alerts, self-healing, a way to query the data efficiently, and a pretty dashboard on top

What your client/boss will expectCalculate the average of several numbers. By the way, they might be MANY numbers. They will probably not fit in memory. They might not even fit in one file or on a single hard drive.

Truth is they will not be in one file, but they will be streamed live from different sensors… In different parts of the world

Some sensors might send a few events per hour, some a few thousands per second… We want not just the total average of all the points, but the moving average every 30 seconds, for every sensor. And the hourly, daily, and monthly averages

Sometimes the sensors will have connectivity issues and will not send their data until later, but of course I want the calculations to still be correct

… is easier

said than

done

Our complete system in 100 lines of Java, of which 90 are mostly

boilerplate and configuration

<= Don’t try to read that. I’ll zoom in later

14

What we needA streaming data pipeline components

Data acquisition

Data validation

Transformation / Aggregation VisualizationStorage/

Analytics

Monitoring and alerts of all the components

Data AcquisitionSending and receiving data at scale

16

Google Cloud Pub/Sub

Google Cloud Pub/Sub brings the scalability, flexibility, and reliability of enterprise message-oriented middleware to the cloud. By providing many-to-many, asynchronous messaging that decouples senders and receivers, it allows for secure and highly available communication between independently written applications.

Google Cloud Pub/Sub delivers low-latency, durable messaging that helps developers quickly integrate systems hosted on the Google Cloud Platform and externally.

Ingest event streams from anywhere, at any scale, for simple, reliable, real-time stream analytics

The spotify proof of conceptCurrently our production load peaks at around 700K events per second. To account for the future growth and possible disaster recovery scenarios, we settled on a test load of 2M events per second.

To make it extra hard for Pub/Sub, we wanted to publish this amount of traffic from a single data center, so that all the requests were hitting the Pub/Sub machines in the same zone. We made the assumption that Google plans zones as independent failure domains and that each zone can handle equal amounts of traffic.

In theory, if we’re able to push 2M messages to a single zone, we should be able to push number_of_zones * 2M messages across all zones.

Our hope was that the system would be able to handle this traffic on both the producing and consuming side for a long time without the service degrading.

https://labs.spotify.com/2016/03/03/spotifys-event-delivery-the-road-to-the-cloud-part-ii/

The spotify proof of concept

https://labs.spotify.com/2016/03/03/spotifys-event-delivery-the-road-to-the-cloud-part-ii/

They pushed 2 million events per second (to two topics) from 29 servers, non-stop, for five days.

“We did not observe any lost messages whatsoever during the test period.”

The no operations advantage

https://labs.spotify.com/2016/03/10/spotifys-event-delivery-the-road-to-the-cloud-part-iii/

Event Delivery System In Cloud

We’re actively working on bringing the new system to production. The preliminary numbers we obtained from running the new system in the experimental phase look very promising. The worst end-to-end latency observed with the new system is four times lower than the end-to-end latency of old system.

But boosting performance isn’t the only thing we want to get from the new system. Our bet is that by using cloud-managed products we will have a much lower operational overhead. That in turn means we will have much more time to make Spotify’s products better.

A truly global network

Pub/Sub now works with Cloud IOT Core

Device Manager

The device manager allows individual devices to be configured and managed securely in a coarse-grained way; management can be done through a console or programmatically. The device manager establishes the identity of a device, and provides the mechanism for authenticating a device when connecting. It also maintains a logical configuration of each device and can be used to remotely control the device from the cloud.

Protocol Bridge

The protocol bridge provides connection endpoints for protocols with automatic load balancing for all device connections. The protocol bridge has native support for secure connection over MQTT, an industry-standard IoT protocol. The protocol bridge publishes all device telemetry to Cloud Pub/Sub, which can then be consumed by downstream analytic systems.

Which is very cool if you are into Arduino, Raspberry PI, Android, or embedded systems

Storage & AnalyticsReliable, fast, scalable, and flexible. And no-ops

25

BigQuery

A database where you can send as much (or as little) data as you want, either batch or streaming, and run any SQL you want, no matter how big your data is.

Even if you have petabytes of data.

Even if you want to join data from different projects or from public data sources.

Even if you want to query external data on Spreadsheets or Cloud Storage.

Even if you want to create your own User Defined Functions in JavaScript.

BigQuery also...

… is serverless and zero configuration. You never have to worry about memory, CPU, network, or disk. You send your data, you send your queries, you get results.

Behind the scenes BigQuery will use up to 2000 CPUs in parallel for your queries, and a huge amount of networked storage. But you don’t care.

You pay for how much data you send and how much data you query. If you are not using the database, you are not paying anything. But it’s always available

Hope you are not easily impressed

How long it would take to read 4 Terabytes from a hard drive at 100 MB/s?

And to filter 100 billion data points using a regular expression for each?

And moving 278 GB across a 1 Gbps network?

Hope you are not easily impressed

How long it would take to read 4 Terabytes from a hard drive at 100 MB/s?

About 11 hours

And to filter 100 billion data points using a regular expression for each?

About 27 hours

And moving 278 GB across a 1 Gbps network?

About 40 minutes

Hope you are not easily impressed

Hope you are not easily impressed

We will use a simple table for our system

Data ValidationApparently simple, but always a pain

34

Google Cloud DataprepAn intelligent cloud data service to visually explore, clean, and prepare data for analysis

Cloud Dataprep: Explorer & Suggestions

Cloud Dataprep: Transforms & Scripts

Cloud Dataprep: No-ops execution & reports

Transformation & Aggregation

Batch and streaming ETL jobs, and data pipelines40

Apache BEAM: An advanced unified programming modelApache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.

Beam is particularly useful for Embarrassingly Parallel data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system.

Apache BEAM: A basic pipeline

Apache BEAM: Streaming is hard

Apache BEAM: Streaming is hard

Averages with BEAM: Overview

Boilerplate

and

configuration

Writing the output to BigQuery

This is the code that actually processes and aggregates the data

Start the pipeline

Averages with BEAM: Config

Averages with BEAM: Output to BigQuery

Averages with BEAM: The processing itself

Transform/Filter. We are just parsing a line of text into multiple fields

Aggregate. We are outputting the mean speed of the last minute per sensor, every 30 seconds

Google Cloud Dataflow: BEAM with no-operations

Google developed internally BEAM as a closed-source product. Then they realised it would make sense to open-source it and they donated it to the Apache community.

Anyone can use BEAM completely for free, and choose the runner in which to execute your pipeline.

Google Cloud Dataflow is a BEAM runner to execute your pipelines with no-operations, with logging, monitoring, auto-scaling, shuffling, and dynamic re-balancing.

It’s like BEAM, but as a managed service.

Demo time

51

Three instances in three continents

Our dataflow pipeline ready to accept data

Let’s start sending some data

Our dataflow pipeline seems to be working fine.

64 elements per second should be easy.

Let’s send much more data from all over!

Our dataflow pipeline starts feeling the heat.

Receiving 2440 elements per second now-

And now we are processing over 20,000 elements per second at the 1st step.

But the lag starts to increase

Auto scaling to the rescue. Three workers now

And lag goes back to normal

And back to just one worker

MonitoringWhat I can’t see, doesn’t exist

64

Google Stackdriver monitoring

Google Stackdriver monitoring

Google Stackdriver alerts

VisualizationBecause an image is worth a thousand logs

68

Google Data Studio

Google Data Studio: my dashboard

Google Data Studio: data sources

Google Data Studio: data sources

Google Data Studio: drag and drop

Google Data Studio: drag and drop

Almost thereJust one more slide

75

Cloud IoT Core

Cloud Dataprep

Stackdriver Monitoring Logging ErrorReporting

All together now!

CHEERS!I’m happy to answer any questions you may have at lunchtime or the coffee breaks.

Or ping me at @supercoco9 on twitter. You got 240 chars now

Demo source code available at:https://github.com/GoogleCloudPlatform/training-data-analyst/tree/master/courses/streaming/process

Javier Ramirez

End-to-end streaming analytics on Google Cloud Platform

From event capture to dashboard to monitoring

Javier Ramirez@supercoco9

Template Design Credits

The Template provides a theme with four basic colors:

The backgrounds were created by Free Google Slides Templates.

The original template for this presentation was provided by, and it’s property of, Free Google Slides Templates - http://freegoogleslidestemplates.com

Vectorial Shapes in this Template were created by Free Google Slides Templates and downloaded from pexels.com and unsplash.com.

Icons in this Template are part of Google® Material Icons and 1001freedownloads.com.

Shapes & Icons Backgrounds

Fonts Color PaletteThe fonts used in this template are taken from Google fonts. ( Dosis,Open Sans )You can download the fonts from the following url: https://www.google.com/fonts/ #93c47dff #0097a7ff

#78909cff #eeeeeeff

#f7b600ff #00ce00e3

#de445eff #000000ff