Dive into Streams with Brooklin · Issues with KMM: didn’t scale well, diﬀicult to operate and...

Dive into Streams with Brooklin

Celia KungLinkedIn

Outline

Background

Scenarios

Application Use Cases

Architecture

Current and Future

Background

Nearline Applications

• Require near real-time response

• Thousands of applications at LinkedIn

○ E.g. Live search indices, Notifications

• Require continuous, low-latency access to data

○ Data could be spread across multiple database systems

• Need an easy way to move data to applications

○ App devs should focus on event processing and not on data access

Heterogeneous Data Systems

Espresso (LinkedIn’s

document store)

MicrosoftEventHubs

Building the Right Infrastructure

• Build separate, specialized solutions to stream data from and to each different system?

○ Slows down development

○ Hard to manage!

MicrosoftEventHubs

Streaming System C

Streaming System D

Streaming System A

Streaming System B

Need a centralized, managed, and extensible service to continuously deliver data in near real-time

Brooklin

• Streams are dynamically provisioned and individually configured

• Extensible: Plug-in support for additional sources/destinations

• Streaming data pipeline service

• Propagates data from many source types to many destination types

• Multitenant: Can run several thousand streams simultaneously

Kinesis

EventHubs

Kinesis

EventHubs

Databases

Messaging SystemsSo

sEspresso

Pluggable Sources & Destinations

Scenarios

Change Data Capture

Scenario 1:

Capturing Live Updates

1. Member updates her profile to reflect her recent job change

2. LinkedIn wants to inform her colleagues of this change

Member DB

Updates

News Feed Service

Member DB

Updates

News Feed Service

Search Indices Service

Member DB

UpdatesNews Feed

Service

queryNotifications

Service

Standardization Service

Member DB

UpdatesNews Feed

Service

queryNotifications

Service

Member DB

UpdatesNews Feed

Service

queryNotifications

Service

• Isolation: Applications are decoupled from the sources and don’t compete for resources with online queries

• Applications can be at different points in change timelines

• Brooklin can stream database updates to a change stream

• Data processing applications consume from change streams

Change Data Capture (CDC)

Messaging System

News Feed Service

Notifications Service

Member DB

Updates

Streaming Bridge

Scenario 2:

● Across…

○ cloud services

○ clusters

○ data centers

Stream Data from X to Y

Streaming Bridge

• Data pipe to move data between different environments

• Enforce policy: Encryption, Obfuscation, Data formats

● Aggregating data from all data centers into a centralized place

● Moving data between LinkedIn and external cloud services (e.g. Azure)

● Brooklin has replaced Kafka MirrorMaker (KMM) at LinkedIn

○ Issues with KMM: didn’t scale well, difficult to operate and manage, poor

failure isolation

Mirroring Kafka Data

Use Brooklin to Mirror Kafka Data

DestinationsSources

Messaging systems

MicrosoftEventHubs

Messaging systems

MicrosoftEventHubs

Databases Databases

Kafka MirrorMaker

Topology

Datacenter B

aggregate tracking

tracking

Datacenter A

aggregate tracking

tracking

aggregate metrics

metrics

aggregate metrics

metrics

Datacenter C

aggregate tracking

tracking

aggregate metrics

metrics

KMM KMM

KMM KMM KMM

KMM KMM KMM KMM KMM KMM

... ... ...

Brooklin Kafka Mirroring Topology

Datacenter A

aggregate tracking

tracking

Brooklin

metrics

aggregate metrics

Datacenter B

aggregate tracking

tracking metrics

aggregate metrics

Datacenter C

aggregate tracking

tracking metrics

aggregate metrics

...Brooklin Brooklin

Brooklin Kafka Mirroring

● Optimized for stability and operability

● Manually pause and resume mirroring at every level

○ Entire pipeline, topic, topic-partition

● Can auto-pause partitions facing mirroring issues

○ Auto-resumes the partitions after a configurable duration

● Flow of messages from other partitions is unaffected

Security

Search Indices

Security

Search Indices

ETL or Data warehouse

Security

Search Indices

Materialized Views or Replication

Security

Search Indices

Materialized Views or Replication

Repartitioning

Adjunct Data

Bridge

Adjunct Data

Bridge

Serde, Encryption, Policy

Adjunct Data

Bridge

Serde, Encryption, Policy

Standardization, Notifications …

Adjunct Data

Architecture

Stream updates made to Member Profile

Example:

Member DB

Updates

News Feed Service

● Scenario: Stream Espresso Member Profile updates into Kafka

○ Source Database: Espresso (Member DB, Profile table)

○ Destination: Kafka

○ Application: News Feed service

Example

• Describes the data pipeline

• Mapping between source and destination

• Holds the configuration for the pipeline

Datastream

Name: MemberProfileChangeStream

Source: MemberDB/ProfileTableType: EspressoPartitions: 8

Destination: ProfileTopicType: KafkaPartitions: 8

Metadata: Application: News Feed service Owner: newsfeed@linkedin.com

1. Client makes REST call to create datastream

Brooklin Client

Load Balancer

ZooKeeper

Brooklin Instance

Datastream Management Service (DMS) Espresso Consumer

Coordinator (Leader)

Kafka ProducerBrooklin Instance

Coordinator

Kafka Producer

Brooklin Instance

Coordinator

Kafka Producer

Member DB

create POST /datastreamNews Feed

service

2. Create request goes to any Brooklin instance

Brooklin Client

Load Balancer

ZooKeeper

Brooklin Instance

Coordinator

Kafka Producer

Brooklin Instance

Coordinator

Kafka Producer

Member DBNews Feed

service

3. Datastream is written to ZooKeeper

Brooklin Client

Load Balancer

ZooKeeper

Brooklin Instance

Coordinator

Kafka Producer

Brooklin Instance

Coordinator

Kafka Producer

Member DBNews Feed

service

4. Leader coordinator is notified of new datastream

Brooklin Client

Load Balancer

ZooKeeper

Brooklin Instance

Coordinator

Kafka Producer

Brooklin Instance

Coordinator

Kafka Producer

Member DBNews Feed

service

5. Leader coordinator calculates work distribution

Brooklin Client

Load Balancer

ZooKeeper

Brooklin Instance

Coordinator

Kafka Producer

Brooklin Instance

Coordinator

Kafka Producer

Member DBNews Feed

service

6. Leader coordinator writes the assignments to ZK

Brooklin Client

Load Balancer

ZooKeeper

Brooklin Instance

Coordinator

Kafka Producer

Brooklin Instance

Coordinator

Kafka Producer

Member DBNews Feed

service

7. ZooKeeper is used to communicate the assignments

Brooklin Client

Load Balancer

ZooKeeper

Brooklin Instance

Coordinator

Kafka Producer

Brooklin Instance

Coordinator

Kafka Producer

Member DBNews Feed

service

8. Coordinators hand task assignments to consumers

Brooklin Client

Load Balancer

ZooKeeper

Brooklin Instance

Coordinator

Kafka Producer

Brooklin Instance

Coordinator

Kafka Producer

Member DBNews Feed

service

9. Consumers start streaming data from the source

Brooklin Client

Load Balancer

ZooKeeper

Brooklin Instance

Coordinator

Kafka Producer

Brooklin Instance

Coordinator

Kafka Producer

Member DBNews Feed

service

10. Consumers propagate data to producers

Brooklin Client

Load Balancer

ZooKeeper

Brooklin Instance

Coordinator

Kafka Producer

Brooklin Instance

Coordinator

Kafka Producer

Member DBNews Feed

service

11. Producers write data to the destination

Brooklin Client

Load Balancer

ZooKeeper

Brooklin Instance

Coordinator

Kafka Producer

Brooklin Instance

Coordinator

Kafka Producer

Member DBNews Feed

service

12. App consumes from Kafka

Brooklin Client

Load Balancer

ZooKeeper

Brooklin Instance

Coordinator

Kafka Producer

Brooklin Instance

Coordinator

Kafka Producer

Member DBNews Feed

service

13. Destinations can be shared by apps

Brooklin Client

Load Balancer

ZooKeeper

Brooklin Instance

Coordinator

Kafka Producer

Brooklin Instance

Coordinator

Kafka Producer

Member DBNews Feed

service

News Feed service

Brooklin Architecture

ZooKeeper

Brooklin Instance

Coordinator

Datastream Management

Service (DMS)

Brooklin Instance

Service (DMS)

Brooklin Instance

Coordinator

Service (DMS)

Brooklin Instance

Current & Future

Current

• Consumers: ○ Espresso○ Oracle○ Kafka○ EventHubs○ Kinesis

• Producers: ○ Kafka○ EventHubs

• APIs are standardized to support additional sources and destinations

• Multitenant: Can power thousands of datastreams across several source and destination types

• Guarantees: At-least-once delivery, order is maintained at partition level

• Kafka mirroring improvements: finer control of pipelines (pause/auto-pause partitions), improved latency with flushless-produce mode

Sources & Destinations Features

38Bmessages/day

2K+datastreams

1K+unique sources

200+applications

Brooklin in Production

Brooklin streams with Espresso, Oracle, or EventHubs as the source

2T+messages/day

200+datastreams

10K+topics

Brooklin in Production

Brooklin streams mirroring Kafka data

Future

• Consumers: ○ MySQL○ Cosmos DB○ Azure SQL

• Producers: ○ Azure Blob storage○ Kinesis○ Cosmos DB○ Azure SQL○ Couchbase

Sources & Destinations Open Source

• Plan to open source Brooklin in 2019 (soon!)

Optimizations

• Brooklin auto-scaling

• Passthrough compression

• Read optimizations: Read once, Write multiple

Thank you

Questions?

Celia Kung

: ckung@linkedin.com : /in/celiakkung/

Dive into Streams with Brooklin · Issues with KMM: didn’t scale well, diﬀicult to operate and...

Documents

User's Guide Apache Kafka Software Release 2 diagram shows the parts (green) of Apache Kafka: Core Apache Kafka (light green), including the Kafka client API and the Kafka broker

Brooklin Town Crier

Franz Kafka - Letter to his · PDF fileFranz Kafka Pictures Home Page Kafka and Judaism The Holocaust photographs Galleries Franz Kafka Biography Franz Kafka-Wax Museum Kafka &

Kafka to the Maxka - (Kafka Performance Tuning)

Kafka Messaging JavaCro2019 IBM pdfv · •Kafka basics •Kafka Use Cases •Java Connector Architecture •Distributed messaging challenges •Reliable Kafka Processing •IBM Event

Partituras the Brooklin Tabernacle Choir - Only to Him

Apache Kafka - RainFocus · Apache Kafka Scalable Message ... Introduction& Motivation Apache Kafka -Scalable Message Processing and more! Apache Kafka -Overview ... • Apache Spark

Kafka Streams: Hands-on Session - ce.uniroma2.it · Kafka Streams Kafka Streams: • Kafka Streams is a client library for processing and analyzing data stored in Kafka • Supports

Kafka Low-Level Design discussion of Kafka Design Kafka …cloudurable.com/ppt/4-kafka-detailed-architecture.pdf · Cassandra / Kafka Support in EC2/AWS. Kafka Training, Kafka Consulting

Follow brooklin

Formatted: Figure [PACKT] cm, Width: 21.59 cm, Height: 27 ... · Kafka 0.7.x Consumer Kafka 0.7.x Cluster Kafka Migration Kafka 0.8 Cluster Kafka 0.8 Producer Producer (Front End)

Paris Kafka Meetup - How to develop with Kafka

Development and evaluation of a score to predict diﬀicult

Apresentacao energy brooklin

Brooklin Boat Yard Center Cockpit Sloop - Nicholson Yachtsnicholsonyachts.com/.../01/Brooklin-Boat-Yard-Center-Cockpit-Sloop.pdf · The flush deck and low cabin trunk ... Cockpit

AN EXTRACT OF THE BROOKLIN MODELS COLLECTORS GUIDE · 2020. 6. 8. · AMERICAN CARS. by Gianluigi Cappi. BROOKLIN MODELS COLLECTORS GUIDE 1 HISTORY OF BROOKLIN MODELS Brooklin Models

The Whitby Perspective€¦ · In 2017, Brooklin Mill Montessori School was one of 14 grant recipients: Brooklin Transportation: Master Plan Receives Approval: The Brooklin Transportation

Kafka Tutorial - Introduction to Apache Kafka (Part 1)

Forcepoint Behavioral Analytics Installation Manual › content › support › library › ... · Kafka kafka 9092-9095 API, Rose Kafka Manager kafka 9000 Administrator Workstation

Kafka blr-meetup-presentation - Kafka internals