13
Streaming Data Platforms Brandon O’Brien @hakczar Expedia, Inc

Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Embed Size (px)

Citation preview

Page 1: Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Streaming Data PlatformsBrandon O’Brien@hakczarExpedia, Inc

Page 2: Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Outline Use Cases

Streaming Data Opportunities & Challenges

Streaming Data Platform at Expedia

Questions?

Page 3: Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Outline Use Cases

Streaming Data Opportunities & Challenges

Streaming Data Platform at Expedia

Questions?

Page 4: Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Use Case: Streaming Analytics

Page 5: Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Outline Use Cases

Streaming Data Opportunities & Challenges

Streaming Data Platform at Expedia

Questions?

Page 6: Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Streaming Data: Opportunities

Data -> Insights -> Products -> $$$

Fresher data = better

However…

Page 7: Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Streaming Data: Challenges Today’s Challenge: System integration

System level: N^2 integrations Team level: Roadmap alignment

Page 8: Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Streaming Data: Solutions Solution:

Kafka with Streaming Data Platform Decoupled systems & roadmaps Easy publish Easy consume Central data depot

Talk Focus: Streaming Data Platform that makes life easier

for Kafka producers and consumers

Page 9: Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Outline Use Cases

Streaming Data Opportunities & Challenges

Streaming Data Platform at Expedia

Questions?

Page 10: Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Platform Architecture

Page 11: Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Streaming Data Platform Self-Service

Centralized Data Depot (logical)

DevOps: Primer, CICD, Cloud

Elastic Components

Monitoring: Velocity, Lag

Page 12: Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Streaming Data Platform cont’d Producers

Publish data to HTTP Ingestor endpoint Ingestor -> Kafka Ingestor -> S3

Consumers Create skeleton apps with built-in Kafka

integration CICD pipeline for deployments

Changes the Conversation

Page 13: Streaming Data Platforms & Building Data-Centric Products (w/ Brandon O'Brien)

Streaming Data PlatformsBrandon O’Brien@hakczarExpedia, Inc

Questions?