Embed Size (px)
Citation preview
• • Kafka
• Exactly Once :
•
2
• 2011/04
• 2015/09
• • Druid (KDP, 2015)
• RDB NoSQL ( , 2016; : HBase )
• ESP8266 Wi-Fi IoT (KDP, 2016)
• • (WebDB Forum 2014)
• Spark Streaming (Spark Meetup December; 2015)
• Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016)
• (FutureOfData; 2016)
• Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016)
• Apache Spark ( Geek Night #11; 2016)
• (BigData-JAWS #6; 2017)
3
Kafka
Kafka
5
Kafka
ACK
→
ACK
→
6
Kafka
Broker
→
7
Kafka
→
→
8
Kafka
At Least Once At Most Once
9
Kafka
At Least Once At Most Once
Exactly Once
9
Exactly Once :
Kafka 0.11.0.0
• 2017/06/28
• KIP-129 → KAFKA-4923: Add Exactly-Once Semantics to Streams • https://issues.apache.org/jira/browse/KAFKA-4923
• Confluent Blog: Exactly-once Semantics are Possible: Here’s How Kafka Does it • https://www.confluent.io/blog/exactly-once-semantics-are-possible-
heres-how-apache-kafka-does-it/ • “In the following posts in this series, we will go into more details …”
• Exactly Once Delivery and Transactional Messaging in Kafka • https://docs.google.com/document/d/
11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/ • KIP-98 → KAFKA-5059: Implement Transactional Coordinator
11
Kafka
13
Kafka : Exactly Once
14
Producer Exactly Once
15
Producer Exactly Once
1. 2. 3. x N 4. →
16
Producer Exactly Once
1. 2. 3. x N 4.
→
→
→ 1
17
Producer Exactly Once
5.Broker Commit Marker
COMMITTED ABORTED ↓ ↓
18
Consumer Exactly Once
19
Consumer Exactly Once
6.
↓
READ_COMMITTED↓
READ_UNCOMMITTED
20
Consumer Exactly Once
7. 8.
- COMMITTED
- ABORTED Transaction List
21
• Kafka API RecordBatch
• Kafka < 0.11 Message MessageSet
API
• Kafka 0.11 MessageSet/Message RecordBatch/Record
•
• Producer Broker Consumer
ID PID Producer ID
22
Exactly Once
• : Exactly Once
• Producer
• Broker
• Commit Marker
• Consumer
•
→ OSS
• Exactly Once • Producer Broker
• Consumer
• 3%
• • Confluent
• OSS
Kafka
• Exactly Once
24
