Upload
jaxlondon2014
View
447
Download
0
Tags:
Embed Size (px)
DESCRIPTION
JAX London Presentation 2014
Citation preview
Dr. James Stanier | Head of Analytics | Brandwatch.com | [email protected]
Coming Up/
© 2014 Brandwatch | www.brandwatch.com 2
• Me, Brandwatch and new problems
• Apache Kafka
• Processing data in Java
• Distributing work with Zookeeper
• Managing state
© 2014 Brandwatch | www.brandwatch.com 3
Who?
© 2014 Brandwatch | www.brandwatch.com 4
Dr. James Stanier Head of Analytics, Brandwatch @jstanier | [email protected]
© 2014 Brandwatch | www.brandwatch.com 5
Brandwatch
© 2013 Brandwatch | www.brandwatch.com 6
Where we are/
• Brighton
• New York
• San Francisco
• Berlin
• Stuttgart
© 2014 Brandwatch | www.brandwatch.com 7
Crawl Store and Index
What Brandwatch does/
© 2014 Brandwatch | www.brandwatch.com 8
Analyse
3
Present
4
• Crawl 70M+ sites including key social networks
• 27 languages
• Powerful search operators
• 20Bn + indexed URLs
• Years of historical data
• Automated topic & sentiment analysis in all 27 languages
• Automate common tasks including alerts
• Advanced analytics modules
• Automatic categorisation with rules
• Custom dashboards
• Reporting & alerts
© 2014 Brandwatch | www.brandwatch.com 9
Brandwatch Analytics
10
Data/ Presentation
© 2014 Brandwatch | www.brandwatch.com 11
Data/ Aggregation
© 2014 Brandwatch | www.brandwatch.com 12
Data/ Classification
© 2014 Brandwatch | www.brandwatch.com 13
Data/ Not just top level metrics
© 2014 Brandwatch | www.brandwatch.com 14
Development/ What do we use?
© 2014 Brandwatch | www.brandwatch.com 15
Data/ The numbers
• 50+ Java Web Crawlers
• 10+ Historical crawlers for new queries
• Twitter via GNIP (now Twitter)
• 70M+ query matches per day
© 2014 Brandwatch | www.brandwatch.com 16
The speed of social
© 2014 Brandwatch | www.brandwatch.com 17
© 2014 Brandwatch | www.brandwatch.com 18
© 2014 Brandwatch | www.brandwatch.com 19
© 2014 Brandwatch | www.brandwatch.com 20
A new challenge
21
The challenge/ The signal from the noise
© 2014 Brandwatch | www.brandwatch.com 22
• 100K+ user queries
• 70M+ mentions per day
• Polling the database for mentions would take
8hrs for one pass
The challenge/ at scale
© 2014 Brandwatch | www.brandwatch.com 23
Crawler 1
Crawler 2
Crawler n-1
Crawler N
Kafka Cluster
Signals Processing
cluster
Signals handler
JVM
DB
Mentions
Signals
Signals
Mentions
The Problem/ How we handled it…
© 2014 Brandwatch | www.brandwatch.com 24
Kafka
Step 1/ Kafka
© 2014 Brandwatch | www.brandwatch.com 25
Crawler 1
Crawler 2
Crawler n-1
Crawler N
Kafka Cluster Mentions
© 2014 Brandwatch | www.brandwatch.com 26
Kafka/ What is it?
• Apache Kafka is a publish-subscribe messaging
system rethought as a distributed commit log
• Apache top level project November 2013
• Started at LinkedIn
© 2014 Brandwatch | www.brandwatch.com 27
Kafka/ is…
• Fast: hundreds of MBs read/write per second
from thousands of clients
• Scalable: clustered, partitioned over many
machines, expanded without downtime
• Durable: messages persisted to disk and
replicated in cluster
© 2014 Brandwatch | www.brandwatch.com 28
Kafka/ Terminology
• Kafka maintains feeds of messages
called topics
• Programs that publish messages are
called producers
• Programs that subscribe to messages
are called consumers
• Kafka is a cluster of servers called brokers
Kafka/ How it’s used...
© 2014 Brandwatch | www.brandwatch.com 29
Producer Producer Producer
Consumer Consumer Consumer
Kafka Cluster
Kafka/ Written to disk?
© 2014 Brandwatch | www.brandwatch.com 30
http://q.acm.org/detail.cfm?id=1563874
Kafka/ Bending, not breaking
© 2014 Brandwatch | www.brandwatch.com 31
http://engineering.gnip.com/tag/kafka/
Kafka/ The anatomy of a topic
© 2014 Brandwatch | www.brandwatch.com 32
0 2 1 5 3 4 6
0 2 1 3 4
0 2 1 5 3 4
Partition 0
Partition 1
Partition 2
Old New
Writes
© 2014 Brandwatch | www.brandwatch.com 33
Kafka/ Warning: ordering
Kafka guarantees a total ordering per partition,
not per whole topic
© 2014 Brandwatch | www.brandwatch.com 34
Kafka/ Try it out! > tar -xzf kafka_2.9.2-0.8.1.tgz!
> cd kafka_2.9.2-0.8.1!
> bin/zookeeper-server-start.sh config/zookeeper.properties!
> bin/kafka-server-start.sh config/server.properties!
!
© 2014 Brandwatch | www.brandwatch.com 35
Kafka/ Try it out! > bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test!
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test!
Hello JAX!!
> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning!
Hello JAX!!
© 2014 Brandwatch | www.brandwatch.com 36
Kafka/ With Java <dependency>!
!<groupId>org.apache.kafka</groupId>!
!<artifactId>kafka_2.10</artifactId>!
!<version>0.8.1</version>!
</dependency>!
© 2014 Brandwatch | www.brandwatch.com 37
Kafka/ With Java Properties props = new Properties();!
props.put("metadata.broker.list", "broker1:9092,broker2:9092");!
props.put("serializer.class", "kafka.serializer.StringEncoder");!
props.put("partitioner.class", "example.producer.SimplePartitioner");!
props.put("request.required.acks", "1");!
ProducerConfig config = new ProducerConfig(props);!
Producer<String, String> producer = new Producer<String, String>(config);!
© 2014 Brandwatch | www.brandwatch.com 38
Kafka/ Partitioning
public class SimplePartitioner implements Partitioner<String> {!
public SimplePartitioner (VerifiableProperties props) {!
!
public int partition(String key, int numberOfPartitions) {!
return md5hash(key) % numberOfPartitions;!
}!
}!
© 2014 Brandwatch | www.brandwatch.com 39
Kafka/ Sending from the crawlers String message = toJson(...);!
KeyedMessage<String, String> message = new!
!KeyedMessage<String, String>("query.mentions", !queryId, message);!
producer.send(message);!
Step 1/ Done
© 2014 Brandwatch | www.brandwatch.com 40
Crawler 1
Crawler 2
Crawler n-1
Crawler N
Kafka Cluster Mentions
© 2014 Brandwatch | www.brandwatch.com 41
Processing
42
Processing/ What’s happening now?
Step 2.1/ One processing JVM
© 2014 Brandwatch | www.brandwatch.com 43
Crawler 1
Crawler 2
Crawler n-1
Crawler N
Kafka Cluster Mentions
Signals processor
Processing/ A wild tweet appears!
© 2014 Brandwatch | www.brandwatch.com 44
Mention date: 14/10/2014 6:10PM pageType: twitter author: @javadude hashtags: [#jaxlondon, #amazingtalk, #greatshoes] mentionedTweeters: [@jstanier] text: “@jstanier is at #jaxlondon tonight #amazingtalk #greatshoes”
Processing/ Storing hashtags
© 2014 Brandwatch | www.brandwatch.com 45
Map<Date, Multiset<String>>!
!
Initialise with the last 24 hours
Processing/ Storing hashtags
© 2014 Brandwatch | www.brandwatch.com 46
Map<Date, Multiset<String>>!
!
Mention date: 14/10/2014 6:10PM hashtags: [#jaxlondon, #amazingtalk, #greatshoes]
Processing/ Storing hashtags
© 2014 Brandwatch | www.brandwatch.com 47
Map<Date, Multiset<String>>!
!
Mention date: 14/10/2014 6:10PM hashtags: [#jaxlondon, #amazingtalk, #greatshoes] add(“#jaxlondon”)!add(“#amazingtalk”)!add(“#greatshoes”)!
© 2014 Brandwatch | www.brandwatch.com 48
Processing/ Cycling the buckets @Scheduled(cron = "0 0 * * * *")!
public void cycleBuckets() {!
Date oldest = buckets.lastKey();!
removeBucket(oldest);!
DateTime newest = new!
! ! !DateTime(buckets.firstKey());!
addBucket(newest.plusHours(1).toDate());!
}!
Processing/ Detecting spikes
© 2014 Brandwatch | www.brandwatch.com 49
• At regular intervals • For each hashtag
• Convert to a timeseries [5, …. 1002, 5499] • Use our super secret detection algorithm • Give a score to it
• If score > threshold, it’s interesting • Send it on a new Kafka topic
Processing/ What we just did
© 2014 Brandwatch | www.brandwatch.com 50
#hashtag data
model
Processing/ But we also track…
© 2014 Brandwatch | www.brandwatch.com 51
author data
model
sentiment data
model
page type data
model
link share data
model
#hashtag data
model
country data
model
volume data
model
Processing/ …for one query
© 2014 Brandwatch | www.brandwatch.com 52
“JAX London” query
author data
model
sentiment data
model
page type data
model
link share data
model
#hashtag data
model
country data
model
volume data
model
Processing/ 100K+ queries and rising
© 2014 Brandwatch | www.brandwatch.com 53
© 2014 Brandwatch | www.brandwatch.com 54
Processing/ We need more JVMs
But how do we share the workload?
© 2014 Brandwatch | www.brandwatch.com 55
Distribution of work
© 2014 Brandwatch | www.brandwatch.com
Crawler 1
Crawler 2
Crawler n-1
Crawler N
Kafka Cluster
Signals Processing
cluster
Mentions Signals
Mentions
Step 2.2/ A cluster of processing JVMs
© 2014 Brandwatch | www.brandwatch.com
Signals Processing
cluster
Distribution/ An atomic unit of work
?
© 2014 Brandwatch | www.brandwatch.com 58
Distribution/ Leader election
A way of deciding who is the leader for a task in a group of distributed nodes
© 2014 Brandwatch | www.brandwatch.com 59
Distribution/ Zookeeper
A way of coordinating and managing distributed applications
© 2014 Brandwatch | www.brandwatch.com 60
Zookeeper/ It’s like a file system
/brandwatch
/feature_2 /feature_1
© 2014 Brandwatch | www.brandwatch.com 61
Zookeeper/ At the command line
© 2014 Brandwatch | www.brandwatch.com 62
Distribution/ In Java
http://curator.apache.org/curator-framework
© 2014 Brandwatch | www.brandwatch.com 63
Distribution/ Recipes
© 2014 Brandwatch | www.brandwatch.com 64
Distribution/ Instantiating Curator !
CuratorFrameworkFactory!
.builder()!
.connectString(zkQuorum)!
.namespace(namespace)!
.build()!
! ! ! !.start();!
© 2014 Brandwatch | www.brandwatch.com 65
Distribution/ Offering jobs
/brandwatch
/signals
/queries
/1268589 /15846
Manager JVM
DB
© 2014 Brandwatch | www.brandwatch.com 66
Distribution/ Adding nodes !
public void createZNode(String queryId) {!
try {!
client.create().forPath(ZK_NODE_PREFIX + queryId);!
} catch (NodeExistsException e) {!
log.debug("Node {} was already created.”, queryId);!
}!
}!
© 2014 Brandwatch | www.brandwatch.com 67
Distribution/ Deleting nodes !
public void removeZNode(String queryId) {!
try {!
client.delete().forPath(ZK_NODE_PREFIX + queryId);!
} catch (NoNodeException e) {!
log.debug("Node {} was already deleted.”, queryId);!
}!
}!
© 2014 Brandwatch | www.brandwatch.com 68
Distribution/ Leader election 101
/brandwatch
/signals
/queries
/1268589 /15846
Processing JVM 1
Processing JVM 2
Processing JVM 3
© 2014 Brandwatch | www.brandwatch.com 69
Distribution/ Leader election 101
/brandwatch
/signals
/queries
/1268589 /15846
1 2 3
Processing JVM 1
Processing JVM 2
Processing JVM 3
© 2014 Brandwatch | www.brandwatch.com 70
Distribution/ The leader dies
/brandwatch
/signals
/queries
/1268589 /15846
2 3
Processing JVM
Processing JVM
© 2014 Brandwatch | www.brandwatch.com 71
Distribution/ The dead rises again
/brandwatch
/signals
/queries
/1268589 /15846
2 3 4
Processing JVM
Processing JVM
Processing JVM
© 2014 Brandwatch | www.brandwatch.com 72
Distribution/ Curator: LeaderLatch !
public LeaderLatch(CuratorFramework client, String latchPath)!
Parameters:!
!
client - the client!
latchPath - the path for this leadership group!
© 2014 Brandwatch | www.brandwatch.com 73
Distribution/ LeaderLatch recipe public class WorkerManager implements PathChildrenCacheListener {!
!
private Map<Integer, LeaderLatch> leaderLatches = newHashMap();!
!
@Override!
public void childEvent(CuratorFramework client, !
PathChildrenCacheEvent event) {!
// Handle adds and removes here!!
}!
}!
© 2014 Brandwatch | www.brandwatch.com 74
Distribution/ Curator: Starting up @PostConstruct!
public void initialise() throws Exception {!
List<ChildData> currentData = newArrayList(initialisePathChildrenCache());!
log.info("Pre creating workers for {} existing queries", currentData.size());!
for (ChildData childData : currentData) {!
int queryId = parseQueryIdFromPath(childData.getPath());!
startLeaderElection(queryId);!
}!
}!
© 2014 Brandwatch | www.brandwatch.com 75
Distribution/ Curator: PathChildrenCache private List<ChildData> initialisePathChildrenCache() throws Exception {!
pathChildrenCache.start(StartMode.BUILD_INITIAL_CACHE);!
pathChildrenCache.getListenable().addListener(this);!
List<ChildData> currentData = pathChildrenCache.getCurrentData();!
return currentData;!
}!
© 2014 Brandwatch | www.brandwatch.com 76
Distribution/ Curator: Adding a node @Override!
public void childEvent(CuratorFramework client, PathChildrenCacheEvent event) { !
ChildData childData = event.getData();!
switch (event.getType()) {!
case CHILD_ADDED:!
queryId = parseQueryIdFromPath(childData.getPath());!
if (!haveLeaderLatchForQuery(queryId)) {!
startLeaderElection(queryId);!
}!
break;!
© 2014 Brandwatch | www.brandwatch.com 77
Distribution/ Curator: Deleting a node // Continued...!
case CHILD_REMOVED:!
queryId = parseQueryIdFromPath(childData.getPath());!
removeLeaderLatchForQuery(queryId);!
break;!
default:!
break;!
}!
}!
© 2014 Brandwatch | www.brandwatch.com 78
We are processing long running jobs
What about workers getting overloaded?
Distribution/ Almost there?
© 2014 Brandwatch | www.brandwatch.com 79
1. Take leadership
2. Hit max queries?
a. No – go to 3
b. Yes – give up leadership, try again
3. Start working
Distribution/ After leader election
© 2014 Brandwatch | www.brandwatch.com 80
Actually, no…
Distribution/ Now we’re almost there?
© 2014 Brandwatch | www.brandwatch.com 81
Distribution/ Infinite election
Processing JVM
Processing JVM Processing JVM
At capacity! At capacity!
At capacity!
Elected for 1328
Elected for 1328
Elected for 1328
© 2014 Brandwatch | www.brandwatch.com 82
Distribution/ Solution
Processing JVM
Processing JVM Processing JVM
At capacity!
Elected for 1328
Elected for 1328
Refused 1328
© 2014 Brandwatch | www.brandwatch.com 83
Distribution/ Solution
Processing JVM
Processing JVM Processing JVM
At capacity! At capacity!
At capacity!
Elected for 1328
Elected for 1328
Elected for 1328
Refused 1328
Refused 1328
Refused 1328
© 2014 Brandwatch | www.brandwatch.com 84
State
© 2014 Brandwatch | www.brandwatch.com 85
State/ CAP theorem
Availability
CA AP
CP
© 2014 Brandwatch | www.brandwatch.com 86
If one worker dies, we want the other to pick up where it left off
Regular snapshotting to HBase
State/ Snapshotting of worker data
© 2014 Brandwatch | www.brandwatch.com 87
Serialise and compress using Kryo
~ 0.5MB per query, but a lot are very small
State/ Serialisation and compression
© 2014 Brandwatch | www.brandwatch.com 88
Crawler 1
Crawler 2
Crawler n-1
Crawler N
Kafka Cluster
Signals Processing
cluster
Signals handler
JVM
DB
Mentions
Signals
Signals
Mentions
Step 2.2/ Done!
© 2014 Brandwatch | www.brandwatch.com 89
Monitoring
© 2014 Brandwatch | www.brandwatch.com 90
Monitoring/ Statsd and Graphite
© 2014 Brandwatch | www.brandwatch.com 91
Closing remarks
© 2014 Brandwatch | www.brandwatch.com 92
© 2014 Brandwatch | www.brandwatch.com 93
Now • Smarter alerts (email, push, in-browser) • Monitoring crises/events as they happen
Underway • Automatic clustering of spikes into events • Historical analysis of trends
Summary/ Using this architecture
Say hello/
© 2014 Brandwatch | www.brandwatch.com 94
@brandwatch | @jstanier
www.brandwatch.com
UK: +44 (0)1273 358 635
© 2014 Brandwatch | www.brandwatch.com 95
Q&A