[Hatsune Miku] Shoot Frieza with Amazon Kinesis ! [EN]

Shoot the Frieza with Amazon Kinesis !

JAWSDAYS 2014.3.15 @shimy_net

I’m AWS Comedian Takayuki SHIMIZU @shimy_net

AWS Comedian have to waste a technical talent for getting a laugh.

My works

Cycling with DynamoDB AWS Automation by

Rube Goldberg machine

EC2 Kickstart by Kinect

Today’s theme

Kinesis

What’s Amazon Kinesis ?

Kinesis

Big data Real-time Processing Visualize

We can visualize everything in real-time.

In short

Got an idea !

If we put the POWER into Amazon Kinesis …

We can defeat Frieza

… I can annihilate you with only half of my full

strength.

Imagine…

Collect the POWER JUST

GIVE ME A LITTLE BIT OF YOUR

POWER..

EARTH…

OCEAN…

ALL LIVING

THINGS…

Put it into Amazon Kinesis

Visualize it !

We can defeat Frieza

(ﾟーﾟ*;) OK

(́Д｀;) Really OK?

How to collect the POWER ?

Tweet Morphological Analysis Emotion Analysis Estimate the POWER

1Tweet ＝ Max 100 pt Convert Tweet into POWER

POWER

Step1. Morphological Analysis

本日は晴天なり

本日 / は / 晴天 / なり

MeCab It's fine today

It / is / fine / today

Step2. Emotion Analysis

本日 / は / 晴天 / なり

Emotion Dictionary -1.0～+1.0 (Negative or Positive)

+0.63 -0.18

fine today

Step3. Estimate the POWER

POWER = 72 pt

本日 / は / 晴天 / なり +0.63 -0.18

fine today

Step4. Create an Energy ball

82pt

37pt 77pt

64pt

93pt

Tweet

Tweet

Tweet

Tweet

Tweet

Break the limit over 530,000pt !!!

Don’t worry. I won’t use all of my power.

It’s 530,000

I wrote a program

Architecture

Kinesis Stream

Twitter4J Kinesis MeCab Dynamo Unity

Visualize Analyze Kinesis Tweet

We fight with #jawsdays’s POWER

Put tweets into Kinesis Kinesis Stream

Twitter4J Kinesis MeCab Dynamo Unity

TwitterStream twitterStream = new TwitterStreamFactory().getInstance();StatusListener listener = new StatusListener() { @Override public void onStatus(Status status) {

String key = RandomStringUtils.randomAlphanumeric(10);String json = DataObjectFactory.getRawJSON(status); PutRecordRequest putRecordRequest = new PutRecordRequest();putRecordRequest.setStreamName(PropUtil.getInstance().getProp("StreamName"));putRecordRequest.setData(ByteBuffer.wrap(json.getBytes(Charset.forName("UTF-8")));putRecordRequest.setPartitionKey(key);PutRecordResult putRecordResult = kinesis.putRecord(putRecordRequest);

}};twitterStream.addListener(listener);FilterQuery filterQuery = new FilterQuery();filterQuery.track(new String[] { PropUtil.getInstance().getProp("Keyword1"), PropUtil.getInstance().getProp("Keyword2") });twitterStream.filter(filterQuery);

Get tweets from Kinesis Kinesis Stream

Twitter4J Kinesis MeCab Dynamo Unity

private void processRecordsWithRetries(List<Record> records) { for (Record record : records) { boolean processedSuccessfully = false; String data = null; for (int i = 0; i < NUM_RETRIES; i++) { try { // Get Data data = decoder.decode(record.getData()).toString(); Status status = DataObjectFactory.createStatus(data ); // MeCab something // ・・・・・・

processedSuccessfully = true; break; } catch (Throwable t) {} }}

}

Analyze tweets Kinesis Stream

Twitter4J Kinesis MeCab Dynamo Unity Tagger tagger = new Tagger("-Ochasen");tagger.parse(status.getText().replaceAll("¥n", ""));Node node = tagger.parseToNode(status.getText().replaceAll("¥n", ""));Double sum = 0.0;int count = 0;for (; node != null; node = node.getNext()) {

String[] mecabArray = node.getFeature().split(","); String originalWord = node.getSurface();if (!mecabArray[6].equals("*")) { originalWord = mecabArray[6]; // 基本形 } String feature = mecabArray[0];if (feature.equals("名詞") || feature.equals("動詞") || feature.equals("形容詞”) || feature.equals("形容動詞") || feature.equals("副詞")) { // Positive or Negative sum += PositiveNegativeJudge.getInstance().getDouble(originalWord + ":" + feature); count++;}

}Double point = Math.floor((sum / count + 1.0) * 100.0 / 2.0);

Store results on Dynamo DB Kinesis Stream

Twitter4J Kinesis MeCab Dynamo Unity

private static PutItemRequest createPutItemRequest(String id, String datetime, String name, String screenName, String text, Double value) {

Map<String, AttributeValue> putItems = new HashMap<String, AttributeValue>();putItems.put("AppName", new

AttributeValue().withS(PropUtil.getInstance().getProp("ApplicationName")));putItems.put("Datetime/SerialNumber", new AttributeValue().withS(datetime+"/"+id));putItems.put("ScreenName", new AttributeValue().withS("@" + screenName));putItems.put("Name", new AttributeValue().withS(name));putItems.put("Text", new AttributeValue().withS(text));putItems.put("Value", new AttributeValue().withN(Double.toString(value)));PutItemRequest putItemRequest = new

PutItemRequest().withTableName(PropUtil.getInstance().getProp("DynamoDbTableNameDetail")).withItem(putItems);

return putItemRequest;}

Store results on Dynamo DB

The POWER

Total POWER Atomic Counter

Output to JSON

Visualize the Power using Unity Kinesis Stream

Twitter4J Kinesis MeCab Dynamo Unity

Hatsune Miku ※Virtual idol

Here we go !

Give me the POWER

In real-time Miku collects the POWER

Miku

Everyone’s tweet

POWER meter

Kamehameha

Frieza

Demonstration

Lose

We fight with #jawsdays’s POWER

Miku falls down

Lose

We need more POWER

We need more POWER !!!

This situation is within my “assumption”

if (tweet.indexOf(“芸人イケメン”)) != -1) {

point = 530000.0;

}

”Comedian Cool”

Give me your POWER !

芸人イケメン #jawsdays

Please tweet “Comedian Cool” and give me your POWER !!

Comedian Cool

Thanks for a lot of tweets

Retry

Kamehameha

Win

Over 530,000 !!!

Frieza

Frieza

Win

We did it !

Comedian Cool Kinesis Cool

Conclusion

Inside story I monitored processes of my program in server-side because I had to accomplish this real-time demonstration within 5 min. But an incident happened. Just before my turn to go on stage, I noticed that a process which retrieve POWER’s data from Dynamo DB was often stopped. ec2-user 14685 0.1 4.2 1625848 71584 ? Sl 02:10 0:27 java KinesisPutter ec2-user 18622 2.2 4.8 1787796 82660 pts/1 Sl 06:01 0:21 java -Djava.library.path=.:/usr/local/bin/mecab-java KinesisGetter ec2-user 18944 22.0 4.2 1618412 71836 ? Sl 06:16 0:08 java DynamoJsonCreator ↓ ec2-user 14685 0.1 4.2 1625848 71584 ? Sl 02:10 0:27 java KinesisPutter ec2-user 18622 2.2 4.9 1839496 83452 pts/1 Sl 06:01 0:21 java -Djava.library.path=.:/usr/local/bin/mecab-java KinesisGetter

Inside story I tried to set an alive-monitoring by “cron” and be able to restart a process automatically when it’s dead. crontab -l */2 * * * * bash /home/ec2-user/KinesisTwitter/check_putter.sh > /home/ec2-user/cron.log.putter.txt 2>&1 */2 * * * * bash /home/ec2-user/KinesisTwitter/check_getter.sh > /home/ec2-user/cron.log.getter.txt 2>&1 */2 * * * * bash /home/ec2-user/KinesisTwitter/check_create_json.sh > /home/ec2-user/cron.log.create.json.txt 2>&1

In the meantime, I found my mistake that Dynamo DB’s throughput set 1. (；´Д｀) oh…

Inside story Nintendo’s presentation had started and the frequency of #jawsdays’s tweet was increasing. Then my program’s process was stopped. I modified Dynamo DB’s throughput to solve a problem. This is the quintessence of AWS.

Nintendo’s presentation start

Modify throughput

References ドラゴンボール Amazon Kinesis　http://aws.amazon.com/jp/kinesis/ Amazon DynamoDB　http://aws.amazon.com/jp/dynamodb/ Twitter4J　http://twitter4j.org/ja/ MeCab　http://mecab.googlecode.com/svn/trunk/mecab/doc/index.html MeCab-java　 単語感情極性対応表　http://www.lr.pi.titech.ac.jp/~takamura/pndic_ja.html Unity　http://japan.unity3d.com/ MikuMikuDcance for Unity　http://mmd-for-unity-proj.github.io/mmd-for-unity/　 Lat式ミク　http://dic.nicovideo.jp/a/lat フリーザ　http://ux.getuploader.com/karota318/ かめはめ波モーション　http://www.nicovideo.jp/watch/sm15093547 歩き・スキップモーション　http://www.nicovideo.jp/watch/sm21263509 待機モーション　http://www.nicovideo.jp/watch/sm18961728 やられモーション　http://www.nicovideo.jp/watch/sm19073965 日常モーション　http://www.nicovideo.jp/watch/sm18015670

Thank you

Well,

goodbye !