girlie_mac@

KittyCam.jsSmile, you’re on camera!Smile, you’re on camera!

Tomomi Imura

girlie_mac@

Hello, I’m Tomomi & I’m a:

● San Francisco Dweller● Front-End Engineer● Open Web + Tech Advocate● N00b Hardware Hacker● Sr. Developer Advocate at Nexmo● Cat lady of InterWeb

girlie_mac@

girlie_mac@

CC-BY-SA 3.0 https://commons.wikimedia.org/wiki/File:Basic_robot.jpg

girlie_mac@

Arduino● MCU-based kit● Open-Source hardware &

software (IDE)● The first developer-friendly

boards

girlie_mac@

Sketch● Language for Arduino● Loosely based on C● Wut, C?● HALP!!1!!!

#include <Servo.h>

#include <Wire.h>

#include <Firmata.h>

#define I2C_WRITE B00000000

#define I2C_READ B00001000

#define I2C_READ_CONTINUOUSLY B00010000

#define I2C_STOP_READING B00011000

#define I2C_READ_WRITE_MODE_MASK B00011000

#define I2C_10BIT_ADDRESS_MODE_MASK B00100000

#define I2C_MAX_QUERIES 8

#define I2C_REGISTER_NOT_SPECIFIED -1

#define MINIMUM_SAMPLING_INTERVAL 1

int analogInputsToReport = 0;

byte previousPINs[TOTAL_PORTS];

byte pinConfig[TOTAL_PINS];

byte portConfigInputs[TOTAL_PORTS];

int pinState[TOTAL_PINS];

unsigned long currentMillis;

unsigned long previousMillis;

unsigned int samplingInterval = 19;

girlie_mac@

Raspberry Pi● $35 single-board computer ● Broadcom chip with

ARM-compatible CPU & GPU● Runs Linux● More language choices: C,

C++, Python...

girlie_mac@

But… I want to code in

girlie_mac@

Johnny-Five● JavaScript robotics framework● Works with

Arduino-compatible Boards● IO plugins for more platform

supports● http://johnny-five.io/

girlie_mac@

Awww, JavaScript all the way!

girlie_mac@

Hello world!

girlie_mac@

KittyCamRaspberry Pi camera with cat facial detection!

● Hardware: Raspberry Pi, camera, and PIR sensor

● Software: Node.js + J5 + More open-source goodies

girlie_mac@

Motion detected!

snap!

Mmm… donut

girlie_mac@

kittyCam.js1. Detect motion 2. Take a photo3. Cat facial detection 4. Store the photo in cloud5. Real-time view on web6. SMS the photo link

HELL,YEAH!

girlie_mac@

girlie_mac@

Raspberry Pi 3

PIR Motion Sensor

Camera Module

F/F Jumper Wires (x3)

girlie_mac@

girlie_mac@

girlie_mac@

1. Raspberry Pi 3

2. 5MP Camera Board Module

3. Pyroelectric Infrared (PIR) motion sensor

4. Female/Female wires

girlie_mac@

Programming Raspberry PiPre-installed on Raspbian OS:

C / C++

girlie_mac@

Programming RPi with Node.js

girlie_mac@

girlie_mac@

Installing Node ARM$ wget

https://nodejs.org/dist/v4.4.5/node-v4.4.5-l

inux-armv7l.tar.xz

$ tar -xvf node-v4.4.5-linux-armv7l.tar.xz

$ cd node-v4.4.5-linux-armv7l

$ sudo cp -R * /usr/local/

girlie_mac@

kittyCam.js1. Detect motion 2. Take a photo3. Cat facial detection 4. Store the photo in cloud5. Real-time view on web6. SMS the photo link

HELL,YEAH!

girlie_mac@

kittyCam.js Tech Stack1. Detect motion w/ Johnny-Five IR.Motion obj2. Take a photo w/ Raspistill, command line tool3. Cat facial detection w/ KittyDar4. Store the photo in Cloudinary5. Publish & subscribe the url to display on web

via PubNub 6. Send a text message via Nexmo

girlie_mac@

$ Raspistill

canvas

catDetect.js

app.js

kittyDar

Motion detected

take a photo

Store the photo if

cats are detected

display photos on web browsers real-time anywhere

Johnny-Five w/ raspi-io

child process

Returns url

Notify with SMS via

girlie_mac@

Johnny-Five w/ Raspi-ioconst five = require('johnny-five');

const raspi = require('raspi-io');

let board = new five.Board({io: new raspi()});

board.on('ready', () => {

console.log('board is ready');

...

});

girlie_mac@

Motionconst five = require('johnny-five');

const raspi = require('raspi-io');

const board = new five.Board({io: new raspi()});

board.on('ready', function() {

// Create a new `motion` hardware instance

const motion = new five.Motion('P1-7');

...

});

a PIR is wired on pin 7 (GPIO 4)

VCC

Ground

Data

girlie_mac@

Raspistill Command Line Tool

$ raspistill -o myPhoto.jpg

girlie_mac@

PIR Sensor > Run Camera const child_process = require('child_process');

board.on('ready', () => {

const motion = new five.Motion('P1-7');

motion.on('motionstart', () => { // Motion detected

let filename = 'photo/image_'+i+'.jpg';

let args = ['-w', '320', '-h', '240', '-o', filename, '-t', '1'];

let spawn = child_process.spawn('raspistill', args);

spawn.on('exit', function() {

console.log('A photo is saved as '+filename);

...

motion detected!Take a photo!

Spawns a new process w/ a given shell command

girlie_mac@

Processing Imagespawn.on('exit', () => {

let imgPath = __dirname + '/' + filename;

// Child process: read the file and detect cats with KittyDar

let args = [imgPath];

let fork = child_process.fork(__dirname+'/detectCatsFromPhoto.js');

fork.send(args);

// the child process is completed

fork.on('message', (base64) => {

if(base64) {

uploadToCloud(base64); // Send to cloud storage

}

}); ...

Create another worker by running a new instance of V8 engine.

girlie_mac@

detectCatsFromPhoto.jsconst kittydar = require('kittydar');

const Canvas = require('canvas');

process.on('message', (m) => {

fs.readFile(m[0], (err, data) => {

...

let canvas = new Canvas(w, h);

let ctx = canvas.getContext('2d');

ctx.drawImage(img, 0, 0, w, h, 0, 0, w, h);

let cats = kittydar.detectCats(canvas);

console.log('There are', cats.length, 'cats in this photo');

if(cats.length > 0) {base64Img = canvas.toDataURL();}

process.send(base64Img);

process.exit(0);

});

Running in an another process

a cat detected!

KittyDar

girlie_mac@

KittyDar● Open-source JavaScript cat facial

detection written by Heather Arthur● Takes a canvas obj & calculate the

locations of cats in the image

let cats = kittydar.detectCats(canvas);

girlie_mac@

KittyDar: Behind the Scene1. Chops up the image up into many “windows”2. Extracts data by measuring a set of gradients,

from light & dark in order to find edges3. Compares the direction of these edges to the

edges found in known cat images

Neural network (JSON w/ vector data) is pre-trained w/ thousands pics of cats & non-cats

girlie_mac@

Cat Facial Detection

http://research.microsoft.com/pubs/80582/ECCV_CAT_PROC.pdf

girlie_mac@

Cat Facial Detection

http://research.microsoft.com/pubs/80582/ECCV_CAT_PROC.pdf

girlie_mac@

Cat Facial DetectionThe annotation data sequence:(Number of points, default is 9), (Left Eye), (Right Eye), (Mouth), (Left Ear-1), (Left Ear-2), (Left Ear-3), (Right Ear-1), (Right Ear-2), and (Right Ear-3)

girlie_mac@

Cat Facial Data

9 247 94 294 92 273 127 213 81 207 29 247 53 286 50 320 20 322 74

00000561_012.jpg.cat00000561_012.jpg

girlie_mac@

Cat Facial Data Collection

girlie_mac@

Neural Network

Positive

Negative

JSON data

hog-descriptor

kittydar.jsCompare data

girlie_mac@

Send the Pic w/ Kitty to Cloudconst cloudinary = require('cloudinary');

// the child process is completed

fork.on('message', (base64) => {

if(base64) {

cloudinary.uploader.upload(base64, (result) => {

// Done! - Get the URL and do more stuff

});

} else deletePhoto(imgPath);

});

girlie_mac@

View Photos Real-time via Socket

const pubnub = require(pubnub);

Publish Subscribe

girlie_mac@

http://www.girliemac.com/RPi-KittyCam/

girlie_mac@

Send SMS via Nexmoconst Nexmo = require(nexmo);

let nexmo = new Nexmo({//config with API keys});

nexmo.message.sendSms(

FROM_NUMBER, TO_NUMBER, ' '+ url, options,

(err, responseData) => {

if (err) console.log(err);

else console.dir(responseData);

});

girlie_mac@

girlie_mac@

QA TeamLead QA:Jamie

Ginger Basil Alice Yugi Venom@kittenVenom

Ozzy

girlie_mac@

github.com/girliemac/RPi-KittyCam

girlie_mac@

Next Steps● Upgrade Hardware

○ Raspberry Pi 3○ NoIR Night Vision Camera

● Upgrade Node (was 0.12) & all dependencies● ES5 to ES6● More features (Maybe)

○ Cat Identification w/ RFID○ Photo Booth w/ Filter effects & props

girlie_mac@

More Cat Projects w/ Node.js

girlie_mac@

https://http.catThanks Rogério Vicente for the .cat domain & the API!

Previously: HTTP Status Cats

girlie_mac@

HTTP Status Cats Slack Bot

Install it at:

http://girliemac.com/slack-httpstatuscats/

Tutorial on Medium.com: https://goo.gl/saOUl0

@girlie_mac

Next Project?● Selfie bot

(à la Mannie the Selfie Cat)

Mannie the Selfie Cat by @yoremahm on Instagram

https://www.instagram.com/yoremahm/

girlie_mac@

Danke schön!@girlie_macgithub.com/girliemac

girlie_mac@

Attribution:Emoji by Emoji-One (CC-BY 4.0)