Cloud Spin - building a photo booth with the Google Cloud Platform

Cloud Spin

Bret McGowenGoogle@bretmcg

Learning about the cloud by dodging bullets, Matrix-style

Bret McGowen

Developer Advocate, GoogleGoogle Cloud PlatformNew York, NY

@bretmcg

Howdy DevFest MN!

I'm a fake startup CEO

Fake startup, real problems

3 weeks to minimum viable product

@bretmcg @googlecloud @googlecloudspin

The user steps into the center of the Cloud Spin

area

The user jumps, and we take a photo from all the

phones

It's pretty tricky

● Taking a photo at the same EXACT SAME INSTANT

● Coordinating the phone cameras

● File storage

● Scaling

● Image processing

● Controlling cost

● Create multiple versions

● Debugging & Logging

Problems to solve

Taking the photos: how would you do it?


It isn’t that straightforward

● Need to coordinate 19 phones to take photosof the same moment!

● Android - random delay when taking a photo!

Hence impossible to take photos of the same moment


What we ended up with?

● Record a video

● 30 fps is like burst mode


More problems

● How do we identify the frames?

● Mark the the frame some how...

● Many Experiments: flash, timestamp

Use an audio marker!


Mic-In Into the Phone

MIC

OMTP

MIC

GND

AHJ / CTIA


Audio can be tricky



Finally - Off the shelf components


Record the audio cue in the video

http://www.youtube.com/watch?v=wEAvymonG6w

How do we coordinate 19 phone cameras?

No, seriously, how do we coordinate 19 phone cameras?


What do we need to coordinate?

● Camera app status & error message

● Exposure ISO & shutter speed

● Internal versus external microphone

● Timestamp to start the recording

● And more...


Let's use Firebase!

● Can power our app's backend

● Stores our data and syncs it in real-time

● JSON NoSQL database with real-time sync

● Updates across devices within milliseconds

// Set this phone's status on Firebase at /cameras/{cameranumber}Firebase firebaseRef = new Firebase(FIREBASE_BASE_URL);Firebase cameraStatusRef = firebaseRef.child("cameras").child(cameraNumber);

CameraStatus cameraStatus = new CameraStatus();cameraStatus.appVersion = BuildConfig.VERSION_NAME;cameraStatus.externalMic = mExternalMic;...cameraStatus.status = status;

cameraStatusRef.setValue(mCameraStatus);

Android: updating camera status with Firebase



Recording video with camera2 API

● camera2 API

● I learned a lot of this API from the android-

Camera2Video GitHub project

Scaling with Google Cloud Platform

Stitching Q

Extraction Q

InputBucket

Stitching Q

Devices

Extraction Q

Notification Processor

Extractor

Stitcher

Video Uploads

Object Change Notifications

Output Bucket

Coordinator

InputBucket


Extractor

Stitcher

Output Bucket


Video Uploads and notification processing

InputBucket

Stitching Q

Devices

Extraction Q


Extractor

Stitcher

Video Uploads

Notification

Output Bucket

Coordinator

InputBucket


Cloud Datastore Cloud SQL Cloud Storage

4.5 trillion! Managed MySQL Cat Videos?Cat Videos?

Jumping

People

// Credentials for Google Cloud Platform.

Credential credential = ...

Storage storage = new Storage.Builder(httpTransport, jacksonFactory, credential)

.setApplicationName("[email protected]")

.build();

StorageObject storageObject = new StorageObject();

storageObject.setBucket(bucket);

InputStream fileUploadStream = new FileInputStream(videoFile);

try {

InputStreamContent content = new InputStreamContent("video/mp4", fileUploadStream);

Storage.Objects.Insert insert = storage.objects().insert(bucket, null, content);

insert.setName(cloudStoragePath);

insert.execute();

} finally {

fileUploadStream.close();

}



Cloud Storage Bucket

[myapp-input-bucket]

App Engine

Object Change Notification

ClientAdd/update/remove object


Edge Caching


Google App Engine

Choice of Runtimes- Java, Python, Go- PHP- Custom, NodeJS!

Easy to develop- Build and test locally- Focus on App Code- Versioning- Traffic Splitting

Trivial to manage- Fully managed- No patches/updates- 24x7 operation by Google SREs- Autoscale



Building Pipelines

InputBucket

Stitching Q

Devices

Extraction Q


Extractor

Stitcher

Video Uploads


Output Bucket

Coordinator

Subscriber Subscriber

Pub/Sub Pub/Sub

Push Subscription Pull Subscription

msg ack msg ack

RPC SendRPC Return


Cloud Pub/Sub (Beta)

Reliable andreal-timemessaging

Designed for Fast Data

Provides reliable, real-time, many-to-many, asynchronous messaging between applications

Send data into processing pipelines and out to other apps, devices and Google Cloud Services

Global by design and highly available. ‘fire-and-forget’ with minimal latency. Redundant forwarding paths

Designed for Google scale


Pub/Sub

Stitcher

Frames Topic

Extractor

Frame Ready Message

Publishers and Subscribers

Original Stitcher Subscription


Pub/Sub

Original Stitcher

Frames Topic

Extractor

Frame Ready Message

Publishers and Subscribers

Original Stitcher Subscription

Variation A Stitcher

Variation A Subscription

Variation B Stitcher

Variation B Subscription


Variations - How Pub/Sub Can Help

Meanwhile, in our secret startup lab

Our first prototype was cheap and scrappy

Like, REALLY scrappy.

We used anything we could find.

Like, REALscrappy(selfie sticks!)



Extracting and stitching


Pipeline Processing

InputBucket

Stitching Q

Devices

Extraction Q


Extractor

Stitcher

Video Uploads


Output Bucket

Coordinator

Extractor

Stitcher

from moviepy.video.io.VideoFileClip import VideoFileClip

import numpy

STD_DEV_SCALAR = 4

def calc_peak_time(clip):

audio_arr = clip.audio.to_soundarray(buffersize=1000)

frames = clip.fps * len(audio_arr) / clip.audio.fps

buckets = numpy.array_split(audio_arr, frames)

mean_arr = [numpy.sqrt(numpy.mean(bucket**2)) for bucket in buckets]

peaks = numpy.where(mean_arr > MEAN_SCALAR * numpy.mean(mean_arr))

if len(peaks) > 0 and len(peaks[0]) > 0:

return peaks[0][0] / float(clip.fps)

else:

return 0.0


Frames extracted


Google Compute Engine

Compute Resources- US, Europe and Asia zones- Fast SDN-Based virtual networking

Consistently Fast- Fast VM Provisioning - Consistent Performance

Cost Effective- Sub-Hour Billing- No IOPS charges for Block Storage- VMs not required for Load balancing


Compute Engine Autoscaler

ManagedInstanceGroup

create/destroy VMs

Instance Group Manager

actuator

monitor

Autoscaler

Cloud Monitoring

utilisation

VMVMVM

Pub/Sub Topic


Next step: Containerization

Extractor Workers

create/destroy VMs

Instance Group Manager

actuator

monitor

Autoscaler

Cloud Monitoring

utilisation

VMVMContainer

VM

Extract Topic


Stitching the frames together

Extract Q

Temp Store

ExtractorStitch QExtractorExtractors

READY

Stitcher

Firebase

All READY?

Retrieve Images

Visualizer

● Taking a photo at the same EXACT SAME INSTANT - video with audio beep

● Coordinating the phone cameras - Firebase

● File storage - Google Cloud Storage

● Scaling - Cloud Pub/Sub, Compute Engine Instance Groups

● Image processing - Compute Engine & Containers

● Controlling cost - Scaling instance group sizes

● Create multiple versions - Cloud Pub/Sub topics

● Debugging & Logging - Google Cloud Logging

Problems to solve

DEMO VIDEO TIME

http://www.youtube.com/watch?v=6oGWFiSrPBc


Productivity!

We built a lot in just 3 weeks

Focus 100% of time coding


• Talk with us!• Firebase• Containers• Kubernetes• Big data• BigQuery

Want to know more?

Bret McGowen@bretmcg@googlecloudspinTHANK YOU

Technology

Cloud Spin - building a photo booth with the Google Cloud Platform