Taras Chaykivskyy - Computer Vision in Front-End

eleks.com

Computer VisionIn Front-End (only)by Taras Chaykivskyy

Q&ABut js is slow … ?

Why ?Cross-platformNo installation

No huge video trafficWorks offline (cached)

JavaScript CV instrumentsARToolkit.jsARUco.jsTracking.jsJSFeat

Results :● QR marker (3 / 30 FPS)● Color QR markers (1 / 5 FPS)● Image matching (0 / 1 FPS)

“Web-Based Augmented Reality:To adopt or not to adopt ?”

* (LG Optimus L5 / Nexus 5 )

Results :● Laptop/Desktop (15-30 FPS)● Smartphone* (5 FPS)

Virtual-Makeup

* Nexus 5

How to boost CV on Web Application ?

*dlib’s implementation**with -O3 optimization flag

SURF* on C++

height 128 256 512

C++** (ms) 0.77 6.2 29.67

asm.jsIs low-level subset of JavaScript.

Emscripten (source-to-source compiler)

clang main.cpp -o mainem++ main.cpp -o mainModule.js

asm.js

C codesize_t strlen(char *ptr) { char *curr = ptr; while (*curr != 0) { curr++; } return (curr − ptr);}

asm.js codefunction strlen(ptr) { ptr = ptr|0; var curr = 0; curr = ptr; while (MEM8[curr]|0 != 0) { curr = (curr + 1)|0; } return (curr − ptr)|0;}

*dlib’s implementation**with -O3 optimization flag

SURF on asm.js

height 128 256 512

C++** (ms) 0.77 6.2 29.67

Asm.js** (ms) 3.5 21.5 104

Call js from C++ code

Run js code from c++#include <emscripten.h>

int main() { EM_ASM( alert('hello!'); throw 'all done'; ); return 0;}

Arguments…EM_ASM_({Module.print('I received:'+$0);}, 100);…

WebGLIs a JavaScript API for rendering interactive 3D computer graphics

WebGL

http://webglstats.com/

WebGL

js codevar gl = canvas.getContext("webgl");...var shaderProgram = gl.createProgram();gl.attachShader(shaderProgram, vertexShader);gl.attachShader(shaderProgram, fragmentShader);gl.linkProgram(shaderProgram);gl.useProgram(shaderProgram);...

*dlib’s implementation**with -O3 optimization flag***upper bound

height 128 256 512

C++** (ms) 0.77 6.2 29.67

Asm.js** (ms) 3.5 21.5 104

+WebGL*** (ms)

~15 ~25 ~75

SURF on asm.js + WebGL

What’s next ?

Thank you !

chaykivskyy.taras@gmail.com

github.com/felistrs