Deeplearning4J

François Garillot, @huitseeker

Neural Networks & Deep Learning

• graphical models w/ inputs and outputs

• represents composition of differentiable functions

• deep learning : expressivity exponential w.r.t depth

Interesting results

• cat paper by Andrew Ng & Goole

• AlexNet by Toronto

• last week CNTK at speech recognition parity with humans

Industrial results

• Autonomous Driving : Drive.ai, Comma.ai + the usual suspects

• Drugs discovery : Deep Genomics (Frey) & Bayer

• Predictive Maintenance : Thales, Bosch

• optimistic pessimism (Moghimi, Manulife Financial Corp.)

DeepLearning in two steps : training, applying

• training tends to require lots of data, (R)

• but applying does not (embedded, etc).

So that applying pre-trained models (Tensorframes) not the technical/business challenge.

Enterprise : have lots of data yourself, what to apply ?

Benchmarks aren't distributed

Training, but how ?

New Amazon GPU instances ?

Deep Learning Training

• Facebook, Amazon, Google, Baidu, Microsoft have this distributed

• But what if you’re not one of them ?

Training, but how ?

Distributing training

• basically distributing SGD (R)

• challenge is AllReduce Communication

• Sparse updates, async communications

Deeplearning4J

• the first commercial-grade, open-source, distributed deep-learning library written for Java and Scala

• Skymind its commercial support arm

Scientific computing on the JVM

• libnd4j : Vectorization, 32-bit addressing, linalg (BLAS!)

• JavaCPP: generates JNI bindings to your CPP libs

• ND4J : numpy for the JVM, native superfast arrays

• Datavec : one-stop interface to an NDArray

• DeepLearning4J: orchestration, backprop, layer definition

• ScalNet: gateway drug, inspired from (and closely following) Keras

Reinforcement learning

Killing the bottlenecks : generic

• swappable net backend : netty -> aeron (Hi Lightbend !)

• better support for binary data : big indexed tablesBinary, columnar, off-heap

• and more (Tamiya Onodera's group @ IBM Japan):http://www.slideshare.net/ishizaki/exploiting-gpus-in-spark

And if you don't care about Deep Learning ?

• Spark-6442 : better linear algebra than breeze, please.(sparse, performant, Java-compatible, and an OK license)

• SystemML got a best paper at VLDB'16, how about helping out on nd4j ?

• ND4J only lacks sparse, but not for long ...

Questions ?