Optimization with Big Data

Peter RichtarikSchool of Mathematics

Optimization with Big Data * in a billion dimensional space on a foggy day

Extreme* Mountain Climbing=

BIG DATA

• digital images & videos• transaction records• government records• health records• defence• internet activity (social media, wikipedia, ...)• scientific measurements (physics, climate models, ...)

BIG Volume BIG Velocity BIG Variety

Sources

BIG Volume BIG Velocity BIG Variety

Western General Hospital(Creutzfeldt-Jakob Disease)

Arup (Truss Topology Design)

Ministry of Defence dstl lab(Algorithms for Data Simplicity)Royal Observatory

(Optimal Planet Growth)

GOD’S Algorithm = Teleportation

If you are not a God...

x0x1

x2 x3

Optimization as Lock Breaking

Setup: Combination maximizing F opens the lock

x = (x1, x2, x3, x4) F(x) = F(x1, x2, x3, x4)

A number representing the

“quality” of a combination

Optimization Problem: Find combination maximizing F

Optimization Algorithm

How to Open a Lock with Billion Interconnected Dials?

F : Rn R# variables/dials = n = 109

x1

x2

Assumption:F = F1 + F2 + ... + Fn

-----------------------Fj depends on the neighbours of xj only

x3

x4

Example:F1 depends on x1, x2, x3 and x4

F2 depends on x1 and x2, ...

xn

Optimization Methods

Computing Architectures• Multicore CPUs• GP GPU accelerators• Clusters / Clouds

• Effectivity• Efficiency• Scalability• Parallelism• Distribution• Asynchronicity• Randomization

Optimization Methods for Big Data

• Randomized Coordinate Descent– P. R. and M. Takac: Parallel coordinate descent

methods for big data optimization, ArXiv:1212.0873 [can solve a problem with 1 billion variables in 2 hours using 24

processors]• Stochastic (Sub) Gradient Descent

– P. R. and M. Takac: Randomized lock-free methods for minimizing partially separable convex functions

[can be applied to optimize an unknown function]• Both of the above

M. Takac, A. Bijral, P. R. and N. Srebro: Mini-batch primal and dual methods for SVMs, ArXiv:1302.xxxx

Theory vs Reality

start

settle for this

holy grail

Parallel Coordinate Descent

TOOLSProbability

Machine LearningMatrix Theory

HPC