Embed Size (px)
Citation preview
A High-Throughput Screening Approach to Discovering Good Forms of Biologically-Inspired Visual RepresentationNicolas Pinto1 , David Cox 2 and James DiCarlo1
1. The McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology2. The Rowland Institute at Harvard
Take home message:
GPU Metaprogramming dramatically accelerates the discovery of bio-inspired vision models that beat state-of-the-art computer vision systems
Abstract1While many models of biological object recognition share a common set of "broad-stroke" properties, the performance of any one model depends strongly on the choice of parameters in a particular instantiation of that model - e.g. the number of units per layer, the size of pooling kernels, expo-nents in normalization operations, etc. Since the number of such parameters (explicit or implicit is typically large, and the computational cost of evaluating one particular parameter set is high, the space of possible model instantia-tions goes largely unexplored. Thus, when a model fails to approach the abili-ties of biological visual systems, we are left uncertain whether this failure is because we are missing a fundamental idea, or because the correct "parts" have not been tuned correctly, assembled at sufficient scale, or provided with enough training.
Here, we present a high-throughput approach to the exploration of such pa-rameter sets, leveraging recent advances in stream processing hardware (high-end NVIDIA graphic cards and the PlayStation 3's IBM Cell Processor). In analogy to high-throughput screening approaches in molecular biology and genetics, we explored thousands of potential network architectures and pa-rameter instantiations, screening those that show promising object recogni-tion performance for further analysis. We show that this approach can yield significant, reproducible gains in performance in across an array of basic object recognition tasks, consistently outperforming a variety of state-of-the-art vision systems from the literature.
As the scale of available computational power continues to expand, we argue that this approach has the potential to greatly accelerate progress in both ar-tificial vision and our understanding of the computational underpinning of biological vision.
The Rowland Institute at HarvardHARVARD UNIVERSITY
High-throughput screening for good models5
Analysis: Why are these models good ?7
Metaprogramming3
A Vast space of models to explore2
Recipe:1) Use scripting (e.g. Python) and templates,2) Instrumentalize your code,3) Let the computer auto-tune it!
Unsupervised Learning Test with a “screening”object recognition task Validate on other tasksChoose the best modelsGenerate Random Models
Unsupervised Learning Screen Distribution
Law & Order “World”
Performance / Cost
Hardware
Relative Speedup 1x 80x4x 544x 222x 1544x 2712x
Relative Perf. / $ 1x 120x2x 272x 833x 772x 1356x
Full System Cost(approx.)
$1,500** $1,000$2,700** $3,000* $400 $3,000* $3,000*
Manufacturer Intel IntelIntel NVIDIA Sony, IBM, Toshiba NVIDIA NVIDIA
Model Q9450 Q9450Q9450 7900 GTX PlayStation 3 8800 GTX GTX 280
Year 2008 20082008 2006 2007 2007 2008
Implementation MATLAB SSE2MATLAB Cg Cell SDK CUDA CUDA
# cores used 1 44 4x96 2+6 4x128 4x240
CPUs GPUs
10-1
100
101
102
103
theoreticalobserved
Proc
essi
ng P
erfo
rman
ce in
GFL
OPS
(lo
g-sc
ale)
Software
LanguagesExtensions / Libraries
Performance / Cost
Relative Speedup$ / GFLOPS
Year of implementation
Full System Cost(approx.)
Hardware
ManufacturerModel# cores used
GFLOPS (max)
5.3
(max
)
0.5
(max
) {
IntelQ9450
MATLAB / CMEX
1
2008
$1,500**
3000
1
0.5
1350
Q9450
21 (m
ax)
2 (m
ax)
{
4
Intel
MATLAB / C
4
2008
$2,700**
MEX
2
25
Q9450
85 (m
ax)
40 (m
ax)
{
4
Intel
CSSE2 / pthread
80
2008
$1,00040
22-11*
187
(max
)
68 (m
ax)
{
NVIDIA7900 GTX
Python / C++OpenGL / Cg
136-544*
2006
$1,500-$3,000*
96
68-272*
4
154
(max
)
111
(max
)
76 (m
ean)
9 (m
in)
{
Sony, IBM, ToshibaPlayStation 3 (Cell)
Python / CCell SDK
222
2007
$400
8 (2+6)
111
8-4*
518
(max
)
193
(max
) {
NVIDIA8800 GTX
Python / CPyCUDA
386-1544*
2007
$1,500-$3,000*
128
193-772*
240
4-2*
933
(max
)
339
(max
) {
NVIDIAGTX 280
Python / CPyCUDA
678-2712*Relative GFLOPS / $ 1 2 120 136-272* 833 386-772* 678-1356*
2008
$1,500-$3,000*339-1356*
num
ber o
f mod
els
50 60 70 80 90 1000
50
100
150
200
250
N=2500
top 5 models
Percent Correct
• bio-inspired architecture• 52 free parameters• >10 models25
Best Models
50
60
70
80
90
100
Perc
ent C
orre
ct
top 5 models (screen)
5 4 3 2 1 blend
di�erent initial weights
50
100di�erent
videos
Cars
vs.
Planes
• Why are the best models better? • How can we adjust the parameter space to get farther?
False True
Preprocessing / Norm / Zero-mean
tcerroc %
Parameter value
3 5 7 9
Layer 2 / Linear Filtering / RF size
Parameter value
16 32 64 128
1 3 5 7
50
60
70
80
90
Layer 3 / Learning / Neighborhood size
9
50
60
70
80
90
50
60
70
80
90
50
60
70
80
90
32.5
coun
t
median of Hamming distances (from 5-choose-2 binarized parameter vectors)
N = 100,000p = 0.136
Parameters similarity (top-�ve vs rest)
coun
t
12828
N = 100,000p = 0.082
median of Euclidean distances(from 5-choose-2 similarity matrices)
Representation similarity (top-�ve vs rest)
>=
coun
t
-ln(p)
Parameter Signi�cance
102 4 6 8
10
20
0.25
0.15
0.05
Frac
tion expected fraction
Filtering ActivationNormalizationPooling
Learning
Parameters by Category
low signi�cance
medium signi�cance
high signi�cance
conv_kernel_template.cu
...mad.rn.f32 $r4, s[$ofs3+0x0000], $r4, $r1mov.b32 $r1, c0[$ofs2+0x0008]mad.rn.f32 $r4, s[$ofs3+0x0008], $r1, $r4mov.b32 $r1, c0[$ofs2+0x000c]mad.rn.f32 $r4, s[$ofs3+0x000c], $r1, $r4mov.b32 $r1, c0[$ofs2+0x0010]mad.rn.f32 $r4, s[$ofs3+0x0010], $r1, $r4
...
version A2x faster!version B
...mad.rn.f32 $r1, s[$ofs1+0x007c], c0[$ofs1+0x0078], $r1mad.rn.f32 $r1, s[$ofs2+0x0000], c0[$ofs2+0x007c], $r1mad.rn.f32 $r1, s[$ofs2+0x0008], c0[$ofs2+0x0080], $r1mad.rn.f32 $r1, s[$ofs2+0x000c], c0[$ofs2+0x0084], $r1mad.rn.f32 $r1, s[$ofs2+0x0010], c0[$ofs2+0x0088], $r1
...
texture<float4, 1, cudaReadModeElementType> tex_float4;__constant__ float constant[$FILTER_D][$FILTER_W][$N_FILTERS];
#define IMUL(a, b) __mul24(a, b)extern "C" {
#for j in xrange($FILTER_H)
__global__ void convolve_beta_j${j}(float4 *input, float4 *output) {
#set INPUT_BLOCK_W = $BLOCK_W+$FILTER_W-1 __shared__ float shared_in[$INPUT_BLOCK_W][4+1];
// -- input/output offsets const uint in_idx = (blockIdx.y+$j)*INPUT_W + blockIdx.x*blockDim.x + threadIdx.x; const uint out_idx = blockIdx.y*OUTPUT_W + blockIdx.x*blockDim.x + threadIdx.x; float4 input_v4;
// -- load input to shared memory#for i in xrange($LOAD_ITERATIONS)#if $i==($LOAD_ITERATIONS-1) if((threadIdx.x+$BLOCK_W*$i)<$INPUT_BLOCK_W)#end if {
input_v4 = tex1Dfetch(tex_float4, in_idx+$BLOCK_W*$i);shared_in[threadIdx.x+$BLOCK_W*$i][0] = input_v4.x;shared_in[threadIdx.x+$BLOCK_W*$i][1] = input_v4.y;shared_in[threadIdx.x+$BLOCK_W*$i][2] = input_v4.z; ...
Cheetah Templating
conv_kernel_4x4x4.cu
20 kB
conv_kernel_8x8x4.cu
64 kB
#include <stdio.h>
texture<float4, 1, cudaReadModeElementType> tex_float4;__constant__ float constant[4][4][4];
#define IMUL(a, b) __mul24(a, b)extern "C" {
__global__ void convolve_beta_j0(float4 *input, float4 *output) {
__shared__ float shared_in[131][4+1];
// -- input/output offsets const uint in_idx = (blockIdx.y+0)*INPUT_W + blockIdx.x*blockDim.x + threadIdx.x; const uint out_idx = blockIdx.y*OUTPUT_W + blockIdx.x*blockDim.x + threadIdx.x; float4 input_v4;
// -- load input to shared memory {
input_v4 = tex1Dfetch(tex_float4, in_idx+128*0);shared_in[threadIdx.x+128*0][0] = input_v4.x;shared_in[threadIdx.x+128*0][1] = input_v4.y;shared_in[threadIdx.x+128*0][2] = input_v4.z;shared_in[threadIdx.x+128*0][3] = input_v4.w;
} if((threadIdx.x+128*1)<131) {
input_v4 = tex1Dfetch(tex_float4, in_idx+128*1);shared_in[threadIdx.x+128*1][0] = input_v4.x;shared_in[threadIdx.x+128*1][1] = input_v4.y;shared_in[threadIdx.x+128*1][2] = input_v4.z;shared_in[threadIdx.x+128*1][3] = input_v4.w;
} __syncthreads();
// -- compute dot products float v, w;
float sum0 = 0; float sum1 = 0; float sum2 = 0; float sum3 = 0; v = shared_in[threadIdx.x+0][0]; w = constant[0][0][0]; sum0 += v*w; w = constant[0][0][1]; sum1 += v*w; w = constant[0][0][2]; sum2 += v*w; w = constant[0][0][3]; sum3 += v*w; v = shared_in[threadIdx.x+1][0]; w = constant[0][1][0]; sum0 += v*w; w = constant[0][1][1]; sum1 += v*w; w = constant[0][1][2]; sum2 += v*w; w = constant[0][1][3]; sum3 += v*w; v = shared_in[threadIdx.x+2][0]; w = constant[0][2][0]; sum0 += v*w; ...
4 GPU performance
Result6Validation
50
60
70
80
90
100
Perc
ent C
orre
ctPe
rcen
t Cor
rect
50
60
70
80
90
100
50
60
70
80
90
100
Cars vs. Planes (validation)
Synthetic Faces MultiPIE Hybrid
v1-like(control)
state-of-the-art(from literature)
top 5 models(high-throughput search)
5SIFTregular + GB PHOG PHOW SLF 4 3 2 1 blend50
60
70
80
90
100
Boats vs. Animals
v1-like(control)
state-of-the-art(from literature)
top 5 models(high-throughput search)
regular + blendSIFT GB PHOG PHOW SLF
v1-like(control)
state-of-the-art(from literature)
top 5 models(high-throughput search)
SIFTregular + GB PHOG PHOW SLF blend
v1-like(control)
state-of-the-art(from literature)
top 5 models(high-throughput search)
regular + blendSIFT GB PHOG PHOW SLF
5 4 3 2 1
5 4 3 2 15 4 3 2 1
We discovered models that beat state-of-the-art computer vision(object and face recognition)
L1
L2
L3
input
Read-out (standard “back-end” classi�er)
number of �lters
kernel
kernel
number of �lters
number of �lters
Learning
kernel
normalization
normalization
normalization
norm strengththresh/sat
norm strengththresh/sat
norm strengththresh/sat
RateTrace“Temp. Adv.”“Auto-reset”
...
Learning
RateTrace“Temp. Adv.”“Auto-reset”
...
Learning
RateTrace“Temp. Adv.”“Auto-reset”
...
