CoreNeuron : Morphologically Detailed Neuron Simulations

Building, Simulating and Optimizing Large Neuron Networks on GPUs

Pramod Kumbhar, Michael Hines& Blue Brain HPC Team7th April 2016, GTC

Understanding Brain..

2013

BRAIN Initiative

2013

Human Brain Project

2014Brain/MINDS

Better understanding of brain

2007

Izhikevic: brain scale simulation on cluster

1 million cells

2009

IBM: Cat’s brain scale simulation on BG-P

1.5 billion cells

See source1

Brain Simulations

Point Neurons Morphologically Detailed Neurons

Molecular Level

See source3 See source4

Blue Brain Project, EPFL

Reverse Engineering Brain (10000 feet view)

See source5

See source6

Reconstruction Workflow

• 15+ years of Experiments

• 14,000 Neurons recorded

and labeled

• 2,052 Classified neurons

• 1,009 Reconstructed

neurons

• 2,000+ Ion channel

recordings

• 4,000+ Electrical recordings

of single neurons

• 5,000+ Synaptic recordings

of pairs of neurons

Markram et al. 2015, Cell

Modeling Neuron

See source8

See source7

HH, 1952

• Prominent components of nervous system

• More than 300 ion channels

Ion Channels

Biologist view:compartment model

Every channel is a compute kernel, no single hotspot!

NMODL: Source to Source Compiler

LexicalAnalyzer

SyntaxAnalyzer

SemanticAnalyzer

~ IntermediateRepresentation

tokens

parse tree

parse tree

OpenACCC Cuda Cyme:SIMDDSL

backends

NMODL

PortablePerformance

wrap OpenACC and vectorisation hints related pragmas

auto-generated kernel

OpenACC API’s to copy the complex data structure

OpenACC Kernels

User defined DS: Major challenge for many application

AoS/SoA, Vectorisation, Memory Coalescing etc..

bksub: for(i = x; i < nodes; i++) {

rhs[i] -= b[i] * rhs[ parent[i] ]rhs[i] /= d[i]

}

GPU: Cell level Parallelism (some kernels)

node : 1 2 3 4 parent[i]: 0 1 2 3

step 0

Step 1

Step 2

Step 3

node : 6 7 8 9 parent[i]: 5 6 7 8

node : 156 157 158 159 parent[i]: 155 156 157 158

thread 0

thread 1

thread 31

Memory addresses0

warp

Stride Depth of Tree

0

1

2

3

4

-1

0

1

2

3

parent_indexnode_index

5

6

7

8

9

-1

5

6

7

8

0

2

4

6

8

-1

0

1

2

3

parent_indexnode_index

1

3

5

7

9

-1

5

6

7

8

0

1

2

3

4

5

6

7

8

9

-1

-1

0

1

2

3

4

5

6

7

all cells root

Cell 0 and Cell 1 nodes interleaved

Roots parent

Cell 0 parent

Cell 1 parent

Nodes Parents

Cell Interleaving

Permutations: ions, synapses, areas, point processes..

Memory addresses0

warp

Spike ExchangeA

20 mV200 ms

bAC

dNAC

cSTUT

bSTUT

cIR

bIR

cAD

cAC

dSTUTcNAC

bNAC

e-ty

pe fr

actio

n fo

r eac

h m

-type

m-ty

pe

e-type

100%

80

60

40

20

0

C

cAC

cNAC

bNAC

dNACB

bAC

cIR

37% 31% 5%

25% 1.5% 1.5%

Figure 4. Markram et al.

e-types me-types me-combinationsL1DAC

L1NGC-DA

L1NGC-SA

L1HAC

L1LAC

L1SAC

L23PC

L23MC

L23BTC

L23DBC

L23BP

L23NGC

L23LBC

L23NBC

L23SBC

L23ChC

L4PCL4SP

L4SS

L4MC

L4BTC

L4DBC

L4BP

L4NGC

L4LBC

L4NBC

L4SBC

L4ChC

L5TTPC1

L5TTPC2

L5UTPC

L5STPC

L5MC

L5BTC

L5DBC

L5BP

L5NGC

L5LBC

L5NBC

L5SBC

L5ChC

L6TPCL1

L6TPCL4

L6UTPC

L6IPC

L6BPC

L6MC

L6BTC

L6DBC

L6BP

L6NGC

L6LBC

L6NBC

L6SBC

L6ChC

cAC

bAC

cNAC

bNAC

dNAC

cSTU

TbS

TUT

dSTU

T cIR bIR cAD

L23 NBC(burst Accommodating)

(continuous Non-accommodating)

(burst Non-accommodating)

(continuous Accommodating)

(delayed Non-accommodating)

(continuous Stuttering)

(burst Stuttering)

(delayed Stuttering)

(continuous Irregular)

(burst Irregular)

(continuous Adapting)

• Electrical diversity: 11 e-types; 207 me-types

• Number of connections increases exponentially

• Different types of events

NetCon List Buffering Mechanism

Compute engine of NEURON simulator

Being developed for large scale simulations (28 racks BG-Q)

Ion Channels: ~ 85% time

Linear Algebra: ~ 5-7%

Spike Exchange: 7-10%

CoreNeuron

Timeline

loadCPU

GPUcurrent solve state

dt

copy initialize

setup

threshold

queue MPI queue

mindelay

Toolchain

Soma

Compartment

hhpas

pas

pas

pas

pas

Simple model…

larger cells

65536 3072 1024

Varying # : Rings, Cells, Branches, Compartments

Model A Model B Model C Model D Model E

Performance

• K20x vs 8-core Xeon• Cray (OpenACC)• Cuda 7• No “hand” tuning yet• Optimized CPU code with

vectorization

Real World Models

Ion channels are 4-8x faster in all models!

Kernels with cell level parallelism, low occupancy!

393216 65536 4096 3072

• 8-core Xeon / 8 MPIs• BG-Q Node / 32-64 threads• Xeon Phi 61 core @ 1.23 GHz,

180-240 threads• K20X GPU• Optimized Xeon/MIC code with

vectorization (XLC issue)

Performance

Varying # : Rings, Cells, Branches, Compartments

Cell Interleaving and Exposing Parallelism

HomogenousHeterogeneous

Ideal

ill - suited

How much parallelism? How much imbalance?

Morphological diversity challenge

“Morphology Aware Scheduling of Kernels using Isomorphic Subtrees”

Resource

Reconstruction and Simulationof Neocortical MicrocircuitryHenry Markram,1,2,19,* Eilif Muller,1,19 Srikanth Ramaswamy,1,19 Michael W. Reimann,1,19 Marwan Abdellah,1

Carlos Aguado Sanchez,1 Anastasia Ailamaki,16 Lidia Alonso-Nanclares,6,7 Nicolas Antille,1 Selim Arsever,1

Guy Antoine Atenekeng Kahou,1 Thomas K. Berger,2 Ahmet Bilgili,1 Nenad Buncic,1 Athanassia Chalimourda,1

Giuseppe Chindemi,1 Jean-Denis Courcol,1 Fabien Delalondre,1 Vincent Delattre,2 Shaul Druckmann,4,5

Raphael Dumusc,1 James Dynes,1 Stefan Eilemann,1 Eyal Gal,4 Michael Emiel Gevaert,1 Jean-Pierre Ghobril,2

Albert Gidon,3 Joe W. Graham,1 Anirudh Gupta,2 Valentin Haenel,1 Etay Hay,3,4 Thomas Heinis,1,16,17 Juan B. Hernando,8

Michael Hines,12 Lida Kanari,1 Daniel Keller,1 John Kenyon,1 Georges Khazen,1 Yihwa Kim,1 James G. King,1

Zoltan Kisvarday,13 Pramod Kumbhar,1 Sebastien Lasserre,1,15 Jean-Vincent Le Be,2 Bruno R.C. Magalhaes,1

Angel Merchan-Perez,6,7 Julie Meystre,2 Benjamin Roy Morrice,1 Jeffrey Muller,1 Alberto Munoz-Cespedes,6,7

Shruti Muralidhar,2 Keerthan Muthurasa,1 Daniel Nachbaur,1 Taylor H. Newton,1 Max Nolte,1 Aleksandr Ovcharenko,1

Juan Palacios,1 Luis Pastor,9 Rodrigo Perin,2 Rajnish Ranjan,1,2 Imad Riachi,1 Jose-Rodrigo Rodrıguez,6,7

Juan Luis Riquelme,1 Christian Rossert,1 Konstantinos Sfyrakis,1 Ying Shi,1,2 Julian C. Shillcock,1 Gilad Silberberg,18

Ricardo Silva,1 Farhan Tauheed,1,16 Martin Telefont,1 Maria Toledo-Rodriguez,14 Thomas Trankler,1 Werner Van Geit,1

Jafet Villafranca Dıaz,1 Richard Walker,1 Yun Wang,10,11 Stefano M. Zaninetta,1 Javier DeFelipe,6,7,20 Sean L. Hill,1,20

Idan Segev,3,4,20 and Felix Schurmann1,201Blue Brain Project, Ecole polytechnique federale de Lausanne (EPFL) Biotech Campus, 1202 Geneva, Switzerland2Laboratory of Neural Microcircuitry, Brain Mind Institute, EPFL, 1015 Lausanne, Switzerland3Department of Neurobiology, Alexander Silberman Institute of Life Sciences, The Hebrew University of Jerusalem, Jerusalem 91904, Israel4The Edmond and Lily Safra Center for Brain Sciences, The Hebrew University of Jerusalem, Jerusalem 91904, Israel5Janelia Farm Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA6Laboratorio Cajal de Circuitos Corticales, Centro de Tecnologıa Biomedica, Universidad Politecnica de Madrid, 28223 Madrid, Spain7Instituto Cajal (CSIC) and CIBERNED, 28002 Madrid, Spain8CeSViMa, Centro de Supercomputacion y Visualizacion de Madrid, Universidad Politecnica de Madrid, 28223 Madrid, Spain9Modeling and Virtual Reality Group, Universidad Rey Juan Carlos, 28933 Mostoles, Madrid, Spain10Key Laboratory of Visual Science and National Ministry of Health, School of Optometry and Opthalmology, Wenzhou Medical College,Wenzhou 325003, China11Caritas St. Elizabeth’s Medical Center, Genesys Research Institute, Tufts University, Boston, MA 02111, USA12Department of Neurobiology, Yale University, New Haven, CT 06510 USA13MTA-Debreceni Egyetem, Neuroscience Research Group, 4032 Debrecen, Hungary14School of Life Sciences, University of Nottingham, Nottingham NG7 2UH, United Kingdom15Laboratoire d’informatique et de visualisation, EPFL, 1015 Lausanne, Switzerland16Data-Intensive Applications and Systems Lab, EPFL, 1015 Lausanne, Switzerland17Imperial College London, London SW7 2AZ, UK18Department of Neuroscience, Karolinska Institutet, Stockholm 17177, Sweden19Co-first author20Co-senior author*Correspondence: henry.markram@epfl.chhttp://dx.doi.org/10.1016/j.cell.2015.09.029

SUMMARY

We present a first-draft digital reconstruction of themicrocircuitry of somatosensory cortex of juvenilerat. The reconstruction uses cellular and synapticorganizing principles to algorithmically reconstructdetailed anatomy and physiology from sparse experi-mental data. An objective anatomical method definesa neocortical volume of 0.29 ± 0.01 mm3 containing!31,000 neurons, and patch-clamp studies identify55 layer-specific morphological and 207 morpho-electrical neuron subtypes. When digitally recon-structed neurons are positioned in the volume andsynapse formation is restricted to biological boutondensities and numbers of synapses per connection,

their overlapping arbors form !8 million connectionswith !37 million synapses. Simulations reproducean array of in vitro and in vivo experiments withoutparameter tuning. Additionally, we find a spectrumofnetworkstateswithasharp transition fromsynchro-nous to asynchronous activity, modulated by physio-logical mechanisms. The spectrum of network states,dynamically reconfigured around this transition, sup-ports diverse information processing strategies.

INTRODUCTION

Since Santiago Ramon y Cajal’s seminal work on the neocortex(DeFelipe and Jones, 1988; Ramon y Cajal, 1909, 1911), a vastnumber of studies have attempted to unravel its multiple levels

456 Cell 163, 456–492, October 8, 2015 ª2015 Elsevier Inc.

Resource

Reconstruction and Simulation of NeocorticalMicrocircuitry

Graphical Abstract

Highlightsd The Blue Brain Project digitally reconstructs and simulates a

part of neocortex

d Interdependencies allow dense in silico reconstruction from

sparse experimental data

d Simulations reproduce in vitro and in vivo experiments

without parameter tuning

d The neocortex reconfigures to support diverse information

processing strategies

AuthorsHenry Markram, Eilif Muller,

Srikanth Ramaswamy,

Michael W. Reimann, ..., Javier DeFelipe,

Sean L. Hill, Idan Segev, Felix Schurmann

Correspondencehenry.markram@epfl.ch

In BriefA digital reconstruction and simulation of

the anatomy and physiology of

neocortical microcircuitry reproduces an

array of in vitro and in vivo experiments

without parameter tuning and suggests

that cellular and synaptic mechanisms

can dynamically reconfigure the state of

the network to support diverse

information processing strategies.

Markram et al., 2015, Cell 163, 456–492October 8, 2015 ª2015 Elsevier Inc.http://dx.doi.org/10.1016/j.cell.2015.09.029

THANK YOU!

Explanatory Graphic Sources

• source 1: dgallery.s3.amazonaws.com

• source 2: clipartbest.com/cliparts, scaryforkids.com, geniusawakening.com

• source 3: developer.humanbrainproject.eu

• source 4: nature.com

• source 5: deviantart.net, squarespace.com

• source 6: lcn.epfl.ch

• source 7: nature.com

• source 8: genesis-sim.org

low occupancy!

Model A Model B Model C Model D Model E

larger cells

65536 2048 1024

Backup