MIT

December 15, 2017

Jeff Hawkinsjhawkins@numenta.com

Have We Missed Half of What the Neocortex Does?

Allocentric Location as the Basis of Perception

1) Reverse engineer the neocortex- an ambitious but realizable goal- seek biologically accurate theories- test empirically and via simulation

2) Enable technology based on cortical theory- active open source community- basis for Machine Intelligence

L2

L3aL3b

L4

L6a

L6b

L6 ipL6 mpL6 bp

L5 ttL5 ccL5 cc-ns

L2/3

L4

L6

L5

Input

The Cortical Column

1) Cortical columns are complex- Twelve or more excitatory cellular layers- Two parallel FF pathways- Parallel FB pathways (not shown)- Numerous intra- and inter-column connections (not shown)- Inhibitory neurons/circuits are equally complex

2) The function of a cortical column must also be complex.

3) Whatever a column does applies to everything the cortex does.

L5: Calloway et. al, 2015L6: Zhang and Deschenes, 1997

Simple

Output, via thalamus

50%10%

CortexThalamus

Output, direct

L5 CTC: Guillery, 1995Constantinople and Bruno, 2013

A Couple of Thoughts

Output

Observation:

The neocortex is constantly predicting its inputs.

How do networks of neurons, as seen in the neocortex,learn predictive models of the world?

Research:

1) How does the cortex learn predictive models of extrinsic sequences?

2) How does the cortex learn predictive models of sensorimotor sequences?

Current research: How do columns compute allocentric location?

- Grid cells in entorhinal cortex solve a similar problem- Big Idea: cortical columns contain analogs of grid cells and head direction cells- Starting to understand the function of numerous layers and connections

“Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex”Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30

- Big Idea: Pyramidal neuron model for prediction- A single layer network model for sequence memory- Properties of sparse activations

“A Theory of How Columns in the Neocortex Learn the Structure of the World”Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25

- Extension of sequence memory model- Big Idea: Columns compute “allocentric” location of input- By moving sensor, columns learn models of complete objects

Proximal synapses: Cause somatic spikesDefine classic receptive field of neuron

Distal synapses: Cause dendritic spikesPut the cell into a depolarized, or “predictive” state

Depolarized neurons fire sooner, inhibiting nearby neurons.

A neuron can predict its activity in hundreds of unique contexts.

5K to 30K excitatory synapses- 10% proximal- 90% distal

Distal dendrites are pattern detectors- 8-15 co-active, co-located synapses

generate dendritic spike- sustained depolarization of soma

HTM Neuron Model

Prediction Starts in the Neuron

Pyramidal Neuron

Major, Larkum and Schiller 2013

Properties of Sparse Activations

L2

L3aL3b

L4

L6a

L6b

L6 ipL6 mpL6 bp

L5 ttL5 ccL5 cc-ns

Example: One layer of cells, 5,000 neurons, 2% (100) active

1) Representational capacity is virtually unlimited(5,000 choose 100) = 3x10211

2) Randomly chosen representations have minimal overlap

3) A neuron can robustly recognize an activation pattern by forming 10 to 20 synapses

4) Unions of patterns do not cause errors in recognition

Hypothesis: Cellular layers use unions to represent uncertainty

Hawkins, Ahmad, 2016Ahmad, Hawkins, 2015

Pattern 1 (100 active cells)

Cell robustly recognizes pattern1by forming synapses to small sub-sample of active cells

UnionPatterns 1-10 (1,000 active cells)

Cell still robustly recognizes pattern 1

A Single Layer Network Model for Sequence Memory

- Neurons in a mini-column learn same FF receptive field.- Neurons forms distal connections to nearby cells.

No prediction Predicted input

(Hawkins & Ahmad, 2016)

(Cui et al, 2016)

- High capacity (learns up to 1M transitions)

- Learns high-order sequences: “ABCD” vs “XBCY”

- Makes simultaneous predictions: “BC…” predicts “D” and “Y”

- Extremely robust (tolerant to 40% noise and faults)

- Learning is unsupervised, continuous, and local

- Satisfies many biological constraints

- Multiple open source implementations (some commercial)

t=0

t=1

Predicted cells fire first and inhibit neighbors

Next prediction t=2

t=0

t=1

1) How does the cortex learn predictive models of extrinsic sequences?

2) How does the cortex learn predictive models of sensorimotor sequences?

Current research: How do columns compute allocentric location?

- Grid cells in entorhinal cortex solve a similar problem- Hypothesis: cortical columns contain analogs of grid cells and head direction cells- Starting to understand the function of numerous layers and connections

“Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex”Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30

- Pyramidal neuron model- A single layer network model for sequence memory- Properties of sparse activations

“A Theory of How Columns in the Neocortex Learn the Structure of the World”Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25

- Extension of sequence memory model- Big Idea: Columns compute “allocentric” location of input- By moving sensor, columns learn models of complete objects

How Could a Layer of Neurons Learn a Predictive Model of Sensorimotor Sequences?

Sequence memory

Sensorimotor sequences

SensorMotor-related context

Hypothesis:By adding motor-related context, a cellular layer can predictits input as the sensor moves.

What is the correct motor-related context?

L2

L3aL3b

L4

L6a

L6b

L6 ipL6 mpL6 bp

L5 ttL5 ccL5 cc-ns

50%

Sensoryfeature

Two Layer Model of Sensorimotor Sequence Memory

Feature @ location

Object Stable over movement of sensor

With allocentric location input, a column can learn models of complete objects by sensing different locations on object over time.

SensorFeature

AllocentricLocation

Pooling

Seq Mem

Changes with each movement

Object

Feature @ LocationLocationon object

Column 1 Column 2 Column 3

Sensorfeature

Sensorimotor Inference With Multiple Columns

Each column has partial knowledge of object.

Long range connections in object layer allow columns to vote.

Inference is much faster with multiple columns.

FeatureFeatureFeatureLocationLocationLocation

Output

Input

Objects Recognized By Integrating Inputs Over Time

FeatureLocationFeatureLocationFeatureLocation

Column 1 Column 2 Column 3

Output

Input

Recognition is Faster with Multiple Columns

Yale-CMU-Berkeley (YCB) Object Benchmark (Calli et al, 2017)

- 80 objects designed for robotics grasping tasks

- Includes high-resolution 3D CAD files

YCB Object Benchmark

We created a virtual hand using the Unity game engineCurvature based sensor on each fingertip4096 neurons per layer per column

98.7% recall accuracy (77/78 uniquely classified)

Convergence time depends on object, sequence of sensations, number of fingers.

Simulation using YCB Object Benchmark

Pairwise confusion between objects after 1 touch

Convergence 1 finger 1 touch

Pairwise confusion between objects after 2 touches

Convergence 1 finger 2 touches

Pairwise confusion between objects after 6 touches

Convergence 1 finger 6 touches

Pairwise confusion between objects after 10

touches

Convergence 1 finger 10 touches

Convergence Time vs. Number of Columns

This is why we can infer complex objects in a single grasp or single visual fixation.

1) How does the cortex learn predictive models of extrinsic sequences?

2) How does the cortex learn predictive models of sensorimotor sequences?

Current research: How do columns compute allocentric location?

- Hypothesis: cortical columns contain analogs of grid cells and head direction cells- Starting to understand the function of numerous layers and connections

“Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex”Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30

- Pyramidal neuron model- A single layer network model for sequence memory- Properties of sparse activations

“A Theory of How Columns in the Neocortex Learn the Structure of the World”Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25

- Extension of sequence memory model- Big Idea: Columns compute “allocentric” location of input- By moving sensor, columns learn models of complete objects

Entorhinal Cortexenvironments

A

B C

X

Y Z

R

S T

Room 3

Room 2Room 1

Location- Encoded by Grid Cells- Unique to location in room AND room- Location is updated by movement

Orientation (of head to room)- Encoded by Head Direction Cells- Anchored to room- Orientation is updated by movement

Location- Unique to location on object AND object- Location is updated by movement

Orientation (of sensor patch to object)- Anchored to object- Orientation is updated by movement

Cortical Columnobjects

Hypothesis:Cortical columns contain analogs of grid cells and head direction cells

A

C

B

X

Y

Z

Stensola, Solstad, Frøland, Moser, Moser: 2012

Location and Orientation are both necessary to learn the structure of rooms and predict sensory input.

Location and Orientation are both necessary to learn the structure of objects and predict sensory input.

L3

L4

L6a

L6b

L5a

L5b

Mapping Orientation and Location to a Cortical Column (most complex slide)

Sensation

Orientation

1) A column is a two-stage sensorimotor model for learning and inferring structure.

2) A column usually cannot infer a Feature or Object in one sensation.- Integrate over time (sense, move, sense, move, sense..)- Vote with neighboring columns

3) This system is most obvious for touch, but it applies to vision and other sensory modalities.

Because this architecture exists throughout the neocortex, it suggests we learn, infer,and manipulate abstract concepts the same way we manipulate objects in the world.

Location

Sensation @ Orientation

Feature

Feature @ Location

Object

Motor updated (HD cell-like)

Motor updated (grid cell-like)

Seq mem

Pooling

Seq mem

Pooling

Meaning Operation

Rethinking Hierarchy

Every column learns complete models of objects. They operate in parallel.

Inputs project to multiple levels at once. Columns operate at different scales of input.

Sense

Simple features

Complex features

Objects

Classic

Objects

Objects

Objects

Sensor array

Proposed

Region 3

Region 2

Region 1

Rethinking Hierarchy

Every column learns complete models of objects. They operate in parallel.

Inputs project to multiple levels at once. Columns operate at different scales of input.

Non-hierarchical connections allow columns to vote on shared elements such as “object” and “feature”.

Sense

Simple features

Complex features

Objects

Classic

Sensor array

Objects

Objects

Objects

Sensor array

vision touch

Proposed

Region 3

Region 2

Region 1

Summary

Goal: Understand the function and operation of the laminar circuits in the neocortex.

Method: Study how cortical columns make predictions of their inputs.

Proposals

1) Pyramidal neurons are the substrate of prediction.Each neuron predicts its activity in hundreds of contexts.

2) A single layer of neurons forms a predictive memory of high-order sequences.(sparse activations, mini-columns, fast inhibition, and lateral connections)

3) A two-layer network forms a predictive memory of sensorimotor sequences.(add motor-derived context and a pooling layer)

4) Columns need motor-derived representations of location and orientation, of the

sensor relative to the object. These are analogous to grid and head direction cells.

5) A framework for the cortical column.- Columns learn complete models of objects as “features at locations”, using twosensorimotor inference stages.

6) The neocortex contains thousands of parallel models, that resolve uncertainty byassociative linking and/or movement of the sensors.

Open Issues

Behaviors: how are they learned, encoded, and applied to objects?

Detailed model of hierarchy including thalamus

How can the model be applied to “Where” pathways, and how do “What” and “Where” pathways work together

Collaborations

There are many testable predictions in this model, a “green field”. We welcome collaborations and discussions.

We are always interested in hosting visiting scholars and interns.

Numenta Team

Subutai AhmadVP Research

Marcus Lewis

Thank You