25
Future Directions in Neuromorphic Computing Steve Furber ICL Professor of Computer Engineering The University of Manchester 1

Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

Future Directions in Neuromorphic

Computing

Steve Furber ICL Professor of Computer

Engineering

The University of Manchester

1

Page 2: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

200 years ago…

• Ada Lovelace, b. 10 Dec. 1815

"I have my hopes, and very distinct ones too, of one day getting cerebral phenomena such that I can put them into mathematical equations--in short, a law or laws for the mutual actions of the molecules of brain. …. I hope to bequeath to the generations a calculus of the nervous system.”

2

Page 3: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

65 years ago…

3

Page 4: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

ConvNets - structure

• Dense convolution kernels • Abstract neurons • Only feed-forward connections • Trained through backpropagation

4

Page 5: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

The cortex - structure

• Spiking neurons • Two-dimensional structure • Sparse connectivity

Feed-forward input

Feedback input

Feed-forward output

Feedback output

5

Page 6: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

ConvNets - GPUs

• Dense matrix multiplications • 3.2kW • Low precision

6

Page 7: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

Cortical models - Supercomputers

• Sparse matrix operations • Efficient communication of spikes • 2.3MW

7

Page 8: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

Cortical models - Neuromorphic hardware

• Memory local to computation • Low-power • Real time • 62mW

8

Page 9: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

http://www.technologyreview.com/featuredstory/526506/neuromorphic-chips/

9

Page 10: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

10 https://agenda.weforum.org/2015/03/top-10-emerging-technologies-of-2015-2/

Page 11: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

REPORT TO THE PRESIDENT

Ensuring Long-Term U.S.

Leadership in Semiconductors

Executive Office of the President

President’s Council of Advisors on

Science and Technology

January 2017

11

Ensuring Long-Term U.S. Leadership in Semiconductors

28

Table A1. Selected component technology vectors that have a high probability of deployment in ten years

(* denotes more speculative deployment within this timeframe)

Component technology vector

Time-frame to first commercial products

Approach to achieving and retaining competitive advantage

Neuromorphic Computing

Available now Continued R&D into new architectures coupled with 3D technologies and new materials, Deep Learning accelerators (for mobile and data center applications), and applications for true brain-inspired computing

Photonics Available now Foundries for tools and materials R&D; integrate photonics with CMOS and other materials

Sensors Available now Foundries for tools and materials R&D; integrate new types/classes of sensors with CMOS and other materials

CMOS (sub 7nm node size or new 3D structures)*

Advances in thermal management available with new process nodes

Deep understanding of transistor physics and chipset architecture and related design know-how; foundries and labs for transistor and materials R&D

Magnetics 1-2 years (MRAM as eFlash), 3 years (as DRAM), 5-7 years (as SRAM)

Foundries for tools and materials R&D; integrate magnetics with CMOS and other materials

3D 2-3 years (wafer-to-wafer stacking), 4-5 years (die-to-wafer stacking), 5-7 (Monolithic 3D)

Deep understanding of applications space and benefits associated use of 3D technologies and design know-how; foundries for tools and materials R&D; design automation tool R&D

Data-flow based architectures

3-4 years Continued architecture R&D, coupled with materials, integration, and manufacturing; build an ecosystem for solutions using data-flow based architectures

Ultra-high performance wireless systems

3 years (5G), 10-12 years (6G)

Continued R&D in new materials and processes, antenna design advances, chipset manufacturing, and integration

Advanced non-volatile memory as SRAM

5+ years Deep understanding of applications space and chipset architectures

Carbon nanotubes and phase change materials*

5-7 years Foundries/labs for materials R&D for hardware architectures; chipset designs to leverage these technologies

Biotech/human health

5-10 years R&D towards low power, highly integrated, high performance processing, high-data rate communications, wireless charging; couple R&D with clinical research to create, build, and evaluate on new materials and interfaces

Quantum Computing

< 10 years Pre-competitive R&D labs for new materials; foundries for new materials and hardware architectures; tools for quantum algorithms and software programming with various architectural paradigms

Point-of-Use Nanoscale 3D printing

Available now Desktop fab capabilities for rapid prototyping, additive manufacturing, moving beyond silicon and interfacing with soft matter, and small batch production

DNA for compute and storage*

10+ years Multi-disciplinary basic research in efficiently and reliably reading and writing and retrieving DNA strands

11

Page 12: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

Start-ups and industry interest

Page 13: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

SpiNNaker project

• A million mobile phone processors in one computer

• Able to model about 1% of the human brain…

• …or 10 mice!

13

Page 14: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

SpiNNaker system

14

Page 15: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

SpiNNaker chip

Multi-chip packaging by

UNISEM Europe

15

Page 16: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

Chip resources

16

Page 17: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

SpiNNaker machines

17

SpiNNaker chip

(18 ARM cores)

SpiNNaker board

(864 ARM cores)

SpiNNaker racks

(1M ARM cores)

• HBP platform

– 1M cores

– 11 cabinets (including server)

• Launch 30 March 2016 – then 500k cores

– 82 remote users

– 4,939 SpiNNaker jobs run

Page 18: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

SpiNNaker machines

18

• 100 SpiNNaker

systems in use

– global coverage

• 4-node boards

– training & small-

scale robotics

• 48-node boards

– insect-scale

networks

• multi-board systems

• 1M-core HBP

platform

sales (40 48-node boards)

loans

Page 19: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

19

Cortical microcolumn 1st full-scale simulation of 1mm2 cortex on neuromorphic & HPC systems

• 77,169 neurons, 285M synapses, 2,901 cores

S.J. van Albada, A.G. Rowley, A. Stokes, J. Senk, M. Hopkins, M. Schmidt, D.R. Lester, M. Diesmann, S.B. Furber, “Performance comparison of the digital neuromorphic hardware SpiNNaker and the Neural network simulation software NEST for a full-scale cortical microcircuit model”, Frontiers 2018.

Page 20: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

20

Constraint satisfaction problems

work by: Gabriel Fonseca Guerra

(PhD student)

G. A. Fonseca Guerra and S. B. Furber,

“Using Stochastic Spiking Neural

Networks on SpiNNaker to Solve

Constraint Satisfaction Problems”,

Frontiers 2018.

S. Habenschuss, Z. Jonke, and

W. Maass, “Stochastic computations in

cortical microcircuit models”, PLOS

Computational Biology,

9(11):e1003311, 2013.

Stochastic spiking neural

network:

• Solves CSPs, e.g. Sudoku

• 37k neurons

• 86M synapses

• also

• map colouring

• Ising spin systems

Page 21: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

Deep Rewiring

• Synaptic sampling as dynamic rewiring for rate-based neurons (deep networks)

• Ultra-low memory footprint even during learning

• Uses PRNG/TRNG, FPU, exp • speed-up 1.5

• Example: LeNet 300-100 • 1080 KB 36 KB • training on local SRAM possible • ≈ 100x energy reduction for training

on SpiNNaker2 prototype (28nm) compared to X86 CPU

• 96.2% MNIST accuracy for 0.6% connectivity

LeN

et 3

00

-10

0 In MNIST 784

Hidden FC 300

Hidden FC 100

Out Softmax 10

G. Bellec et al., “Deep rewiring: Training very sparse deep networks”, arXiv, 2018 Chen Liu et al., “Memory-efficient Deep Learning on a SpiNNaker 2 prototype”, submitted 21

Page 22: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

SpiNNaker2 Chip Overview

22

• 160 ARM-based processing elements (PEs) • 8 GByte LPDDR4 DRAM • 7 energy efficient chip-to-chip links

Page 23: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

New Hardware Feature Innovations

23

Feature Output/Benefit Patents Publications

Dynamic Neuromorphic Power Management

Reduce power consumption by up to x5 Examples: • Reward-based learning • Synfire chain • Event based vision sensor processing

DE102017128711.6 [Höppner2017]

Exponential Function Accelerator Enhance performance by x2 Examples: • Reward-based learning • STDP • BCPNN

[Partzsch2017] [Bauer2017] [Mikaitis2018]

Pseudo and true random number generation

Enhance performance by >x2 for pseudo random numbers Enable true randomness Examples: • Synaptic sampling • Deep-rewiring

EP3147775A1 US20170083289

[Neumärker2017]

exp exp

ARMexp

unitFIFOAHB

Page 24: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

SpiNNaker2 Testchip JIB

24

• Taped out Testchip “JIB” • 8 processing elements (PE) • SpiNNaker Router, Chip2Chip Links • 9mm² in GLOBALFOUNDRIES 22FDX

technology • Low voltage operation down to 0.40V

• Power analyses for neuromorphic and

machine learning • Analysis results (scaled to SpiNNaker2)

• CPU Efficiency: • 47GOPS @<20µW/MHz

• Machine Learning • 4.6TOPS @ 6.4 Tops/W

• Scaling compared to SpiNNaker1

• x15 CPU performance • x2 performance by hardware

accelerators • >x30 better efficiency

PE

QPE

SpiNNaker Router

Shared SRAM

Host IF

Ch

ip2

Ch

ip IO

Ch

ip2

Ch

ip IO

QPE

Page 25: Future Directions in Neuromorphic Computing...Neuromorphic Computing Available now Continued R& D into new architectures coupled with 3D technologies and new mater ials, Deep Learning

Conclusions • SpiNNaker:

• has been 20 years in conception…

• …and 10 years in construction, • and is now ready for action!

• ~90 boards with groups around the world

• 1M core machine built

• HBP is supporting s/w development

• Industrial AI uses 2nd generation (non-spiking) neural nets

• there is an expectation that neuromorphics will contribute • especially energy-efficiency?

• SpiNNaker is the ideal research platform to explore this space • SpiNNaker2 will be 10x better!

25