• Home

  • Documents

  • Experimental demonstration of software-trained neural ... · Experimental demonstration of...
8
Experimental demonstration of software-trained neural network inferencing in analog memristor crossbar arrays Miao Hu, Qiangfei Xia*, J. Joshua Yang*, R. Stanley Williams, and John Paul Strachan Hewlett Packard Labs, Hewlett Packard Enterprise, Palo Alto CA *UMass Amherst

Experimental demonstration of software-trained neural ... · Experimental demonstration of software-trained neural network inferencing in analog memristorcrossbar arrays Miao Hu,

  • Upload
    others

  • View
    6

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Experimental demonstration of software-trained neural ... · Experimental demonstration of software-trained neural network inferencing in analog memristorcrossbar arrays Miao Hu,

Experimental demonstration of software-trained neural network inferencing in analog memristor crossbar arrays

Miao Hu, Qiangfei Xia*, J. Joshua Yang*, R. Stanley Williams, and John Paul StrachanHewlett Packard Labs, Hewlett Packard Enterprise, Palo Alto CA*UMass Amherst

Page 2: Experimental demonstration of software-trained neural ... · Experimental demonstration of software-trained neural network inferencing in analog memristorcrossbar arrays Miao Hu,

2

• Parallel multiply & add through Kirchoff and Ohm’s law

1961, K. Steinbuch “Die Lernmatrix“– suggests using “ferromagnetic toroids”

• Memristors as highly scalable, tunable analog resistors

High ON/OFF ratio (~105), supporting multiple levels HPE differentiator vs competing accelerator designs

• Advantages: • Well suited for streaming workloads; Key advantage is in-memory processing; Many ways to scale up

• Many Teams have been working in this field: IBM, GeorgiaTech (Hasler), U Michigan (W.Lu), ASU (S. Yu), Duke (H.Li), and many others

IjO= ∑j Gij

. ViI

Input Voltage vector

Output current

DPE: Memristor arrays for computing

Is this true in the real world?

• But what actually does a crossbar do?

Page 3: Experimental demonstration of software-trained neural ... · Experimental demonstration of software-trained neural network inferencing in analog memristorcrossbar arrays Miao Hu,

3

Dot-product Engine demonstrator

3

Probe Card PCB:Programing,Reading,& Computing

Computer:Algorithm,

Data analysis

• Flexible peripheral circuit platform to study the behavior of actual memristor crossbars for in-memory computing.

Integrated Tantalum Oxide memristors

Page 4: Experimental demonstration of software-trained neural ... · Experimental demonstration of software-trained neural network inferencing in analog memristorcrossbar arrays Miao Hu,

4

64x64 = 4096 memristors (TaOx)~6 bits at each memristor (full range of accessible resistance)

Programming full memristor arrays

-100 -80 -60 -40 -20 0 20 40 60 80 100

G ( S)

0

50

100

150

200

250

300

350

Cou

nt

Histogram of error around zero

Page 5: Experimental demonstration of software-trained neural ... · Experimental demonstration of software-trained neural network inferencing in analog memristorcrossbar arrays Miao Hu,

5

MNIST Pattern recognition demonstration

5

Patte

rn v

ecto

r

10 values, entry with maximum value is the prediction

Wei

ght m

atrix

1 layer softmaxNeural network

Neural networkWeight matrix

Target Conductance (uS)

Programmed Conductance (uS)

Partition and program (100 uS to 700 uS) Programming error distribution

Page 6: Experimental demonstration of software-trained neural ... · Experimental demonstration of software-trained neural network inferencing in analog memristorcrossbar arrays Miao Hu,

6

Computing accuracy of a 64x64 crossbar

6

• Crossbar parameters:• Wire per segment ≈ 1 ohm• Input/output resistance ≈ 1ohm• Device resistance: 1.4k to 10k ohm (100 uSto 700 uS)

• Computing accuracy• 150k (2.5k * 60) data points.

• Memristor is <4 bit for the given range

• Output accuracy is ~4 bit.

• Noise is nonlinear due to circuit parasitics.

Page 7: Experimental demonstration of software-trained neural ... · Experimental demonstration of software-trained neural network inferencing in analog memristorcrossbar arrays Miao Hu,

7

Pattern “7” recognition

7

Output current per column Output current per class

Pattern “7” prediction

Part1 + Part2 + Part3 + Part4 + Part5 + Part6 = Current per class

Significant DegradationDue to circuit parasitic

Simulation matches experiments

G_IDEAL current = Vin * G_IDEAL; G_PROG current = Vin * G_PROG

Experiment

G_PROGcurrent

G_IDEAL current

Page 8: Experimental demonstration of software-trained neural ... · Experimental demonstration of software-trained neural network inferencing in analog memristorcrossbar arrays Miao Hu,

8 8

MNIST pattern recognition accuracy• Using a software-trained weight matrix, a single 64x64 crossbar achieves 85% accuracy (90% is ideal) for MNIST with post processing

• Single-layer NN highly sensitive to even a few defects

• Next steps:

• Better matrix to conductance mapping:

• Implement the “conversion algorithm” taking non-idealities into account

• Use Multi-layer NNs more resilient to defects:


Suitability of Artificial Neural Network to Text Document ... · of neural network approach to text summarization. The neural network is trained using paragraphs for identifying sentences

Suitability of Artificial Neural Network to Text Document ... · of neural network approach to text summarization. The neural network is trained using paragraphs for identifying sentences

Documents
Input-Dependent Neural Network trained by Real-Coded ... · Input-Dependent Neural Network trained by Real-Coded Genetic Algorithm and its Industrial Applications . ... networks such

Input-Dependent Neural Network trained by Real-Coded ... · Input-Dependent Neural Network trained by Real-Coded Genetic Algorithm and its Industrial Applications . ... networks such

Documents
Neural Circuits Trained with Standard Reinforcement ... · Neural Circuits Trained with Standard Reinforcement Learning Can Accumulate Probabilistic Information during Decision Making

Neural Circuits Trained with Standard Reinforcement ... · Neural Circuits Trained with Standard Reinforcement Learning Can Accumulate Probabilistic Information during Decision Making

Documents
Synthetically Trained Neural Networks for Learning Human ... · Fig. 2. System overview. As a user performs a demonstration in the real world, the perception networks detect the pose

Synthetically Trained Neural Networks for Learning Human ... · Fig. 2. System overview. As a user performs a demonstration in the real world, the perception networks detect the pose

Documents
Discovering Active Flow Control strategies using Neural ... · Artificial Neural Networks (ANNs) trained through Deep Reinforcement Learning (DRL) have recently proven efficient

Discovering Active Flow Control strategies using Neural ... · Artificial Neural Networks (ANNs) trained through Deep Reinforcement Learning (DRL) have recently proven efficient

Documents
Bacterial foraging trained wavelet neural networks ......1 Bacterial foraging trained wavelet neural networks: Application to bankruptcy prediction in banks Project Report Institute

Bacterial foraging trained wavelet neural networks ......1 Bacterial foraging trained wavelet neural networks: Application to bankruptcy prediction in banks Project Report Institute

Documents
Analyzing the Internal Representations of Trained Neural ...jxb/PUBS/AIRTNN.pdfAnalyzing the Internal Representations of Trained Neural Networks John A. Bullinaria Centre for Speech

Analyzing the Internal Representations of Trained Neural ...jxb/PUBS/AIRTNN.pdfAnalyzing the Internal Representations of Trained Neural Networks John A. Bullinaria Centre for Speech

Documents
A Comparison of Sequence-Trained Deep Neural Networks · PDF fileDeep Neural Networks and Recurrent Neural Networks Optical Modeling For ... Théodore Bluche, Hermann Ney, Christopher

A Comparison of Sequence-Trained Deep Neural Networks · PDF fileDeep Neural Networks and Recurrent Neural Networks Optical Modeling For ... Théodore Bluche, Hermann Ney, Christopher

Documents
Interpreting Computational Neural Network QSAR Models: A ... · Neural network models also have a number of shortcomings. First, neural network models can be over-trained. Thus, it

Interpreting Computational Neural Network QSAR Models: A ... · Neural network models also have a number of shortcomings. First, neural network models can be over-trained. Thus, it

Documents
Explaining Trained Neural Networks with Semantic Web ...ceur-ws.org/Vol-2003/NeSy17_paper4.pdf · Explaining Trained Neural Networks with Semantic Web Technologies 3 for our approach,

Explaining Trained Neural Networks with Semantic Web ...ceur-ws.org/Vol-2003/NeSy17_paper4.pdf · Explaining Trained Neural Networks with Semantic Web Technologies 3 for our approach,

Documents
Rule Extraction From Trained Neural Networks

Rule Extraction From Trained Neural Networks

Documents
IPOD Neural Impulse Demonstration. Brain and Behavior Introduction

IPOD Neural Impulse Demonstration. Brain and Behavior Introduction

Documents
Implementation of Pid Trained Artificial Neural Network Controller for

Implementation of Pid Trained Artificial Neural Network Controller for

Documents
Neural Networks - University of Illinois at Chicagodasgupta/resume/publ/... · Artificial neural networks have been proposed as a ... In these roles, a neural network is trained

Neural Networks - University of Illinois at Chicagodasgupta/resume/publ/... · Artificial neural networks have been proposed as a ... In these roles, a neural network is trained

Documents
Treetop Detection using Convolution Neural Networks Trained ......2019/11/20  · 1 Treetop Detection using Convolution Neural Networks Trained through Automatically Generated Pseudo

Treetop Detection using Convolution Neural Networks Trained ......2019/11/20  · 1 Treetop Detection using Convolution Neural Networks Trained through Automatically Generated Pseudo

Documents
Neural Networks Trained on Natural Scenes Exhibit Gestalt ...mozer/Research/Selected Publications... · Neural Networks Trained on Natural Scenes Exhibit Gestalt Closure Been Kim,

Neural Networks Trained on Natural Scenes Exhibit Gestalt ...mozer/Research/Selected Publications... · Neural Networks Trained on Natural Scenes Exhibit Gestalt Closure Been Kim,

Documents
Convolutional Neural Networks Applied to High-Frequency ...repository.essex.ac.uk/21296/1/Convolutional Neural... · An artificial neural network (ANN) can be trained to classify

Convolutional Neural Networks Applied to High-Frequency ...repository.essex.ac.uk/21296/1/Convolutional Neural... · An artificial neural network (ANN) can be trained to classify

Documents
Interpreting Adversarially Trained Convolutional Neural ...Interpreting Adversarially Trained Convolutional Neural Networks shuffled images, then evaluate the classification accuracy

Interpreting Adversarially Trained Convolutional Neural ...Interpreting Adversarially Trained Convolutional Neural Networks shuffled images, then evaluate the classification accuracy

Documents
Data Mining Using a Genetic Algorithm Trained Neural Networkpeople.missouristate.edu/RandallSexton/datamining.pdf · 1 Data Mining Using a Genetic Algorithm Trained Neural Network

Data Mining Using a Genetic Algorithm Trained Neural Networkpeople.missouristate.edu/RandallSexton/datamining.pdf · 1 Data Mining Using a Genetic Algorithm Trained Neural Network

Documents
IPOD Neural Impulse Demonstration

IPOD Neural Impulse Demonstration

Documents
Query Intent Detection using Convolutional Neural Networkspeople.cs.pitt.edu/~hashemi/papers/QRUMS2016_slides.pdf · Convolutional Neural Networks Use pre-trained word vector representations

Query Intent Detection using Convolutional Neural Networkspeople.cs.pitt.edu/~hashemi/papers/QRUMS2016_slides.pdf · Convolutional Neural Networks Use pre-trained word vector representations

Documents
Neural Program Synthesis from Diverse Demonstration Videos

Neural Program Synthesis from Diverse Demonstration Videos

Documents
Rainfall Prediction by Articial Neural Networks Trained

Rainfall Prediction by Articial Neural Networks Trained

Documents
Evaluating Deep Neural Networks Trained on Clinical Images

Evaluating Deep Neural Networks Trained on Clinical Images

Documents
ImageNet Classification with Deep Convolutional Neural Networks · ImageNet Classification with Deep Convolutional Neural Networks Abstract(We! trained! a large, deep convoluti onal!

ImageNet Classification with Deep Convolutional Neural Networks · ImageNet Classification with Deep Convolutional Neural Networks Abstract(We! trained! a large, deep convoluti onal!

Documents
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks

Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks

Education
NSGA-II-trained neural network approach to the estimation ... · NSGA-II-Trained Neural Network Approach to the Estimation of Prediction Intervals of Scale Deposition Rate in Oil

NSGA-II-trained neural network approach to the estimation ... · NSGA-II-Trained Neural Network Approach to the Estimation of Prediction Intervals of Scale Deposition Rate in Oil

Documents
9. Lecture Neural Networks - uni-saarland.de · trained by neural networks. A neural network connected serially ... Hybrid Neuro-Fuzzy-systems: simple neural networks that uses "fuzzy

9. Lecture Neural Networks - uni-saarland.de · trained by neural networks. A neural network connected serially ... Hybrid Neuro-Fuzzy-systems: simple neural networks that uses "fuzzy

Documents
Convexified Convolutional Neural Networks · 2018-04-15 · than CNNs trained by backpropagation, SVMs, fully-connected neural networks, ... over a fully-connected neural network:

Convexified Convolutional Neural Networks · 2018-04-15 · than CNNs trained by backpropagation, SVMs, fully-connected neural networks, ... over a fully-connected neural network:

Documents
Airplane Vortex Encounters Identification Using Multilayer ... · neural networks. Multilayer feedforward neural networks are trained using the back-propagation learning algorithm

Airplane Vortex Encounters Identification Using Multilayer ... · neural networks. Multilayer feedforward neural networks are trained using the back-propagation learning algorithm

Documents