MOS AK

DEVICEMODELINGFORNEUROMORPHICAPPLICATIONS

L.E.Calvet,C.Bennett,D.Querlioz,CenterforNanoscienceandNanotechnology,

UniversitéParis-Sud,France

T.Krauss,U.Schwalke,TUDarmstadt,Germany

A.KloesandM.Schwarz

NanoP,TechnischeHochschuleMittelhessen,Germany

MOS AK DresdenSept.3,2017 2/29

Outline

•  Introduction•  Architecturesimulations•  DeviceModeling•  Combiningdevicemodelingandarchitectures•  Conclusions

MOS AK DresdenSept.3,2017 3/29

TheBrainvs.TheCluster3

Brain~20W~10-3m3

Cluster>105W>1m3

MOS AK DresdenSept.3,2017 4/29

ThebiologicaladvantageBRAIN

Microelectronics Biology

Ultrafastbasicdevices(

MOS AK DresdenSept.3,2017 5/29

NeuralComputing

•  Signalstravelelectro-chemically•  Neuronscomputeandcommunicate•  Synapsestransferandstoreinformation

IonChannelsarethecurrentsource

~5nm

MOS AK DresdenSept.3,2017 6/29

Neural-inspiredComputing

Neurons: Decidewhethertotransmitinformation

Synapses:Memoryelementthatcanlearn

MOS AK DresdenSept.3,2017 7/29

Electronicidealization:Neuron

WinnerTakesAll

MAXΣ+

NEURON1 NEURON2

MOS AK DresdenSept.3,2017 8/29

Typesofsynapticdevices 8

FilamentarydevicesBarbaraetalACSNano2015

FerroelectricSynapsesChanthboualaetalNatMat2012

AtomicSwitchesOhnoetalNatMat2011

PhaseChangeSynapsesTumaetalNatNano2016

SpintorqueSynapsesVincentetalIEEETransBiomedCirSys2015

FloatingGatetransistorsMeadetalIEEETrans1996

MOS AK DresdenSept.3,2017 9/29

LearningRules

SynapticLearning:δrule

? Donothing

? CHANGEBYδ

MOS AK DresdenSept.3,2017 10/29

II.Architectures

MOS AK DresdenSept.3,2017 11/29

Thecrossbararrayforneuralcomputing

Synapse

CMOSinputneurons

•  CMOSNeuronsinputdata

•  Synapsesstoretheweights

•  Kirchoff’slawprovidessummation

•  CMOSwinnertakesallneuroncircuitcalculatestheoutput

ANSWER=max(∑↑▒𝑤↓𝑖 𝑥↓𝑖 +𝑏 )

X1X2X3X4b

C1C2C3

WinnerTakesAll

MOS AK DresdenSept.3,2017 12/29

Learning

•  Computethegradientdescentofthesquareerrortogetthedeltarule:

•  SUPERVISED:TeacherprovidesadatasetXwithknowntargetsT

•  TRAINING:Compute(∑↑▒𝑤↓𝑖 𝑥↓𝑖 +𝑏 )jforeachsample𝑥↓𝑖 andeachcategorycj

X1X2X3X4b

C1C2C3

WinnerTakesAll

δ=η(w⋅x-t)

•  ‘adaptivelinearneuron’(adalinerule)

SINGLELAYERNETWORKSKNOWNAS‘PERCEPTRONS’

MOS AK DresdenSept.3,2017 13/29

Nopropagationmultilayerperceptron

WTA

X1X2X3X4b

h1 h2h3h4 h5h6

C1C2C3

β-matrix:min/maxvaluesrandomlyassignedandfixedthroughoutlearning

Uselearningrule:δ=η(wj⋅h-tj)

ANSWER=max(∑↑▒𝑤↓𝑖 ℎ↓𝑖  )

•  Inspiredby‘ExtremeLearningMachine’Algorithm•  Avoidsbackpropagationcircuitryattheexpenseofhavingalargernumberofhiddenlayers

β

w

MOS AK DresdenSept.3,2017 14/29

IrisClassification

•  150samplesfrom3speciesofIris•  4Features:Lengthandwidthofthesepalsandpetals(incm)

Classificationresultsusingsoftwareandideallearning:•  Perceptron:97%

MOS AK DresdenSept.3,2017 15/29

IrisClassification

Classificationresultsusingsoftwareandideallearning:•  Perceptron:97%

BACKPROP

•  Back-prop:98%at4hiddennodes

MOS AK DresdenSept.3,2017 16/29

IrisClassification

Classificationresultsusingsoftwareandideallearning:•  Perceptron:97%

η=0.001

OnlineELMAverage:92.4%

BACKPROP

98%

•  Back-prop:98%at4hiddenlayers•  OnlineELM:92.4%average

MOS AK DresdenSept.3,2017 17/29

TransferingToHardware

Softwarelearningrule:δ=η(wj⋅h-tj)η  =0.02

Hardwarelearningrule:δ  =ΔG*sign(wj⋅h-tj)levels=200

AVERAGE:86.8% AVERAGE:83.0%

MOS AK DresdenSept.3,2017 18/29

TransferingToHardware

Softwarelearningrule:δ=η(wj⋅h-tj)η  =0.02

Hardwarelearningrule:δ  =ΔG*sign(wj⋅h-tj)levels=200

AVERAGE:86.8% AVERAGE:83.0%

MOS AK DresdenSept.3,2017 19/29

III.DeviceModeling

MOS AK DresdenSept.3,2017 20/29

SONOSTransistorasasynapse

G

S D

FloatingGate

•  SONOSallowssuperiornumberofread/writecyclesthanconventionalfloatinggateMOSFETs

•  Engineerstructuretooptimizethetiming

MOS AK DresdenSept.3,2017 21/29

ModelingusingTCADSynopsys

•  ChargeinjectedintothegateviaFowlerNordheimtunneling

•  Finddevicecharacteristicsanddeviceoperationalrange

•  Determineconductancerangeofsynapses,compactmodelofdevicebehavior

DEVICEPARAMETERS:Tunneloxide:18ÅSi3N4layer: 80Å Topoxide: 40ÅLg: 130 nm

MOS AK DresdenSept.3,2017 22/29

DeviceCharacteristics

•  Togeneratetwocurves:300pulses+6V(+charge)300pulses-3.75V(-charge)•  ChooseVgrange

•  Find:GmaxGmin,#levels

Gmax

Gmin

MOS AK DresdenSept.3,2017 23/29

SimulationofPulses

Ateachpulse±6V,anIdvsVgistaken.ThisgraphplotsthefractionchangeoftheevolutionofVg=0.75V

MOS AK DresdenSept.3,2017 24/29

IV.CombiningDeviceandArchitectureSimulations

MOS AK DresdenSept.3,2017 25/29

Overiewofmethodology

DeviceSimulationstoobtainGmax,Gmin,ΔG,

Architecturesimulationstoobtainpulsesequences

DeviceSimulationstoobtainweight

matrices

Architecturesimulationswith

weightsfromdevicesimulations

Compactmodel

Pulsesequencessimulatedweights

MOS AK DresdenSept.3,2017 26/29

LearningwithaSONOStransistor(linear)

UseLinearRegime

Takeweights=Vt

WTA

X1X2X3X4b

h1 h2h3h4 h5h6

C1C2C3

β

w

InputsappliedtoVds

Learning:•  EachpulsechangesbyΔVt.•  ΔG=ΔVt.

𝐼↓𝑑 = 𝜇𝐶↓𝑜𝑥 𝑤/𝐿 (𝑉↓𝑔𝑠 − 𝑉↓𝑡 − 𝑉↓𝑑𝑆 /2 )𝑉↓𝑑 

linear

Highpowerconsumption

MOS AK DresdenSept.3,2017 27/29

LearningwithaSONOStransistor(exponential)

𝐼↓𝑑 = 𝐼↓𝑜 𝑒𝑥𝑝[𝛼[𝑉↓𝑔 − 𝑉↓𝑡 ]]Subthresholdcurrent:

Takew=Vt

WTA

X1X2X3X4b

h1 h2h3h4 h5h6

C1C2C3

β

w

InputsappliedtoVg

Learning:•  EachpulsechangesbyΔVt.•  ΔG=𝑒𝑥𝑝[−𝛼∆𝑉↓𝑡 ]

Exponentialweights

Lowpowerconsumption

MOS AK DresdenSept.3,2017 28/29

FirstResults

DeviceSimulationstoobtainGmax,Gmin,ΔG,

Architecturesimulationstoobtainpulsesequences

DeviceSimulationstoobtainweight

matrices

Architecturesimulationswith

weightsfromdevicesimulations

MOS AK DresdenSept.3,2017 29/29

Conclusions

•  Devicemodelingofsynapsesinneuromorphicarchitecturescanoptimizeclassification

•  Floatinggatedevicesbasedonalayeroftraps/defectsareidealforsuchsimulationsbecausewidevariationsinthetrappingpropertiesofdifferentmaterials

MOS AK DresdenSept.3,2017 30/29

Thankyouforyourattention!