OptoNeuro

Optogenetic visual cortical prosthesis

Dr. Patrick DegenaarReader in Neuroprosthesis

Conflicts of interest: I have filed a number of patents and CANDO may have commercial impact. But currently no commercial interests.

Neuroprosthetics @ Newcastle

Stimulate

Process

Record

http://www.optoneuro.eu

Epilepsy brain prosthesis

Visual prosthesis

Spinal cord injury

Hand/Leg bionics

Multi-EMG recording

Electrodiagnostics

http://www.cando.ac.uk

Closed loop neural

interfaces!!

Blue light is used to activate neurons expressing

channelrhodopsin. Other opsins can silence activityOptogenetic control of behaviour.

Gradinaru et al. J Neurosci (2007)

The optogenetics revolution

High radiance optoelectronics for optogenetics:Grossman N et al. IEEE TBME 2011, 58(6), 1742-1751.McGovern B et al IEEE TBCAS 2010, 4(6, part 2), 469-476.Grossman N et al, J. Neural Engineering 2010, 7(1), 016004.…

Optoelectronics for optogenetics

CMOS

µlens optics

Gallium

Nitride µLED

3D bump bond

CMOS driver

GaN MQW

p-contact

Solder bond

2010 2011 20132009 2012 2014

64x64, 64x1

passive µLED arrays16 x 16 MPW CMOS

chip driven µLED arrays90x90 Retina

wafer

2015 2016 2017 2018

CANDO 1

Optrode

CANDO 3

Wafer

Temporal

Primary visual

cortex

Optics (cornea/lens)

Sense/code (retina)

Transmission (Optic Nerve)

Sorting(Thalamus)

Interpretation(Visual Cortex)

Pulvinar nucleus

Lateral Geniculate

Nucleus

Superior

Colliculus

Optical

radiation

5

The visual system

6

PrevalenceAccording to WHO, about

180 million people worldwide

have a visual impairment

Legal blindness:

“medically diagnosed central

visual acuity of 20/200 or less

in the better eye with the best

possible correction, and/or a

visual field of 20 degrees or

less”

Cataracts

(48%)

Others

(16%)

AMD

(9%)

Corneal opacities (5%)

Diabetic retinopathy (5%)

Childhood blindness (4%)

Glaucoma

(12%)

Trauma

(?%)

Drugs and gene therapy becoming increasingly

useful.. But cannot restore lost vision

Primary visual

prosthesis target

conditions

Visual loss: statistics

From: WHO

RP (1%)

Optics (cornea/lens)

Sense/code (retina)

Transmission (Optic Nerve)

Sorting(Thalamus)

Interpretation(Visual Cortex)

Retinitis Pigmentosa

(<1% of blindness)

Glaucoma

Trauma

Other severe blindness

(~30% of blindness)

Sub/epi retinal

prosthesis

Optic nerve

prosthesis

LGN

prosthesis

Cortical

prosthesis

For invasive procedures only!

Very expensive!

Will be enormously difficult!

But can treat much larger cohort

Disorders of the visual system

Optics (cornea/lens)

Sense/code (retina)

Transmission (Optic Nerve)

Sorting(Thalamus)

Interpretation(Visual Cortex)

Implementing a visual prosthesis

Wireless

power/data

Visual

processing

Image

acquisition

Stimulator

Easy??

Brindley and Lewin did it in

1968!!!• Brindley,G.S.,Lewin,W.S.,1968.The visual sensations produced by electrical stimulation of the medial

occipital cortex. J Physiol.19454-5P.

• Brindley,G.S.,Lewin,W.S.,1968.The sensations produced by electrical stimulation of the visual

cortex.J.Physiol.196,479–493.

Challenge 1: Sheer scale of challenge

1960’s: You could build a device on a Monday

and put it in a patient on a Friday

Now: A project to create an active implantable

device requires £10-50M and 5-10 years

worth of regulatory effort to demonstrate

safety before implantation is allowed

As such, almost the entire field moved to retinal

prosthesis in the 1990’s as it was a much easier target!!

5 x 5 (25) pixels10 x 10 (100)33 x 33 (1000)70 x 70 (5000)100 x 100 (10,000)Normal vision

Asian/pictographic

scripts:

European/phonetic

scripts:

~ 1000 pixels

~ 5000 pixels

~ 100 pixels

European/phonetic

characters:

11

Challenge 2: Visual acuity

From Retina AG

1500 electrodes

– but only a few

dozen can be on

at any moment in

time.

Lessons from retinal prosthetics

Original image

Assume greyscale retina

@ 64 x 64 resolution

Add noisy retina

effect

100:1 contrast ratio 50:1 contrast ratio

With insufficient

stimulus contrast it

is like whispering

through a

typhoon!

13

Challenge 3: Effective contrast

1. Image capture

Rods/cones

2. smoothing

Horizontal cells

3. DerivativeBipolar cells

Amacrine cells

Ganglion cells

5. Spike coding

OFF ON

14

The retina

Ligtht

4. Temporal edge

The retina has 60 different cell types. If we are bypassing the natural processing, we need to replicate the key features!

Challenge 4: The eye is not a camera!

Annu. Rev. Biomed. Eng.

10:275–309, 2008

IEEE Transactions on Neural Systems and

Rehabilitation , Vol. 12, No. 2, June 2004

Electrode

insertion

point

Accelerated ageing, 1000

cycles at 1.7V/s

Journal of The Electrochemical

Society, 152 (7) J85-J92 ~2005!

Glial scarring

Ideal Real

Electrode

degradation and

adverse tissue

reactions are a major

issue for smaller

stimulating

electrodes because

of high charge

density requirements

15

Challenge 5: Biocompatibility

Where to start?

0.18 mm

0.18 mm

0.46 mm

0.41 mm

0.31 mm

0.68 mm

2.2 mm

From: Atlas of Cytoarchitectonics

- Von Economo

Developing the stimulator

Surface stimulation

(Brindley, Dobelle)

Penetrating stimulation

Can we make the communication bi-directional?

Can we treat communication as a control problem?

`

0.18 mm

0.18 mm

0.46 mm

0.41 mm

0.31 mm

0.68 mm

2.2 mm

From: Atlas of Cytoarchitectonics

- Von Economo

Penetrating stimulation

Can we make the communication bi-directional?

Can we treat communication as a control problem?

Light stimulus

Electrical recording

Optical

Electricall

Optogenetics & closed loop control

19

£10M project: Clinical trials aimed for 2021

Controlling Abnormal Network Dynamics with Optogenetics

Noise cancellation of epileptic

seizures – but adaptable to

vision

Connecting

lead

Central control unit

inc wireless comms

Brain

implant unitBrain implant unit (Actually

different position)

Control unit

External

headset

Principles:

1. Custom brain implant starting as 4 x

4 x 4 LEDs can be utilised

2. The control unit is quite different –

higher power, no battery

3. Cabling can be simpler

CANDO - Epilepsy Visual brain prosthesis

Skin Fat Muscle

Average 5 mm

2mm 1mm 4mm

TX RX

Adapting to visual brain prosthetics

Brain stimulator unit

Interface unit

Mounting

plate

Optrode

Active electronics

Communications

Recording Circuits

Stimulation Circuits

Digital control unit (FSM)

Diagnostic Circuits

Power

Recording Electrode

Photonic Stimulation

site

Optrodeshaft

Optrode head

1. Ramezani et al. submitted to IEEE TCAS-I, June 2017

2. Zhao et al. Implantable optrode GB1410886.4

3. Dekhoda et al. Temperature sensor GB1522790.3

4. Degenaar Optical stimulation arrangement GB1616725.6

5. Constandinou et al On Chip Random ID generation GB1702623.8

Vin

Vref 50C LNA

C

VIP VIN

VONVOP

C

6C Gm

C=100fF

VINVIPVOUT

Front end amplifier Programmable gain amplifier

Recordings from non-human primate brain

Recording microelectronics

LED driver circutry

DAC

LED drive power LED efficiency

• Amplitude modulation

• PWM modulation

• Diagnostic and methods

0 200 400 600 800 1000 0 200 400 600 800 1000 0 200 400 600 800 1000

0 200 400 600 800 1000

1

10

100

0.1

0.01

0.001

1

10

100

0.1

0.01

0.001

0.0001

0.00001

Tissue penetration depth (μm)

Op

tica

l P

ow

er

(mW

)

Threshold = 1mW/mm2 Threshold = 1mW/mm2 Threshold = 1mW/mm2

40μm60μm80μm100μm120μm

Op

tica

l P

ow

er

(mW

)

LED diameter

0 200 400 600 800 1000 0 200 400 600 800 1000

Tissue penetration depth (μm)

Threshold = 1mW/mm2 Threshold = 0.1mW/mm2 Threshold = 0.01mW/mm2

Lambertian emitter Shaped emitter Pseudo-collimated emitter

Pseudo-collimatedShapedLambertian

LED type

(d) (e) (f)

(g) (h) (i)

(a)

(b)

(c)

Dong et al. For submission to J. Biomedical Photonics 2017

How much light do we need?

Key take home message:

1. Size doesn’t matter to penetration depth!

2. Need to penetrate at least 100-200µm for long term implantables due to gliosis

3. More collimated light will give better penetration

Maximum allowed duty ratio (%)Maximum allowed pulse width

(e) (f)

1ms 10ms100ms 1s 10s 1ms 10ms100ms 1s 10s

1ms 10ms100ms 1s 10s 1ms 10ms100ms 1s 10s

ΔT (°C)

(a)

ΔT (°C)

ΔT (°C) ΔT (°C)

Optical stimulus Optical stimulus

Optical stimulus Optical stimulus

(b)

(c) (d)

1 LED on

1 LED on

8 LEDs on

8 LEDs on

2.5mW

5mW

10mW

10mW5mW2.5mW

10mW

5mW

2.5mW 2.5mW

5mW

10mW

0.01

0.1

1

10

0.01

0.1

1

10

0

3

0

10

2

4

6

8

1

2

10s

1s

0.1s

10ms

1ms

Thermal power (mW)

0 10 20 30 40 50 60

Thermal power (mW)

0 10 20 30 40 50 60

20

40

60

80

100

1 LED on

8 LEDs on 8 LEDs on

1 LED on

0.000

0.1000

0.2000

0.3000

0.4000

0.5000

0.6000

0.7000

0.8000

0.9000

1.000

S

0 . 0 0 0

0 . 1 0 0 0

0 . 2 0 0 0

0 . 3 0 0 0

0 . 4 0 0 0

0 . 5 0 0 0

0 . 6 0 0 0

0 . 7 0 0 0

0 . 8 0 0 0

0 . 9 0 0 0

1 . 0 0 0

0 . 0 0 0

0 . 1 0 0 0

0 . 2 0 0 0

0 . 3 0 0 0

0 . 4 0 0 0

0 . 5 0 0 0

0 . 6 0 0 0

0 . 7 0 0 0

0 . 8 0 0 0

0 . 9 0 0 0

1 . 0 0 0

0 . 0 0 0

0 . 1 0 0 0

0 . 2 0 0 0

0 . 3 0 0 0

0 . 4 0 0 0

0 . 5 0 0 0

0 . 6 0 0 0

0 . 7 0 0 0

0 . 8 0 0 0

0 . 9 0 0 0

1 . 0 0 0

EL

P

P

S

L

EL

1.00.80.60.40.20

ΔT (K)

2.01.51.00.50

2.01.51.00.50

ΔT (K)

ΔT (K)

No coating 20um polymer coating

25um in the tissue100um in the tissue

On-optrode On-optrode

S

P

ΔT(

K)

1000

20

0

200

Optrode profile

100μm

100μm

100μm

10

00μ

m

(a)

(b)

(c)

(d) I

(e) II

(f) III

0.5

0

1.0

1.5

2.0

0 1.0 2.0 3.0

0

0.5

1.0

1.5

2.0

ΔT(K)

Dimension (μm)

Dim

ensi

on

(μ

m)

ΔT(

K)

Dimension (μm)

L: LED E: Electrode P: Polymer coating S: Silicon substrate

III

II

I

Target LED characteristics:

Driving current: 1mA

Driving voltage: 4-5V

Light requirement: 1mW

Efficiency: >20%

Typical micro-LED efficiencies from

literature ~ 1-5%

Wu et al. Neuron 2015: 0.8%

Kim et al. Science 2013: 0.2%

McAlinden al: Opt Lett 2013: 5%

McGovern et al: PLOS ONE 2013: 5%

McGovern et al: IEEE TBCAS 2010:1%

Dong et al. For submission to J. Biomedical Photonics 2017

LED thermal impact

Dong et al. For submission to J. Biomedical Photonics 2017

Wu et al Neuron 2015

10µm30µmOur LED

320 x 240!100µm @ 1mA

LED efficiencies

“Official” threshold is

0.7mW/mm2 for dissociated

culture. But what about in

practice???

In NHP experiments, We saw

responses at 1mm with emission

intensities of only 0.5mW. i.e. At

that depth the irradiance <<

0.1mW/mm2

Caveat: we cannot separate

neural transmission!

Modelling indicates a lower

effective threshold of 0.1 - 0.3

mW/mm2

Luo et al. Submitted to JNE 2017Currently Unpublished work

Light requirement in-vivo

b)

Schematic design

3D assembly

CAD layout Wafer fabrication

Probe assembly

Light-

emitting

diodes

Recording

sites

Control

electronics

We are developing a fully manufacturable implant!!

Assembling the optrodes

Camera

RX

Embedded

SoC

BLE

module

Power

module

Power

trans/Amp

BLE

module

Embedded

SoC

Power RX/

AC->DC

Voltage

regulator

External control unit Internal control unit

Power and

Control unit

Optrodes

Optrode unit

To receiver

antennae

Internal

control unit

Ontogenetically

Sensitized tissue

Optrode

Implant

Base plate

Receiver

antennae Transmitter

antennae

Batt and cam

3.7V

USB

TX

Fattah & Soltan et al. in preparation for IEEE TBME submission

Developing the control system

System software - preprocessing

Song of the Machine!

Simplification Cartoonization

Visual augmentation

The vision that we willreturn to patients withvisual prosthesis will berudimentary. We aretherefore using patientswith partially degeneratevision as a simulation ofwhat can be achieved.

Camera

External control unit Internal control unit

Power and

Control unit

To receiver

antennae

Internal

control unit

Ontogenetically

Sensitized tissue

Optrode

Implant

Base plate

Receiver

antennae Transmitter

antennae

Fattah & Soltan et al. in preparation for IEEE TBME submission

Full software process

Compression

Decompression Pulse coding

Video/Image

acquisition

Simplification Retinal processing

Transmission

LED reconstruction

Visual cortical Headset

Implantable control unit

0

10

20

30

40

50

60

70

80

60 40 20

MC

Pe

rfro

man

ce (

fps)

Frequency (MHz)

44% 61% 75% 86%

20

30

40

50

60

70

80

60 40 20

Po

we

r (m

W)

Frequency (MHz)

We are here

CANDO First

in human trial

Current

funding ends

(Aug 2021)

CANDO Engineering

Visual cortical trial??

CANDO trial

CANDO

Regulatory &

ethical

submission

Timeline

OptoNeuro

Acknowledgments

CANDO project

Tim Constandinou (Imperial College), Nick Donaldson

(UCL), Andrew Sims (Newcastle RVI (NHS)), Andrew

Jackson (Newcastle IoN), Mark Cunningham

(Newcastle IoN), Anthony O’Neil (Newcastle EEE),

+ Others:

OptoNeuro FP7 project

Mark Neil (Imperial College), Ernst Bamberg (Max

Planck Institute), Christian Bamann (Max Planck

Institute), Botond Roska (FMI, Switzerland), Pleun

Maaskant (Tyndall Institute), David Rogerson

(Scientifica), Mark Johnson (Scientifica)