ADVANCING MOTOR NEURAL PROSTHESIS ROBUSTNESS AND NEUROSCIENCEshenoy/Theses/Stavisky.pdf · 2018-05-26 · The frontier challenges that must be solved before brain-machine interfaces

ADVANCING MOTOR NEURAL PROSTHESIS

ROBUSTNESS AND NEUROSCIENCE

A DISSERTATION

SUBMITTED TO THE NEUROSCIENCES PROGRAM

AND THE COMMITTEE ON GRADUATE STUDIES

OF STANFORD UNIVERSITY

IN PARTIAL FULFILLMENT OF THE REQUIREMENTS

FOR THE DEGREE OF

DOCTOR OF PHILOSOPHY

Sergey D. Stavisky

March 2016

http://creativecommons.org/licenses/by-nc/3.0/us/

This dissertation is online at: http://purl.stanford.edu/hq855qg2603

© 2016 by Sergey Stavisky. All Rights Reserved.

Re-distributed by Stanford University under license with the author.

This work is licensed under a Creative Commons Attribution-Noncommercial 3.0 United States License.

ii



http://purl.stanford.edu/hq855qg2603

I certify that I have read this dissertation and that, in my opinion, it is fully adequatein scope and quality as a dissertation for the degree of Doctor of Philosophy.

Krishna Shenoy, Primary Adviser


William Newsome


Anthony Norcia


Jennifer Raymond

Approved for the Stanford University Committee on Graduate Studies.

Patricia J. Gumport, Vice Provost for Graduate Education

This signature page was generated electronically upon submission of this dissertation in electronic format. An original signed hard copy of the signature page is on file inUniversity Archives.

iii

Abstract

The frontier challenges that must be solved before brain-machine interfaces (BMIs) can be used

as clinically useful motor prostheses differ depending on the degree of function being restored.

Two-dimensional cursor control (i.e., for communication) has recently reached high levels of peak

performance in pre-clinical studies (e.g. Gilja et al., 2015), but translation is hampered by less than

reliable performance due to unstable neural signals (despite substantial inroads, e.g., Jarosiewicz

et al. 2015). Meanwhile, control of robotic arms remains poor, despite some impressive glimpses at

what the future could be (e.g. Collinger et al., 2013b), because we lack fundamental understanding

of how the brain incorporates the BMI into its motor schema. This hampers our ability to accurately

decode intended arm movements. My dissertation focused on both sets of problems in pre-clinical

macaque BMI studies. Chapters 2 and 3 provide neural engineering solutions for improving BMI

robustness. Chapter 4 looks ahead to challenges that will be encountered when BMI-controlled limbs

operate in the physical world by describing how movement error signals impact ongoing BMI control.

Chapter 2 describes a machine learning approach to building decoder algorithms that are robust

to the changing neural-to-kinematic mappings that plague translational BMI efforts. We developed

a multiplicative recurrent neural network decoder that could exploit the large quantities of data

generated by a chronic BMI — data that has heretofore gone unused. This novel decoder was

markedly more robust to both artificial and naturally occurring recording condition changes than an

existing state-of-the-art decoder. This robustness benefit came without a performance tradeoff under

more ideal conditions.

Chapter 3 describes a neural engineering approach for increasing the device lifespan of a BMI

by providing high performance control even after losing spike signals. I developed a method for

decoding local field potentials (LFPs) as a longer-lasting alternative or complimentary BMI control

signal. This led to the highest-performing LFP-driven BMI and the first ’hybrid’ BMI which decoded

kinematics from spikes and LFPs together. This study also demonstrated that a decoding method’s

control loop latency can result in dramatic differences between offline and closed-loop performance,

and that this should serve as a cautionary lesson to guide future BMI design.

Chapter 4 focuses on one project that exemplifies my efforts to understand the neuroscience

of BMI use by studying it as a feedback control system. BMIs used in real-world settings will

iv

encounter unexpected perturbations due to, for example, collisions with other objects. Fortunately,

BMI use, like natural movement, is a closed-loop process in which sensory feedback conveying

errors is transformed into motor corrections. But this presents a potential problem in areas such as

motor cortex, which both receive error signals and generate efferent motor commands: error-related

modulation needs to not ”leak out” to the effector unless it would generate the appropriate corrective

movement. While this is the case for natural movement, in which motor cortex indirectly affects

muscles, what happens when motor cortical activity directly drives the effector via a BMI? I tested

this by perturbing the kinematics of monkeys performing a BMI-controlled cursor task. Visual

feedback drove cortical responses as early as 70 ms after the perturbation, but this initial modulation

did not cause deleterious motor feedback. Rather, it was limited to a neural null space in which

activity does not affect the BMI. When neural activity changed in output-potent dimensions starting

115 ms after perturbation, it caused corrective BMI movement. This elegant arrangement may hint

at a broader computational strategy by which error processing is separated from output.

v

Acknowledgments

My graduate training, which has culminated in completing this dissertation, owes its success to many

people. First I would like to thank my advisor, Professor Krishna Shenoy. Krishna is an amazing

person and mentor. In addition to being a brilliant and creative scientist, he is also an extraordinarily

kind and caring mentor who has time after time put his people first, despite everything he’s had to

go through. He’s taught me to be the best scientist I can be, and to do so with integrity, purpose,

courage, and humility. The success of his group, as measured by its scientific contributions and the

subsequent achievements of those who have come through it, are a testament to the amazing research

and teamwork culture that Krishna has cultivated. I am incredibly fortunate, and grateful, to have

had this opportunity to learn from him.

My research also owes a great deal to the prior work, help, and advice of the Shenoy group, past

and present: Jonathan Kao, Dr. Paul Nuyujukian, Dan O’Shea, Dr. Cora Ames, Eric Trautmann, Dr.

Joline Fan, Dr. Justin Foster, Dr. Werapong Goo, Megan Wang, Nir Even-Chen, Xulu Sun, Jessie

Verhein, John Aguayo, Clare Sherman, Erica Morgan, Liana Yates, Dr. Stephen Ryu, Mackenzie

Mazariegos, Michelle Wechsler, Steven Lin, Shannon Smith, Rosie Steinbach, Beverly Davis, Dr.

Boris Oskotsky, Dr. Chandramouli Chandrasekaran, Dr. David Sussillo, Dr. Mark Churchland, Dr.

Matthew Kaufman, Dr. Vikash Gilja, Dr. Cindy Chestek, Dr. Ilka Siester, Dr. Chethan Pandarinath,

and Dr. Paul Kalanithi, as well as our close collaborator, Professor Maneesh Sahani, and his group.

You’re a group of the smartest people I’ve ever been around, and I wouldn’t have learned half as

much as I did without you all. You’re also good, generous, kind, funny and warm people and I

treasure our scientific family. A few of these names deserve additional mention. Paul N. was the

senior student on the prosthetics team when I joined the group, and helped train me to work with our

macaques and understand the experimental rig’s software. Jonathan was my teammate throughout

our graduate experience, and I couldn’t ask for a more brilliant, humble, and helpful colleague. Matt

stepped up as the acting laboratory leader during Krishna’s prolonged health-related absence, and

provided me with crucial mentoring and advice during my second year of graduate school, which

lived up to its notoriety of existential difficulty. Mackenzie is an unmatched laboratory manager and

monkey czar, and I’m extraordinarily grateful for her above-and-beyond care of our research animals.

Knowing she could be called upon to expertly deal with any emergency gave me peace of mind. I

vi

also want to explicitly acknowledge and thank Jonathan, Paul N., David, Stephen, and Krishna, who

were my collaborators on the projects described here and co-authors on the resulting journal papers.

My thesis committee — Professors Bill Newsome, Jennifer Raymond, and Tony Norcia — provided

highly helpful scientific and career advice throughout my graduate training. Thank you for being

my sounding-board and cheerleaders. I’m also grateful to Professor Tom Clandinin for chairing my

defense, and to Professors Steve Baccus and Tony Norcia for mentoring me during my first-year

rotations in their laboratories.

Outside the lab, my science has benefited from the excellent neurosciences community at Stanford.

I’m thankful to Yvette Fisher, Emily Ferenczi, Diogo Peixoto, Hannah Payne, Vince McGinty, David

Kastner, Georgia Panagiotakos, Niru Maheswaranathan, Lane McIntosh, Amy Braun, Becca Krock,

Kiah Hardcastle, Jordan Sorokin, for their feedback, which helped me become a better student and

scientist. My training experience in Stanford’s Neurosciences graduate program has been excellent

not only due to the superb faculty and students, but also because the program is very well run:

its administration is both efficient and shows personalized concerned for each individual’s success.

This excellence is due to the hard work of the program directors (Professors Tony Ricci and John

Huguenard) and staff (Kalai Diamond, Ross Colvin, Katie Johnson, and Sheila Quimpo).

Stanford has been an exhilarating intellectual and professional journey, but the past five and a half

years have also been immensely interesting and joyous outside of the lab. For this, I must thank my

new friends from within my Program and from varied and serendipitous meetings outside it. Thank

you all for making the Bay Area my home. It is a rare thing in academia to be able to stay in the

same area after finishing one’s Ph.D, and I feel extraordinarily fortunate to be able to stay close to

you in this most beautiful of lands.

At the same time, I’ve also been fortunate to keep in close touch with my friends from childhood

and University, most of whom are scattered across the East Coast. It’s been a joy to host you in San

Francisco and to visit you through frequent trips to New York, Boston, and D.C. Though a part of

me is sad not to return east for my postdoc, I look forward to the numerous upcoming weddings

(and occasional vacations) as opportunities to see you all, as I miss you dearly.

Lastly, I thank my family for giving me such a loving foundation from which to strive for the

Ph.D. and a career in science. I am especially grateful to my parents, Natalia and Dmitry Stavisky,

for sacrificing their own scientific careers to emigrate from the Soviet Union and build a new and

better life for us in the United States. It’s humbling to think that in the time it’s taken me to finish

my degree, you’d already built a second career after having finished your scientific education, while

also raising me.

Sergey D. Stavisky

Stanford University, California, USA, March 2016

vii

Contents

Abstract iv

Acknowledgments vi

1 Introduction 1

1.1 The premise and promise of motor neural prostheses . . . . . . . . . . . . . . . . . . 1

1.2 Key challenges to clinically viable motor neural prostheses . . . . . . . . . . . . . . . 3

1.3 Dissertation overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.4 Animal model and general experimental setup . . . . . . . . . . . . . . . . . . . . . . 5

2 Harnessing months of previously collected data to improve brain-machine interface

robustness to future neural variability 8

2.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.3.1 An MRNN can leverage large amounts of training data to improve decoder

performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.3.2 Robustness to unexpected loss of the most informative electrodes . . . . . . . 15

2.3.3 Robustness to naturally occurring recording condition changes . . . . . . . . 16

2.3.4 High performance BMI using the MRNN decoder . . . . . . . . . . . . . . . . 18

2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.4.1 A robust and high-performing BMI decoder . . . . . . . . . . . . . . . . . . . 23

2.4.2 Key innovations enabling the MRNN’s robustness . . . . . . . . . . . . . . . 24

2.5 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.5.1 Animal model and neural recordings . . . . . . . . . . . . . . . . . . . . . . . 25

2.5.2 Behavioral tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.5.3 Multiplicative Recurrent Neural Network (MRNN) decoder . . . . . . . . . . 27

2.5.4 Training the MRNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.5.5 MRNN training data augmentation . . . . . . . . . . . . . . . . . . . . . . . . 29

viii

2.5.6 Controlling the BMI cursor with MRNN network output . . . . . . . . . . . . 30

2.5.7 Feedback Intention Trained Kalman Filter (FIT-KF) decoder . . . . . . . . . 30

2.5.8 Mutual information for determining electrode dropping order . . . . . . . . . 31

2.5.9 Principal angles of neural subspaces analysis . . . . . . . . . . . . . . . . . . 32

2.6 Project acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

3 A high performing brain-machine interface driven by low-frequency local field

potentials alone and together with spikes 34

3.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

3.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

3.3.1 Selection of local motor potential as a control signal based on offline evaluation

of candidate LFP features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

3.3.2 Control experiment showing LMP is not a movement artifact . . . . . . . . . 38

3.3.3 Closed-loop BMI performance driven by LMP . . . . . . . . . . . . . . . . . . 42

3.3.4 Fixed LMP decoders work across multiple days without retraining . . . . . . 46

3.3.5 BMI control using a hybrid spikes and LMP decoder . . . . . . . . . . . . . . 46

3.3.6 LMP decoding has a longer control loop latency than spikes decoding . . . . 48

3.3.7 Hybrid decoding can rescue performance when spikes are lost . . . . . . . . . 51

3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

3.4.1 Choosing LMP for BMI use based on offline decode performance . . . . . . . 55

3.4.2 Closed-loop LMP-driven BMI performance . . . . . . . . . . . . . . . . . . . 57

3.4.3 Hybrid decoding using both LMP and spikes . . . . . . . . . . . . . . . . . . 59

3.4.4 Decoding LMP to mitigate losing spikes signals . . . . . . . . . . . . . . . . . 60

3.4.5 Differences between offline and closed-loop results . . . . . . . . . . . . . . . 61

3.5 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

3.5.1 Subjects and neural recording . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

3.5.2 Behavioral tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

3.5.3 Signal processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

3.5.4 Neural decoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

3.5.5 Performance measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

3.5.6 Electrode preferred direction and decoder contribution . . . . . . . . . . . . . 70

3.5.7 Comparing the contribution of LMP and spikes in the hybrid decoder . . . . 71

3.6 Project Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

4 Effect of Sensory Error Signals on BMI Use 73

4.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

4.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

ix

4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

4.3.1 Perturbations evoked an early motor cortical response

without accompanying cursor velocity output . . . . . . . . . . . . . . . . . . 77

4.3.2 Initial error-related neural activity was confined to a decoder-null space . . . 77

4.3.3 Perturbation-specific information in the early error-related response . . . . . 84

4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

4.4.1 BMI error correction latency . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

4.4.2 Early BMI error-related signals are present in motor cortex but do not interfere

with ongoing BMI control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

4.4.3 Computing in an output-null subspace provides a mechanism by which BMI

output can ignore early error signals . . . . . . . . . . . . . . . . . . . . . . . 87

4.4.4 Differences between responses to different jump location conditions . . . . . . 88

4.4.5 Implications for future BMI design . . . . . . . . . . . . . . . . . . . . . . . . 89

4.5 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

4.5.1 Cursor jump task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

4.5.2 Neural recording and decoding . . . . . . . . . . . . . . . . . . . . . . . . . . 91

4.5.3 Analysis of responses to cursor jumps . . . . . . . . . . . . . . . . . . . . . . 92

4.5.4 Neural response from the decoder’s perspective . . . . . . . . . . . . . . . . . 93

4.5.5 Electrode neural push direction and decoder contribution weight . . . . . . . 94

4.6 Project Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

5 Conclusion 96

5.1 Summary of motor neural prosthesis advances . . . . . . . . . . . . . . . . . . . . . . 96

5.2 Next steps: testing in a human clinical trial . . . . . . . . . . . . . . . . . . . . . . . 97

5.3 Additional graduate work not covered in this dissertation . . . . . . . . . . . . . . . 98

5.3.1 Drawing scientific conclusions about neural population activity without sorting

single neuron activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

5.3.2 Observed BMI cursor position only weakly affects motor cortical activity . . 100

5.3.3 Single trial motor cortical correlates of a rapidly adapting visuomotor internal

model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

5.3.4 Rapid but limited adaptation to a change in BMI velocity gain . . . . . . . . 101

5.3.5 Do motor cortical dynamical rules constrain the activity that can be generated

for BMI control? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

x

List of Tables

2.1 Network and training parameters used for the closed-loop MRNN BMI decoder . . . 28

xi

List of Figures

1.1 Schematic of a motor neural prosthesis . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Rhesus macaque motor cortical regions recorded from . . . . . . . . . . . . . 6

2.1 Our strategy for training a decoder robust to recording condition changes 10

2.2 An MRNN decoder can harness large training datasets . . . . . . . . . . . . 12

2.3 Similar neural recording conditions can often be found on other days . . . 14

2.4 Robustness to unexpected loss of the most important electrodes . . . . . . 16

2.5 Artificially dropped electrodes were active in the training data . . . . . . . 17

2.6 Robustness to naturally occurring recording condition changes . . . . . . . 18

2.7 Training data spike rate perturbations improve MRNN performance . . . 19

2.8 Additional tests showing that FIT Old typically performs poorly . . . . . . 20

2.9 The MRNN outperforms the FIT Kalman filter under “ideal” conditions 21

2.10 Closed-loop MRNN decoder performance on the Random Target Task . . 22

3.1 Overview of local motor potential (LMP) BMI signal processing . . . . . . 37

3.2 Comparison of offline decode accuracy using different neural features . . . 39

3.3 Offline decode analysis of additional candidate LFP features . . . . . . . . . 40

3.4 Directional tuning of LMP overlaid on electrode location in motor cortex 41

3.5 LMP-driven BMI performance on the Continuous Random Target Task . 43

3.6 Performance of the LMP decoder on a Radial 8 Task . . . . . . . . . . . . . 44

3.7 Performance on a Continuous Random Target Task with short hold time 45

3.8 Stable performance using a fixed LMP decoder . . . . . . . . . . . . . . . . . 47

3.9 Closed-loop comparisons of different neural control signals . . . . . . . . . . 49

3.10 LMP underperforms online due to longer control loop latency . . . . . . . 50

3.11 Performance of spikes-only and hybrid decoders after electrode dropping 52

3.12 Cumulative electrode contribution to the spikes-only and LMP-only decoders 54

3.13 Closed-loop LMP decoding benefited from half-wave rectification . . . . . 66

4.1 Should we worry about error signals affecting ongoing control? . . . . . . . 74

xii

4.2 BMI cursor jump task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

4.3 Rapid motor cortical perturbation response without deleterious BMI output 78

4.4 Perturbations evoked both increases and decreases of firing rates . . . . . 79

4.5 Examples of neural push following a cursor jump . . . . . . . . . . . . . . . . 80

4.6 Decoder null to decoder potent neural dynamics hypothesis . . . . . . . . . 82

4.7 Initial error-related modulation is restricted to the decoder’s null space . 83

4.8 Responses in the potent and null space of an observation-trained decoder 84

4.9 Electrode decoder contribution weight curves . . . . . . . . . . . . . . . . . . 85

4.10 Summary schematic of early error response confined to an output-null space 86

xiii

Chapter 1

Introduction

1.1 The premise and promise of motor neural prostheses

Motor neural prostheses are an emerging type of medical device with the potential to restore motor

function to people with paralysis (Figure 1.1). The core premise of motor neural prosthesis, a type of

brain-machine interfaces (BMI), is to decode neural activity to allow a user to directly control an

effector such as a computer cursor (Ganguly & Carmena, 2009; Gilja et al., 2012) or robotic arm

(Carmena et al., 2003; Taylor et al., 2002).

Inspired by initial studies in non-human primates, intracortical BMIs are now being tested in

pilot clinical trials to restore communication (Bacher et al., 2015; Pandarinath et al., 2014) and

movement (Collinger et al., 2013b; Hochberg et al., 2012). This technology can work across a wide

range of neurological damage etiologies because it decodes movement intentions directly from the

brain, upstream of most strokes, neurodegeneration, spinal cord injury, or amputations that cause

movement impairment (Anderson, 2004; Foundation, 2009). The eventual goal is to not only be

able to control external prostheses, but to use stimulation to restore movement of the patient’s own

musculature (Ethier & Miller, 2014; Ethier et al., 2012; Schearer et al., 2014) and to write sensory

information from the robotic arm (or the real arm, if sensory afferents are injured) into the brain

(Bensmaia & Miller, 2014; Dadarlat et al., 2014; O’Doherty et al., 2011; Tabot et al., 2013). In

other words, the long-term vision for motor neural prostheses is to restore near-natural function by

bi-directionally bypassing the injury.

The underlying operational principle of these systems is to identify a mapping between recorded

neural activity and desired movement intention such that when the neural correlates of a particular

movement intention are detected, the appropriate control signal is output to the effector. This

operation is called decoding (Kao et al., 2014) and forms one of the three key neuroscientific and

engineering domains of the broader neural prosthetics endeavor. It is in this domain that I have

focused my work as a neuroscientist and neural engineer. The other two core challenges — acquisition

1

CHAPTER 1. INTRODUCTION 2

Chronically Implanted Sensor

Neural Actvitiy

Sensory Feedback

Prosthesis

Decoded

Movement

Command

Figure 1.1: Schematic of a motor neural prosthesisIn paralyzed or amputee patients, there is damage somewhere along the normal efferent pathwaybetween the brain and muscles. Nonetheless, neural correlates of the person’s movement intentioncan be detected using implanted sensors (here, one 96-electrode BlackRock Microsystems array isshown). This activity is then decoded and transformed into a control signal sent to an effector suchas a robotic arm. The neural prosthesis is a closed-loop system: the user receives sensory feedbackabout the position and movements of the prosthesis.


of the neural signals in the first place, and design of the output effector — are the subject of

focused research and development efforts by neurosurgeons, neurologists, materials scientists, bio-

and electrical engineers, prosthetists and roboticists. The scope of motor neural prosthetics is large

and inherently lends itself to multidisciplinary collaboration. This is well reviewed in (Ethier et al.,

2015; Gilja et al., 2011; Green & Kalaska, 2011; Hatsopoulos & Donoghue, 2009; Homer et al., 2013;

Nicolelis & Lebedev, 2009; Schwartz, 2004; Tsu et al., 2015).

Although there are a variety of neural signals that can potentially be decoded from both within

and outside the brain (Andersen et al., 2004; Moran, 2010), to date the highest-performing BMIs

have been driven by decoded action potentials (‘spikes’) and local field potential signals recorded

from within cortex (reviewed in Gilja et al., 2012; Stavisky et al., 2015a). Given the technology

currently available and on the horizon, I believe that accurate control of complex effectors such as

prosthetic arms will only be possible with intracortical systems, and this perspective has directed my

training and research. That said, since intracortical recording necessitates the non-zero risk and cost

of neurosurgery, I would love to be proved wrong.

Although I have motivated the development of BMIs based on their potential medical value,

they have a closely related, but distinct, purpose as a tool for basic motor neuroscience. In a

BMI, the relationship between neural activity and the effector that the brain is controlling (e.g.,

computer cursor) are completely defined by the experimenter. This provides a uniquely rigorous

and parameterized technique for studying how the motor system learns to alter the activity of

BMI-connected neurons in order to generate particular patterns of output for accomplishing a given

task (e.g. Fetz, 1969; Jarosiewicz et al., 2008; Orsborn et al., 2014; Sadtler et al., 2014). My Chapter

4 exemplifies the use of BMIs to study motor neuroscience.

1.2 Key challenges to clinically viable motor neural prostheses

The key challenges that need to be overcome in order for BMIs to become clinically useful motor

prostheses differ depending on the degree of function being restored. Here I will review two sets of

challenges that directly motivated my dissertation research.

Two-dimensional cursor control has recently reached high levels of peak performance in pre-clinical

studies (Gilja et al., 2012; Nuyujukian et al., 2015), but translation to human participants is often

hampered by unreliable performance due to instability of the neural signals from which motor

intentions are decoded These changing recording conditions consist of both moment-by-moment and

day-to-day changes in the recorded spike signals (Aflalo et al., 2015; Perge et al., 2013, 2014; Simeral

et al., 2011; Wodlinger et al., 2015), as well as longer-term degradation of the chronically implanted

electrodes’ ability to record spikes (Barrese et al., 2013; Chestek et al., 2011; Kruger et al., 2010;

Simeral et al., 2011). The former problem interrupts normal use of the BMI and necessitates decoder

recalibration, whereas the latter shortens the useful lifespan of the device. To be clinically useful,


motor neural prostheses will need to be more robust both to short-term and long-term neural input

changes. Only then will they provide the reliable, long-lasting benefit necessary to justify their risks.

Recent human studies such as Gilja et al. 2015 and Jarosiewicz et al. 2015 have achieved improved

performance and robustness by adopting and extending methods developed in non-human primates;

these studies demonstrate the utility of working on these problems in an animal model to generate a

pipeline for further translational improvements.

BMI-driven control of robotic arms remains poor both in humans (Hochberg et al., 2012; Wodlinger

et al., 2015) and pre-clinical monkey models (Velliste et al., 2008) because we lack fundamental

understanding of what neural signals mean when a user is trying to control a BMI arm. Factors such

as intended muscle forces (Ethier & Miller, 2014; Evarts et al., 1983; Flint et al., 2012a; Lillicrap

& Scott, 2013; Todorov, 2000), actual and anticipated sensory feedback (Archambault et al., 2011;

Georgopoulos et al., 1983; Omrani et al., 2014; Pruszynski et al., 2014; Shadmehr et al., 2010),

grasp-related activity (Castiello, 2005; Pohlmeyer et al., 2007; Vargas-Irwin et al., 2010, 2015), and

changing expectations about the physical dynamics of the arm (Shadmehr & Mussa-Ivaldi, 1994;

Wolpert et al., 1995) all change moment-by-moment during the complex process of moving a physical

arm that is interacting with other objects. This hampers our ability to accurately decode arm control

intention. Further progress in BMI capabilities (e.g. dexterous reach and grasp) will require greater

scientific understanding of how the brain controls a BMI and incorporates it into its motor schema

and how to best decode prosthesis control signals amidst these many intermixed neural signals.

1.3 Dissertation overview

Chapter 2 addresses the BMI robustness problem due to fluctuating spike input signals. It describes

a machine learning method in which a recurrent neural network decoder is trained to exploit the large

quantity of data generated by a chronic BMI. These data, which have heretofore gone unused, include

a variety of previously encountered neural recording conditions. Our intuition was that a decoder

which has learned a variety of neural-to-kinematic mappings will be more robust to encountering

similar recording conditions in the future. We experimentally tested this novel decoding strategy and

showed that it is indeed more robust than a current state-of-the-art decoder to two different forms of

recording condition changes.

This work is currently under review, with the authors listed as David Sussillo*, Sergey D. Stavisky*,

Jonathan C. Kao*, Stephen I. Ryu, and Krishna V. Shenoy, where the asterisk represents equal

contribution. J.C.K and I previously presented it at the 2014 Society for Neuroscience annual

meeting (Kao,* Stavisky*, Sussillo*, et al., 2014). As co-equal lead author, my contributions included

co-conceiving the study, helping develop the closed-loop MRNN decoder, performing the experiments

with the help of J.C.K. and D.S., and writing the manuscript with help from J.C.K. and D.S. Professor

Shenoy provided mentorship throughout the process.


Chapter 3 also addresses BMI robustness, but focuses on how to render the BMI robust to

longer-term loss of the sensor’s ability to record spike signals. To do so, I developed methods for

decoding local field potentials as a longer-lasting complimentary and alternative BMI control signal.

When tested in closed-loop, these techniques resulted in the highest performance to date of a local

field potential-driven BMI, as well as the first closed-loop BMI controlled using spikes and LFPs

together. This work has been published in the Journal of Neural Engineering, with the title “A

high performing brain-machine interface driven by low-frequency local field potentials alone and

together with spikes”, by Sergey D. Stavisky, Jonathan C. Kao, Paul Nuyujukian, Stephen I. Ryu,

and Krishna V. Shenoy (Stavisky et al., 2015a). Prior to that, I presented it at the 2014 IEEE

Engineering in Medicine and Biology Society Conference, which was published as a peer-reviewed

conference proceeding (Stavisky et al., 2014). As lead author, I conceived the study, performed all

analyses, conducted the experiments with the help of J.C.K., and wrote the paper. Professor Shenoy

provided mentorship throughout the process.

Chapter 4 lays the groundwork for robust closed-loop control of prosthetic limbs by studying the

neuroscience of how sensory feedback related to perturbation of the BMI effector effects motor cortical

activity and ongoing BMI control. This feedback presents a potential problem because it could affect

the decoded output commands, resulting in unwanted prosthesis movement due to a deleterious

feedback loop. Anticipating this issue, I designed an experiment that captured the essential elements

of perturbing a BMI in a simpler, more controlled 2D cursor control experiment: I applied a step

perturbation to the position of a BMI cursor and measured how the neural response affected the BMI.

I found that although visual feedback did indeed produce early error-related motor cortical responses,

this initial modulation was limited to ‘output-null’ neural dimensions (as defined in Kaufman et al.

Nat Neurosci 2014) which did not cause deleterious feedback into the BMI output. Only later did

the neural activity enter ‘output-potent’ dimensions that led to the correct BMI output.

This work is currently being prepared for submission, and will have the authors listed as Sergey D.

Stavisky, Jonathan C. Kao, Stephen I Ryu, and Krishna V. Shenoy. I presented a preliminary report

of a similar task at the 2015 IEEE Neural Engineering and Repair meeting, which was published

as a peer-reviewed conference paper (Stavisky et al., 2015b). As the lead author of this study, my

contributions included conceiving the study, implementing the task, co-conducting the experiments

with J.C.K, analyzing the data, and creating the written description of the work that is presented

here. Professor Shenoy provided mentorship throughout the process.

1.4 Animal model and general experimental setup

All three of these research projects were performed using a non-human primate pre-clinical animal

model. Rhesus macaques have a close motor and sensory system homology to humans and are


D

V

AP

Primary Motor Cortex (M1)

Dorsal Premotor

Cortex (PMd)

Figure 1.2: Rhesus macaquemotor cortical regionsrecorded fromI recorded from twoclosely-related motor corticalregions, the primary motorcortex (M1) and the dorsalpremotor cortex (PMd) using96-electrode arrays. A =Anterior, P = Posterior, D =Dorsal, V = Ventral.

the best animal model currently available for studying the motor system (Churchland, 2015) and

developing neural prostheses rapidly without endangering humans (Nuyujukian et al., 2011). These

monkeys are capable of learning complex motor tasks, capable of precise repeated behavior, and have

brains large enough to implant the same sensors that are currently used in human BMI clinical trials

(Barrese et al., 2013; Homer et al., 2013). As such, able-bodied monkeys provide the best available

compromise between the unethical use of human research subjects in early research, and the ethical

and veterinary care difficulties posed by a paralyzed monkey model. Still, unlike the target patient

population that this research has in mind for its long-term impact, our monkeys were able-bodied.

This is an important difference that imposes additional uncertainty on how well pre-clinical findings

will translate, but I note that human BMI studies in which participants imagined making movements

found similar patterns of cortical activity to that of able-bodied monkeys reaching (Hochberg et al.,

2006; Aflalo et al., 2015; Gilja et al., 2015; Pandarinath et al., 2015).

In all three projects, BMI control was driven by neural activity from two motor-related brain

areas: the arm/hand related areas of the primary motor cortex (M1) and dorsal premotor cortex

(PMd), diagrammed in Figure 1.2.

Neural activity in these areas is strongly related to movement preparation (Tanji & Evarts, 1976;

Bastian et al., 1998; Crammond & Kalaska, 2000; Churchland et al., 2012) and generation (Evarts,

1968; Kalaska & Drew, 1993; Kandel et al., 2000; Rathelot & Strick, 2009). M1 and PMd neurons

exhibit strong correlations with both kinematics (Georgopoulos et al., 1988; Kalaska et al., 1989;

Moran & Schwartz, 1999) and kinetics (Cabel et al., 2001; Evarts, 1968; Kalaska et al., 1989; Morrow

& Miller, 2003; Suminski et al., 2015) of movement. This richness of movement-related activity,

combined with the surgical accessibility of their surface location, has made M1 and PMd common

sites for recording control signals for motor neural prostheses (e.g. Taylor et al., 2002; Schwartz, 2004;

Ganguly & Carmena, 2009; Hatsopoulos & Donoghue, 2009; Hochberg et al., 2006; Santhanam et al.,


2006; Velliste et al., 2008). That said, there are ongoing investigations of decoding neural activity

from other regions (e.g. Andersen et al., 2004; Musallam et al., 2004; Pasley et al., 2012; Graf &

Andersen, 2014; Aflalo et al., 2015).

Although there are distinctions between M1 and PMd, most notably a gradient of increasingly

more motor preparation as one records in more anterior sites, within the cortical areas I recorded

from these differences are more of degree than a stark qualitative difference (Schwartz et al., 1988;

Riehle & Requin, 1989; Wise et al., 1998; Crammond & Kalaska, 1996; Messier & Kalaska, 2000). For

the purposes of the studies described in this dissertation, I did not explicitly distinguish between M1

and PMd activity. This does not preclude there being interesting anterior vs. posterior differences

worthy of future investigation, particularly for the sensory feedback-related analyses (Chapter 4).

Chapter 2

Harnessing months of previously

collected data to improve

brain-machine interface robustness

to future neural variability

2.1 Summary

A major hurdle to clinical translation of brain-machine interfaces (BMIs) is that current decoders,

which are trained from a small quantity of recent data, are rendered ineffective when neural recording

conditions subsequently change. Meanwhile, a typically large quantity of data accumulated from

chronic BMI use has heretofore gone unused. We tested the hypothesis that a decoder trained to

handle a variety of recording conditions sampled from months of previously collected data would

be more robust to future neural variability. To do so, we developed a new BMI decoder that can

successfully learn a large variety of neural-to-kinematic mappings in a single artificial recurrent neural

network. When tested in a non-human primate preclinical BMI model, this decoder was robust under

conditions that disabled a state-of-the-art Kalman filter based decoder. These results validate a new

BMI strategy in which accumulated data history is effectively harnessed, and may facilitate reliable

daily BMI use by reducing the need for decoder retraining downtime.

8

CHAPTER 2. ROBUSTNESS TO NEURAL SIGNAL VARIABILITY USING AN MRNN 9

2.2 Introduction

Brain-machine interfaces (BMIs) can restore motor function and communication to patients with

paralysis (Collinger et al., 2013b; Hochberg et al., 2006, 2012). Progress has been particularly strong

towards enabling two-dimensional computer cursor control, which may allow versatile communications

prostheses (Bacher et al., 2015; Gilja et al., 2015; Jarosiewicz et al., 2015). Cursor control performance

has approached that of the native hand in recent macaque studies (Gilja et al., 2012; Nuyujukian

et al., 2015), but this was accomplished under favorable laboratory conditions where neural recordings

are often stable both during and across BMI sessions (Ganguly & Carmena, 2009; Flint et al., 2012a;

Nuyujukian et al., 2014). In contrast to these pre-clinical studies, one of the major challenges

impeding BMI use by human clinical trial participants is the high degree of within- and across-day

variability in neural recording conditions (Simeral et al., 2011; Perge et al., 2013, 2014; Wodlinger

et al., 2015; Aflalo et al., 2015), illustrated in Figure 2.1a. I use the term ‘recording condition’ to

broadly encompass the combination of factors which together determine the relationship between

observed neural activity and intended kinematics. These factors include the relative position of the

electrodes and surrounding neurons (diagrammed in Figure 2.1b), variability in sensor properties

such as impedance or wiring quality, noise sources, and biological factors such as cognitive state or

medications. Existing neural decoding algorithms are poorly suited to handle this input variability,

resulting in intermittent performance and a need for frequent decoder retraining (Bacher et al., 2015;

Jarosiewicz et al., 2013, 2015; Perge et al., 2013).

The clinical viability of BMIs would be much improved by making decoders robust to recording

condition changes (Huggins et al., 2011; Collinger et al., 2013a), and several recent studies have

begun to focus on this problem. We can broadly divide the conditions that a BMI will encounter

into one of two types: 1) conditions that are completely different from those previously encountered,

and 2) conditions that share some commonality with those previously encountered. For existing

BMI methods, both of these situations necessitate some interruption of function while the decoder is

updated to handle the new condition. One strategy for minimizing this interruption is to use adaptive

decoders rather than collecting new training data for a de novo decoder (Dangi et al., 2013, 2014;

Bishop et al., 2014; Orsborn et al., 2014; Nuyujukian et al., 2014; Pohlmeyer et al., 2014; Shenoy &

Carmena, 2014; Jarosiewicz et al., 2015) . In principle, however, BMI interruption could be avoided

altogether in the second case by a decoder capable of exploiting the overlap between the current and

previously encountered conditions (Figure 2.1c).

We were motivated to try this complimentary strategy because chronic BMI systems do typically

encounter recording conditions in which there is some commonality with past recording conditions

(Ganguly & Carmena, 2009; Chestek et al., 2011; Flint et al., 2013; Perge et al., 2013, 2014; Bishop

et al., 2014; Orsborn et al., 2014; Nuyujukian et al., 2014; Stavisky et al., 2015a). Furthermore,

these systems generate and store months, or even years, of neural and kinematic data as part of

their routine use. Almost all of this past data is left unused in existing BMI systems: decoders are


D5D

1D

1D

2D

1D

3D

4D

1D

5D

5D

1D

1D

2D

2D

3D

1D

3D

6

?

Traditional

Decoding

day 1 day 2 previous day now

…

Decoder Trained

From Chronic

Recordings

Dall

Time

(lost)

Electrodes

Recording Condition 1 Recording Condition 2 Recording Condition 1 Recording Condition 3

(added)

(broken)

Time (min)10020 40 60 80

0

100

% T

ria

ls C

orr

ect

0

20

Firin

g R

ate

(h

z)

a

b

c

Figure 2.1: Our strategy for training a decoder robust to recording condition changesa) Example data from a BMI clinical trial showing sudden decoder failure caused by a recordingcondition change. The black trace shows the participant’s closed-loop performance over the course ofan experiment using a fixed Kalman filter. An abrupt drop in performance coincides with a reductionin the observed firing rate (red trace) of a neuron with a high decoder weight. Both the neuron’sfiring rate and decoder performance spontaneously recover 40 minutes later. Adapted from Figure 7of (Perge et al., 2013).(b) A cartoon depicting one hypothetical cause of the aforementioned change: micro-motion of theelectrodes leads to Recording Condition 2, in which spikes from the red-shaded neuron are lost.BMI recovery corresponds to a shift back to Condition 1. Over time, further changes will result inadditional recording conditions; for example, Condition 3 is shown caused by a disconnected electrodeand an additional neuron entering recording range.(c) Recording conditions (schematized by the colored rectangles) will vary over the course of chronicintracortical BMI use. We hypothesize that oftentimes new conditions are similar to ones previouslyencountered (repeated colors). Typically, decoders are fit from short blocks of training data and areonly effective under that recording condition (decoders D1,D2, ...). Consider the goal of traininga decoder for use at time “now” (black rectangle on right). Standard practice is to use decoderD1 trained from the most recently available data (e.g. from the previous day or the start of thecurrent experiment). D1 will perform poorly if the recording condition encountered differs from itstraining data. To increase the likelihood of having a decoder that will perform well given the currentrecording condition, we tested a new class of decoder, Dall, trained using a large collection of previousrecording conditions.


trained using the most recently available data, typically from a block of calibration trials at the

start of that day’s experiment, or from a recent previous experiment (Homer et al., 2013). Using

this historical data would be difficult for most state-of-the-art BMI decoders, as they are linear

(e.g. Collinger et al., 2013b; Gilja et al., 2012). Although linear decoders have other advantages

(for instance, their operation is easy to understand and they can be fit quickly), they are prone

to underfitting heterogeneous training sets, such as those that might be sampled from months of

data. To overcome this limitation, an essential aspect of our approach is to use a nonlinear and

computationally powerful decoder capable of learning a diverse set of neural-to-kinematic mappings.

Specifically, we tested whether one could gain traction on the decoder robustness problem by

exploiting this idle wealth of stored data using an artificial recurrent neural network (RNN). We did

this with a three-pronged approach. The first was the aforementioned use of the nonlinear RNN.

The second was to train the decoder from many months of previously recorded data. Third, to

‘harden’ the decoder against being too reliant on any given electrode’s inputs, we artificially injected

additional variability into the data during decoder training. We did this by perturbing the training

data in severe ways, for example by zeroing or doubling the firing rates of the electrodes.

Our strategies to enhance the typical BMI training set provide a large and variable set of training

data that most traditional decoding methods are unable to handle due to their limited computational

power (Kao et al., 2014). We therefore developed a new BMI decoder using a powerful RNN

variant called the multiplicative recurrent neural network (‘MRNN’, Sutskever et al., 2011). Several

properties of an MRNN make it attractive for this application. Its recurrent connectivity leads to rich

nonlinear internal dynamics that are well-suited to decoding the time-varying, complex relationships

between neural firing rates and kinematics (Churchland & Shenoy, 2007; Sussillo et al., 2012). Its

‘multiplicative’ architecture increases computational power by allowing the neural inputs to strongly

influence the internal dynamics of the RNN by changing the connectivity weights (Figure 2.2a).

Loosely speaking, this allows the MRNN to learn a ‘library’ of different neural-to-kinematic mappings

that are contextually appropriate to the current recording conditions (the mathematical details

are described in the Methods). This general nonlinear approach is likely to work well with many

out-of-the-box RNN variants, such as a standard RNN (e.g. Sussillo et al., 2012) or LSTM (Hochreiter

& Schmidhuber, 1997) and is also completely complementary to adaptive decoding.

We evaluated decoders using two non-human primates implanted with chronic multielectrode

arrays similar to those used in ongoing clinical trials. In this chapter, I first show that the MRNN

trained with more data from previous recording sessions leads to improved accuracy when decoding

new neural data, and that a single MRNN can be trained to accurately decode hand reach velocities

across hundreds of days. I next present closed-loop results showing that an MRNN trained with

many days’ worth of data is much more robust than a state-of-the-art Kalman filter based decoder

(the Feedback Intention Trained Kalman filter ‘FIT-KF’, Fan et al., 2014) to two types of recording

condition changes likely to be encountered in clinical BMI use: the unexpected loss of signals from


1 1024

0

1

1 663

0

1

De

co

de

Accu

racy (

r2)

Recording DayRecording Day

MRNN

FIT LongFIT Sameday

0

1 ***

0

1 ***

1 14 34 47 55

0

1

50 Training Days38 Training Days25 Training Days17 Training Days11 Training Days 7 Training Days 5 Training Days 3 Training Days 2 Training Days 1 Training Day

Training Days

1 7 51 59

0

1

37 Training Days25 Training Days 17 Training Days 11 Training Days 7 Training Days 5 Training Days 3 Training Days 2 Training Days 1 Training Day

Test Day

De

co

de

Accu

racy (

r2)

Test Day

Training Days

vy(t)

vx(t)

px(t)

py(t)

b Monkey R Monkey L

c

a

Figure 2.2: An MRNN decoder can harness large training datasets(a) Schematic of the experimental setup and MRNN decoder. A monkey performed a targetacquisition task while spikes were recorded. Data from many days of this task were used to traintwo MRNNs such that velocity and position could be read out from the state of their respectiveinternal dynamics. These internal dynamics are a function of the binned spike counts; thus, theMRNN can conceptually be thought of as selecting which internal dynamics are active at any giventime based on the input signals. During closed-loop use, decoded velocity and position outputs wereblended together to update the on-screen cursor. (b) The MRNN was better prepared for futuredays’ recording conditions after being trained with more previously collected data. We evaluated itsability to reconstruct offline hand velocity after training with increasing numbers of previous days’datasets. Training data were added by looking further back in time. In monkey R the early test daysalso contributed training data (with the test trials held out). In monkey L, the training datasetsstart with the day prior to the first test day. More training data (darker colored traces) improveddecode accuracy, especially when decoding more chronologically distant recording conditions (latertest days). For comparison, I also plot performance of a traditional decoder (FIT Kalman filter)trained from each individual day’s training data (‘FIT Sameday’, light blue). (c) An MRNN couldsuccessfully learn even larger datasets spanning many more recording days. An MRNN (red trace)was trained with data from 154 (monkey R) and 250 (monkey L) recording days spanning manymonths. Its offline decoding accuracy on held-out trials was compared to that of the FIT Sameday(light blue). We also tested a single FIT-KF trained using the same large dataset as the MRNN(‘FIT Long’, dark blue). Gaps in the connecting lines denote recording gaps of more than ten days.(Insets) mean ± std decode accuracy across all recording days. The MRNN outperformed both typesof FIT-KF decoders on every day’s dataset.


highly-informative electrodes, and day-to-day changes. Finally, I show that this robustness does

not come at the cost of reduced performance under more ideal (unperturbed) conditions: in the

absence of artificial challenges, the MRNN provides excellent closed-loop BMI performance and

slightly outperforms the FIT-KF. To my knowledge, this is the first attempt to improve robustness

by using a large and heterogeneous training dataset: we used roughly two orders of magnitude more

data than in previous closed-loop studies.

2.3 Results

2.3.1 An MRNN can leverage large amounts of training data to improve

decoder performance

We first tested whether training the MRNN with many days’ worth of data can improve offline

decoder performance across a range of recording conditions. This strategy was motivated by our

observation that the neural correlates of reaching — as recorded with chronic arrays — showed

day-to-day similarities (Figure 2.3). For a typical recording session, the most similar recording came

from a chronologically close day, but occasionally the most similar recording condition was found

in chronologically distant data. MRNN decoders were able to exploit these similarities: Figure

2.2b shows that as more days’ data (each consisting of 500 point to point reaches) were used to

train the decoder, the accuracy of reconstructing reach velocities of test datasets increased (positive

correlation between number of training days and decoded velocity r2, p < 0.001). Two results

from this analysis were particularly encouraging. First, using more training data substantially

increased the decode accuracy for the “hard” days that challenged decoders trained with only a few

days’ data (e.g., test day 51 for monkey R). Second, this improvement did not come at the cost of

worse performance on the initially “easy” test days. These results demonstrate that larger training

datasets better prepare the MRNN for a variety of recording conditions, and that learning to decode

additional recording conditions did not diminish the MRNN’s capability to reconstruct kinematics

under recording conditions that it had already “mastered”. There was not a performance versus

robustness trade-off. Finally, it is worth noting that after training with just one day’s data, the

MRNN’s offline performance on the next several days exceeded that of the FIT-KF trained from

held out test day reaches.

We then tested whether the MRNN’s computational capacity could be pushed even further by

training it using data from 154 (250) different days’ recording sessions from monkey R (L), which

spanned 22 (34) months (Figure 2.2c). Across all these recording sessions’ held-out test trials, the

MRNN’s offline decode accuracy was r2 = 0.81± 0.04 (mean ± std, monkey R) and r2 = 0.84± 0.03

(monkey L). For comparison, we tested the decode accuracy of the FIT-KF trained in two ways:

either specifically using reaching data from that particular day (‘FIT Sameday’), or trained on the

same large multiday training dataset (‘FIT Long’). Despite the multitude of recording conditions


1 252

252

Min

. Prin

cipa

l Ang

le (°

)

3

36Recording Session #1 154

154

Recording Session #

Min

. Prin

cipa

l Ang

le (°

)

2

Rec

ordi

ng S

essi

on #

22

Monkey R Monkey L

b

c

a

1 1024Recording Day 3

36

1 663Recording Day 2

22

Min

. Prin

cipa

l Ang

le (°

)

sess

ion

#2

sess

ion

#52

sess

ion

#103

sess

ion

#153

sess

ion

#2

sess

ion

#84

sess

ion

#168

sess

ion

#251

1 50 100 150

78%

10%

Number of Recording Sessions Apart1 20 40 60 80 100 120O

ccur

ence

Fre

quen

cy

Number of Recording Sessions Apart

62%

14%

Figure 2.3: Similar neural recording conditions can often be found on other daysChronologically close days tend to have more similar neural recordings, but for a given day there areoccasional similar recordings from more distant days. (a) Between-day variability of the structureof neural activity recorded during center-out 8-target reaches over the course of many months (154recording sessions over a 663 day period in monkey R, and 252 sessions spanning 1024 days in monkeyL; these sessions correspond to the same data as in 2.2c). The color at the intersection of row iand column j corresponds to how differently the observed neural activity covaried during recordingsessions i and j. Specifically, we have plotted the minimum principal angle between subspacesspanned by the top 10 eigenvectors of each day’s mean-activity-subtracted covariance matrix (seeMethods). These 10 eigenvectors captured on average 51 (46)% of single-trial variance for monkeysR (L). Sharp “block” structure transitions typically correspond to a long (many weeks’) intervalbetween consecutive recording sessions. (b) Four slices through each monkey’s principal angle matrixshow that for these example days, there were similar recording conditions on other days both nearand far in time. Each color series shows the minimum principal angle between every recording day’sdata and the reference day marked with the arrow of this color in panel a. Note that the horizontalaxis, which spans the same time range as in panel a, is now in units of (calendar) days rather thansession number. Each series’ reference day is marked with a vertical line. (c) Histograms showing thedistribution, across each monkey’s recordings, of how many recording sessions away (either forwardor back in time) we observed the most similar neural correlates of reaching as measured by minimumprincipal angle.


that the MRNN had to learn, on each and every test day each monkey’s single MRNN outperformed

that day’s FIT Sameday filter (monkey R: FIT-Sameday r2 = 0.57 ± 0.05, p < 0.001 paired t-test

comparing FIT-Sameday to MRNN; monkey L: r2 = 0.52 ± 0.05, p < 0.001). Unsurprisingly, a

linear FIT-KF did not benefit from being trained with the same large multiday training set and

also performed worse than the MRNN (monkey R: FIT Long r2 = 0.56, p < 0.001 for a paired t-test

comparing FIT Long to MRNN; monkey L: r2 = 0.46± 0.05, p < 0.001).

2.3.2 Robustness to unexpected loss of the most informative electrodes

We next performed closed-loop BMI cursor control experiments to test the MRNN’s robustness to

recording condition changes. The first set of experiments challenged the decoder with an unexpected

loss of inputs from multiple electrodes. The MRNN was trained with a large corpus of hand reaching

training data up through the previous day’s session (119 - 129 training days for monkey R, 212 - 230

days for monkey L). Then, its closed-loop performance was evaluated on a Radial 8 Task while the

selected electrodes’ input firing rates were artificially set to zero. By changing how many of the most

informative electrodes were dropped (“informative” as determined by their mutual information with

reach direction; see Methods), we could systematically vary the severity of the challenge. Since this

experiment was meant to simulate sudden failure of electrodes during BMI use (after the decoder

had already been trained), we did not retrain or otherwise modify the decoder based on knowledge

of which electrodes were dropped. There were no prior instances of these dropped electrode sets

having zero firing rates in the repository of previously collected training data (Figure 2.5). Thus,

this scenario is an example of an unfamiliar recording condition (zero firing rates on the dropped

electrodes) having commonality with a previously encountered condition (the patterns of activity on

the remaining electrodes).

We found that the MRNN was robust to severe electrode dropping challenges. It suffered only a

modest loss of performance after losing up to the best 3 (monkey R) or 5 (monkey L) electrodes

(Figure 2.4). We compared this to the electrode-dropped performance of a FIT-KF decoder trained

with calibration data from the beginning of that day’s experiment according to previously reported

best practices (Gilja et al., 2012; Fan et al., 2014). This FIT Sameday decoder’s performance worsened

dramatically when faced with this challenge. Across all electrodes-dropped conditions, Monkey R

acquired 52% more targets per minute using the MRNN, while Monkey L acquired 92% more targets.

Although the past datasets used to train the MRNN never had these specific sets of highly

important electrodes disabled, our technique of artificially perturbing the true neural activity during

MRNN training did generate training examples with reduced firing rates on various electrodes (as

well as examples with increased firing rates). The MRNN had therefore been broadly trained to be

robust to firing rate reduction on subsets of its inputs. Subsequent closed-loop comparisons of MRNN

electrode dropping performance with and without this training technique (Figure 2.7) confirmed its

importance.


Monkey R Monkey L

Top N Electrodes Dropped0 3 5 7 10

30

0 5/8 8/11 5/7 5/5 1/43/110/8 0/7 0/5 0/4

Top N Electrodes Dropped

Targ

ets

Per M

inut

e

0 2 3 5 7 10

0

30

6/9 7/7 6/7 2/7 1/7 1/71/9 1/7 1/7 1/70/7 0/7

**

* ***

Figure 2.4: Robustness to unexpected loss of the most important electrodesClosed-loop BMI performance using the MRNN (red) and FIT Sameday (blue) decoders whilesimulating an unexpected loss of 0 to 10 electrodes by setting the firing rates of these electrodes tozero. The mean and SEM of performance across experimental sessions is shown for each decoderas a function of how many electrodes were removed. Stars denote conditions for which the MRNNsignificantly outperformed FIT Sameday across sessions (p < 0.05, paired t-test). The fractionsabove each condition specify for how many of the individual evaluation days each decoder performedsignificantly better according to success rate and time to target. Electrode-dropping order wasdetermined by the mutual information between that electrode’s spike count and target directionduring arm-controlled reaches.

2.3.3 Robustness to naturally occurring recording condition changes

The second set of closed-loop robustness experiments challenged the MRNN with naturally occurring

day-to-day recording condition changes. In contrast to the highly variable recording conditions

encountered in human BMI clinical trials, neural recordings in our laboratory setup are typically

quite stable on the timescale of days (Figure 2.3; Nuyujukian et al., 2014). Therefore, in order to

challenge the MRNN and FIT-KF decoders with greater recording condition variability, we evaluated

them after withholding the most recent several months of recordings from the training data. I refer

to this many month interval between the most recent training data day and the first test day as the

training data “gap” in these ‘stale training data’ experiments. We repeated the experiment with two

different training data gaps for each monkey.

For each gap, we trained the MRNN with a large dataset consisting of many months of recordings

preceding the gap and compared it to two different types of FIT-KF decoders. The ‘FIT Old’ decoder

was trained from the most recent available training day (i.e. the day immediately preceding the gap);

this approach was motivated under the assumption that the most recent data were most likely to

be similar to the current day’s recording condition. The ‘FIT Long’ decoder was trained from the

same multiday dataset used to train the MRNN and served as a comparison in which a conventional

decoder is provided with the same quantity of data as the MRNN. The logic underlying this FIT

Long approach was that despite the Kalman filter being ill-suited for fitting multiple heterogeneous


Recording Session #

Ele

ctr

od

e

20 40 60 80 100 120 140 160 180 212 231

20

40

60

80

10

20

30

40

50

60

Recording Session #

Ele

ctr

od

e

20 40 60 80 100 129

20

40

60

80

100

120

140

160

10

20

30

40

0 0

1 1

192

96

ExperimentsPrevious Training Data

Ave

rag

e F

irin

g R

ate

(H

z)

Ave

rag

e F

irin

g R

ate

(H

z)

119

ExperimentsPrevious Training Data

Monkey R

Monkey L

Dropped

Electrode

Figure 2.5: Artificially dropped electrodes were active in the training dataThese plots show each electrode’s average firing rate during each dataset used to train theMRNN; electrodes are ordered by descending average firing rate across all recording sessions.Recording sessions numbered in gray were only used for training data. The electrode droppingexperiments (Figure 2.4) were conducted during the sessions numbered in black. Zero firing rates (i.e.non-functional electrodes) are shown in purple for emphasis, while electrodes selected for dropping ona particular day are shown in red (note that although on a given test session we evaluated differentnumbers of electrodes dropped, this plot shows each day’s broadest dropped set). These droppedelectrodes rarely recorded zero firing rates in the training data sessions, and the specific sets ofdropped electrodes used to challenge the decoders never all had zero firing rates in the training data.

datasets, this “averaged” decoder might still perform better than the FIT-KF Old trained using a

single distant day.

We found that the MRNN was the only decoder that was reliably usable when trained with stale

data (Figure 2.6). FIT Old performed very poorly in both monkeys, failing completely in 4/6 monkey

R experimental sessions and 6/6 monkey L sessions. Further tests of additional FIT Old decoders

confirmed that they generally perform poorly (Figure 2.8). FIT Long performed better than FIT

Old, but its performance was highly variable — it was usable on some test days but not on others.

In Monkey R, the across-days average acquisition rate was 105% higher for the MRNN than FIT Old

(p < 0.01, paired t-test). Monkey L’s MRNN did not perform as consistently well as Monkey R’s, but

nevertheless demonstrated a trend of outperforming FIT Long (32% improvement, not significant).

Although monkey L’s FIT Long outperformed the MRNN on one test day, on all other test days FIT

Long was either similar to, or substantially worse than, MRNN. Moreover, whereas the MRNN was

controllable every day, FIT Long was not even capable of acquiring targets on some days. The lack


Targ

ets

Per M

inut

e

0

30

FIT Old

MRNNFIT Long

Decoder Set 2 Training Data

FIT Old

MRNNFIT Long


706 810 964Day 1 Set 1 Test

{ {

Set 2 Test 384 447 558Day 1

0

30

Targ

ets

Per M

inut

e

Arm Control

FIT Sameday

FIT Old

MRNNFIT Long


FIT Old

MRNNFIT Long


Set 1 Test{ {

Set 2 Test

Monkey R Monkey L

Figure 2.6: Robustness to naturally occurring recording condition changesRobustness to a natural sampling of neural input variability. We created decoder evaluation conditionsin which the neural inputs were likely to be different from much of the training data by withholdingaccess to the most recent several months of data. Each circle corresponds to the mean closed-loopBMI performance using these “stale” MRNN (red), FIT Long (dark blue), and FIT Old (teal)decoders when evaluated on six different experiment days spanning 7 (13) days in monkey R (L).Each test day, these three decoders, as well as a FIT Sameday decoder trained from that day’s armreaches, were evaluated in an interleaved block design. The legend bars also denote the time periodsfrom which training data for each stale decoder came from. We repeated the experiments for asecond set of decoders to reduce the chance that the results were particular to the specific trainingdata gap chosen. The training data periods contained 82 and 92 datasets (monkey R), and 189 and200 training datasets (monkey L). Both monkeys were only able to consistently perform the taskusing the MRNN decoder. To aid the interpretation of these stale decoder performances, we showthe average performance across the six experiment days using arm control (gray dashed line) or aFIT Sameday decoder (blue dashed line).

of consistent usability by any of the FIT-KF decoders demonstrates that having access to a large

repository of stale training data does not enable training a single Kalman filter that is robust to

day-to-day variability in recording conditions. In contrast, an MRNN trained with this large dataset

was consistently usable.

2.3.4 High performance BMI using the MRNN decoder

Finally, we asked whether the MRNN’s robustness to challenging recording conditions came at the

cost of reduced performance under more ideal conditions. We evaluated the MRNN’s closed-loop

performance when trained using several months’ data up through the previous day, without electrode

dropping or any other artificial challenge. In this scenario, the MRNN enabled both monkeys to

accurately and quickly control the cursor. We compared the MRNN’s performance to that of a

FIT-KF Sameday decoder. Figure 2.9a shows representative cursor trajectories using each decoder,

as well as under hand control. The MRNN outperformed the FIT-KF in both monkeys (Figure 2.9b).

Across nine experimental sessions and 4,000+ trials with each decoder, Monkey R acquired targets

7.3% faster with the MRNN (619 ± 324 ms mean ± std vs. 668 ± 469 ms, p < 0.001, two-sided


Top N Electrodes Dropped0 3 5 7 10

0

30

Top N Electrodes Dropped

Ta

rge

ts P

er

Min

ute

0 2 3 5 7 10

0

30Monkey R Monkey La

b

Block

400

800

1200

1600

Monkey L

0Block

200

400

600

800

0Block

0

200

400

600

800

Tim

e to

Ta

rge

t (m

s)

Monkey R

Training Set 1 Training Set 1Training Set 2

Figure 2.7: Training data spike rate perturbations improve MRNN performance(a) Robustness to electrode dropping. We evaluated the closed-loop BMI performance of the MRNNdecoder trained with (red) and without (brown) the training data spike count perturbations. Bothdecoders were evaluated on the same days with firing rates on the top 0 to 10 electrodes set to zero(similar to Figure 2.4). Each circle corresponds to a decoder’s mean performance on a given evaluationday. Filled circles denote days where there was a significant performance difference (p < 0.05 binomialtest on success rate, followed, if success rate not significantly different, by two-sample t-test of timesto target) between the two decoders tested. The line shows the grand mean performance for eachdecoder. The MRNN trained with perturbed firing rates consistently outperformed the MRNNtrained without these training data perturbations when encountering electrode-dropped neural input.(b) Stale training data experiments. MRNNs were trained without access to recent training data, asin Figure 2.6, either with (red) or without (brown) training data spike rate perturbations. We traineddecoders from both of monkey R’s stale training data periods and from monkey L’s longer staletraining data period. Closed-loop BMI performance using these decoders was then compared on thesame evaluation day in alternating blocks. The plot shows the mean time to target for each block. Forall three pairs of decoders compared, the MRNN with spike rate perturbations had significantly fastertimes to target (p < 0.05, two-sample t-test aggregating trials across blocks). Datasets R.2014.03.21& L.2014.04.04.


Days Between Training Data and Decoder Test155 272 965 978

0

35

Days Between Training Data and Decoder Test

Ta

rge

ts P

er

Min

ute

31 559

0

35 Monkey R Monkey L

FIT O

ld fr

om 2

014-

02-0

3

FIT O

ld fr

om 2

013-

10-2

2

FIT O

ld fr

om 2

013-

07-1

0

FIT O

ld fr

om 2

011-

08-0

4

FIT O

ld fr

om 2

013-

10-0

9

FIT O

ld fr

om 2

013-

11-2

1

FIT O

ld fr

om 2

013-

09-1

9

FIT O

ld fr

om 2

013-

04-1

6

FIT O

ld fr

om 2

012-

09-2

1

Figure 2.8: Additional tests showing that FIT Old typically performs poorlyWe investigated whether our result that three of the four different FIT Old decoders tested in themain stale training data experiments (Figure 2.6) failed was due to a particularly unlucky choiceof FIT Olds. To better sample the closed-loop performance of FIT-KF decoders trained usingold training data, we trained FIT Old decoders from 3 (monkey R) and 2 (Monkey L) additionalarbitrarily chosen arm reaching datasets from the monkey’s prior experiments. We evaluated all 5(4) FIT Old decoders on a number of additional days over the course of the current study (8 totaltest days for monkey R, 13 total test days for monkey L). Each point shows the performance of aparticular FIT Old decoder on one test day. Different days’ evaluations of the same FIT Old decoderare shown in the same color. Black borders denote data points and black underlines denote decodersthat are shared with Figure 2.6.

t-test). Monkey L acquired targets 10.8% faster with the MRNN (743 ± 390 ms vs. 833 ± 532 ms,

p < 0.001, two-sided t-test) across 8 sessions and 2,500+ trials using each decoder. These online

results corroborate the offline results presented in Figure 2.2c; both show that an MRNN trained

from many days’ recording conditions outperforms the current state-of-the-art FIT Kalman filter

trained from training data collected at the start of the experimental session.

A potential risk inherent to a computationally powerful decoder such as the MRNN is that

it will overtrain to the task structure of the training data and fail to generalize to other tasks.

Most of our MRNN training data were from arm reaches on a Radial 8 Task similar to the task

used for evaluation (albeit with 50% further target distance). We therefore also verified that the

MRNN enabled good cursor control on the Random Target Task, in which the target could appear

in any location in a 20× 20 cm workspace (Figure 2.10). Monkey R performed the Random Target

Task on two experimental sessions and averaged a 99.4% success rate over 649 trials, with mean

distance-normalized time to target of 0.068 s/cm. Monkey L performed one session of this task at a

100% success rate across 207 trials with mean normalized time to target of 0.075 s/cm.


0 250 500 750

Trial Time (ms)

0

4

8

0 250 500 750

0

4

8

Trial Time (ms)

Dis

tan

ce

to

Ta

rge

t (c

m)

Monkey R Monkey L

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

1

2

3

4

56

7

8

4 cm

Arm Control MRNN FIT Sameday

b

a

Figure 2.9: The MRNN outperforms the FIT Kalman filter under “ideal” conditions(a) We compared cursor control using the MRNN (red) trained from many datasets up throughthe previous day to the FIT-KF (blue) trained from data collected earlier the same day, withoutany artificial challenges (i.e. no electrodes dropped). Cursor trajectories are shown for eightrepresentative and consecutive center-out-and-back trials of the Radial 8 Task. Gray boxes showthe target acquisition area boundaries, and the order of target presentation is denoted with greennumbers. For comparison, cursor control using the hand is shown in gray. From dataset R.2014.04.03.(b) Mean distance to target, during the Radial 8 Task, as a function of trial time using each cursorcontrol mode. Thickened portions of each trace correspond to ‘dial-in time’, i.e. the mean timebetween the first target acquisition and the final target acquisition. The MRNN and FIT Samedaydata corresponds to the drop 0 electrodes condition in Figure 2.4.

2.4 Discussion

We developed the MRNN decoder to help address a major problem hindering the clinical translation

of BMIs: once trained, decoders can be quickly rendered ineffective due to recording condition

changes. A number of complementary lines of research are aimed at making BMIs more robust,

including improving sensors to record from more neurons more reliably (e.g. Seo et al., 2014); decoding

multiunit spikes (Fraser et al., 2009; Chestek et al., 2011; Nuyujukian et al., 2014) or local field

potentials (Flint et al., 2013; So et al., 2014; Stavisky et al., 2015a) which appear to be more stable

control signals than single unit activity; and using adaptive decoders that update their parameters

to follow changing neural-to-kinematic mappings (Wu & Hatsopoulos, 2008; Dangi et al., 2013, 2014;

Jarosiewicz et al., 2013, 2015; Bishop et al., 2014; Orsborn et al., 2014; Pohlmeyer et al., 2014; Shenoy


200

1100

20002001-5000

Tim

e to

ta

rge

t (m

s)

fail4 cm

Time to target (ms)0 1000 2000 3000 4000 fail

# T

ria

ls0

20

40

60

80 median = 692 ms

Time to target (ms)0 1000 2000 3000 4000 fail

# T

ria

ls

0

100

200

300 median = 562 ms

Monkey R Monkey La

b

Figure 2.10: Closed-loop MRNN decoder performance on the Random Target TaskBoth monkeys were able to use this decoder to acquire targets across a broad workspace in whichtargets often appeared at locations that differed from the target locations dominating the trainingdatasets.(a) Histograms of times to target (time of final target entry minus target onset time, not includingthe 500 ms target hold period) for both monkeys.(b) Task workspace plots showing the location of each Random Target Task trial’s target. Eachpoint corresponds to the center of one trial’s target, and its color represents the time it took themonkey to acquire this target. The location of the one failed trial (for monkey R) is shown with ablack ‘x’. The acquisition area boundaries of the nine Radial 8 Task targets used for the majorityof the training data are shown as black squares. Monkey R’s data are aggregated across the twoexperimental sessions in which he performed this task. Monkey L’s data are from one session.


& Carmena, 2014). Here we’ve taken the approach of building a fixed decoder that is trained to

be inherently robust to recording condition changes it has previously encountered as well as new

conditions that have some similarity to a previously encountered condition. I stress that these

approaches are complementary in several respects. For example, a decoder that is inherently more

robust to neural signal changes, such as the MRNN, would still benefit from improved sensors and

decoder adaptation, and could operate on a mix of input signal types including single- and multiunit

spikes and field potentials.

2.4.1 A robust and high-performing BMI decoder

We demonstrated the MRNN’s robustness to two types of recording condition changes. These changes

were chosen because they capture key aspects of the changes that commonly challenge BMI decoders

during clinical use. The stale training data experiments showed that the MRNN was usable under

conditions where the passage of time would typically require recalibration of conventional decoders

such as the FIT-KF. I do not mean to suggest that in a clinical setting one would want to — or

would often have to – use a BMI without any training data from the past several months. Rather, we

used this experimental design to model recording condition changes that can happen on the timescale

of hours in human BMI clinical trials (Perge et al., 2013). Possible reasons for the greater recording

condition variability observed in human participants compared to non-human primates include:

more movement of the array relative to the human brain due to larger cardiovascular pulsations

and epidural space; greater variability in the state of the BMI user (health, medications, fatigue,

cognitive state); and more electromagnetic interference from the environment. The MRNN can

take advantage of having seen the effects of these sources of variability in previously accumulated

data; it can therefore be expected to become more robust over time as it builds up a “library” of

neural-to-kinematic mappings under different recording conditions.

The electrode dropping experiments, which demonstrated the MRNN’s robustness to an unexpected

loss of high-importance electrodes, are important for two reasons. Firstly, sudden loss of input

signals, for instance due to a electrode connection failure (Gold et al., 2006; Barrese et al., 2013),

is a common BMI failure mode that can be particularly deleterious to conventional BMI decoders

(Kao et al., 2013). The MRNN demonstrates considerable progress in addressing this so-called

“errant unit” problem. Secondly, these results demonstrate that the MRNN trained with artificially

perturbed neural data can be relatively robust even to a recording condition change that has not

been encountered in past recordings.

The MRNN’s robustness did not come at the cost of diminished performance under more ideal

conditions. This result is nontrivial given the robustness-focused decisions that went into its design

(e.g. perturbing the input spike trains in the training set). Instead, we found that the MRNN was

excellent under favorable conditions, outperforming a state-of-the-art same day trained FIT-KF

decoder. Taken together, these results demonstrate that the MRNN exhibits robustness to a variety of


clinically relevant recording condition changes, without sacrificing peak performance. These advances

may help to reduce the onerous need for clinical BMI users to collect frequent retraining data.

One disadvantage of this class of nonlinear decoders trained from large datasets, when compared

to traditional linear decoders trained on smaller datasets, is the longer training time required to

fit their parameters. In the present study, which we did not optimize for fast training, this took

multiple hours. This could be substantially sped up by iteratively updating the decoder with new

data instead of retraining de novo and by leveraging faster computation available with e.g. graphics

processing units, cloud computing, or custom hardware.

2.4.2 Key innovations enabling the MRNN’s robustness

The MRNN decoder’s robustness was due to the combination of a large training data corpus,

deliberate perturbation of the training data, and a computationally powerful architecture that was

able to effectively learn this diverse training data. While it may seem obvious that a decoder that

successfully learns more training data is better, this is not necessarily true. Older data only helps if

some of these past recordings capture neural-to-kinematic relationships that are similar to that of the

current recording condition. Our offline and closed-loop MRNN robustness results suggest that this

was indeed the case for the two monkeys used in this study. While there are indications that this

will also be true in human BMI studies (Perge et al., 2014), validating this remains an important

future question. The relevance of old data to present recording conditions also enables a different

robustness-enhancing approach: store a library of different past decoders and evaluate each to find a

decoder well-suited for the current conditions (e.g. Nuyujukian et al., 2014). However, since offline

analyses are poor predictors of closed-loop performance (Fraser et al., 2009; Cunningham et al., 2010;

Koyama et al., 2010; Stavisky et al., 2015a), this approach necessitates a potentially lengthy decoder

selection process. Using a single decoder (such as the MRNN) that works across many recording

conditions avoids switching-related downtime.

In addition to training with months of previous data, we improved the MRNN’s robustness by

intentionally perturbing the training neural data. In the present study we applied random Gaussian

firing rate scaling based on a general assumption that the decoder should be broadly robust to

both global and private shifts in observed firing rates. This perturbation type proved effective, but

this approach (called data augmentation in the machine learning community) can potentially be

much more powerful when combined with specific modeling of recording condition changes that the

experimenter wants to train robustness against. For example, data augmentation could incorporate

synthetic examples of losing a particularly error-prone set of electrodes; the effects of known sources

of electrical noise; recording changes predicted by models of array micro-movement or degradation;

and perhaps even the predicted interaction between kinematics and changes in cognitive state or

task context. I believe this is an important avenue for future research.


I view the success of our specific MRNN decoder implementation as a validation of the more

general BMI decoder strategy of training a computationally powerful nonlinear decoder to a large

quantity of data representing many different recording conditions. This past data need not be have

been collected explicitly for the purpose of training as was done in this study; neural data and

corresponding kinematics from past closed-loop BMI use can also serve as training data (Nuyujukian

et al., 2014; Jarosiewicz et al., 2015). It is likely that other decoding algorithms will also benefit from

this strategy, and that there are further opportunities to advance the reliability and performance of

BMIs by starting to take advantage of these devices’ ability to generate large quantities of data as

part of their regular use.

2.5 Methods

2.5.1 Animal model and neural recordings

All procedures and experiments were approved by the Stanford University Institutional Animal Care

and Use Committee. Experiments were conducted with adult male rhesus macaques (‘R’ and ‘L’),

implanted with 96-electrode Utah arrays (Blackrock Microsystems Inc., Salt Lake City, UT) using

standard neurosurgical techniques. Monkeys R and L were implanted 30 months and 74 months

prior to the primary experiments, respectively. Monkey R had two electrode arrays implanted, one

in PMd and the other in M1, as estimated visually from anatomical landmarks. Monkey L had one

array implanted on the border of PMd and M1. Spike counts were collected by applying a single

negative threshold, set to 4.5 × root mean square of the spike band of each electrode. Behavioral

control and neural decode were run on separate PCs using the xPC Target platform (Mathworks,

Natick, MA), enabling millisecond-timing precision for all computations. Neural data were initially

processed by Cerebus recording system(s) (Blackrock Microsystems Inc., Salt Lake City, UT) and

were available to the behavioral control system within 5± 1 ms.

2.5.2 Behavioral tasks

We trained the monkeys to acquire targets with a virtual cursor controlled by either the position of

the hand contralateral to the arrays or directly from neural activity. Reaches to virtual targets were

made in a 2D fronto-parallel plane presented within a 3D environment (MSMS, MDDF, USC, Los

Angeles, CA) generated using a Wheatstone stereograph fused from two LCD monitors with refresh

rates at 120 Hz, yielding frame updates within 7± 4 ms (Cunningham et al., 2011). Hand position

was measured with an infrared reflective bead tracking system at 60 Hz (Polaris, Northern Digital,

Ontario, Canada). During BMI control, we allowed the monkey’s reaching arm to be unrestrained

(Nuyujukian et al., 2011; Shenoy & Carmena, 2014) so as to not impose a constraint upon the monkey


that during BMI control he must generate neural activity that does not produce overt movement

(Kaufman et al., 2014).

In the ‘Radial 8 Task’ the monkey was required to acquire targets alternating between a center

target and one of eight peripheral targets equidistantly spaced on the circumference of a circle. For

our closed-loop BMI experiments, the peripheral targets were positioned 8 cm from the center target.

In hand reaching datasets used for decoder training and offline decode, the targets were either 8 cm

or 12 cm (the majority of datasets) from the center. In much of Monkey L’s training data, the three

targets forming the upper quadrant were placed slightly further (13 and 14 cm) based on previous

experience that this led to decoders with improved ability to acquire targets in that quadrant. To

acquire a target, the monkey had to hold the cursor within a 4 cm × 4 cm acceptance window

centered on the target for 500 ms. If the target was acquired successfully, the monkey received a

liquid reward. If the target was not acquired within 5 s (BMI control) or 2 s (hand control) of target

presentation, the trial was a failure and no reward was given.

We measured closed-loop BMI performance on the Radial 8 Task using two metrics. Target

acquisition rate is the number of peripheral targets acquired divided by the duration of the task. This

metric is calculated across all trials of a certain experimental condition (i.e., which decoder was used)

and holistically reflects cursor control ability because, unlike time to target, it is negatively affected

by failed trials and directly relates to the animal’s rate of liquid reward. As a second measure of

performance that is more sensitive when success rates are high and similar between decoders (such

as the “ideal” conditions where we presented no challenges to the decoders), we compared times to

target. This measure consists of the time between when the target appeared and when the cursor

entered the target acceptance window prior to successfully acquiring the target, but does not include

the 500 ms hold time (which is constant across all trials). Times to target are only measured for

successful trials to peripheral targets.

In the Random Target Task each trial’s target appeared at a random location within a 20 cm × 20

cm region centered within a larger workspace that was 40 × 30 cm. A new random target appeared

after each trial regardless of whether this trial was a success or a failure due to exceeding the 5 s

time limit. The target location randomization enforced a rule that the new target’s acceptance area

could not overlap with that of the previous target. Performance on the Random Target Task was

measured by success rate (the number of successfully acquired targets divided by the total number of

presented targets) and the normalized time to target. Normalized time to target is calculated for

successful trials following another successful trial, and is the duration between target presentation

and target acquisition (not including the 500 ms hold time), divided by the straight-line distance

between this target’s center and the previously acquired target’s center (Suminski et al., 2010).


2.5.3 Multiplicative Recurrent Neural Network (MRNN) decoder

At a high level, the MRNN decoder transforms inputs u(t), the observed spike counts on each

electrode at a particular time, into a cursor position and velocity output. This is accomplished by

having first trained the weights of an artificial recurrent neural network such that when the network

is provided a time series of training neural data inputs, the training data kinematic outputs can be

accurately “read out” from this neural network’s state. The rest of this section will describe the

architecture, training, and use of the MRNN for the purpose of driving a BMI.

The generic recurrent network model is defined by an N -dimensional vector of activation variables,

x, and a vector of corresponding “firing rates”, r = tanh(x). Both x and r are continuous in time

and take continuous values. In the standard RNN model, the input affects the dynamics as an

additive time-dependent bias in each dimension. In the MRNN model, the input instead directly

parameterizes the artificial neural network’s recurrent weight matrix, allowing for a multiplicative

interaction between the input and the hidden state. One view of this multiplicative interaction is

that the hidden state of the recurrent network is selecting an appropriate decoder for the statistics of

the current dataset. The equation governing the dynamics of the activation vector is of the form

suggested in (Sutskever et al., 2011), but adapted in this study to continuous time in order to control

the smoothness to MRNN outputs,

τ x(t) = −x(t) + Ju(t) r(t) + bx. (2.5.1)

The N ×N ×|u| tensor Ju(t) describes the weights of the recurrent connections of the network, which

are dependent on the E-dimensional input, u(t). The symbol |u| denotes the number of unique values

u(t) can take. Such a tensor is unusable for continuous valued u(t) or even discrete valued u(t) with

prohibitively many values. To make these computations tractable, the input is linearly combined

into F factors and Ju(t) is factorized (Sutskever et al., 2011) according to the following formula:

Ju(t) = Jxf · diag(Jfu u(t)) · Jfx, (2.5.2)

where Jxf has dimension N × F , Jfu has dimension F × E, Jfx has dimension F ×N , and diag(v)

takes a vector, v, and returns a diagonal matrix with v along the diagonal. One can directly control

the complexity of interactions by choosing F . Additionally, the network units receive a bias bx. The

constant τ sets the time scale of the network, so we set τ on the order of hundreds of milliseconds to

allow meaningful interactions. The output of the network is read out from a weighted sum of the

network firing rates plus a bias, defined by the equation

z(t) = WTOr(t) + bz, (2.5.3)

where WO is an N ×M matrix, and bz is an M -dimensional bias.


Monkey R Monkey L∆t 20ms 20-30msτ 100ms 100msN 100 50F 100 50σtrial 0.045 0.045σelectrode 0.3 0.3gxf 1.0 1.0gfu 1.0 1.0gfx 1.0 1.0E 192 96days of training data 82-129 189-230years spanned 1.59 2.77β 0.99 0.99

Table 2.1: Network and training parameters used for the closed-loop MRNN BMI decoder

2.5.4 Training the MRNN

We began decoder training by instantiating MRNNs of network size N = 100 (monkey R) and N = 50

(monkey L) with F = N in both cases (see Table 2.1 for all key parameters). For monkey R, who

was implanted with two multielectrode arrays, E = 192, while for monkey L with one array, E = 96.

The non-zero elements of the non-sparse matrices Jxf , Jfu, Jfx are drawn independently from a

Gaussian distribution with zero mean and variance gxf/F , gfu/E, and gfx/N , with gxf , gfu, and

gfx set to 1.0 in this study. The elements of WO are initialized to zero, and the bias vectors bx and

bz are also initialized to 0.

The input u(t) to the MRNN (through the matrix Ju(t)) is the vector of binned spikes at each

time step. Concatenating across time in a trial yields training data matrix, Uj , of binned spikes

of size E × T j , where T j is the number of times steps for the jth trial. Data from five consecutive

actual monkey-reaching trials are then concatenated together to make one ‘MRNN training’ trial.

The first two actual trials in an MRNN training trial were used for seeding the hidden state of the

MRNN (i.e., not used for learning), whereas the next three actual trials were used for learning. With

the exception of the first two actual trials from a given recording day, the entire set of actual trials

are used for MRNN learning by incrementing the actual trial index that begins each training trial by

one.

The parameters of the network were trained offline to reduce the averaged squared error between

the measured kinematic training data and the output of the network, z(t). Specifically, we used the

Hessian-Free (HF) optimization method (Martens, 2010; Martens & Sutskever, 2011) for RNNs (but

adapted to the continuous-time MRNN architecture). HF is an exact 2nd order method that uses

back-propagation through time to compute the gradient of the error with respect to the network

parameters. The set of trained parameters is {Jxf ,Jfu,Jfx, bx, bz}. The HF algorithm has three


critical parameters: the minibatch size, the initial lambda setting, and the max number of conjugate

gradient iterations. We set these parameters to one-fifth the total number of trials, 0.1, and 50,

respectively. The optimizations were run for 200 steps and a snapshot of the network was saved every

10 steps. Among these snapshots, the network with the lowest cross-validation error on held-out data

was used in the experiment.

We independently trained two separate MRNN networks to each output a 2-dimensional (M = 2)

signal, z(t). The first network learned to output the normalized hand position through time in both

the horizontal (x) and vertical (y) spatial dimensions. The second MRNN learned to output the

hand velocity through time, also in the x and y dimensions. We calculated hand velocities from the

positions numerically using central differences.

In this study we trained a new MRNN whenever adding new training data; this allowed us to

verify that the training optimization consistently converged to a high-quality decoder. However, it

is easy to iteratively update an MRNN decoder with new data without training from scratch. By

adding the new data to the training corpus and using the existing decoder weights as the training

optimization’s initial conditions, the MRNN will more rapidly converge to a new high-quality decoder.

2.5.5 MRNN training data augmentation

A critical element of achieving both high performance and robustness in the MRNN decoder was

training the decoder using data from many previous recording days spanning many months. When

training datasets included data from more than one day, we randomly selected a small number of

trials from each day for a given minibatch. In this way, every minibatch of training data sampled the

input distributions from all training days.

A second key element of training robustness to recording condition changes was a form of data

augmentation in which we intentionally introduced perturbations to the neural spike trains that

were used to train the MRNN. The concatenated input, U = [ui, · · · ,ui+4] was perturbed by adding

and removing spikes from each electrode. We focus on electrode c of the jth training trial, i.e., a

row vector of data Ujc,:. Let the number of actual observed spikes in Uj

c,: be njc. This number was

perturbed according to

njc = ηjηcnjc, (2.5.4)

where both ηj and ηc are Gaussian variables with a mean of one and standard deviations of σtrial and

σelectrode, respectively. Conceptually, ηj models a global firing rate modulation across all electrodes

of the array (e.g., array movement, arousal), while ηc models electrode by electrode perturbations

such as electrode dropping or moving baselines in individual neurons. If njc was less than zero or

greater than 2njc, it was resampled, which kept the average number of perturbed spikes in a given

electrode and training trial roughly equal to the average number of true (unperturbed) spikes in the

same electrode and training trial. Otherwise, if njc was greater than njc, then njc − njc spikes were


added to random time bins of the training trial. If njc was less than njc, then njc − njc spikes were

randomly removed from time bins of the training trial that already had spikes. Finally, if njc = njc,

nothing was changed.

The process of perturbing the binned spiking data occurred anew on every iteration of the

optimization algorithm, i.e. in the HF algorithm, the perturbation njc = ηjηcnjc occurs after each

update of the network parameters. Note that these input data perturbations were only applied

during MRNN training; when the MRNN was used for closed-loop BMI control, true neural spike

counts were provided as inputs. Figure 2.7 shows the closed-loop control quality difference between

the MRNN trained with and without this data augmentation.

2.5.6 Controlling the BMI cursor with MRNN network output

Once trained, the MRNNs were compiled into the embedded real-time operating system and run in

closed-loop to provide online BMI cursor control. The decoded velocity and position were initialized

to 0, as was the MRNN hidden state. Thereafter, at each decode time step the parallel pair of

MRNNs received binned spike counts as input and had their position and velocity outputs blended to

yield a position estimate. This was used to update the drawn cursor position. The on-screen position

that the cursor moves to during BMI control, dx(t), dy(t), is defined by

dx(t) = β(dx(t−∆t) + γvvx(t−∆t)∆t) + (1− β)γppx(t) (2.5.5)

dy(t) = β(dy(t−∆t) + γvvy(t−∆t)∆t) + (1− β)γppy(t), (2.5.6)

where vx, vy, px, py are the normalized velocity and positions in the x and y dimensions and γv, γp

are factors that convert from the normalized velocity and position, respectively, to the coordinates of

the virtual-reality workspace. The parameter β sets the amount of position versus velocity decoding

and was set to 0.99. In effect, the decode was almost entirely dominated by velocity, with a slight

position contribution to stabilize the cursor in the workplace (i.e., offset accumulated drift). Note

that when calculating offline decode accuracy (Figure 2.2), we set β to 1 to more fairly compare the

MRNN to the FIT-KF decoder, which decodes velocity only.

2.5.7 Feedback Intention Trained Kalman Filter (FIT-KF) decoder

We compared the performance of the MRNN to FIT-KF (Fan et al., 2014). The FIT-KF is a Kalman

filter where the underlying kinematic state, z(t), comprises the position and velocity of the cursor as

well as a bias term. Observations of the neural binned spike counts, u(t), are used to update the

kinematic state estimate. With ∆t denoting bin width (25 ms in this study), the FIT-KF assumes

the kinematic state gives rise to the neural observations according to the following linear dynamical


system:

z(t+ ∆t) = Az(t) + w(t) (2.5.7)

u(t) = Cz(t) + q(t), (2.5.8)

where w(t) and q(t) are zero-mean Gaussian noise with covariance matrices W and Q respectively.

The Kalman filter is a recursive algorithm that estimates the state z(t) using the current observation

u(t) and the previous state estimate z(t−∆t). Previous studies have used such decoders to drive

neural cursors (e.g. Kim et al., 2008; Sussillo et al., 2012; Bacher et al., 2015).

The parameters of this linear dynamical system, A,W,C,Q, are learned in a supervised fashion

from hand reach training data as previously reported (Wu et al., 2003; Gilja et al., 2012). The

FIT-KF then incorporates two additional innovations. First, it performs a rotation of the training

kinematics using the assumption that at every moment in time, the monkey intends to move the

cursor directly towards the target. Second, it assumes that at every time step, the monkey has

perfect knowledge of the decoded position via visual feedback. This affects Kalman filter inference in

two ways: first, the covariance of the position estimate in Kalman filtering is set to 0, and secondly,

the neural activity that is explainable by the cursor position is subtracted from the observed binned

spike counts. These innovations are further described in (Gilja et al., 2012; Fan et al., 2014).

2.5.8 Mutual information for determining electrode dropping order

When testing the decoders’ robustness to unexpected electrode loss, we determined which electrodes

to drop by calculating the mutual information between each electrode’s binned spike counts and the

reach direction. This metric produced a ranking of electrodes in terms of how statistically informative

they were of the reach direction; importantly, this metric is independent of the decoder being used.

Let p denote the distribution of a electrode’s binned firing rates, u denote the binned spike counts

lying in a finite set U of possible binned spike counts, M denote the number of reach directions, and

xj denote reach direction j. The set U comprised {0, 1, 2, 3, 4, 5+} spike counts, where any spike

counts greater than or equal to 5 were counted towards the same bin (‘5+‘, corresponding to an

instantaneous firing rate of 250 Hz in a 20 ms bin). We calculated the entropy of each electrode,

H(U) = −∑u∈U

= p(u) log p(u), (2.5.9)

as well as its entropy conditioned on the reach direction,

H(U |X) = −M∑j=1

p(xj)∑u∈U

p(u|xj) log p(u|xj). (2.5.10)


From these quantities, we calculated the mutual information between the neural activity and the

reach direction as Idrop(X;U) = H(U)−H(U |X). We dropped electrodes in order from highest to

lowest mutual information.

2.5.9 Principal angles of neural subspaces analysis

For a parsimonious scalar metric of how similar patterns of neural activity during reaching were

between a given pair of recording days (used in Figure 2.3), I calculated the minimum principal angle

between the neural subspaces of each recording day. I defined the neural subspace on a recording

day as the top K principal components of the neural coactivations. Put more simply, I asked how

similar day i and day j’s motifs of covariance between electrodes’ activity were during arm reaching.

Specifically, I started with a matrix Ui from each day i consisting of neural activity collected while

the monkey performed 200 trials of a Radial 8 Task (8 cm distance to targets) using arm control;

this task has been run at the start of almost every experimental session conducted using both

monkeys R and L since array implantation. Ui is of dimensionality E × T , where E is the number of

electrodes and T is the number of non-overlapping 20 ms bins comprising the duration of this task. I

next subtracted from each row of Ui that electrode’s across-days mean firing rate (I also repeated

this analysis without across-days mean subtraction and observed qualitatively similar results, not

shown). To obtain the principal components, I performed eigenvalue decomposition on the covariance

matrix UiUTi (note, Ui is zero mean), and defined the matrix Vi as the first K eigenvectors. Vi had

dimensions I ×K, where each column k is the vector of principal component coefficients (eigenvector)

corresponding to the kth largest eigenvalue of the decomposition. Figure 2.3 was generated using

K = 10, i.e., keeping the first 10 PCs, but the qualitative appearance of the data was similar when

K was varied from 2 to 30 (not shown). Finally, the difference metric between days i and j was

computed as the minimum of the K subspace angles between matrices Vi and Vj . Subspace angles

were computed using the subspacea MATLAB function (Knyazev & Argentati, 2002).

2.6 Project acknowledgements

I thank Chethan Pandarinath and the members of the Shenoy group for very helpful discussions;

Mackenzie Mazariegos, Michelle Wechsler, John Aguayo, Clare Sherman, Erica Morgan, and Liana

Yates for surgical assistance and veterinary care; Beverly Davis, Sandy Eisensee, and Evelyn Castaneda

for administrative support; Boris Oskotsky for information technology support.

This work was supported by the National Science Foundation Graduate Research Fellowship

(J.C.K., S.D.S.); NSF IGERT 0734683 (S.D.S.); Christopher and Dana Reeve Paralysis Foundation

(S.I.R. and K.V.S.); and the following to K.V.S.: Burroughs Welcome Fund Career Awards in the

Biomedical Sciences, Defense Advanced Research Projects Agency Reorganization and Plasticity

to Accelerate Injury Recovery N66001-10-C-2010, US National Institutes of Health Institute of


Neurological Disorders and Stroke Transformative Research Award R01NS076460, US National

Institutes of Health Director’s Pioneer Award 8DP1HD075623-04, US National Institutes of Health

Director’s Transformative Research Award (TR01) from the NIMH 5R01MH09964703, and Defense

Advanced Research Projects Agency NeuroFAST award from BTO W911NF-14-2-0013.

Chapter 3

A high performing brain-machine

interface driven by low-frequency

local field potentials alone and

together with spikes

3.1 Summary

Current high performance BMIs are driven by action potentials (spikes), but access to this signal often

diminishes as sensors degrade over time. Decoding local field potentials (LFPs) as an alternative or

complementary BMI control signal may improve performance when there is a paucity of spike signals.

Only a small handful of LFP decoding methods have been tested online; there remains a need to test

different LFP decoding approaches and improve LFP-driven performance. Until this project, there

had also not been a reported demonstration of a hybrid BMI that decodes kinematics from both

LFP and spikes. We recorded spikes and LFP from two macaques implanted with multielectrode

arrays in primary and premotor cortex while they performed a reaching task. I first evaluated a

BMI driven by the local motor potential (LMP), a low-pass filtered time-domain LFP amplitude

feature. I then combined decoding of both LMP and spikes to implement a hybrid BMI. We compared

closed-loop BMI control using biomimetic decoders driven by LMP, spikes, or both signals together.

LMP decoding enabled quick and accurate cursor control which surpassed previously reported LFP

BMI performance. Hybrid decoding of both spikes and LMP improved performance when spikes

signal quality was mediocre to poor. These findings show that LMP is an effective BMI control signal

which requires minimal power to extract and can substitute for or augment impoverished spikes

34

CHAPTER 3. ROBUSTNESS THROUGH DECODING LOCAL FIELD POTENTIALS 35

signals. Use of this signal may lengthen the useful lifespan of BMIs and is therefore an important

step towards clinically viable BMIs.

3.2 Introduction

To date, the highest-performing BMIs have been driven by spike activity recorded with intracortical

multielectrode arrays; these have enabled non-human primates to accurately control computer cursors

(Serruya et al., 2002; Ganguly & Carmena, 2009; Gilja et al., 2012), robotic limbs (Carmena et al.,

2003; Velliste et al., 2008), or the subject’s own musculature (Ethier et al., 2012). This research is

now being translated to early-stage clinical studies in people with paralysis (Hochberg et al., 2012;

Collinger et al., 2013b; Bacher et al., 2015). A critical challenge that must be overcome to enable

clinically viable BMIs is to improve the device’s useful lifespan by maintaining high performance over

a long period of time, thereby improving its risk-benefit balance (Ryu & Shenoy, 2009). Chronically

implanted sensors often degrade over time and gradually lose their ability to record action potentials

(Kruger et al., 2010; Simeral et al., 2011; Chestek et al., 2011; Barrese et al., 2013). One approach to

mitigate this is to decode multiunit spikes instead of well-isolated single unit activity (Fraser et al.,

2009; Chestek et al., 2011; Gilja et al., 2012; Hochberg et al., 2012; Nuyujukian et al., 2014). An

alternative — or complementary — strategy is to make use of neural signals other than spikes that

contain information about movement intention and are available from the same sensors. The local

field potential (LFP) is such a signal: it is obtained by low-pass filtering the same raw voltage signal

from which spikes are high-pass filtered, and it carries information about the kinematics of planned

(O’Leary & Hatsopoulos, 2006) and executed (Mehring et al., 2003; Rickert et al., 2005; Scherberger

et al., 2005; O’Leary & Hatsopoulos, 2006; Bansal et al., 2011a) reaching movements.

Despite many offline LFP decoding studies (Pesaran et al., 2002; Mehring et al., 2003; Rickert

et al., 2005; Scherberger et al., 2005; O’Leary & Hatsopoulos, 2006; Asher et al., 2007; Markowitz

et al., 2011; Bansal et al., 2011a; Flint et al., 2012a,b; Zhuang et al., 2010; Hwang & Andersen, 2013;

Wang et al., 2014; Perge et al., 2014) and a few closed-loop demonstrations (Kennedy et al., 2004;

Hwang & Andersen, 2009), only very recently have there been reports of effective closed-loop cursor

control driven by LFP (Flint et al., 2013; So et al., 2014). While encouraging, the performance

achieved by these first forays into LFP-driven BMIs is low compared to spike-driven performance and

leaves open the question of how viable LFP is as an alternative and complimentary control signal.

My study aimed to address this gap by showing considerably improved LFP-driven performance and

by showing, for the first time, that under certain conditions LFP can be beneficially combined with

available spikes to improve closed-loop BMI control. LFP can be processed into a variety of different

features, and these two previous studies made specific design choices to decode LFP power in multiple

frequency bands as well as — in (Flint et al., 2013) — low-frequency LFP amplitude known as local

motor potential (‘LMP’, which has also been referred to as movement-evoked potential in earlier


studies). Closed-loop performance using other choices of LFP features had yet to be characterized.

Based on the results of my own offline evaluation of decoder performance using various LFP features,

I identified that the LMP was the best candidate feature. We subsequently evaluated the closed-loop

performance of a BMI design that differs from that of previous studies by only extracting from each

electrode a half-wave rectified variant of the LMP, rather than the unrectified LMP or LFP power

in various frequency bands. Two macaques successfully used this LMP-driven BMI to perform a

2D target acquisition task and, to my knowledge, demonstrated higher performance than in any

previously reported online LFP BMI study.

After showing that this LMP feature is an effective alternative BMI control signal, I set out to

test whether combining LMP and spikes decoding could improve closed-loop performance. LFP

reflects a spatial averaging of synaptic and other currents in the vicinity of the electrode and carries

information that is distinct from spikes (Belitski et al., 2008; Katzner et al., 2009; Buzsaki et al.,

2012; Einevoll et al., 2013). Furthermore, LFP can be informative even on electrodes which do not

record spikes (Bokil et al., 2006; Heldman et al., 2006; Flint et al., 2012b; Hwang & Andersen, 2013;

Wang et al., 2014). For these reasons it has long been hoped that decoding LFPs could augment

BMIs, especially in cases where few electrodes have spikes available. I demonstrated the first use of a

hybrid LFP and spikes decoder for continuous BMI cursor control and showed that this approach

can indeed improve performance compared to spikes-only decoding. The benefit of hybrid decoding

increased as we reduced the number of electrodes with informative spike activity. However, we also

found that in our second monkey, who had more electrodes recording tuned spikes activity and worse

overall LMP-driven performance, hybrid decoding was only beneficial if spikes were made unavailable

from the majority of electrodes.

3.3 Results

3.3.1 Selection of local motor potential as a control signal based on offline

evaluation of candidate LFP features

In order to choose which LFP-derived feature(s) to evaluate as the control signal for closed-loop BMI

experiments, I first tested a number of neural features offline to determine how well the velocity

of point to point arm reaches could be decoded from each feature. I compared spike counts in

non-overlapping 50 ms bins, the local motor potential (‘LMP’, i.e. low-pass filtered LFP amplitude),

and LFP power in the 1-4 Hz, 3-10 Hz, 12-23 Hz, 27-38 Hz, and 50-300 Hz bands. I also tested

decode performance using combinations of spike counts and one or more LFP-derived features. For

this analysis I used five datasets of the Radial 8 Task from each of two adult male rhesus macaques.

A velocity Kalman filter was trained to decode 2D hand endpoint velocity from each type of neural

feature or combination of features. Neural data came from 192-electrode recordings from contralateral

M1 and PMd. Half of each dataset’s 500 trials were used for training the decoder and the remaining


d

cb

a

Analog Filter 0.3 to 7500 Hz bandpass

digital filter500 Hz lowpass

0 1

-250

0

150

time (s)

Vo

lta

ge

(µ

v)

Raw Neural Signal

Decode

Velocity

a

time (s)-0.3 0 1.1

-80

0

80

0

80

vo

lta

ge

(µ

V)

vo

lta

ge

(µ

V)

0°45°90°

135°180°225°270°315° Half-Wave Rectified LMP

50 ms boxcar filterExtract LMP

LMP

vo

lta

ge

(µ

V)

0 1

-150

0

150

time (s)

LFP

LMP

Figure 3.1: Overview of local motor potential (LMP) BMI signal processing(a) A macaque sat in a primate chair and observed a cursor and target in a virtual-reality environment.During arm control, he controlled the cursor position by reaching with his unrestrained arm. DuringBMI control, the cursor was driven by a velocity command decoded from neural activity.(b) Neural signals were recorded from 96-electrode arrays in M1 and PMd. Electrical signals wereamplified, analog filtered from 0.3 to 7500 Hz, and digitized. A 1 s sample of this raw signal is shown;both lower frequency components and high frequency multiunit spikes are visible.(c) The LFP (shown in grey) was then obtained by low-pass filtering the raw digital signal below 500Hz. LMP, shown in black, was extracted by boxcar filtering the most recent 50 ms of LFP.(d) (Top) Example LMPs trial-averaged over 32 arm reaches to each of 8 peripheral targets. Tracecolor corresponds to reaches to targets in the specified direction. The target was presented at time =0. (Bottom) Before decoding, LMP was half-wave rectified: negative voltage samples were rectifiedwhile positive voltage samples were set to 0. Dataset R.2013.07.30, electrode 1 (M1 array).


half were used to test how accurately the decoder reconstructed the true hand velocity. Kinematics

and neural features were binned every 50 ms, and decode performance using different lags (i.e., neural

data aligned to or leading kinematics) was evaluated; I report how accurately the testing data was

decoded using the best lag for each particular feature as fit from the training data.

Figure 3.2 shows the average of the Pearson’s linear correlation coefficients between true and

reconstructed x- and y-velocities using each neural feature(s). I found that LMP outperformed the

various LFP power features, and was the only LFP-derived feature to match or exceed the offline

decoding accuracy of spikes (monkey R spikes r = 0.73± 0.022 [mean ± standard deviation], LMP

r = 0.78± 0.010, p = 0.007 [paired t-test for a difference in means]; monkey J spikes r = 0.80± 0.032,

LMP r = 0.77± 0.005, p = 0.065). Combining LMP with spikes improved offline decoding accuracy

in both animals compared to decoding spikes alone (monkey R ∆r = 0.11, p < 0.001; monkey

J ∆r = 0.054, p = 0.003). Although decoding both gamma LFP power (50-300 Hz) and spikes

together also led to a small increase in accuracy, this effect was smaller than when combing LMP

and spikes. Furthermore, adding both LMP and gamma to spikes did not substantially improve

the decode accuracy over just using spikes + LMP (monkey R ∆r = 0.0026, p = 0.034; monkey J

∆r = 0.0002, p = 0.78). LMP was also a better offline decode feature than signal amplitude in the

other bands, or acausally filtered power in the other bands (Figure 3.3).

Velocity could be decoded from LMP because this signal evolved differently over time depending

on which direction the animal was reaching. This can be seen in Figure 3.1d, which shows larger

amplitude LMP deflections during reaches towards a subset of the 8 targets on an example M1

electrode. Across electrodes and monkeys I observed a variety of LMP waveforms in terms of shape,

number of peaks, and tuning for velocity (not shown). Figure 3.4 shows that the LMP recorded on

most electrodes was significantly tuned for reach direction, and that the preferred directions (PDs) of

LMP recorded on nearby electrodes tended to have similar tuning. Despite these local correlations,

across all electrodes the PDs spanned both dimensions required for 2D cursor control. I wish to

point out that monkey R’s M1 array is an example of an array that records very poor spikes signals

(offline decode accuracy of r = 0.11± 0.042 [mean ± standard deviation] using just this array’s spikes

to perform the same analysis as in Figure 3.2), but has significant LMP tuning on a number of

electrodes. When using LMP measured only from this array, I observed an improved offline decode

accuracy of r = 0.46± 0.029. Based on these offline results I decided to evaluate LMP alone and

spikes + LMP together as control signals for a closed-loop BMI.

3.3.2 Control experiment showing LMP is not a movement artifact

Prior to conducting closed-loop experiments, I first wanted to convince myself that the LMP we

were decoding was not artifactually correlated with the monkey’s arm movement due to factors such

as microphonic pickup of electronic noise in the room (despite heavily electromagnetically shielded

walls) or mechanical vibration of the recording equipment (despite the ruggedly built experimental


*** ***

*** ***

***

*** **

*** **

0

0.2

0.4

0.6

0.8

1 Monkey J

spikes

LMP +

50−

300H

zLM

P

power

1−4

Hz

power

3−1

0Hz

power

12−

23Hz

power

27−

38Hz

power

50−

300H

z

LMP

power

1−4

Hz

power

3−1

0Hz

power

12−

23Hz

power

27−

38Hz

power

50−

300H

z

spikes +

**

*** ******

***

***

** ****

***

0

0.2

0.4

0.6

0.8

1

***

spikes

LMP +

50−

300H

zLM

P

power

1−4

Hz

power

3−1

0Hz

power

12−

23Hz

power

27−

38Hz

power

50−

300H

z

LMP

power

1−4

Hz

power

3−1

0Hz

power

12−

23Hz

power

27−

38Hz

power

50−

300H

z

spikes +

Monkey R

*

De

co

de

Accu

racy (

r)

1 2 3 4 5 6 7 8 9 10

0

0.1

0.2

0.3

Frequency (Hz)

No

rma

lize

d V

olta

ge example monkey R M1 electrode

example monkey J PMd electrode

a

c

0 20 40 60 80

−40

−30

−20

−10

0

Frequency (Hz)

Ma

gn

itu

de

(d

B)

LMP

1-4 Hz 3-10 Hz 12-23 Hz 27-38 Hz 50-300 Hzb

Figure 3.2: Comparison of offline decode accuracy using different neural features(a) Two-dimensional hand velocity from Radial 8 Task reaches was reconstructed offline with aKalman filter that decoded either multiunit threshold crossing spikes (red), various LFP features(blue), or the combination of spikes and one (violet) or two (striped) LFP features. The Pearsoncorrelation (r) between true and decoded velocity was averaged across all trials and both x- and y-dimensions. We show the grand mean and standard error of measurement across five datasets foreach monkey. Stars denote signals or combinations of signals whose decode performance differedsignificantly from that of decoding spikes only (paired t-test, * p < 0.05, ** p < 0.01, *** p < 0.001).(b) Amplitude response of the filter cascade used to generate each of the evaluated LFP frequencybands. The periodic dip in the response of LMP is due to the 50 ms boxcar filtering operation. Thehigh frequency cutoff of the 50-300 Hz filter extends beyond the panel; it is flat until 200 Hz androlls off to -4.7 dB at 300 Hz. (c) To illustrate the actual spectral content of the half-wave rectifiedLMP signal, we have plotted the frequency spectra of this feature during closed-loop use of the LMPdecoder from a representative electrode from each monkey. Datasets R.2013.09.09 electrode 1 andJ.2013.09.11 electrode 122.


monkey R monkey J

0

0.2

0.4

0.6

0.8

1

acausal pow

er 3-1

0Hz

De

co

de

Accu

racy (

r)

spike

sLM

P

|LM

P|

power 1−4Hz

bandpass 1−4Hz

power 3−10Hz

bandpass 3-1

0Hz

power 12-2

3Hz

bandpass 1

2-23Hz

power 27-3

8Hz

bandpass 27-3

8Hz

power 50-3

00Hz

bandpass 50-3

00Hz

spike

sLM

P

|LM

P|

power 1−4Hz

bandpass 1−4Hz

power 3−10Hz

bandpass 3-1

0Hz

power 12-2

3Hz

bandpass 1

2-23Hz

power 27-3

8Hz

bandpass 27-3

8Hz

power 50-3

00Hz

bandpass 50-3

00Hz 0

0.2

0.4

0.6

0.8

1

acausal pow

er 3-1

0Hz

Figure 3.3: Offline decode analysis of additional candidate LFP featuresOffline decode analysis of additional candidate LFP features. Solid bars are the same features asin Figure 3.2a, plotted here for comparison; outlined bars with bolded labels show offline decodeaccuracy using additional features. The offline decode was performed using the same data andmethods as in Figure 3.2. None of these additional features performed as well as the LMP feature.‘|LMP|’ is the absolute value of the LMP signal; note that this is not the same as the half-waverectified LMP, which had similar offline decode accuracy to the LMP results shown here. The ‘acausalpower 3-10Hz’ feature was computed by filtering the signal both forwards and backwards in timeto yield a zero phase delay filter. While this improved offline decode accuracy, we note that thisacausal filter would be impractical for closed-loop decoding because of the required delay introducedinto the control loop. We also evaluated the other frequency bands’ power after filtering in thisacausal manner (including the LMP), and found that these features’ offline decode accuracy (notshown) were very similar to their causally filtered counterparts. The ‘bandpass’ features are thetime-domain signal amplitude (sampled every 50 ms) after filtering the LFP using the same filter asthe corresponding band power feature.


Cen

tral

Sul

cus

M1

Precentral

DimplePMd

Central Sulcus

M1

PMd

Precentral

Dimple

M

L

A P

4 mm

Monkey R Monkey J

Figure 3.4: Directional tuning of LMP overlaid on electrode location in motor cortexThe locations of the two arrays in each monkey are drawn along with major anatomical landmarksbased on intraoperative surgical photographs. Each dot with a colored vector coming out of itcorresponds to an electrode and the preferred direction (PD) of the LMP feature recorded on thatelectrode, as computed from 250 center out arm reaches. Both the vector’s color and angle correspondto PD, while the vector length signifies 1− (p value) of tuning (ANOVA). Electrodes that were notsignificantly tuned (p > 0.01) are shown in greyed out color. Note that the PDs shown are in thecoordinate system of the two-dimensional task and are unrelated to the anatomical coordinate systemused to map the physical location of each electrode in the brain. PMd = dorsal premotor cortex, M1= primary motor cortex, A = anterior, P = posterior, L = lateral, M = medial. Monkey R’s M1array quality was poor both for LMP and especially for spikes even from the earliest post-implantrecordings. Datasets R.2013.09.09 J.2013.09.11.


apparatus being connected to massive load-bearing metal walls via structural metal). Both of these

noise sources could appear to be “tuned” to reach kinematics because arm movements can cause

very small but specific movements of the experimental apparatus. We therefore conducted a control

experiment in which similar arm movements were made without a corresponding neural movement

intention. We sedated the monkey with ketamine and dexmedetomidine, gently placed him in

the primate chair, and recorded neural signals while the monkey’s hand was moved by a human

experimenter to complete the same Radial 8 Task. Mean hand velocity reconstruction accuracy using

LMP dropped from r = 0.82 (monkey R), r = 0.79 (monkey J) when the animals were awake and

making volitional reaches, to r = 0.05 (monkey R) and r = 0.01 (monkey J) using the sedated control

data collected later the same day. This indicates that the LMP signal is not an artifact related to

the monkey’s arm movements.

3.3.3 Closed-loop BMI performance driven by LMP

Both monkeys demonstrated high performance using an LMP-driven BMI. For each experimental

session, we trained a velocity Kalman filter using LMP from 500 arm reaches. I evaluated performance

by measuring the success rate, normalized time to target, and path length ratio on a Continuous

Random Target Task in which 5× 5 cm targets appeared anywhere in a 20× 20 cm workspace and

had to be acquired within 8 seconds (Figure 3.5). Monkey R did the task at a 99% success rate

(11,429 trials across 6 experiment sessions) with 0.08 s/cm median normalized time to target and

median path length ratio of 2.09. Monkey J had an 86% success rate using the LMP decoder across

2,065 trials over 8 sessions, with median normalized time to target of 0.095 s/cm and path length

ratio 2.47. Chance performance levels were 21% success rate, 0.33 s/cm time to target for monkey

R, and 34% success rate, 0.31 s/cm time to target for monkey J. Note that I had given monkey J

an easier task (300 ms target hold time compared to monkey R’s 500 ms hold time) to keep him

engaged in the task because he had worse LMP-driven BMI performance.

Because there have only been a few published online LFP BMI studies, we were able to also

measure performance on tasks closely matched to those used in two prior reports to allow for a more

direct comparison. When tested on a Radial 8 Task similar to that of (So et al., 2014), monkeys R

and J achieved considerably higher success rates: 98% for monkey R and 85% for monkey J (Figure

3.6), compared to 78% and 72% success rates reported in that study. In terms of successful target

acquisitions per minute, monkey R’s 16.1 targets/min and monkey J’s 9.8 targets/min compare

favorably to the 10.6 and 5.8 targets/min in (So et al., 2014). We also had both monkeys perform a

Continuous Random Target Task with smaller targets and 100 ms hold times as in (Flint et al., 2013)

(Figure 3.7). Monkey R performed this task with 79% grand mean success rate, normalized time to

target = 0.070 s/cm, index of performance = 2.71 bits/s, and path length ratio = 1.53. Monkey J’s

metrics were: 73% success rate, normalized time to target = 0.090 s/cm, index of performance =

2.16 bits/s, and path length ratio = 1.79. Compared to (Flint et al., 2013), this represents a slightly


TrialTrial

Su

cce

ss

Ra

teN

TT

T (

s/c

m)

Pa

th L

en

gth

(cm

/cm

)

a

b

c

Monkey R Monkey J

0.86

1 221 490 765 1041 1325 1605 1876 2065 0 0.2 0.4 0.6 0.8 1

1 221 490 765 1041 1325 1605 1876 2065 0 0.1 0.2 0.3 0.4 0.5

1 221 490 765 1041 1325 1605 1876 2065 1 2 3 4 5 6

2.47

0.1530.095

0.99

1 2702 6622 7891 9057 10155 11429 0 0.2 0.4 0.6 0.8 1

1 2702 6622 7891 9057 10155 11429 0 0.1 0.2 0.3 0.4 0.5

1 2702 6622 7891 9057 10155 11429 1 2 3 4 5 6

2.46

0.0810.0810.039

1.18

0.0380.038

1.28

Figure 3.5: LMP-driven BMI performance on the Continuous Random Target TaskEach performance metric is smoothed over the previous 50 trials and is shown across six and eightdays’ experiments for monkeys R and J, respectively. Vertical gray lines separate trials from differentdays. Grand mean (success rate) or median (NTTT and path length) performance for each plot ismarked with a black arrow and value. Dashed gray lines and gray numbers show performance onthis task using hand control.(a) Success rate is the fraction of trials where the target was acquired within 8 seconds.(b) Normalized time to target measures how long it took to successfully acquire the target, normalizedby the straight-line distance between the current and previous targets. The final hold time is notincluded in NTTT.(c) Path length ratio measures the distance the cursor travelled to acquire the target divided by thestraight-line distance; closer to 1 is more direct (better).


Monkey R

Pe

rce

nt S

ucce

ssfu

l

trial

98.3% overall successful

1 203 458 763 1066 1371 1679

0

100

Monkey J

Pe

rce

nt S

ucce

ssfu

l

trial

85.1% overall sucessful

1 170 385 557

0

1000 4 8

0

0.27

Time to Target (s)

Pro

po

rtio

n o

f tr

ials

Monkey R

0 4 8

0

0.30

Time to Target (s)

Monkey J

Pro

po

rtio

n o

f tr

ials

a b

Figure 3.6: Performance of the LMP decoder on a Radial 8 TaskTargets were pseudorandomly positioned at one of 8 locations evenly spaced along a 13 cm diametercircle. The monkey initiated the trial by first acquiring a central target, after which the targetjumped to a peripheral location. To successfully acquire the target, the cursor had to be movedwithin an acquisition region of width 3.4 cm and held there for a contiguous 300 ms. (a) Runningmean of success rate for the previous 50 trials across 6 days for monkey R and 3 days for monkey J.Vertical grey lines mark transitions between experiment days. Overall percent successful across allexperimental sessions’ trials is shown with a black arrow. (b) Histograms showing the distributionof times to target, also known as reach times, for all trials of this task. The 300 ms hold time is notincluded in this calculation of time to target.

better success rate and over twice as fast target acquisition times. Note that these success rates are

lower than in the standard Continuous Random Target Task because I posthoc counted as failures

those trials in which the cursor exited the target and then successfully reacquired it (which were

successful as far as the monkey was concerned). I did this to be consistent with the methods used by

Flint and colleagues; otherwise, the success rates would have been 99% and 86% for monkeys R and

J, respectively.

Since we evaluated biomimetic decoders trained using arm reaching neural activity, we did not

restrain the monkey’s contralateral (with respect to the arrays) arm during BMI use. I observed that

both monkeys continued to move their arms during LMP-driven cursor control. The monkeys were

not required to keep the reflective bead taped to their fingers visible to the tracking system when

using the BMI; consequently, I was only able to quantify the relationship between hand and cursor

movement for 78% (monkey R) and 53% (monkey J) of LMP-driven time samples. During these

epochs, the grand mean correlation between hand and LMP-driven cursor velocities was r = 0.736

(p < 0.001) for monkey R and r = 0.668 (p < 0.001) for monkey J (averaging across x- and y- velocity,

and across all Continuous Random Target Task sessions).


a

b

c

d

Monkey R Monkey J

1 2 3

0

0.2

0.4

0.6

0.8

1

1 2 3

0

0.1

0.2

0.3

0.4

0.5

0.6

1 2 3

0

1

2

3

1 2 3

0

1

2

3

4

5

6

Su

cce

ss R

ate

1 2 3 4 5 6

0

0.2

0.4

0.6

0.8

1

NT

TT

(s/c

m)

1 2 3 4 5 6

0

0.1

0.2

0.3

0.4

0.5

0.6

IP (

bits/s

ec)

1 2 3 4 5 6

0

1

2

3

Pa

th L

en

gth

(cm

/cm

)

1 2 3 4 5 6

0

1

2

3

4

5

6

DayDay

Figure 3.7: Performance on aContinuous Random Target Task withshort hold timeTargets randomly appeared anywhere in a20× 20 cm region. The monkey had up to 10seconds to move the cursor into a 4× 4 cmtarget area and hold it there for 100 ms. Tobe consistent with the metrics used in (Flintet al., 2013), we restricted our definition of asuccessful trial to those where the cursorstayed within the target for 100 ms the firsttime they entered the target region; that is,trials with ‘dial-in’ (leaving the target within100 ms and then reacquiring it) were countedas failures. Note that this is a posthocrestriction of what constitutes a successfultrial, and that our monkeys were trained toexpect that dial-in is allowed and performedthe task under this reward contingency.(a) Mean success rate for each day when thistask was run is shown in filled circles. Opencircles shown the success rate had wecounted trials where the target was acquiredafter dial-in as successful.(b) Normalized time to target.(c) IP = Index of Performance, also knownas Fitts bits per second.(d) Path length ratio.


3.3.4 Fixed LMP decoders work across multiple days without retraining

We also evaluated how well a fixed LMP decoder performed without any retraining for just over

three weeks. A decoder was trained on day 1 of these experiments and then loaded and evaluated for

at least 200 trials at the beginning of subsequent days’ experimental sessions. No recalibration or

re-normalization was done; the decoder weights and entire signal processing pipeline was identical

between collection of the training dataset on day 1 and during evaluation of the fixed decoder. The

fixed LMP decoder enabled both monkeys to perform the Continuous Random Target Task across the

entire tested period (Figure 3.8a-d), although with worse performance than using same day-trained

decoders. Monkey R’s daily success rates using the day 1 decoder on subsequent days ranged from

90.4% to 99.8%, with an accuracy of 93.0% on the last day tested (day 22), while monkey J’s daily

success rate ranged from 69.8% to 86.4%, and was 72.7% on day 23.

To investigate whether the ability to use a fixed decoder over several weeks was due to stable

LMP signals or the animal adapting to a potentially mismatched decoder during closed-loop BMI use,

I compared the fixed decoder’s closed-loop performance with its offline decode accuracy. I performed

an offline decode of hand velocities recorded during Radial 8 Task arm reaches at the start of each of

these experimental sessions using either the fixed day 1 decoder or a decoder trained from that same

day’s reaches. The fixed day 1 decoder was only slightly worse than the same day decoder (Figure

3.8e), suggesting that the LMP signals were largely stable across days. For monkey J, who exhibited

a wider divergence in BMI performance between the fixed and same-day decoders, closed-loop day

1 decoder success rates correlated with the day 1 decoder’s offline accuracy (r = 0.79, p < 0.05,

Pearson correlation). For monkey R, who had little variation in fixed decoder offline accuracy, this

regression was not significant against any of the closed-loop performance metrics.

A potential concern is that the monkey’s familiarity with the fixed decoder could have interfered

with his ability to control new decoders provided on subsequent days. Aside from a brief pause of

the task, no explicit cue was provided to indicate that the decoder had been switched. Nonetheless,

we found that the monkeys rapidly adjusted to each day’s newly trained decoder after switching

from the fixed decoder. The data shown in Figure 3.5 begin following a brief (10-20 trial) “transition”

period after the switch from the fixed decoder, and do not show within-session improvement that

would indicate overcoming interference from the first decoder.

3.3.5 BMI control using a hybrid spikes and LMP decoder

We next evaluated whether simultaneously decoding both LMP and spikes would improve performance

compared to a BMI driven by spikes alone. Figure 3.9 shows aggregated data from eight datasets

in monkey R and five datasets in monkey J where the hybrid decoder and spikes-only decoders

were evaluated in an interleaved ‘ABAB’ block format. When LMP features were added to spikes,

the hybrid decoder outperformed the spikes-only decoder in monkey R (12% improvement in mean

normalized time to target averaged over all datasets, p < 0.001; difference significant at p < 0.01


1 2 3 7 8 9 10 23

0

0.2

0.4

0.6

0.8

1

1 2 3 7 8 9 10 23

0

0.1

0.2

0.3

0.4

0.5

1 2 3 7 8 9 10 23 0

1

2

1 2 3 7 8 9 10 23

1

2

3

4

5

6

1 2 3 4 5 6 10 11 16 20 22

0

0.2

0.4

0.6

0.8

1

1 2 3 4 5 6 10 11 16 20 22

0

0.1

0.2

0.3

0.4

0.5

1 2 3 4 5 6 10 11 16 20 22

0

1

2

1 2 3 4 5 6 10 11 16 20 22

1

2

3

4

5

6

1 2 3 7 8 9 10 231 2 3 4 5 6 10 11 16 20 22

Su

cce

ss

Ra

teN

TT

T

(s/c

m)

IP

(bits/s

ec)

Pa

th L

en

gth

(cm

/cm

)

a

b

c

d

Monkey R Monkey J

Experiment Day Experiment Day

trained same daytrained day 1

e

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

Offline

De

co

de

Accu

acy (

r)

Figure 3.8: Stable performance using a fixed LMP decoderClosed-loop performance of a fixed LMP decoder trained on day 1 of the experiment and thenevaluated on subsequent days. Each gray point shows performance of this fixed decoder for eachsubsequent day’s experiment. On some days we also trained a new decoder from arm reaches; theperformance of this same-day trained decoder is shown in black. On days 2-6 for monkey R thesame day and day 1 decoders performed similarly, such that the black points obscure the gray points.Same day performance corresponds to the data shown in Figure 3.5. (a) Mean success rate. (b)Median normalized time to target. (c) Median Index of Performance (IP), also known as Fittsbits per second. Faster acquisition times and/or accomplishing a harder task yields higher IP. (d)Median path length ratio. (e) Offline decode accuracy when predicting each day’s arm reachinghand velocities from LMP using either the fixed day 1 decoder (gray) or a decoder trained from thatday’s data with 10-fold cross-validation (black)


in 5/8 individual sessions). In monkey J, however, the hybrid decoder was slightly worse than the

spikes-only decoder (8% slower normalized times to target, p = 0.007, difference not significant at

p < 0.01 in any individual session).

To provide context for these results, I also compared each monkey’s spikes-only and hybrid

decoding performance to his performance on the same task using either his hand (3 and 4 datasets

from monkey R and J, respectively) or the LMP-only decoder (9 and 7 datasets from monkey R and

J, respectively); these data came from different experimental sessions. Figure 3.9 shows that in both

monkeys, BMI performance was better using a spikes-only decoder than an LMP-only decoder. This

difference was much larger for monkey J (298% faster normalized times to target) than monkey R

(51% faster). Using an LMP-only decoder, both monkeys were slower to initially reach the target

and were worse at holding the cursor inside the target acquisition area for the requisite hold time.

Monkey R’s mean time to initial target entry was 50% slower for LMP than spikes (937 ms vs. 624

ms) and his dial-in time was 61% slower (LMP: 672 ms, spikes: 417 ms). Monkey J’s initial target

entry was 273% slower for LMP than spikes (1789 ms vs. 479 ms), and his dial-in was 313% slower

(LMP: 517 ms, spikes: 125 ms). All of these differences were significant at p < 0.001 (two-sided

t-test). I also note that monkey J had very good spike-driven performance, acquiring targets only

18% slower than with his hand.

3.3.6 LMP decoding has a longer control loop latency than spikes decoding

To better understand why closed-loop hybrid decoding only modestly helped in monkey R and did

not help in monkey J, I compared how spikes and LMP each contributed to the hybrid decoder both

during online BMI control and during offline decoding of neural data recorded while the monkey did

the same Continuous Random Target Task with his hand. At each decode update time step, spike

counts and LMP features from all electrodes were input to the decoder to generate the instantaneous

velocity component (‘neural push’) due to each signal (Figure 3.10a). In both monkeys, LMP

contributed more than spikes to the BMI cursor’s movement, accounting for 67% of the decoded

velocity in monkey R and 54% in monkey J (Figure 3.10b). This ruled out the possibility that LMP

was simply not being substantially used in the hybrid decoder. However, the online decode velocity

component from LMP pointed less directly towards the target than the spikes’ velocity component

(Figure 3.10c). Note that one should not interpret monkey J’s smaller absolute LMP angular error

to mean that his LMP signal was more accurate than monkey R’s; rather, monkey J’s more direct

hybrid-driven cursor trajectories (even if they were straighter largely due to better spikes decoding)

also brought down the LMP’s absolute errors. These online results contrast with their offline decode

counterparts. Offline, decoded LMP aimed the cursor more accurately towards the target than

decoded spikes and, not surprisingly, was weighted heavily by the decoder (Figure 3.10e,f).

This observation that the LMP aimed the decoded velocity more accurately than spikes towards

the target during hand reaches, which are direct and precise, but was less accurate than spikes during


a

b

d

c

1.4

4

1.6

5

5.6

7

1.6

8

1.7

0

2.6

3

4.0

3

2.1

7

0

1

3

6

1.0

0

1.0

0

0.9

9

1.0

0

***

Su

cce

ss R

ate

0

1

NT

TT

(s/c

m)

Pa

th L

en

gth

(cm

/cm

)

hand spikes LMP hybrid

Monkey R Monkey J

1.0

0

1.0

0

0.8

5

1.0

0

*** 1

hand spikes LMP hybrid

5

4

2

0

1

3

6

5

4

2

0.0

45

0.0

53

0.2

09

0.0

57

0

0.1

0.2

0.0

43

0.0

94

0.1

41

0.0

82

0

0.1

0.2

0

25

12

5

51

7

10

1

250

500

750

0

40

41

7

67

2

24

8

0

250

500

750

Dia

l-in

Tim

e (

ms)

Figure 3.9: Closed-loopcomparisons of different neuralcontrol signalsComparison of spikes-only,LMP-only, and hybrid spikes + LMPdecoders. Mean and SEM of eachperformance metric was computedacross trials from multiple datasetsin which the Continuous RandomTarget Task was performed witheither the hand (gray), or a BMIdriven by spikes (red), LMP (blue),or hybrid spikes + LMP (violet).(a) Success rate. The LMP decoderwas significantly worse than othercontrol modes (p < 0.001).(b) Normalized time to target. Alldifferences significant at p < 0.01.(c) Dial in time. All differencessignificant at p < 0.05.(d) Path length ratio. All differencessignificant at p < 0.05 exceptmonkey J spikes vs. hybrid.


***

0

10

20

30

40

online

err

or

ang

le (d

eg

rees)

spikes

LMP

***

spikes

LMP

0 80

−60

120

Y Position (mm)

Y P

ositio

n (

mm

)a b Monkey R Monkey J Monkey R Monkey J

d

c

e f

***

spikes

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Fra

ctio

n D

eco

de C

ontr

ibutio

n O

nline

***p < 0.001

LMP

spikes

LMP

50ms

100ms

250 0 −200Lag (ms)

50ms

100ms

250 0 −200Lag (ms)

20

90

Err

or

An

gle

(D

eg

ree

s)

Monkey R Monkey J

0

10

20

30

40

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Fra

ctio

n D

eco

de C

ontr

ibutio

n O

ffline

Offline

Err

or

Ang

le (D

eg

rees)

spikes

LMP

spikes

LMP

spikes

LMP

spikes

LMP

Monkey R Monkey J Monkey R Monkey J*** ***

***

***

Figure 3.10: LMP underperforms online due to longer control loop latency(a) Example trial showing BMI cursor position (gray circles) in successive 50 ms steps as it movedtowards the target (black disk). The cursor and target are drawn at half-size. Instantaneouscontributions of the spikes (red) and LMP (blue) signals to decoded velocity are shown as vectorsoriginating from the cursor. Filled cursor positions denote the analysis epoch from 200 ms aftertarget onset until when the cursor first entered the target (dashed box). From dataset R.2013.09.27.(b) Mean and SEM of each signal’s relative contribution to the decoded velocity. (c) Angular errorwas computed between each signal’s decoded velocity and the direction pointing directly towardsthe target. (d) We repeated the angular error analysis but varied the lag between each signal’sinstantaneous neural push and “correct” angle to estimate control loop latency. For example, +100ms lag means that error was computed against a vector pointing towards the target from the cursor’sposition 100 ms prior. Mean and SEM angular error for the velocity decoded from spikes (red) andLMP (blue) are shown for different lags. (e) Relative decode contribution of spikes and LMP signalsin the hybrid decoder was computed as in (b) but using data recorded while the monkey performedthe same task with his hand. (f) Error angle was computed as in (c) but using hand control neuraldata. (b-d) are averaged across all trials from full-channel hybrid decoder datasets. (e,f) are averagedacross all hand control trials.


closed-loop cursor control, which is noisier and requires more online correction, led me to suspect

that the LMP signal was slower than spikes to reflect corrective changes in the animal’s movement

intention. To look for evidence of this, I compared the ‘control loop latency’ for spikes and LMP

during closed-loop BMI control by finding the temporal lag that minimized the error angle between

each signal’s instantaneous neural push and a vector pointing directly towards the target from the

lagged cursor position (Figure 3.10d). In other words, I asked how delayed the response of each

signal appeared to be under the (simplistic) assumption that the monkey’s control strategy was

always to push the cursor directly towards the target. For both monkeys, the LMP control loop

latency estimated in this manner was twice as long as that of spikes (100 ms versus 50 ms). These

error-minimizing lags were the same when closed loop BMI data driven by LMP-only or spikes-only

decoders were analyzed (not shown).

3.3.7 Hybrid decoding can rescue performance when spikes are lost

In the previous section, I reported whether combining LMP with spikes improved BMI performance

when both signals were available from all electrodes. We next investigated whether hybrid decoding

could increase the BMI’s robustness to a loss of available spikes information, as would be encountered

with degrading arrays. To simulate this scenario, we performed channel-dropping experiments

in which we trained hybrid and spikes-only decoders after removing the most contributing spikes

electrodes (see Methods for a description of the electrode ranking metric). Closed-loop performance

using these channel-dropped decoders was then evaluated on the Continuous Random Target Task

(Figure 3.11). Due to the previously described differences in spikes-only, LMP-only, and all-electrodes

hybrid performance between the two monkeys, I treated LMP differently for each monkey’s electrode

dropped hybrid decoders. For monkey J, hybrid decoders used spikes from the reduced subset of

electrodes and LMP from all of the electrodes. This simulates a situation where spikes can no longer

be detected on some electrodes, but LMP is still available. For monkey R, I ran a more aggressive

electrode dropping protocol where both LMP and spikes were removed from dropped electrodes; this

simulates complete failure of the electrodes.

My motivation for using these two different protocols is as follows. Recall that monkey J’s

spikes decoder was much better than his LMP decoder, and his all-electrodes hybrid decoder did

not improve upon spikes-only decoding. Monkey J’s arrays recorded good spikes signals from almost

every electrode, which resulted in a shallow spikes electrode contribution curve (Figure 3.12). Even

with many electrodes removed, his spikes decoder still outperformed the all-electrodes LMP decoder

(Figure 3.11). Given this, and my earlier observation that adding LMP to much better spikes could

be deleterious during closed-loop hybrid decoding, it was not surprising that in pilot studies we

found that dropping electrodes for both signals — even when ordered by spikes contribution — still

resulted in monkey J’s spikes-only decoders always beating hybrid decoders (up to the point where

both failed). I therefore concluded that a more informative monkey J electrode dropping experiment


*0/5 *3/5 *3/5 *3/5 *5/5

0 40 80 120 160

fail

1

Electrodes Removed

(spikes only)

*5/5 *4/5 *4/5 *5/5

0 40 80 120 160 0

0.35

*0/5

2x 5x

Su

cce

ss R

ate

*3/7 *2/6 *5/6 *7/7 *4/4

0 20 30 40 60 80

fail

1

Electrodes Removed

(both spikes & LMP)

NT

TT

(s/c

m)

*5/8 *5/7 *4/6 *6/6 *6/7 *3/4

0 20 30 40 60 80 0

0.35

3x 3x

0.5 0.5

*0/8

Monkey R Monkey Ja

b

Figure 3.11: Performance of spikes-only and hybrid decoders after electrode droppingSpikes-only (red) and hybrid spikes + LMP (purple) decoders were trained and evaluated with thespecified number of electrodes removed in a “worst case scenario” by descending order of contributionto the spikes decoder. For monkey R, we removed both spikes and LMP features from these electrodes.For monkey J only spikes were removed. Spikes and hybrid decoder performances with a givennumber of electrodes removed were compared in the same session. Their mean performances areshown as a red and purple point on the plot; filled points represent a significant difference (p < 0.01)for the paired comparison between decoders on that day. Starred fractions summarize how oftenthis difference was significant out of the total number of days when that condition was tested. Solidlines show the grand mean performance of each decoder as a function of electrode-drop condition.Days when a decoder “failed” (success rate below 0.5) were not included in this grand mean. Dashedlines extend mean lines into the all-failure conditions for illustrative purposes. All-electrodes LMPperformance metrics (mean from Figure 3.9) are shown with horizontal blue lines spanning thoseconditions where all LMP electrodes were available to the hybrid decoder. (a) Success rate. (b)Normalized time to target.


would ask: will hybrid decoding ever be helpful if his spikes recording degrades but his mediocre

all-electrodes LMP is still available?

For monkey R, removing only spikes electrodes (i.e., the monkey J protocol) would not have

been very informative given the following observations. Monkey R’s spikes performance was roughly

1.5 times better than his LMP-only performance (a much smaller difference than in monkey J)

and relied heavily on a small subset of electrodes (Figure 3.12). Once the best few electrodes

were removed, monkey R’s spikes-only decoder became worse than the all-electrodes LMP decoder

(figure 8). Furthermore, from the all-electrodes hybrid experiments I knew that the two signals

already combined beneficially in monkey R’s hybrid decoder, and that this decoder was dominated

by LMP even with all spikes electrodes available. Thus, applying the protocol used in monkey J

to monkey R would have rapidly converged to comparing a reduced-electrode spikes decoder to an

all-electrodes LMP decoder (plus a minor contribution from remaining spikes). This comparison

can be approximated with just a spikes-only electrode dropping experiment (which the data does

include). By instead comparing reduced electrode spikes-only decoders to hybrid decoders in which

both signals are removed from dropped electrodes, I was able to additionally test how much hybrid

decoding could rescue performance in the more dire scenario of completely losing electrodes that had

the best spikes signals.

We tested monkey J with spikes removed from 0, 40, 80, 120, and 160 electrodes and LMP still

available from all electrodes. We found that with 40 or 80 electrodes removed, the hybrid decoder

performed worse than a spikes-only decoder (Figure 3.11). Hybrid decoding started to outperform

spikes-only decoding with 120 electrodes removed, performing superiorly in 3/5 sessions. This 120

electrodes removed condition also corresponds to when spikes-only decoding became worse than

the average all-electrodes LMP-only performance. Once 160 electrodes were removed, the spikes

decoder consistently failed but the hybrid decoder was consistently usable (77-91% trials successful).

However, I note that in the conditions where hybrid decoding outperformed spikes-only decoding, it

was no better, on average, than LMP-only decoding. Thus, in monkey J hybrid decoding improved

robustness to the loss of many spikes electrodes, but not beyond the improvement afforded by just

decoding LMP.

In monkey R, hybrid decoding substantially improved robustness to a complete loss of signals

from electrodes containing the best spikes. I compared hybrid and spikes-only decoders with 0, 20, 30,

40, 60, or 80 electrodes disabled for both spikes and LMP. Hybrid decoders increasingly outperformed

their spikes-only counterparts as more electrodes were removed. This difference was consistently

significant (6/6 sessions) once 40 electrodes were removed; for this condition, success rate was 25%

higher and times to target were 38% faster (6 sessions’ grand mean) using the hybrid decoder,

not including one session where the spikes decoder failed outright. The hybrid decoder remained

consistently usable even with 80 electrodes removed (77-91% trials successful), whereas the spikes-only

decoder started to become unusable once 40 electrodes were removed. Three factors contributed to


Monkey R Monkey J

Electrode Rank1 192

0

1

Cu

mu

lative

De

co

de

r C

on

trib

utio

n

Electrode Rank1 192

0

1

0 0.040

0.04 Correlation r = 0.16

Spikes weight

LM

P W

eig

ht

0 0.10

0.1 Correlation r = 0.36

Spikes Weight

LM

P W

eig

ht

Figure 3.12: Cumulative electrode contribution to the spikes-only and LMP-onlydecodersThe contribution of each electrode to a decoder is calculated based on that electrode’s weight inthe steady-state velocity Kalman filter multiplied by the average magnitude of the signal from thiselectrode in the training data set. Electrodes are ranked from greatest to least contribution, and thecumulative decoder contribution is plotted as a function of electrode rank, starting from the rank 1(most contributing) electrode. We show the cumulative electrode contribution curves for spikes-only(dark red) and LMP-only (dark blue) decoders built from the arm reaching training data of eachexperiment day in which an all-electrodes hybrid decoder was evaluated (8 datasets for monkey R, 5datasets for monkey J). Bright red and bright blue lines show the means of these individual days’contribution curves. On average, half of monkey R’s spikes decoder weight was contributed by thetop 13 electrodes, compared to 30 electrodes for the LMP decoder. For monkey J, on average 31electrodes contributed half the spikes decoder weight, compared to 50 LMP electrodes. Inset scatterplots show the fractional contribution of each electrode to the spikes decoder and LMP decoders;each point corresponds to one electrode from one session. The mean Pearson’s correlation betweenelectrodes’ fractional contribution to the spikes and LMP decoders is summarized in the inset title.This figure shows that LMP decoder weights were more distributed across electrodes than spikesdecoder weights, and that a given electrode’s importance in the LMP decoder was poorly correlatedwith its importance in the spikes decoder.


why the improvement afforded by hybrid decoding increased as I removed more electrodes. First,

I dropped electrodes in order of descending spikes decoder contribution; second, there was a low

correlation between how important a given electrode was for spikes and LMP decoding (Figure 3.12

insets); and third, decoder weight was much more evenly distributed across the LMP electrodes than

spikes electrodes (Figure 3.12).

3.4 Discussion

The goal of this study was to address the need for motor neural prostheses to be robust to the

loss of recordable spikes. The present work substantially advances this goal by demonstrating the

effective use of LFP as an alternative or complementary BMI control signal. We decoded a velocity

command either solely from multielectrode LMP, or from a hybrid control signal consisting of both

binned spike counts and LMP. We demonstrated that LMP alone is a viable alternative control signal

that enabled effective cursor control in two monkeys. The LMP BMI did not require daily decoder

recalibration, and its performance substantially exceeded that of previously reported LFP-driven

systems. This was the first study to evaluate BMI control using a single type of LFP feature, in

this case the time-varying LFP amplitude — LMP — which is dominated by power below 5 Hz and

requires minimal electrical and computational power to extract. We presented new evidence that

this signal is not a movement-related artifact by showing that hand velocity could not be decoded

from the LMP of sedated monkeys whose arms were passively moved. Finally, we demonstrated for

the first time continuous cursor control using a hybrid decoder combining spikes and LMP. We found

that while hybrid decoding did not improve BMI performance when many electrodes were able to

detect spikes, if this spike-based information was removed from some electrodes (simulating sensor

degradation), then providing LMP signals to the decoder could rescue performance. Altogether, these

results demonstrate that even if multielectrode arrays eventually lose their ability to record spikes

on some or even all electrodes, an intracortical BMI may be able to continue to function effectively

as long as LMP can still be recorded. The methods described here may therefore extend the useful

lifespan of BMI systems, thereby increasing clinical viability.

3.4.1 Choosing LMP for BMI use based on offline decode performance

There are myriad ways to process LFP signals into decodable features, and it would not have

been experimentally practical to develop and test online BMIs driven by each candidate feature

set. Existing offline studies examining the relationship between LFP and reaching kinematics give

differing recommendations as to which features are best for BMI use. One study (So et al., 2014) has

compared closed-loop performance using LFP power in different frequency bands, but that study

reported that the best frequency band was different for each monkey and did not evaluate LMP.

Therefore, to maximize our chance of choosing the right LFP feature(s), I first performed my own


offline comparisons which found that LMP was the best of the tested LFP-derived features. This is in

accordance with past studies showing that LMP recorded from primary motor and premotor cortices

is informative about kinematics (Mehring et al., 2003; Rickert et al., 2005; O’Leary & Hatsopoulos,

2006; Stark & Abeles, 2007; Bansal et al., 2011a,b; Flint et al., 2012a,b). Interestingly, both our study

and previous studies (Pesaran et al., 2002; Scherberger et al., 2005; Zhuang et al., 2010; Markowitz

et al., 2011) found that power in low-frequency LFP bands is a poor decode feature. This suggests

that even though low-frequency components dominate the LMP, the loss of phase information when

estimating low-frequency power results in a loss of useful information.

The second best LFP feature I tested, power in the 50-300 Hz band, performed considerably worse

than LMP in the offline decode comparison. This contrasts with (Bansal et al., 2011a; Flint et al.,

2012b) which found that high-frequency LFP power was comparable to or better than LMP. Such

differences might be expected if the relative information contained in LMP versus high frequency

LFP power is sensitive to precisely where the signals are recorded. I am unaware of any study that

methodically mapped how the quality of LMP information changes as a function of recording location,

but one study (Markowitz et al., 2011) showed that decoding performance using higher frequency

spectral features depends strongly on cortical recording depth. Similar studies to optimize recording

location for LMP-based decoding would be very useful.

I were surprised to find that my offline decode accuracy using LMP was comparable to decoding

multiunit spikes. Although previous studies have shown better or equal offline decoding using LFP

features rather than spikes from a small number of electrodes (Pesaran et al., 2002; Mehring et al.,

2003; Bansal et al., 2011a,b; Flint et al., 2012a; Hwang & Andersen, 2013), many of the same studies

(Bansal et al., 2011a,b; Flint et al., 2012a; Hwang & Andersen, 2013) and others (Stark & Abeles,

2007; Flint et al., 2012b), reported that at higher electrode counts, multielectrode spike activity is

more informative than multielectrode LFP. My 192-electrode study falls within the upper electrode

count range of these past reports. However, differences in electrode placement, signal processing and

decoding techniques, and the nature of the kinematics being decoded obfuscate direct comparisons

between studies and may explain why my offline decode of velocity was better using LFP than using

spikes. Consistent with previous reports (Mehring et al., 2003; Stark & Abeles, 2007; Bansal et al.,

2011b; Hwang & Andersen, 2013), I found that offline decoding of both LFP and spikes together led

to a modest but significant accuracy improvement. This improvement was greater using LMP than

50-300 Hz power. Although my results do not speak to this directly, one reason why high frequency

LFP power may not add much information to what can already be decoded from multiunit spikes is

that this signal may itself be contaminated by spiking activity (Asher et al., 2007; Waldert et al.,

2013).

Taken together, my offline decode results led me to conclude that LMP was the most promising

candidate LFP feature both as an alternative and complementary signal to spikes for BMI control.

These data also address several concerns that have been raised about the utility of low-frequency


LFP signals for BMI use. One such concern is that decoding low-frequency LFP may be hindered by

high correlations of both signal and noise between electrodes (O’Leary & Hatsopoulos, 2006; Bansal

et al., 2011b; Flint et al., 2012b; Hwang & Andersen, 2013). While I did observe that LMP from

nearby electrodes had similar tuning, there were a variety of PDs across the two 96-electrode arrays

and I was able to accurately decode single-trial velocity. A second concern is that the signal is an

artifact caused by the animal’s movement. Our sedated animal decode control experiment adds a new

piece of evidence on top of the already-persuasive case made by prior studies (Hwang & Andersen,

2009; Bansal et al., 2011b) that low-frequency LFP is not merely a mechanical or electrical artifact.

A third concern is that LMP would be ill-suited to serve as a neural prosthetic signal if it lags behind

kinematics (i.e., contains mainly sensory information); this has been reported in motor (Bansal et al.,

2011b) and parietal (Asher et al., 2007) cortices. Contrary to these studies, I found that kinematics

were best decoded when LMP was time-shifted to precede kinematics by approximately 100 ms. My

results may differ from (Asher et al., 2007) because they recorded from a different cortical region, and

from (Bansal et al., 2011b) because they appear to have recorded from an area with more sensory

information (they also found that most single units’ optimal decoding lags followed kinematics).

My results are consistent with a different set of studies which found that that LMP is tuned before

movement onset (Mehring et al., 2003; Hwang & Andersen, 2009; Flint et al., 2012a).

3.4.2 Closed-loop LMP-driven BMI performance

This study demonstrated that decoding LMP from two multielectrode arrays can enable effective

BMI cursor control. It adds to the small existing body of closed-loop LFP-driven cursor control work

(Flint et al., 2013; So et al., 2014) by demonstrating improved performance while also characterizing

the quality of control using a single LFP-derived feature (LMP). Differences in monkeys and signal

recording quality inevitably limit how much can be inferred by comparing the performance of different

BMI methods across studies. Nonetheless, I made a best effort to directly compare the performance

of our LMP-driven BMI to that of two previous closed-loop LFP-driven BMI studies on closely

matched tasks. We observed target acquisition rates roughly 50% higher than (So et al., 2014) and

over twice as fast as (Flint et al., 2013). Thus, the work described here represents, to the best of

my knowledge, the highest-performing LFP-driven BMI system to date. I believe that two design

features contributed to better performance using our BMI system. First, we decoded neural activity

from more electrodes than previous studies. We recorded from two 96-electrode arrays (one in M1

and one in PMd), whereas So and colleagues decoded LFP from twenty electrodes randomly chosen

from a 128-electrode array spanning M1 and PMd, and Flint and colleagues recorded from a single

96-electrode array in M1. Second, we decoded LMP rather than spectral power, which according to

my offline decoding analysis better predicts reach velocity. Flint and colleagues also decoded LMP as

one of their features, but did not implement something akin to our half-wave rectification. If either

of their monkeys exhibited the same deleterious LMP after-potential that I observed in our monkey


R (perhaps partially compensated for by other decoded LFP power features), this may have limited

performance. So and colleagues’ decoders used multiple LFP power bands between 0 and 150 Hz,

but did not decode LMP.

This substantially better LFP-driven performance serves as an important existence proof that

high-quality BMI control is possible using alternative (non-spike) signals from chronically implanted

multielectrode arrays. Importantly, however, we found that our LMP-driven BMI performance was

not as good as spike-driven performance using the state-of-the-art FIT-KF algorithm (Fan et al.,

2014). Thus, while our results demonstrate that LMP is a viable alternative control signal to spikes,

I do not suggest employing LFP-only decoding if good spikes are available. I do note, however,

that the low sampling rate and minimal signal processing required to extract LMP, combined with

its reasonable performance as a closed-loop control signal, may make LMP attractive for power-

or bandwidth-constrained applications such as fully implanted BMI systems (these constraints are

overviewed in e.g., Dethier et al. 2013). The low-pass filtering properties used to extract LMP in

this study can be approximated with a passive analog filter followed by half-wave rectification via a

very low power analog rectification circuit. Downstream processing would only need to sample the

resulting voltage 40 times per second (the decode update time used in this study). This contrasts

dramatically with the order 103 samples per second used to identify single unit spikes. Generally

speaking, recording sorted spikes has the highest power demands, followed by threshold crossings

and higher frequency LFP band power, and finally LMP.

A desirable property of a neural prosthetic control signal is not just the performance that it

affords, but also its stability. BMI use without daily decoder recalibration has been shown when

decoding well-isolated single units (Ganguly & Carmena, 2009), multiunit spiking (Flint et al., 2013;

Nuyujukian et al., 2014), and a broad set of LFP-derived features including both LMP and higher

frequency power (Flint et al., 2013). We found that our LMP-only decoder can also be used without

retraining for at least three weeks. Unlike in (Flint et al., 2013), we did not see long-term performance

improvement when using a fixed decoder. This may be because we did not hold the decoder stable

for as many days or for as long during each day, or because the higher performance in our study

reduced the pressure on the animal to improve his performance.

Our study was similar to that of Flint and colleagues (Flint et al., 2013), but differed from the So

and colleagues study (So et al., 2014), in that we allowed the monkeys to continue to move their arm

during BMI use. This arm-free BMI animal model is better suited for evaluating biomimetic decoders

trained using arm reaching data (Nuyujukian et al., 2011), and unsurprisingly we observed that

monkeys continued to move their arms when using the LMP-driven BMI. In this regard LMP-driven

BMI use is similar to spike-driven control — both in this and past studies such as (Gilja et al., 2012)

— during which monkeys typically continue to move their arms if allowed to. As with any neural

prosthetics method tested using an able-bodied animal model, it remains to be seen how well our

approach will work in paralyzed or amputee patients. While I am encouraged by a recent report


of movement intention-evoked transient low-frequency LFP modulation in humans with tetraplegia

implanted with similar arrays (Perge et al., 2014), validation of my LMP decoding approach will

require closed-loop evaluation of this method in clinical trial participants.

3.4.3 Hybrid decoding using both LMP and spikes

Decoding spikes or LFP need not be an either-or decision. While it has often been predicted that

hybrid decoding of both signals could improve BMI performance, this is the first closed-loop study

to decode continuous kinematics from spikes and LFP together. In doing so, I faced two decisions:

how should I combine these two signals in a single decoder, and what spikes-only decoder should I

compare the resulting hybrid decoder against? I felt that it would be most impactful to try to improve

performance and robustness above that of a current state-of-the-art spikes decoder. I therefore used

as our baseline the FIT-KF algorithm (Fan et al., 2014) and then extended it to decode both spikes

and LMP. When both spikes and LMP were decoded from all available electrodes, hybrid decoding

enabled a modest improvement in closed-loop performance in monkey R, but was marginally worse

than spikes-only decoding in monkey J. An analysis of each signal’s contribution to the closed-loop

velocity decode provided insight into why adding LMP did not improve closed-loop performance

as much as might have been expected from offline analyses: when the hybrid decoder is fit, LMP

features are heavily weighted because collectively they are as well as or better correlated than spikes

with the open-loop training data kinematics. During closed-loop BMI use, however, even modest

decode errors must be corrected in order to acquire the target; consequently, the longer control loop

latency characteristic of LMP leads to worse control. As a result, the LMP’s contribution to a hybrid

decoder with excellent spikes signals does little to improve — and can even worsen — closed-loop

cursor control. Consequently, achieving a benefit from online hybrid decoding is not a given, but

rather will depend on the relative closed-loop quality of the spikes and LMP signals.

I note two caveats related to my design choices of building the hybrid decoder by extending the

FIT-KF spikes decoder and then comparing it against the spikes-only variant. First, while combining

the spikes and LMP Kalman filters was a principled application of existing high-performing decoding

techniques to the hybrid decoding problem, I make no claim that the resulting decoder — which

determines the relative weighting of spikes and LMP features based on how well each signal regressed

against training data kinematics and how the resulting residuals covaried — is optimal for closed-loop

BMI use. I view this work as an encouraging first foray into hybrid decoding but recognize a need for

future work looking at whether there are better ways to combine spikes and LFP signals. For example,

there may be an opportunity to improve performance by taking into account the different control

loop latencies of LMP compared to spikes, or by decreasing the decoder weights assigned to LMP

features to account for their inferior closed-loop utility. Second, by comparing the hybrid decoder

to what was already a very high performing spikes-only decoder, I set a high bar for performance

improvement. Monkey J, in particular, had many good electrodes with spikes and controlled the


cursor almost as well with the all-electrodes FIT-KF spikes decoder as with his hand. With these

caveats in mind, the present results suggest that when many electrodes with spikes information are

available, hybrid decoding offers little to no improvement over state-of-the-art spikes-only decoding.

3.4.4 Decoding LMP to mitigate losing spikes signals

Thus far I have discussed how LMP and hybrid decoding compares to spikes-based decoding in the

case of arrays with excellent (monkey J) or mediocre (monkey R) spike signals. But what about

when spike signal quality is poor and insufficient for accurate BMI control? This is oftentimes the

case when arrays degrade over the years following implantation (Barrese et al., 2013; Wang et al.,

2014) and is one of the major impediments to long-lasting, clinically viable neural prostheses. We

experimentally simulated this situation with electrode dropping experiments, and found that the

performance drop resulting from losing spikes information can be mitigated by making use of LMP.

The degree to which LMP provides robustness to spikes loss depends on the relative quality of the

LMP and remaining spikes, as I will now describe.

In monkey J, LMP was in effect a BMI control signal of last resort. This subject had excellent

spike signals on most electrodes, which enabled much better cursor control using a spikes-driven

BMI than with an LMP-driven BMI. Only after a majority of spikes electrodes were dropped did his

spikes-only control become worse than all-electrodes LMP-only control. At that point, both hybrid

decoding and LMP-only decoding (both of which had LMP available from all electrodes) provided

usable cursor control, whereas the spikes-only decoder failed. Importantly, across the electrode

dropping conditions we tested, the hybrid decoder did not outperform whichever individual signal

type (spikes-only or LMP-only) worked best. It is somewhat counterintuitive that adding additional

LMP information to monkey J’s electrode-dropped spikes signal actually reduced performance in the

conditions where spikes still outperformed all-electrodes LMP. This (suboptimal) hybrid decoding

result can be understood in light of the previously discussed observation that LMP and spikes were

comparable for offline decoding but that LMP provided worse closed-loop control. When monkey J’s

most contributing spikes electrodes were removed, the regression to fit a hybrid decoder increasingly

weighed the LMP over the remaining spikes because of LMP’s high offline explanatory power. This

resulted in a hybrid decoder that was (overly) dominated by the inferior LMP signal rather than the

more controllable remaining spikes.

In contrast, utilizing LMP improved monkey R’s BMI control even when spikes were available

from all electrodes, and the LMP became increasingly important for maintaining performance as

spike signals were lost. Monkey R had worse spikes-driven control and better LMP-driven control

than monkey J, such that all-electrode LMP decoding outperformed spikes-only decoding once twenty

or more spikes electrodes were lost. This suggests that when spikes’ performance advantage over

LMP is not large (and is tenuous insofar as it depends on a handful of key electrodes), a modest

degree of array spikes signal decay will result in LMP decoding becoming a preferred alternative to


spikes decoding. Furthermore, monkey R’s hybrid decoder outperformed the spikes-only decoder

across all electrode drop conditions tested, demonstrating that when closed-loop LMP-driven control

is comparable to (or better than) spikes-driven control, LMP can complement spikes and improve

BMI performance.

Taken together these results provide experimental validation that utilizing LFP (in this study,

specifically the LMP feature) as a BMI control signal can increase a neural prosthesis’ robustness to

the loss of spikes signal. At a minimum, LMP can provide an alternative control signal to spikes (as

in monkey J’s case), which is useful because there is considerable experimental evidence suggesting

that arrays’ LFP signals persist longer than spikes (Asher et al., 2007; Bokil et al., 2006; Flint et al.,

2012b; Heldman et al., 2006; Hwang and Andersen, 2013; Perge et al., 2014). In the more fortuitous

case where LMP-driven performance is quite good (such as in monkey R), LMP can be used not just

as an alternative to spikes, but as a complementary signal for a hybrid decoder that both improves

maximal performance, and slows down the rate at which performance decays when spikes signals are

lost. I am particularly encouraged that in this study, decoding LMP enabled effective cursor control

in both monkeys under conditions where spikes-only decoding was non-functional.

There are several electrode dropping experiment permutations that this study did not explore

but which are worth considering. I did not test monkey R in the case that spikes were lost from

dropped electrodes but LMP was still available from all electrodes. However, the observation that his

all-electrodes LMP-only decoder was better than any of the electrode-dropped spikes-only decoders,

and that LMP and spikes combined beneficially in the all-electrodes hybrid case, strongly predicts

that hybrid decoding would be even better if more LMP electrodes are available. Since the main

goal of this study was to use available LMP signals to increase robustness to a potential loss of

spikes signals, I also did not explicitly explore how these decoders would perform when, instead,

LMP electrodes are lost (i.e., LMP-only electrode dropping experiments, or hybrid-decoder electrode

dropping ordered by LMP contribution). Based on the high correlations between LMP on nearby

electrodes and the shallow LMP decoder contribution curves, I predict that such experiments would

find that LMP and hybrid decoders would degrade slower (as a function of number of electrodes

removed) than spikes-driven decoders. This prediction remains to be tested in future closed-loop

experiments.

3.4.5 Differences between offline and closed-loop results

My experiences carrying out this work add to the growing body of literature reporting that offline

decoding results do not always predict closed-loop BMI performance (e.g. Chase et al., 2009;

Cunningham et al., 2010). Given the field’s heavy reliance on offline decode analyses, I believe it

is worthwhile to enumerate these differences in order to draw attention to factors that may affect

how offline performance translates to closed-loop results. Although my offline metrics showed that

decoding LMP was comparable to or even better than decoding spikes, during closed-loop evaluation


spikes-driven cursor control was consistently superior. Qualitatively, the LMP-only decoder appeared

more “ballistic” and less precisely controllable than the spikes-only decoder: if the monkey missed

the target with his initial reach, his corrective response was slower and less smooth. This impression

is supported by several observations: LMP-only decoding had longer times to target and dial-in time;

during hybrid decoding, the decoded LMP aimed the cursor less accurately than spikes did; and

the estimated control loop latency was longer for LMP than for spikes. I therefore attribute LMP’s

reduced closed-loop controllability to this signal being slower to reflect changing movement intention

— a limitation that is likely to be shared by other low-frequency signals. To better predict such offline

versus closed-loop differences in the future, I suggest that the datasets used to evaluate candidate

BMI control signals include conditions in which rapid reach corrections are made. This would

enable estimation of the candidate signal’s control lag. LMP’s inferior closed-loop controllability also

explains a second difference between offline and closed-loop decoding: offline, hybrid spikes and LMP

decoding helped in both monkeys, but during closed-loop control this benefit was only realized in

monkey R.

A third example is that decoding LMP without half-wave rectification worked well offline, but

in one monkey half-wave rectification was needed to enable closed-loop performance. This signal

processing step removed an LMP ‘afterpotential’ (visible at t = 0.6 s in Figure 3.1d) which occurred

shortly after the monkey reached towards the target. Without rectification, this afterpotential

generated a substantial cursor velocity during the time period when the monkey was typically trying

to keep the cursor still over the target. After I discovered this problem during closed-loop BMI testing,

I looked more closely at my offline decode results and noticed that the decoded velocities did slightly

increase at the end of the target hold period. However, this deleterious effect was poorly captured in

my offline decode metrics because the afterpotential happened near the end of the trial. Consequently,

the decoded hand velocity was just beginning to increase when the trial ended, preventing further

accumulation of decode error. The next trial’s decode then began with the Kalman state reset to

zero (a common practice to prevent offline decoding errors from accumulating over the course of

many trials), minimizing the afterpotential’s effect on this next trial. In short, the trial structure of

the data I used for the offline study largely hid the problem, but in closed-loop the afterpotential’s

effect was strong enough to dramatically hinder target acquisition.

Offline studies are invaluable for suggesting promising BMI design approaches, but these differences

underscore the need for careful closed-loop validation. Thus, while I specifically investigated

closed-loop LMP-driven performance because this LFP feature worked best offline, these results

do not rule out that other features cannot yield even better closed-loop performance. The nascent

subfield of LFP-driven BMIs would benefit from more studies evaluating other choices of LFP features

and decoding algorithms in closed-loop experiments.


3.5 Methods

3.5.1 Subjects and neural recording


and Use Committee. Experiments were conducted with adult male rhesus macaques (monkeys ‘R’

and ‘J’) previously implanted with two 96-channel multielectrode arrays (1 mm electrodes spaced

400 µm apart; Blackrock Microsystems, Salt Lake City, UT) using standard neurosurgical techniques

as described in (Santhanam et al., 2006). The arrays were placed in M1 and PMd contralateral

to the reaching arm. Monkey R’s arrays were implanted 24-25 months prior to these closed-loop

and ‘sedated reaching’ (see next paragraph) experiments and 16-20 months prior to collection of the

offline decode datasets. Monkey J’s arrays were implanted 49-50 months before the closed-loop and

sedated reaching experiments and 24 months prior to the offline datasets.

I used an “arm not restrained, not visible” animal model for BMI study, meaning that the monkey’s

reaching arm was unrestrained, but visually occluded by the display, during both arm controlled and

BMI controlled tasks (Shenoy & Carmena, 2014). I believe that this is a good BMI animal model

because it allows training biomimetic decoders from natural reach data, and subsequently allows

the animal to control the BMI cursor by generating whatever neural activity he wants without an

additional constraint that this activity must not cause movement of the arm (Nuyujukian et al.,

2011). As a control experiment to test if LMP could be merely an arm movement artifact, we also

collected datasets in which each monkey was sedated while an experimenter performed the Radial

8 Task by moving the monkey’s hand from target to sequential target. The animal was sedated

with ketamine 2 mg/kg plus dexmedetomidine 0.04 mg/kg IM and positioned in the experimental

rig in the same fashion as during awake-behaving experimental sessions. The experimenter was

electrically insulated from the animal and moved the monkey’s hand so as to approximately match

the animal’s typical reach trajectories; meanwhile, the behavioral task software was running, thereby

allowing for identical analysis of this control experiment data. Neural and kinematics recording,

signal processing, and subsequent data analysis were identical to that of our standard awake-behaving

reaching experiments.

3.5.2 Behavioral tasks

The monkeys were trained to sit head-restrained in a primate chair and perform reaching tasks with

one hand, as shown in Figure 3.1a. A virtual cursor and target were displayed in a 3D environment

(MSMS, MDDF, USC, CA, USA) in front of the monkey by two 120 Hz LCD displays visually fused

by mirrors. However, all of the cursor tasks in this study were confined to a 2D vertical plane.

Custom experiment control software written on an xPC Target platform (Simulink, The Mathworks,

Natick, MA, USA) controlled the behavioral task and updated the monitors with latency of 7 ± 4

ms. The monkey controlled the cursor position either by moving his hand, which was tracked with


an infrared reflective bead-tracking system at 60 Hz (Polaris, Northern Digital, Canada), or with a

velocity command from decoded neural activity (see below). In all tasks, the monkey earned a liquid

reward after each successful target acquisition.

At the beginning of each experimental session the monkey performed a block of the Radial 8 Task.

In this task, the location of the target alternated on each trial between the center of the workspace

and a peripheral location pseudorandomly chosen from eight locations equally spaced along a circle of

radius 12 cm. To successfully complete a trial, the monkey had to keep the cursor inside a 4 × 4 cm

acquisition area around the target for a ‘target hold period’ of 500 consecutive ms. A new center trial

immediately followed a successful trial or after failure to acquire the target within a 2 s time limit.

These data were used for decoder training during closed-loop experiments, and previously collected

Radial 8 Task datasets were used for offline analyses comparing different LFP-derived neural features.

Unless stated otherwise, we evaluated closed-loop performance on the Continuous Random Target

Task. At the start of each trial, the target appeared in a random location anywhere in a 20 × 20

cm box centered in a bounded workspace that was itself 40 × 32 cm. The monkey had up to 8 s to

acquire the target by keeping the cursor within a 5 × 5 cm acquisition area for a target hold period

of 500 ms (monkey R) or 300 ms (monkey J).

We also evaluated performance on two additional tasks to compare our BMI to those of prior

LFP studies. The first was a variant of the Continuous Random Target Task with target size reduced

to 4 × 4 cm, trial time limit of 10 s, and hold time reduced to 100 ms, as in (Flint et al., 2013).

Due to differences in animal training, unlike in that study we did not fail a trial if the cursor left

the target within 100 ms and later came back to make a successful acquisition. Instead, I post-hoc

labeled these trials as failures. This is a more punitive metric of our subjects’ performance because

the animals did not have feedback to strongly incentivize them to avoid leaving the target. The

second task was a variant of the Radial 8 Task similar to (So et al., 2014) with 3.4 cm wide targets

located around a 13 cm diameter circle. Acquiring the center target initiated a trial but only reaches

to the peripheral targets were analyzed. I based our variant of the So and colleagues study’s task on

an earlier report (Dangi et al., 2013) and therefore used a 300 ms hold time and 8 second time limit.

I note that in (So et al., 2014) the hold time was actually 400 ms and the time limit was 10 seconds.

3.5.3 Signal processing

Voltage from each of 192 electrodes was initially sampled at 30 ksps by a Cerebus system (Blackrock

Microsystems) and analog filtered from 0.3 to 7500 Hz with a 3rd order Butterworth filter (Figure

3.1b). To extract multiunit spikes (also known as ‘threshold crossings’), the raw neural data from

each electrode was digitally filtered from 250 to 7500 Hz (3rd order Butterworth). A spike was

detected whenever the signal went below a threshold set at the beginning of each recording session

to be -4.5 times the root mean squared value of the electrode’s voltage (Chestek et al., 2011). Spike

counts in non-overlapping bins from each electrode formed a vector of spike features computed at


each decoder time step. For all offline analyses this decoder time step was 50 ms. Prior studies

have found that such short time steps lead to better closed-loop control than the longer values more

commonly used in offline decoding studies (Cunningham et al., 2010; Gilja et al., 2012). For monkey

R’s closed-loop experiments we used the same 50 ms time step, while for monkey J we used a time

step of 25 ms based on the observation that his spikes-only performance improved with this even

shorter time step. Spikes extraction from disconnected electrodes or electrodes with no spikes was

disabled; depending on the dataset, 168-174 electrodes recorded spikes in monkey R and 188-192 in

monkey J.

LFP was obtained by digitally low-pass filtering the same raw neural data below 500 Hz (4th

order Butterworth) and then downsampling to 2 ksps. For my offline decoding analysis, I evaluated

a number of candidate LFP frequency bands in addition to the time-domain local motor potential

(LMP), which is described in the next paragraph. I created a vector of LFP power features at each

50 ms time step for each channel by taking the mean power of the causally band-pass filtered signal

(3rd order Butterworth) from the previous 50 ms of LFP. Because a 50 ms window does not capture

a full period of the signal for bands with minimum frequency fmin below 20 Hz, for these features

I used overlapping bins of 1/fmin seconds slid every 50 ms. The amplitude response of the filter

cascades used to generate each candidate LFP feature, including the effects of the shared initial

signal processing described above, are shown in Figure 3.2b. These were computed by simulating

the analog and digital filters applied by the Cerebus signal processor, and then analyzing the series

of these filters, plus the LFP frequency band filter of interest, using MATLAB’s cascade and filter

visualization (fvtool) tools.

To obtain the LMP feature from a given channel, LFP samples were first clipped at ±300µV to

mitigate intermittent noise bursts. The LMP was then computed as the sliding mean voltage from

the past 50 ms of LFP (Figure 3.1c). The LMP is thus the low-pass filtered LFP amplitude (Figure

3.2b). For offline decoding the LMP was decoded directly as described above, but in pilot closed-loop

studies in monkey R I found that an additional step was needed to enable effective continuous cursor

control. I observed that on most electrodes, the negative voltage deflection at the start of the reach

was followed by a positive ‘after-potential’, as seen in the top panel of Figure 3.1d. During closed-loop

BMI use this led to the cursor “springing back” in the opposite direction shortly after a movement

was initiated. This markedly impaired performance (Figure 3.13). I therefore applied a half-wave

rectification step where all LMP input values below 0 µV were rectified while positive input values

were set to 0, as illustrated in the bottom panel of Figure 3.1d. While zeroing positive voltages on all

electrodes worked well across the four arrays used in this study, I anticipate that in some situations

the sign of this rectification step may need to be set on a electrode-by-electrode basis depending on

each electrode’s initial LMP polarity. To provide the reader with further intuition about the LMP

signal, Figure 3.2c shows the spectral content of the LMP used to control the BMI on two different

electrodes from representative experimental sessions. These spectra were generated by applying the


MATLAB fft function to the full duration of each electrode’s LMP during BMI task. The spectra

were binned into 1 Hz bins, with figure tick locations correspond to the maximum frequency of each

bin. The signal magnitude within each frequency bin was then normalized by total magnitude over

all frequencies.

Block Within Session

0

0.2

0.4

0.6

0.8

1

Monkey J

26

0 tria

ls

Block Within Session

0

0.2

0.4

0.6

0.8

1

Su

cce

ss R

ate

Monkey Rwith half-wave LMP rectificationwithout rectification

25

6 tria

ls

25

5 tria

ls

25

8 tria

ls

28

7 tria

ls

25

4 tria

ls

25

7 tria

ls

25

7 tria

ls

22

0 tria

ls

23

7 tria

ls

25

2 tria

ls

26

2 tria

ls

Figure 3.13: Closed-loop LMP decoding benefited from half-wave rectification

We evaluated performance on the Continuous Random Target Task using LMP decoders either with

the half-wave rectification step (black) or without this step (grey). When used, half-wave rectification

was applied both to the training data and during closed-loop use. The two decoders were compared

in alternating blocks within the same experimental session. Success rates for each block are shown

in chronological order within an experiment session. The number of trials in each block is labeled.

Monkey R’s overall success rate was significantly higher with half-wave rectification (p < 0.001,

binomial test), but monkey J’s was not (p > 0.05).

3.5.4 Neural decoding

A velocity Kalman filter (KF) was used to decode cursor velocity from neural activity for both offline

decoding and closed-loop BMI experiments. The details of the velocity KF as applied to BMIs are

well described in Wu et al. (2006); Kim et al. (2008); Gilja et al. (2012); here I enumerate specific

design choices used for the present study. Briefly, the KF estimates the state of a linear Gaussian

dynamical model where the kinematics state at time t, x(t) evolves from the previous time step’s

state and relates to neural activity y(t) according to,

x(t) = Ax(t− 1) + w(t) w(t) ∼ N (0,W) (kinematics state update) (3.5.1)

y(t) = Cx(t) + q(t) q(t) ∼ N (0,Q) (neural observation) (3.5.2)


where x(t) is the velocity vector [vx(t), vy(t), 1]T (the 1 is an offset term to accommodate non-zero-mean

neural data), y(t) is a vector consisting of the neural feature(s) from every channel at time t, A is the

matrix describing the linear kinematics dynamics, C is the observation matrix mapping kinematics

state to neural activity, and w(t) and q(t) are Gaussian noise. A,C,W, and Q are fit from binned

velocity and neural feature(s), from the start to the end of each successful trial in the training dataset.

Once the model is learned, decoded kinematics can be inferred at every time step using the KF,

which is a closed-form recursion. When regressing neural data against kinematics to fit C, I allowed

the decoder to try different causal lags (i.e., neural data either aligned with or leading kinematics)

for each type of neural feature: spikes, LMP, or LFP power in different frequency bands (which were

computed using either 50 ms bins or, for lower frequency bands, longer bins slid every 50 ms). This

lag was applied globally to all electrodes. For the final decoder I fit parameters with whichever lag

minimized the residual. For LMP decoding the best lag was 100-125 ms (depending on the dataset),

while for spikes decoding it was 50 ms in monkey R and 150 ms in monkey J. However, when fitting

spikes decoders for closed-loop evaluation I used a 0 ms lag so as to be consistent with the methods

used in (Gilja et al., 2012; Fan et al., 2014). The offline spikes decoding difference between using the

best lag and 0 lag was small and not significant: correlation between true and reconstructed velocity

r = 0.73 vs. r = 0.72 for monkey R, and r = 0.80 vs. r = 0.77 for monkey J (p > 0.05 for a difference

in means, paired t-test). For closed-loop LMP decoding, I only applied the 100-125 ms lag to the

training data used to fit the models; no lag was used during closed-loop use. This choice follows with

the general control systems design principle that minimizing control loop lag improves performance

when feedback is used to change the control signal (Kuo, 1987), as is the case for closed-loop BMI

use. Reducing lag has been empirically shown to be best practice in the context of motor control

(MacKenzie & Ware, 1993) and BMI use specifically (Cunningham et al., 2010; Gilja et al., 2012;

Willett et al., 2012). By applying a lag in the training data only, I was trying to better align neural

correlates of movement intention with kinematics. During closed-loop use, I wanted the observation

of similar neural activity to then generate intended cursor velocity with as little delay as possible.

I also applied an additional operation during LMP-driven BMI use to help stabilize the cursor in

the workspace. A ‘centering velocity‘ pointing towards the center of the workspace was always added

to the decoded velocity, with magnitude 0.15 · r s-1, where r is the distance from the cursor to the

workspace center. Thus, the intervention was minimal when the cursor was near the workspace center

but increased when the cursor was near the edges. I emphasize that this assistance is not dependent

on target location; the centering velocity always moved the cursor towards coordinate (0,0), whereas

targets could appear throughout the workspace. This was helpful because a velocity-only decoder can

accrue a position bias over time; the centering velocity ameliorates this bias. The average centering

velocity applied during LMP-driven cursor control was up and to the left in both monkeys (monkey

R: −1.34± 0.19 mm/s x-velocity, 2.33± 0.30 mm/s y-velocity, grand mean ± std across datasets;

monkey J: −1.14 ± 0.23 mm/s x-velocity, 3.32 ± 0.50 mm/s y-velocity. All y-velocity means and


monkey R’s x-velocity mean were significantly different [p < 0.01] from zero by two-sided t-test). This

indicates that there was a down and to the right decoder bias. The centering velocity gain value of

0.15 was chosen based on a cursory parameter sweep during pilot experiments but was not thoroughly

optimized, either in magnitude or in directionality (i.e., the centering velocity was symmetrical

around the center and thus nave to the particular subject’s typical bias); a more sophisticated bias

correction scheme is a potential avenue for future work. I also sometimes applied this centering

velocity when evaluating hybrid decoders in the electrode-dropping experiment described later. To

prevent this from potentially giving hybrid decoders an undue advantage over spikes-only decoders,

I evaluated spikes-only decoders both with and without the same centering velocity and recorded

the higher performance of the two. Note that this conservative methodology slightly favors the null

hypothesis that hybrid decoding does not perform better than spikes-only decoding.

To evaluate whether decoding both LMP and spikes together would improve closed-loop performance,

I chose to start with a state-of-the-art spikes decoder and then compare it to a hybrid spikes + LMP

decoder. For the spikes-only decoder I used our previously reported Feedback Intention Trained

Kalman filter (FIT-KF) (Fan et al., 2014). The FIT-KF improves upon a standard velocity KF by

adjusting kinematics of the training data to attempt to better match the subject’s true intent: cursor

velocity is rotated to always point towards the target and is set to zero during target hold epochs.

Unlike in (Fan et al., 2014), here I kept a number of design choices from the original ReFIT-KF

decoder (Gilja et al., 2012) from which FIT-KF is descendent: I trained the decoder using Radial 8

Task reaching data, and used data starting from the beginning of each trial.

The hybrid spikes + LMP decoder was built by combining the spikes FIT-KF and LMP velocity

KF models, which were fit from the same training data. This decoder operated on a stacked feature

vector y = [yspikes;yLMP] which was mapped to kinematics by a stacked Chybrid = [Cspikes; CLMP].

Since Aspikes was fit from intention-estimated velocities, it differed slightly from ALMP; I set Ahybrid =

Aspikes and Whybrid = Wspikes. A combined covariance matrix Qhybrid was then calculated from the

training data and Chybrid, yielding the complete KF parameters {Ahybrid,Whybrid,Chybrid,Qhybrid}.Note that during closed-loop use, position feedback subtraction (Gilja et al., 2012) was performed on

the observed spike counts as in FIT-KF, but not on the observed LMP.

3.5.5 Performance measures

The primary metrics I used to quantify closed-loop task performance on the Continuous Target

Acquisition Task were success rate and normalized time to target (NTTT), which directly affect the

animal’s rate of liquid reward. Success rate is defined as the number of trials in which the target was

successfully acquired, divided by the total number of trials. NTTT is the time duration between

trial start and successful target acquisition (not including the required target hold time), divided

by the straight-line distance between this trial’s target center and the previous trial’s target center

(Suminski et al., 2010). NTTT is only computed for consecutively successful trials. Unless otherwise


stated, when comparing two different decoders, I first compared success rates using each decoder

(binomial test). If success rates were not significantly different, I then compared the NTTT with

two-tailed t-tests for a difference in means. A threshold of p < 0.01 was used for both comparisons.

Chance performance levels were calculated under the null hypothesis that the cursor’s movements

were not goal-directed towards acquiring the displayed target. I did this by replaying the time series

of actual cursor trajectories recorded during the BMI session while randomly selecting virtual target

locations drawn from the set of all targets presented during the experiment. A virtual trial ended

successfully if the cursor stayed within the target for the requisite hold time, or unsuccessfully if the

8 s time limit was exceeded. At the end of each virtual trial the next random target was selected and

the kinematics playback continued. Chance success rate and median NTTT were then calculated

by averaging across 1,000 repeats of this target-shuffled simulation. This method of calculating

chance performance differs from the two methods used in (Flint et al., 2013) and (So et al., 2014); in

particular, it yields a comparatively higher chance success rate when trial time limits are long, which

is tempered by long chance times to target. I chose to use this method because it is less prone to

underestimating chance success rates of undirected kinematics which nonetheless traverse much of

the workspace.

I also computed three additional metrics of successful trials’ kinematics. Index of performance (IP)

is a throughput metric for human-computer interface devices motivated by Fitts’ law (Fitts, 1954)

and is specified in the ISO 9241 standard (Douglas et al., 1999). It was adapted to two-dimensional

tasks (MacKenzie & Buxton, 1992) and adopted by the BMI field (e.g. Simeral et al., 2011; Gilja

et al., 2012). IP is calculated for each trial and rewards faster target acquisition while taking into

account differences in target size and distance:

IP =ID

(time to target), (3.5.3)

where ID is the index of difficulty which depends on this trial’s reach distance, D, which is the

Euclidean distance between the cursor’s position at the start of the trial and its position when the

target is successfully held (Simeral et al., 2011), and the width of the target, W :

ID = log2

(DW

+ 1). (3.5.4)

‘Dial-in time’ is calculated for each trial and measures the time between when the cursor first entered

the target acquisition area and when it last entered the target area prior to successfully staying inside

the target area for the requisite hold time 2012a. A dial-in time of 0 ms means that the monkey

reached and successfully held the cursor inside the target on his first attempt. A longer dial-in time

signifies that the cursor left the target acquisition area too soon, which required the monkey to

reacquire the target. Lower mean dial-in times signify better ability to stop the cursor over the target

and hold it there.


Path length ratio (sometimes also referred to as ‘distance ratio’ or the inverse of ‘path efficiency’)

is the ratio of the distance the cursor actually travelled from the beginning of the trial to when it

entered the target acquisition area, divided by the straight-line distance between the cursor’s position

at the start of the trial and the edge of the target acquisition area. Path length ratios closer to 1

signify straighter cursor trajectories.

3.5.6 Electrode preferred direction and decoder contribution

Data from the hand-controlled Radial 8 Task were used to compute the directional tuning of LMP, as

well as each electrode’s contribution to both the LMP and spikes velocity decoders fit from this data.

To compute a simple measure of preferred direction and tuning significance for a given electrode, I

first averaged this channel’s LMP over an analysis epoch spanning 200 to 600 ms into each trial. For

context, the median and standard deviation of movement start times (defined as when hand speed

reached 10% of the trial’s peak speed) was 144± 90.3 ms for monkey R and 222± 68.6ms for monkey

J. The hand entered the target after 511± 196 ms (monkey R) and 495± 141ms (monkey J). The

electrode’s LMP averaged over the analysis epoch yielded a single data point per trial. I then grouped

trials by reach target and performed a one-way ANOVA to test whether LMP was significantly

different for reaches towards at least one of the eight targets. A measure of each electrode’s preferred

direction was computed by summing eight individual vectors that pointed in the prompted reach

directions, where each vector’s length equaled this channel’s average LMP across all trials where the

monkey reached in that direction.

The magnitude of each electrode’s contribution to the decoder (Nuyujukian et al., 2014) was

used both for determining electrode dropping order and for comparing the cumulative electrode

contribution curves of spikes and LMP decoders. To find each electrode’s contribution to a particular

decoder I first converted that Kalman filter to a closed-form steady-state (Malik et al., 2011),

x(t) = M1x(t− 1) + M2y(t), (3.5.5)

where the jth column of M2 is the decoder weight vector wj = [wx,j , wy,j , 0]T that describes how

yj(t), the neural activity observed on electrode j, contributes to x- and y- velocity. Electrode j’s

decoder contribution is then computed as that electrode’s average effect on cursor velocity:

(contribution of electrode j) =∥∥E[wj · yj(t)]

∥∥ (3.5.6)

where the expectation is taken over time. An electrode’s relative contribution to the decoder is then

calculated by dividing that electrode’s contribution by the sum of all electrodes’ contributions. In

the electrode dropping experiments, for each condition we removed the top n electrodes ranked in

order of most to least contribution to the all-electrodes spikes decoder.


3.5.7 Comparing the contribution of LMP and spikes in the hybrid decoder

I dissected LMP’s and spikes’ respective contributions in the hybrid decoder using the steady-state

decoder form described above. At each 50 ms time step, the instantaneous contribution to decoded

velocity (‘neural push’) x(t) was divided into:

xspikes(t) = M2 · [yspikes(t); 0], (3.5.7)

xLMP(t) = M2 · [0;yLMP(t)], (3.5.8)

where yspikes(t) and yLMP(t) are the spikes and LMP feature vectors, respectively, and yspikes(t)

incorporates the position subtraction operation of the ReFIT-KF (Gilja et al., 2012). The relative

contribution of each signal (LMP or spikes) was then computed by dividing the magnitude of its

velocity contribution by the sum of both signals’ velocity contribution magnitudes. This metric

was averaged across all time steps from 200 ms after target presentation until when the cursor first

acquired the target.

I also used the partial decoded velocities xspikes(t) and xLMP(t) to find the angular error between

the direction pointing directly from the cursor towards the target and the direction that each signal

pushed the cursor. I report the mean angular error for LMP and spikes averaged across all time steps

included in the decoded velocity contribution analysis described above. This decode error metric

assumes that at each time step the animal’s true intended velocity points directly from the current

cursor position towards the target. However, due to delays in the control loop (of both biological and

technical origin), the neural signals observed at time t likely reflect an intended velocity towards

the target from the cursor’s earlier position at time t− τ . I therefore also computed angular error

as described above across different lag values of τ . I was particularly interested in the difference

in error-minimizing τ between the spikes and LMP signals because this could reveal differences in

how quickly the animal modulates each type of control signal. For this analysis I used each trial’s

data starting from 300 ms after target presentation (rather than 200 ms, as above) to allow testing a

wider range of τ .

For the analyses of closed-loop hybrid decoder use, all experimental sessions evaluating the

all-electrodes hybrid decoder were included. For the matched offline hybrid decoding comparisons of

Figure 3.10e,f,, I analyzed data from experimental sessions where the monkey performed the same

Continuous Random Target Task with his hand while neural data was recorded (812 trials from 2

sessions for monkey R, 1721 trials from 4 sessions for monkey J).

3.6 Project Acknowledgements

This work was supported by National Science Foundation’s Graduate Research Fellowships (SDS, JCK)

and IGERT award 0734683 (SDS), Stanford Medical Scientist Training Program (PN), NIH Pioneer


Award 8DP1HD075623 (KVS), NIH T-RO1 award NS076460 (KVS), and DARPA REPAIR award

N66001-10-C-2010 (KVS). I thank Mackenzie Mazariegos, John Aguayo, Michelle Wechsler, Claire

Sherman, Erica Morgan, Liana Yates for expert surgical assistance veterinary care, Chandramouli

Chandrasekaran for Simulink real-time code to access LFP data, Boris Oskotsky for IT support, and

Beverly Davis, Evelyn Casteneda for administrative assistance.

Chapter 4

Effect of Sensory Error Signals on

BMI Use

4.1 Summary

Movement is typically a closed-loop process in which sensory feedback conveying errors is transformed

into motor corrections. This creates a constraint for areas such as motor cortex, which both receive

error signals and generate efferent motor commands: error-related modulation must not leak out to

the effector unless it would generate the appropriate correction. While this is the case for natural

movement, in which motor cortex indirectly affects muscles, what happens when motor cortical

activity directly drives the effector via a brain-machine interface (BMI)? We tested this by perturbing

the kinematics of monkeys performing a BMI-controlled cursor task. Visual feedback drove cortical

responses as early as 70 ms after the perturbation, but this initial modulation did not cause deleterious

motor feedback. Rather, it was limited to an output-null subspace which does not effect the BMI

effector. When neural activity later changed in the output-potent dimensions (115 ms), it caused

corrective BMI movement.

4.2 Introduction

Motor cortex plays a causal role in generating output during both natural motor control and

intracortical BMI control. But, a major difference exists between these two contexts: during BMI use,

cortical activity directly affects the BMI with no intermediate processing except that imposed on the

recorded signals by the experimenter (schematized in Figure 4.1). In contrast, motor cortical output

to the real arm is modulated by other cortical, subcortical, cerebellar, and spinal intermediates (Wise

et al., 1997; Scott, 2004; Ramnani, 2006; Cisek & Kalaska, 2010; Shmuelof & Krakauer, 2011) as

73

CHAPTER 4. EFFECT OF SENSORY ERROR SIGNALS ON BMI USE 74

BMI Control

Perturbation

computation

sensoryfeedback

directmotoroutput

computation

sensoryfeedback

Perturbation

indirectmotoroutput

Natural Arm Control

Figure 4.1: Should we worry about error signals affecting ongoing control?During natural motor control, cortical output to the arm is modulated by multiple intermediaries.In contrast, during BMI use motor cortex directly drives the BMI and also participates in ongoingfeedback control computations. The principal question of this study is whether the direct feedbackloop between motor cortex, BMI output, and sensory information about the state of the effector willcause error-related cortical modulation to deleteriously affect ongoing control.

well as internal output-gating mechanisms (Kaufman et al., 2014). What are the consequences of

this difference in light of the fact that motor cortex is also strongly modulated by sensory feedback

carrying motor error signals (Georgopoulos et al., 1983; Archambault et al., 2011; Pruszynski et al.,

2014)? During natural arm use, sensory signals are transformed into the appropriate outputs before

they affect muscles; motor cortical activity related to error signals does not deleteriously affect

ongoing control. Put more plainly, despite motor cortex responding vigorously when something

bumps into the arm, this does not cause immediate and undirected arm flailing.

It remains unknown whether during BMI use, error-related modulation of the same cortical

population driving the BMI output will feed back out to the effector as a maladaptive command.

If this is the case, this unwanted feedback will present an obstacle to real-world use of physical

BMI-driven prostheses such as robotic arms, which will interact and collide with other objects in

the environment. To answer this question, we used the longstanding motor neuroscience strategy

of applying a carefully controlled, discrete perturbation to the motor system and observing how

it responds to — and corrects for — errors (Wolpert & Miall, 1996; Scott, 2004; Shadmehr et al.,

2010; Pruszynski et al., 2011b). We introduced perturbations to the comparatively better-studied

experimental preparation in which monkeys perform a 2D BMI cursor control task. Specifically, we

added a sudden cursor position offset (‘cursor jump’) at different phases of the trial while the monkey

was trying to bring the cursor to a target. Although several groups have studied how the sensorimotor

system responds to a continual perturbation consisting of changing the neural-to-kinematic mapping

of the BMI decoder (Jarosiewicz et al., 2008; Chase et al., 2012; Sadtler et al., 2014; Golub et al.,

2015), to our knowledge this is the first time that the response to a step perturbation has been

studied. These discrete and severe perturbations allowed us to observe how neural dynamics and


BMI output evolved following the error-causing event and answer key questions about when and how

sensory feedback leads to corrective BMI output.

4.3 Results

We investigated the effect of error signals on ongoing BMI control using a ‘Cursor Jump Task’. Two

monkeys (‘J’ and ‘L’) performed a 2D target acquisition task in which we occasionally perturbed their

neurally-controlled cursor by introducing a large instantaneous 6 cm position offset orthogonal to the

12 cm vector from the workspace center to the target (Figure 4.2a). The perturbation could be in

one of two directions and, if a particular trial was randomly chosen for perturbation, the jump could

happen after one of four ‘jump location conditions’ was met: after the cursor traveled 3, 6, or 10 cm

along the ‘Task Axis’ connecting the center start location to the radial target, or 250-300 ms into the

500 ms target hold period required for successfully completing the trial. Since the context of these

jump location conditions was rather different (starting movement, mid movement, acquiring target,

or trying to keep the cursor still), we analyzed trials for each jump location condition separately.

Both monkeys performed the BMI Cursor Jump Task well, with individual dataset success rates

ranging from 99.0% to 99.9% in monkey J and 96.0% to 99.2% in monkey L. Figure 4.2b shows

perturbed trial cursor trajectories from a representative dataset. Following perturbations during

movement, trajectories initially continue along the Task Axis. After a reaction time delay, they curve

towards the target. After jumps during the hold period, the cursor takes a mostly straight trajectory

back towards the target. Although we will examine the changes in neural activity in more detail later,

these cursor trajectories already suggest that there is little to no deleterious cursor deviation due to

sensory feedback signaling the error interfering with ongoing BMI control. Cursor jumps increased

the time it took the monkeys to acquire the target (Figure 4.2c), with the penalty increasing for the

cursor jump conditions occurring later. Monkey J’s time to target penalty (compared to unperturbed

trials) ranged from 16% (jump after 3 cm) to 58% (jump during hold), and monkey L’s penalty

ranged from 19% to 59%. This ordering by jump location is not surprising, given that the geometry

of the task meant that earlier perturbations allowed the monkeys to “cut the corner” when making

corrections back to the target.

No additional training was needed for these monkeys — who had years of prior BMI-controlled

cursor task experience — to adjust to the Cursor Jump Task; my impression was that they were

completely nonplussed by the perturbations. This is corroborated by there being no significant

correlation between chronological experimental session number and mean time to target for any jump

location condition (p > 0.05, linear regression, for both monkeys).


12

85

15

27

16

84

20

23

20

44

0

1000

2000

10

43

12

11

13

42

16

46

16

47

Tim

e T

o T

arg

et (m

s) Monkey J Monkey L

X Position (cm)

Y P

ositio

n (cm

)

10630-3-6-10

-10

0

10

Task Axis

Pe

rtu

rba

tio

n A

xis

Threshold Crossings

DecodedVelocity

Cursor Position

VisualFeedback

∑

KalmanFilter

Perturbation

Time

Δ P

ositio

n

{

MotorCortex

a

c

b

No jumpJump at 3 cmJump at 6 cmJump at 10 cmJump during hold

Figure 4.2: BMI cursor jump task(a) We observed how a BMI user respondsto perturbations of their BMI-controlledcursor. Monkeys were implanted withmultielectrode array(s) and trained toperform a 2D cursor acquisition task.Cursor velocity commands were decodedfrom multiunit spikes using a Kalman filter.After decoder training, the monkeysperformed a two-targetcenter-out-and-back task. On a random25% of outward trials, we applied aperturbation consisting of a 6 cm positionstep between successive monitor updates.Perturbations were randomly selected tooccur at one of four location/times duringthe trial, and in one of two directions.(b) Cursor trajectories from one exampledataset showing the cursor jump trials(colored traces) and 100 of the unperturbedtrials (gray traces). The perturbation wasapplied once the cursor had traveled 3 cm(green), 6 cm (orange), or 10 cm (blue)along the Task Axis towards the target, orduring the target hold period (violet).Dataset J.2015.01.19.(c) Perturbations added a modest time totarget penalty that was larger for laterperturbations. Bars show the mean SEMof time to target, not including the finalhold time and not including the “stolen”hold time for jump during hold trials. Alldifferences were significant (p < 1e-5,two-sample t-test) except Jump after 10cm vs Jump During Hold.


4.3.1 Perturbations evoked an early motor cortical response

without accompanying cursor velocity output

We now examine in more detail how motor cortical neural activity changed following a cursor jump

perturbation. Figure 4.3a shows the change in firing rates, across the recorded population, between

perturbed and unperturbed trials to the same target. We could detect statistically significant firing

rate changes (p < 0.001, shuffle test) as early as 69 ms after the perturbation in monkey J and as

early as 111 ms in monkey L. This error-related response took the form of an initial transient which

diminished somewhat before being followed by a larger change that persisted for the duration of the

cursor trajectory correction. These changes involved both firing rate increases and decreases across

the neural ensemble, with a preponderance of increasing rates (Figure 4.4).

As described so far, our analysis of the error-related response captures the magnitude of changes

across all electrodes, regardless of how these changes affected the BMI. We next examined the same

neural activity changes through the lens (projection matrix, to be more literal) of the BMI decoder.

Since the relationship between recorded neural activity and velocity was completely specified, we

could examine exactly how the perturbation-evoked neural response affected decoder output by

analyzing its ‘neural push’ (see Methods). Analysis of this neural push (Figure 4.3b and Figure

4.5) allowed us to make two key observations. First, the early change in firing rates did not affect

decoded velocity: neural push did not deviate from that of unperturbed trials until at least 35 ms

after the early firing rate changes (the precise time of decoder-potent response is calculated in the

next section). Second, once the neural push did start to change, it was a corrective push opposing

the perturbation (positive deflection of the Perturbation Axis traces in Figure 4.3b). The concurrent

negative deflections in neural push along the Task Axis (most visible for monkey L) correspond

to a reduced push along that dimension due to turning towards the target. The exception to this

was monkey J’s jump during hold condition, in which there was a brief and small initial push away

from the target before the error-related response began to push the cursor towards the target. This

maladaptive response, which was nonetheless still well after the earliest firing rate change for this

condition, may reflect a slight “leak” in what otherwise appears to be strong separation between

early error-related processing and BMI output. We now discuss this separation in more detail.

4.3.2 Initial error-related neural activity was confined to a decoder-null

space

Why was the early error-related change in firing rates invisible from the perspective of the BMI

output? We hypothesized that the changing neural activities were canceling out, i.e. occurring in

the null space of the decoder’s mapping from firing rates to velocity (schematized in Figure 4.6a).

Inspecting the neural responses to specific perturbation conditions on individual electrodes (e.g.

Figure 4.6b,c) suggested that different electrodes’ early firing rate changes generate velocity push


0 200 400 600

−1

0

3

Time After Cursor Jump (ms)

Δ N

eura

l P

ush T

ow

ard

s

Targ

et

(cm

/s)

0 200 400 600

−1

0

6

Task AxisPerturbation Axis

0 200 400 600-2

0

10

111 ms

∆ F

irin

g R

ate

(H

z)

0 200 400 600-2

0

6

69 ms


a

b

Monkey J Monkey L

Figure 4.3: Rapid motor cortical perturbation response without deleterious BMIoutput(a) Change in multiunit firing rates attributable to the cursor jump was computed by subtractingunperturbed trials’ mean firing rates from the responses following the jump. Each trace shows thevector norm of this difference across the 192 (monkey J) or 96 (monkey L) electrodes, averaged acrossall target/jump-direction/dataset conditions. Colors correspond to different perturbation conditionsas in Figure 4.2. Vertical lines mark the earliest detectable change in neural activity (p < 0.001).(b) Change in ‘neural push’ following the perturbation, i.e. how the neural response contributedto change in cursor velocity (the actual cursor trajectory also depended on the momentum termintroduced by the Kalman filter). Neural push is projected separately onto the Perturbation Axis(solid traces) and Task Axis (dashed trace). To aid visual comparison, the vertical lines marking firstfiring rate change from panel a are repeated here. Note that these earliest firing rate changes are notaccompanied by output change as far as the BMI is concerned. Note also that when decoded velocitydoes change, it pushes the cursor towards the target.


Monkey LMonkey J

0 200 400 600

0 200 400 600

0 200 400 600

0 200

0 200 400 600

-10

0

10

0 200 400 600

-10

0

10

0 200 400 600

-10

0

10

0 200

-10

0

10

Time After Cursor Jump (ms) Time After Cursor Jump (ms)

Jump during hold

Jump at 3 cm

Jump at 6 cm

Jump at 10 cm

∆ F

irin

g R

ate

Fo

llow

ing

Pert

urb

atio

n (H

z)

Figure 4.4: Perturbations evoked both increases and decreases of firing ratesEach panel shows a 2D histogram of the distribution of perturbation-evoked firing rate changesfor a given jump location condition. These changes were calculated by subtracting the meanunperturbed response from the mean perturbed response for a given condition, as in Figure 4.3. Agiven dataset/target/jump direction condition (such as the examples shown in Figure 4.5) contributedone datum for each electrode at each time point. These were then counted in bins that are 1 mswide and 1 Hz tall. The color scale maps linearly from 0 (blue) to the maximum number of points inany bin for a given panel (yellow). Distribution means (gray traces) and medians (black traces) areshown for every time point. Distribution tails (|∆firing rate| > 10 Hz) are cut off so that the bulk ofthe distribution can be seen clearly.


0 200 400 600

-1

0

1

2

3

0 200 400 600

-1

0

1

2

3

4

5

0 200 400 600

-2

0

2

4

6

0 200 400 600

-2

0

2

4

6

-1.5

-01

-0.5

0

0.5

1

1.5

0

1

2

3

4

2

4

6

8

10

-2

0

2

4

6

-1

0

1

2

3

4

5

-1

0

1

2

3

-4

-2

0

2

4

-4

-2

0

2

4

X Position (cm)

Y P

ositio

n (cm

)

2-10

-3

14

2-14

-9

5

14-3-9

2

8-2

-14

3a

b

c

d


Δ N

eu

ral P

ush

(cm

/s)

Task A

xis

(c

m/s

)P

ert

urb

atio

n A

xis

(cm

/s)

unpertubed

after perturbation

Jump at 3 cm

J.2015.01.20 J.2015.01.12 L.2015.01.07 L.2015.01.14

Jump at 6 cm Jump at 10 cm Jump During Hold

Δ Neural Push

Figure 4.5: Examples of neural push following a cursor jumpEach figure column corresponds to the neural push analysis of trials from one example condition,which consists of a particular target, cursor jump direction, jump location, and experimental session.All four targets and jump locations are represented, and there are two examples from each monkey.(a) Cursor trajectories for all perturbed trials of each example dataset/condition that met the trialinclusion criteria (see Methods). The cursor started near coordinate (0,0) and proceeded towardsthe target whose acceptance boundaries are denoted by the gray square. (b) The neural pushcomponent along the Perturbation Axis; positive push corresponds to the direction opposing thecursor jump. Unperturbed responses were generated by aligning to ‘faux jumps’, i.e. when the jumpwould have happened in an unperturbed trial had that trial been selected for perturbation. Notethat the duration of the Jump During Hold unperturbed trace is limited because these trials endedshortly after the faux jump. (c) Same as (b) but for the neural push component along the Task Axis.Positive corresponds to push in the target direction (gray arrows in (a)). (d) Difference in neuralpush was calculated as the neural push of jump trials minus the mean neural push for unperturbedtrials to a given target. Neural push is plotted separately along the Perturbation Axis (solid line)and Task Axis (dashed line). Figure 4.3b was generated in the same manner, but averaged across alltarget locations, jump directions, and datasets.


contributions which oppose each — but this intuition had to be verified at the population level. To

do so, we once again leaned on the analytical power afforded by having full knowledge of the mapping

from motor cortex to the BMI output. Figure 4.7a shows the error-related neural response projected

into the decoder-potent subspace (this is the same as the vector norm of the neural push shown in

4.6b) and an orthogonal decoder-null subspace (which is the null space of the Kalman filter gain

matrix). When viewed from this angle, the decoder-potent error-related response (i.e. the corrective

response) began 117 ms after the cursor jump in monkey J (147 ms in monkey L). Decoder-null

error-related responses began significantly earlier: 69 ms after the perturbation in monkey J (111 ms

in monkey L). This suggests that the intuition from Figure 4.6c is correct: there was no net neural

push when early error-related activity was summed across the full neural ensemble contributing to

the BMI output.

We wondered whether this phenomenon of the initial error-related responses being confined to

decoder-null dimensions was a consequence of the ReFIT decoder training protocol (Gilja et al., 2012

and Methods). Perhaps the refitting process optimized the decoder weights such as to minimize the

effect of error-related neural activity evoked during the “jittery” cursor movements of the closed-loop

calibration block? To control for this, we performed the same analysis, now projected into the

decoder-potent and decoder-null subspaces of a velocity Kalman filter fit from the initial passive

observation block. In this first training step, the monkey watched automated, perfectly straight

cursor movements to targets (Figure 4.8). Error-related responses projected into these dimensions

exhibited similar decoder-null followed by decoder-potent dynamics as observed for the final ReFIT

decoder, ruling out the decoder recalibration process as the cause of this effect.

However, the results described thus far could also be explained by a mechanism more trivial than

the “canceling out” arrangement I’ve suggested: there could be a separate neural subpopulation

receiving error signals that is distinct from the BMI-controlling subpopulation. If this were the case,

then the early decoder-null activity could be decoder-null simply because it is restricted to electrodes

that did not affect the BMI. To test for this, I repeated the decoder-potent, decoder-null projection

analysis for a subset of the electrodes that contributed substantially to the decoded velocity (see

Methods for the decoder weight calculation). Depending on the dataset, this data came from 24-30

electrodes in monkey J (cumulative decoder weight 0.5) and 28-38 electrodes in monkey L (cumulative

weight 0.8; Figure 4.9 shows each monkey’s decoder weight curves). Figure 4.7b shows that in monkey

J, the early initial transient was still present in these high-weight electrodes’ decoder-null subspace,

with the first decoder-null activity change occurring 79 ms after perturbation. This effect was weaker

in monkey L, with the first decoder-null activity 137 ms after perturbation, only 10 ms before the

decoder-potent correction. Thus, we can rule out the separate-subpopulations alternate hypothesis

definitively in monkey J but with less confidence in monkey L.

Interestingly, in monkey J removing low-weight electrodes from the analysis also had the effect

of increasing the earliest component of the decoder-potent neural push. This suggests that despite


0 500

-20

0

80

-30

0

130


Decoder Push Direction

Neuron 2

Neuron 1

Δ F

irin

g R

ate

(H

z)

2 cm

Dec

oder-N

ull

Decoder-P

otent

Correct Left

Correct Right

Δ Firing Rate Neuron 1

Δ F

irin

g R

ate

Neuro

n 2

a

b

c

Figure 4.6: Decoder null to decoder potent neural dynamics hypothesis(a) Cartoon view of a mechanism by which early error-related neural activity could avoid affectingthe BMI. Consider two neurons whose increased firing push the cursor velocity in opposite directions.If both neurons increase or decrease their rates (blue line), the net effect on the decoder output iszero. But if one neuron increases its rate and another decreases its rate (e.g. green line), this willchange the cursor’s velocity. These combinations of neuron 1 and neuron 2 activity occupy orthogonaldecoder-null and decoder-potent dimensions. For a BMI driven by E electrodes with a 2-D velocityoutput, there will be an E − 2 dimensional decoder-null space and a 2-D decoder-potent plane. Wehypothesized that error-related signals first evolved in this decoder-null space, and that changes onlyentered the decoder-potent plane when the activity was such as to generate the desired correctiveBMI output.(b) Perturbed trials’ cursor trajectories for an example dataset/condition (upward target, leftwardperturbation after 10 cm) corresponding to the neural data in (c). From Dataset J.2015.01.14.(c) Hand-picked example perturbation-evoked neural response changes more concretely illustrate thedecoder-potent and decoder-null mechanism hypothesis. The arrows show the direction of decodervelocity contribution due to increases of firing rates on each electrode. Shading illustrates an initialdecoder-null epoch (blue) in which these electrodes’ neural activity roughly cancels out as far as thedecoder is concerned, followed by a decoder-potent epoch (green) when the net output is rightwardneural push. Although real data is shown, it is presented as an illustrative cartoon; the magnitudeand time courses of these two electrodes’ first phase of activity does not fully cancel out. Indeed,it would be unlikely to find two example electrodes that truly exemplify what is in actuality ahigh-dimensional, population-level phenomenon.


0 50 100 150 200 250


0 50 100 150 200 250

Perturbation Direction Differences

76 ms 146 ms

Jump at 3 cmJump at 6 cmJump at 10 cmJump during hold

0

∆ D

eco

der-

Po

tent

(AU

)∆

Deco

der-

Nu

ll (A

U)

High Decoder Weight Electrodes Only

79 ms 137 ms

107 ms 185 ms

69 ms 111 ms

147 ms117 ms

a

b

c

Monkey J Monkey L

Figure 4.7: Initial error-related modulation is restricted to the decoder’s null space(a) The initial neural responses to cursor jump perturbations were confined to the decoder-nullsubspace, and only later entered decoder-potent dimensions. We plot change in neural activityprojected into decoder-potent (upward-going traces) and decoder-null (plotted downward-going forvisual clarity) dimensions. Unperturbed trials’ neural projections into each space were subtractedfrom the condition-matched perturbed trials’ projections, and we plot the vector norm of thesedifferences. Vertical lines mark the first time of significant neural activity change (p < 0.001).(b) Same as (a) but restricted to electrodes with high decoder weights. The initial decoder-nullresponse is still present, indicating that early error-related information is not restricted to a neuralsubpopulation with minimal influence on the BMI output.(c) Instead of plotting the difference between perturbed and unperturbed trials, here we plot the normof the population response difference between cursor jumps in opposite directions. This difference isonly shown in the decoder null space, since decoder-potent jump direction difference is essentiallythe same as decoder-potent perturbation response. This analysis reveals that the early decoder-nulltransient contains direction-specific perturbation information.


Jump at 3 cmJump at 6 cmJump at 10 cmJump during hold

200 25010050 15002500 50 100 150 200


147 ms

113 ms69 ms

120 ms

0

∆ D

eco

der-

Po

tent

(AU

)

∆ D

eco

der-

Null

(AU

)Monkey J Monkey L

Figure 4.8: Responses in the potent and null space of an observation-trained decoderThe data shown in Figure 4.7d were re-analyzed with the decoder-potent and decoder-null subspacescalculated with respect to a velocity Kalman filter trained using data from the first block of the ReFITdecoder training protocol, in which the monkey passively watches automated error-free movements ofthe cursor to targets. Cursor perturbation-evoked neural responses projected into these dimensionsare very similar to those shown in Figure 4.7d, which were projected into the null- and potent-dimensions of the final (closed-loop re-calibrated) ReFIT decoder.

having a low weight in the decoder, these electrodes still played a role in the delicate balance which

minimized the effect of early error-related activity on decoder output during closed-loop BMI use.

4.3.3 Perturbation-specific information in the early error-related response

Finally, to better understand what information was present in this early decoder-null motor cortical

response, we took advantage of having presented the monkeys with cursor jumps in two different

directions at any given perturbation location (e.g., jump up or jump down after traveling 3 cm

towards the rightward target). If this early transient was agnostic to the details of the perturbation

(for instance, if it was a generalized surprise or error signal), we would expect no early difference in

the response to jumps in different directions. If, however, this response carried information about

more specific aspects of the perturbation, such as the cursor’s position or a target-cursor error vector,

then we would expect to see a divergence in responses to jumps in different directions. Figure 4.7c

shows the decoder-null difference in post-jump neural activity between responses following different

jump directions, averaged across all target/jump-location/dataset conditions. In monkey J, there is

clear jump directional information as early as 76 ms after the perturbation. For monkey L, whose

data is noisier, the effect is small, but there was still a significant jump direction difference starting

at 146 ms for the jump after 3 and 10 cm conditions. For the 3 cm condition this was substantially

earlier than the decoder-potent perturbation response.


961 400

0.1

Cu

mu

lati

ve

Deco

der

Weig

ht

Electrode Rank

0

1

1921 100

Fra

cti

on

D

eco

der

Weig

ht

0

0.05

0

1

Monkey J Monkey L

0.5 0.8

Figure 4.9: Electrode decodercontribution weight curvesRanked electrode decoder contributionweights are shown in red, whilecumulative decoder weight for the top Eelectrodes are showed in black.Electrodes are sorted in order ofdescending decoder weight. Thinnerfaded traces show the weight curves forindividual experimental sessions; thickertraces show the means of these individualdays’ weight curves. Dashed vertical linesmark when the mean cumulative weightexceeds the threshold used for inclusionin Figure 4.7b.

4.4 Discussion

When we introduced cursor jump perturbations while monkeys used a BMI, we observed error-related

modulation in motor cortex with latency consistent with the earliest visual feedback to this area. The

presence of this activity in the same neural population directly driving the BMI raised the concern

of a deleterious feedback loop by which this response would “leak out” into the BMI output before

it was appropriately shaped to properly direct the corrective movement. Instead, we found that

the early error-related activity was confined to the decoder’s null space and thus did not interfere

with BMI control. Neural responses became output-potent tens of milliseconds later, at which time

they directed the cursor towards the target. This beneficial phenomenon allowed the monkeys to

effectively correct for BMI perturbations. I will now discuss the significance of these results for

understanding BMI use (and motor control more broadly) as a feedback control process, as well as

implications for the future design of BMIs.

4.4.1 BMI error correction latency

Control loop latency is a critical characteristic of any control system, and provides a lower limit

on how well the system will be able to correct for unanticipated errors. Our results provide the

most direct measurement to date of the control loop latency of a skilled BMI user relying on visual

feedback: 117 and 147 ms for the two monkeys. For simplicity, and given that this was a conservative

estimate limited by our statistical power and monitor refresh latency, we will henceforth refer to our

estimate of earliest BMI corrective response as occurring as early as 115 ms. This measurement is

consistent with the 100 – 133 ms range reported in (Golub et al., 2015), which was calculated based

on reaction time to target presentation rather than perturbation correction. These measurements also

provide a benchmark against which to measure reductions in feedback loop time that will hopefully


115 ms

X Output DimensionError dim.

Y O

utpu

t Dim

.

Rotate

X Output Dimension

Y Output Dim.Erro

r Dim

ensi

on

70 ms

Unperturbed

Perturbation Up

Perturbation DownX

Y

Neural Trajectory

Cursor Trajectorya

b

c

Figure 4.10: Summary schematic ofearly error response confined to anoutput-null space(a) Cursor trajectory for this example. Arightward cursor movement to the target isperturbed (red circle) either upward (blacktrajectory) or downward (red trajectory).(b) Neural activity viewed such that adecoder-null dimension occupied byerror-related activity is on the vertical axis, anda decoder-potent dimension corresponding tox-axis velocity is on the horizontal. For visualclarity, this horizontal dimension alsoapproximates time (as if the cursor is also beingpushed faster and faster). Modulation in thedecoder-null dimension begins approximately 70ms after the perturbation, with perturbedneural trajectories (red and black) divergingboth from the unperturbed (gray) neuraltrajectory and from each other.(c) The same neural activity, now rotated 90degrees out of the page such that the verticalaxis is a second decoder-potent dimensionwhich corresponds to decoded y velocity. Theerror-related neural modulation does not affecty velocity output until roughly 115 ms after theperturbation, at which point the red and blacktrajectories also diverge along the y velocitydimension. There is a 45 ms period in whichmotor cortical activity is changing (andpossibly performing computations related toerror correction) without “leaking” this activityto the BMI effector.


be achieved by artificially writing proprioceptive and somatosensory feedback from the BMI into the

user’s brain (O’Doherty et al., 2011; Bensmaia & Miller, 2014; Dadarlat et al., 2014).

4.4.2 Early BMI error-related signals are present in motor cortex but do

not interfere with ongoing BMI control

The entire question of whether or not error-related signals interfere with ongoing BMI control hinges

on such signals actually being present in the motor cortical areas from which we decoded the BMI

control signal. The fact that BMI use is a closed-loop process implies that error signals must, of

course, eventually affect motor cortical output. But until this study, it was unknown whether this

activity is present early after the perturbation, or whether it only enters these areas later after it

is fully processed and in the form of a corrective output command. Our data support the former

hypothesis: we observed error-related modulation in motor cortex that arrived as early as 70 ms

after the perturbation, well before BMI output correction. This is consistent with the earliest

visually-driven re-targeting responses reported in motor cortex during arm control (Georgopoulos

et al., 1983; Archambault et al., 2011) but is considerably slower than the 50 ms latency reported for

proprioception-mediated error responses in M1 (Pruszynski et al., 2011a). Fortunately, we also found

that this early response did not affect the BMI output and did not appear to interfere with ongoing

control. We will discuss the how — and speculate about the why — of this in the next section.

4.4.3 Computing in an output-null subspace provides a mechanism by

which BMI output can ignore early error signals

To explain the mechanism — at least at a computational Marr level (Marr & Poggio, 1976) — of

how the initial error-related activity did not affect BMI output, I adapted Kaufman and colleagues’

‘muscle null’ hypothesis for explaining how motor cortical activity during reach planning does not

affect muscles until after the go cue is given (Kaufman et al., 2014). My analysis was made drastically

easier by the fact that, unlike in that study, partitioning neural activity into BMI decoder-potent and

decoder-null subspaces is both much easier and more accurate than approximating a linear mapping

between cortical activity and the muscles. Figure 4.10 summarizes my working model of how BMI

error-related neural activity transforms from an early response confined to decoder-null dimensions

into a later decoder-potent corrective response.

This study adds to a growing body of knowledge suggesting a computational motif in which

balanced changes in motor cortical activity permit network changes related to motor planning

(Kaufman et al., 2014), motor-reported sensory detection (Zagha et al., 2015), or motor learning

(Li et al., 2001) to occur without immediately affecting the efferent output. Although our study

can only directly speak to this computational strategy being used during a motor BMI task, we

speculate that a similar mechanism is at work following perturbations during natural arm control.


We believe that experiments to test this prediction would be of high value, but note that they will be

made more challenging by the aforementioned difficulty in definitively identifying the muscle-null

and muscle-potent neural dimensions.

The possibility that these output-null to output-potent neural dynamics are shared with natural

motor control also presents a hypothesis about why motor cortex exhibits this advantageous behavior

during BMI use — seemingly without the animals having had to learn to squelch the error-related

signal into output-potent dimensions. A slightly inverted way of asking this question is, why were the

decoder’s weights such that early error-related responses occurred in their null space? I suspect this

was because the decoders used in our study were “biomimetic” in the sense that we sought, to the

best of our ability, to match them to an intuitive pre-existing neural-to-kinematic mapping (Jackson

& Fetz, 2011; Shenoy & Carmena, 2014). In these experiments, the decoder was initiated based on

the patterns of neural activity evoked by the monkeys observing automatic cursor movements and

“following along” with some sort of motor imagery or mental rehearsal. This leads me to speculate

that error-related signals were in the output-null space of the decoder because it is similar to the

output-null space of the muscles. If the motor system has evolved or learned to sequester error

processing into output-null neural dimensions, then biomimetic BMI use might be making use of this

same arrangement. In other words, it could be that the intrinsic manifold (Sadtler et al., 2014) of

intuitive BMI mappings exploits not only the pre-existing output-generation neural machinery, but

also its closely coupled input-processing properties.

However, another possibility which we cannot rule out is that the monkeys’ years of previous

BMI experience, which did not include cursor jumps but did involve spontaneous decoding errors,

led to network changes which minimized undesired effects of sensory feedback. The question of

whether the brain can learn to sequester error-related signals into an output-null subspace could

be answered with a future learning study in which the decoder is intentionally modified such that

the early transient does affect output. This decoder could then be held fixed as the monkey uses it

for a prolonged period, allowing tracking of both BMI output and underlying neural changes as in

(Ganguly & Carmena, 2009; Ganguly et al., 2011).

4.4.4 Differences between responses to different jump location conditions

Deeper investigation of the following two comparisons between how the monkeys corrected BMI

perturbations occurring at different phases of the trial are outside the scope of the current report, but

we feel they are worth remarking upon. First, we were surprised that neural push did not increase

more rapidly for later cursor jump locations, as one might expect under many optimal feedback

control cost functions (Todorov & Jordan, 2002; Scott, 2012). This could be evidence for there

being a limit to how rapidly the motor cortical network can change its firing rates in the service of

error correction goals. On the other hand, it could simply reflect a lack of motivation on the part

of the monkeys to correct these later mistakes “extra hard”, perhaps because the trial time limits


were generous. Second, decoder-potent activity seemed to ramp up slower (in magnitude slope, not

in earliest detectable deviation) for the jump during hold condition. I speculate that this may be

because the neural network is in a less labile/responsive state when it is explicitly trying to hold the

cursor still than when it is in the dynamic phase of movement to the target.

4.4.5 Implications for future BMI design

The primary focus of this study was to address a previously raised concern (e.g. Green & Kalaska,

2011) that the unnatural arrangement of performing a motoric task while bypassing normal cortical

and subcortical circuits may incur a penalty when trying to make rapid error corrections. Our finding

no deleterious feedback loop, as well as the overall proficiency with which the monkeys corrected for

the perturbation, suggests that this may not be as problematic as one might think — at least for

2D cursor tasks. This came as a pleasant surprise; I had expected to see deleterious feedback and

anticipated having to develop methods to adjust the decoder to mitigate this feedback. However, our

study does demonstrate that early visually-driven, error-related signals are very much present in

homologues of the cortical areas currently being used for human robotic arm control studies (Vogel

et al., 2010; Hochberg et al., 2012; Collinger et al., 2013b; Wodlinger et al., 2015). Our findings should

therefore serve as a warning to be on guard for deleterious error signals, as it as of yet unknown

whether this decoder-null, decoder-potent separation holds up 1) in humans and 2) during high

degree-of-freedom arm control in which the output-potent subspace is larger. That said, our results

also provide hope that these initial error-related signals can be sequestered in an output-null neural

subspace. This helpful arrangement might fall out “for free”, especially if a biomimetic approach is

taken, or it could emerge after longer-term neuroprosthetic skill learning (Ganguly & Carmena, 2009;

Orsborn et al., 2014), perhaps thanks to strategic decoder adjustments to guide beneficial learning

(Shenoy & Carmena, 2014).

A second major point of this study concerns BMI control loop latency. As mentioned earlier, this

latency is a critical control system property with profound consequences for the controllability of

the BMI (Chase et al., 2009; Cunningham et al., 2011; Stavisky et al., 2015a). Although we found

that the monkeys did not begin to correct perturbations for at least 115 ms, error-related motor

cortical signals contained information about the cursor jump direction as early as 76 ms after the

perturbation. This presents an opportunity for future work investigating whether one could decode

this early signal in order to automatically correct for the error faster than the BMI user would

otherwise be able to. This proposed motor performance error auto-correction has parallels to our

ongoing work to detect and correct for (binary) task outcome errors (Even-Chen et al., 2015).


4.5 Methods

4.5.1 Cursor jump task


and Use Committee. Two rhesus macaques (monkeys ‘J ’and ‘L’) were trained to sit head-fixed in

a primate chair and perform 2D target acquisition tasks by controlling an on-screen cursor with

either their hand position or via a BMI in order to obtain a liquid reward. The task was displayed in

virtual reality with a latency of 7± 4 ms as described in (Gilja et al., 2012). For this study, we used

a “hand restrained training, hand restrained BMI control” paradigm (Shenoy & Carmena, 2014) in

which the monkey’s arms were gently restrained during open-loop BMI training, closed-loop BMI

training, and the main task.

The Cursor Jump Task at the core of this study was a two-target variant of the traditional

center-out-and-back Radial 8 Task. We used only two targets to increase the number of repetitions of

each target/perturbation condition. The monkey controlled the velocity of a cursor to acquire targets

by bringing the cursor within a 4× 4 cm target acquisition area and holding it there for a continuous

500 ms. The target changed color to signify that the cursor was within the acquisition zone. Leaving

the target during this hold period reset this timer but did not result in immediate failure. If the

monkey acquired the target within a 5 s time limit, he received a liquid reward. Otherwise, he heard

a failure tone and was presented with the center target again. The task always began with the

center target being presented. After a successful center acquisition, one of the two radial targets was

presented. The two possible radial targets were arranged either along the horizontal axis or vertical

axis (the ‘Task Axis’), and the sequence of targets was random. To increase the number repetitions

of a given perturbation type per dataset, the orientation of the Task Axis was fixed within a given

experimental session, but varied across experimental sessions. The next target was presented 20 ms

after either success or failure, and the cursor was under continuous neural control.

The key experimental manipulation was that on a random 25% of trials towards radial targets, a

perturbation was applied. This perturbation consisted of offsetting (“jumping”) the cursor position

by 6 cm perpendicular to the Task Axis (we call this dimension the ‘Perturbation Axis’, see Figure

4.2b). These perturbations were applied on the next decode update time step after one of four

randomly selected and equally likely ‘jump location conditions’ were met. Three of the conditions’

had a spatial criterion: the jump happened after the cursor travelled either 3 cm, 6 cm, or 10 cm

towards the target along the Task Axis. In the fourth condition, the cursor jump happened on the

decode time step prior to 300 ms into the target hold period. Empirically, the mean times when

this happened was 287.0 ms into the hold for monkey J (25 ms decode time steps) and 274.0 ms

for monkey L (50 ms decode time steps). Only one perturbation could happen per trial, and the

different perturbation conditions were interleaved throughout each session. The time when the cursor


jump command was sent to the display computer was recorded with 1 ms resolution, after which it

appeared at the next 120 Hz monitor update.

The data consist of 8 experimental sessions (1 per day) in monkey J and 9 sessions in monkey L.

We will refer to one experimental session’s data as a ‘dataset’. When dataset identifiers are provided

(e.g., “J.2015.01.19”), they are of the format <monkey initial>.<year>.<month>.<date>.

4.5.2 Neural recording and decoding

Monkeys J and L were implanted with 96-electrode arrays (1 mm electrodes spaced 400 µm apart,

Blackrock Microsystems) using standard neurosurgical techniques. Monkey J had one array implanted

into primary motor cortex (M1) and the other into dorsal premotor cortex (PMd) of the same

hemisphere 64 months prior to these experiments. Monkey L was implanted with a single array at

the M1/ PMd border 83 months prior to the experiments. Voltage signals from each of the electrodes

were band-pass filtered from 250 to 7500 Hz and then processed to obtain multiunit ‘threshold

crossings’ spikes. A spike was detected whenever the voltage crossed below a threshold set at the

beginning of each day to be -4.5 × rms voltage (Chestek et al., 2011). We did not spike sort to

assign spikes to individual putative neurons and instead grouped together threshold crossings on

a given electrode; the population activity will therefore include both single- and multiunit activity.

We believe that this is justified because 1) threshold crossings are the becoming the standard spike

signal used for prosthetics applications (e.g. Gilja et al., 2012, 2015; Collinger et al., 2013b), and 2)

we are making claims about information present in the population activity and lower-dimensional

projections of this activity, rather than specific responses of individual neurons (Trautmann et al.,

2016). All three arrays recorded good signals, with spikes detected on almost every electrode.

We used the ‘recalibrated feedback-intention trained’ Kalman filter (ReFIT) training protocol

and decoding algorithm (Gilja et al., 2012). This involved a two-stage decoder training process to

first initialize and then refine a decoder in the absence of overt arm movements. At the start of each

experiment, the monkey first observed 250 automated straight center-out-and-back cursor movements

between the center target and one of eight radial targets located 12 cm from the workspace center.

When training Monkey L’s decoder, the three targets forming the upper quadrant were placed slightly

further (13 and 14 cm) based on previous experience that this improved his ability to acquire targets

in that quadrant following decoder training. We used the neural and automated kinematic data from

these trials to train a position + velocity Kalman filter. This first-pass decoder was used to complete

an additional 250 center-out-and-back trials of the Radial 8 Target task under BMI control. This

closed-loop data was then used to fit the final ReFIT velocity decoder after several modifications

of the kinematics training data to improve the estimation of the animal’s true intent, described in

(Gilja et al., 2012). However, because we experimentally manipulated cursor position via the cursor

jump, we did not apply the ‘position subtraction’ ReFIT operation (otherwise the cursor jump would

directly affect decoded velocity).


During decoder training and BMI use, no distinction was made between M1 and PMd electrodes.

For the purpose of the subsequent analyses in this study, they were analyzed together, with the

exception that we also replicated the main decoder-potent vs. decoder-null analyses for monkey J

using just his M1 array and saw similar results (not shown). I emphasize that this does not preclude

there being more nuanced differences based on electrode location, which may be deserving of future

study.

4.5.3 Analysis of responses to cursor jumps

Both monkeys almost always successfully acquired targets in the Cursor Jump Task, with failures

likely attributable to the monkey disengaging from the task. We thus restricted our analysis to

successful trials. We quantified cursor control performance using time to target, which measured how

long it took to acquire the target, not including the 500 ms required final hold time, but including

incomplete earlier holds. For the jump during hold condition, we also did not include the initial hold

time preceding the cursor jump, since this time was “stolen” in the sense that there was no way for

the animal to minimize this penalty. To exclude trials where the monkey is likely to have briefly

disengaged from the task, we removed outlier trials with times to target more than three standard

deviations longer the mean (between 0.7% and 2.0% of trials across individual datasets). For neural

analyses in which responses were averaged across trials of the same condition, we also excluded any

jump-location/target/jump-direction/dataset conditions with fewer than 5 trials. Combining across

monkey J’s eight datasets, we analyzed 8,297 unperturbed trials, 743 jump after 3 cm trials, 726

jump after 6 cm trials, 698 jump after 10 cm trials, and 552 jump during hold trials. Across monkey

L’s nine datasets, we analyzed 5,039 unperturbed trials, 427 jump at 3 cm trials, 399 jump at 6 cm

trials, 440 jump at 10 cm trials, and 370 jump during hold trials.

To measure how neural activity evolved after a cursor jump perturbation, we calculated the firing

rate of all trials using a 25 ms boxcar window slid by 1 ms steps. Whenever we present data, its

timestamp corresponds to the last millisecond of the window, thereby conservatively estimating

neural response latencies. We were interested in the change in neural activity specifically due to the

perturbation. Therefore, when analyzing perturbation-evoked responses, we first subtracted away

the mean responses of otherwise similar unperturbed trials. This served to largely remove underlying

firing rate changes related to performing the BMI task. More specifically, for every perturbation

trial j, we aligned trial j’s firing rate to the time of the cursor jump. We then subtracted away the

mean firing rate of all unperturbed trials to the same target in trial j’s dataset. These unperturbed

trials’ responses were aligned to ‘faux jumps’ by assigned them a jump time based on when a jump

would have happened on that trial if trial j’s jump location condition had been in effect. We then

calculated a mean jump-evoked firing rate difference for each ‘dataset/condition’ by averaging across

all jump trials in a given dataset that were to the same target, had perturbations at the same

jump location, and perturbed the cursor in the same jump direction. Thus, each dataset typically


contributed (4 jump locations) × (2 targets) × (2 jump directions) = 16 trial-averaged time-varying

neural change traces. Unless otherwise noted, all results presented are averaged across all of the

monkey’s dataset/conditions of a particular jump location. This was done so that every dataset

contributed roughly equally to the final analysis, removing the risk that our results were unduly

weighted by a few dataset/conditions with the most trials.

To calculate overall change of firing rates across the E electrodes, I took the vector norm of

this E-dimensional difference vector at each time point. This captures neural changes due to both

increases and decreases in firing rates. Since difference norms must be non-negative, there will be

some baseline firing rate differences just due to taking differences between firing rate time series that

differ due to single-trial variability unrelated to the difference between unperturbed and perturbed

trials. I estimated this baseline noise with a shuffle test in which I performed the same processing on

data that we knew had no perturbation-related activity: I assigned P unperturbed trials to be labeled

as perturbed trials, where P was the number of actual perturbed trials for this dataset/condition,

and used the remaining unperturbed trials for the mean unperturbed response subtraction. This

shuffled analysis was repeated 10,001 times. The mean of the resulting shuffled difference norms was

subtracted from the true data’s difference norm to yield time series with baseline norms close to

zero as shown in Figure 4.3a. The shuffled data also provided a nonparametric statistical test for

significant post-perturbation firing rate change. Specifically, the p-value of the significance of firing

rate change was calculated at each time point by counting how many of the shuffled norms the real

data was larger than: p < 0.001 denotes that the real neural difference was larger than 10 of the

10,001 shuffled differences.

4.5.4 Neural response from the decoder’s perspective

To assess how these changes in neural activity affected BMI output, I took advantage of the fact that

the closed-loop Kalman filter almost immediately converges to a steady-state linear decoder (Malik

et al., 2011) of the form:

x(t) = M1x(t− 1) + M2y(t), (4.5.1)

where x(t) is a 5× 1 state vector of x- and y- position and 1 for bias offset; y(t) is a E × 1 vector

of each electrode’s spike counts in the current time step; and M1 is a state dynamics matrix that

smooths velocity over time and lawfully integrates velocity to change position. The Kalman gain

matrix, M2, applies no neural contribution to position but has two rows of weights that linearly

map each electrode’s firing rate to an x- and y- velocity component. The effect of observed neural

activity on time step t onto velocity is the summation of these velocity components. We therefore

projected firing rates through the M2 matrix to calculate the ‘neural push’, i.e. how instantaneous

neural activity affects BMI output velocity. Firing rates projected through M2 are decoder-potent


activity. Note that neural push is an offline analysis measurement and is not quite the same thing

as the true velocity of the closed-loop BMI cursor. This difference has two causes. First, true

cursor velocity was subject to a momentum-like term imposed by the M1 matrix. Second, true

cursor velocity is computed once every (non-overlapping) decode time step, with on-screen velocity

interpolated between decode steps. Offline neural push can be computed for any given millisecond

using firing rates computed using arbitrary time windows, and should be interpreted as showing how

the instantaneous neural activity would affect velocity had the decoder updated at that time. A

useful analogy is that neural push is similar to instantaneous muscle force, which is closely related to

— but is distinct from — the movement of the arm, which is subject to physical dynamics.

Perturbation-evoked changes in neural push along the Perturbation Axis and Task Axis (Figures

4.3a and 4.5) is a signed quantity; therefore, no shuffle baseline subtraction was needed since

random variation in neural push would tend to average out to zero. This is not to be confused with

subtracting the unperturbed trials’ neural push, which we did do as shown in Figure 4.5. However,

the magnitude of decoder-potent activity change (Figure 4.7a,b), i.e. the vector norm of the difference

between perturbed and unperturbed trials’ decoder-potent neural activities, is again a non-negative

quantity. As such, this measurement has the same source of baseline noise as described above for

perturbation-evoked changes in firing rates. We therefore performed the same shuffled baseline

subtraction and nonparametric significance testing. To compute the perturbation-evoked response

changes in the decoder-null subspace, we analyzed the data exactly as for the decoder-potent subspace,

except that instead of projected firing rates through M2, we projected them into the null space of

M2.

We also examined the differences between decoder-null responses to cursor jumps that happened

at the same jump location during movements to the same target, but which displaced the cursor

in opposite directions (Figure 4.7c). To do so, we calculated the difference between projected

jump-aligned neural activity for a given dataset/condition and that of the dataset/condition with the

opposite jump direction. We then took the vector norm of this difference at each millisecond. Since

vector norms are non-negative, we once again did a shuffle control to subtract the baseline noise. We

performed the same analysis after randomly permuting the jump direction labels, repeated 10,001

times. At every time point, the mean vector norm across these permuted datasets was subtracted

from the true norm. Significance of jump-direction related difference was calculated at each time

point by counting how many of the shuffled norms the real data norm was larger than. To aggregate

results across targets and datasets, each dataset/jump condition/target contributed one trial-averaged

time-series, which were then averaged across datasets/targets to yield the final result.

4.5.5 Electrode neural push direction and decoder contribution weight

The decoder training data were used to compute each electrode’s directional “tuning” and its decoder

contribution weight. We used the M2 Kalman gain matrix described earlier, where the jth column of


M2 is the decoder weight vector wj = [wx,j , wy,j , 0]T that describes how yj(t), the neural activity

observed on electrode j, contributes to x- and y- velocity. Electrode j’s decoder contribution is then

computed as its average effect on cursor velocity:

(contribution of electrode j) =∥∥E[wj · yj(t)]

∥∥ (4.5.2)

where the expectation is taken over every bin of the training dataset. An electrode’s relative

contribution to the decoder is then calculated by dividing that electrode’s contribution by the sum of

all electrodes’ contributions. Its decoder push direction is determined by the angle specified by wj .

4.6 Project Acknowledgements

I thank the members of the Shenoy group for helpful discussions; Jordan Sorokin for helping examine

earlier cursor jump data during his rotation in the laboratory; Mackenzie Mazariegos, Michelle

Wechsler, Clare Sherman, Liana Yates, Rosie Steinbach, and Shannon Smith for surgical assistance

and veterinary care; Beverly Davis, and Evelyn Castaneda for administrative support; Boris Oskotsky

for information technology support.

This work was supported by the National Science Foundation Graduate Research Fellowship

(J.C.K., S.D.S.); NSF IGERT 0734683 (S.D.S.); Christopher and Dana Reeve Paralysis Foundation

(S.I.R. and K.V.S.); and the following to K.V.S.: Burroughs Welcome Fund Career Awards in the

Biomedical Sciences, Defense Advanced Research Projects Agency Reorganization and Plasticity

to Accelerate Injury Recovery N66001-10-C-2010, US National Institutes of Health Institute of

Neurological Disorders and Stroke Transformative Research Award R01NS076460, US National

Institutes of Health Director’s Pioneer Award 8DP1HD075623-04, US National Institutes of Health

Director’s Transformative Research Award (TR01) from the NIMH 5R01MH09964703, and Defense

Advanced Research Projects Agency NeuroFAST award from BTO W911NF-14-2-0013.

Chapter 5

Conclusion

5.1 Summary of motor neural prosthesis advances

My doctoral research has laid a foundation for addressing several key challenges to clinically useful

motor neural prostheses through experiments in a pre-clinical animal model. For this technology to

provide a substantial quality of life improvement for people with paralysis, it must function reliably

and have a long device lifespan. The recurrent neural network decoding method detailed in Chapter

2 provides a technique for building BMIs that become more robust to changing recording conditions

as they accumulate more data with every passing day. Furthermore, by learning context-dependent

neural mappings and allowing for data augmentation, this technique may also improve decoding

performance across a wider range of tasks and settings. This will help create BMIs that provide

consistently high performance on a minute-by-minute and day-by-day basis.

The local field potential decoding methods detailed in Chapter 3 address the need for longer

device lifespan by providing useful levels of BMI performance even without spike signals. Since local

field potentials can often be recorded long after sensor degradation renders chronically implanted

electrodes unable to record spikes, this can reduce the need for surgical re-implantation of the sensor.

This will help positively tilt the risk-benefit tradeoff of motor neural prostheses, especially since there

does not currently appear to be a path to controlling high-degree-of-freedom effectors such as robotic

arms using non-invasive methods.

The longer-term vision for motor neural prostheses includes restoring control of arm reach-and-grasp

function. Amongst the many additional complexities introduced by controlling physical limbs, rather

than computer cursors, is that they will operate in physical environments where they will encounter

perturbations. The effect of the resulting sensory feedback on neural signals must be understood if

we are to build BMIs that work well when interacting with the physical world. Chapter 4 describes

my detailed investigation of error-related feedback on BMI control. This first-in-the-field study also

96

CHAPTER 5. CONCLUSION 97

revealed a sensorimotor computational mechanism that we might be able to exploit in order to

prevent sensory feedback from deleteriously interfering with BMI output.

5.2 Next steps: testing in a human clinical trial

This research was performed in able-bodied monkeys, a pre-clinical animal model that is as close as

feasible to the human clinical case while still allowing for the higher risk, greater precision, and faster

paced research needed to maximally advance the engineering and science of motor neural prostheses.

Nonetheless, for this work to generate a tangible societal benefit, it will need to be translated to

human clinical studies. I believe that there are clear applications for each of the described projects

that are worth testing in human BMI users.

The MRNN decoder may prove particularly useful for handling the variable recording conditions

often encountered in human BMI users. Ongoing clinical trials are already collecting the necessary

large corpus of previous data; thus, there is a fairly straightforward path to testing whether this

method will improve participants’ BMI reliability. This technique may also prove useful for decoding

control signals for prosthetic arms. In this application, the neural-to-kinematic mapping may vary

depending on the position of the arm and its interaction state with objects. The MRNN might be

well-suited to accommodate these changes because its flexible neural-to-kinematic mapping depends

on the recent history of neural inputs, and can be extended to explicitly depend on the effector state.

Decoding local field potentials may be of high value in participants whose electrode arrays no

longer record good spike signals. Although I hope that recordings conditions will not necessitate

this approach — since this would mean that a participant has lost the better spikes signals — there

is no reason why LFP-only and hybrid decoding can’t be tested in dedicated experiments sooner,

subject only to the constraints of experimental timelines and priority queues. In the meantime, data

already collected can be analyzed with a specific focus on the low-frequency local motor potential

feature that I found performed well in a monkey animal model. That said, my LFP decoding work

specifically emphasized that offline decoding analyses can only go so far in predicting the actual

utility of a BMI method. Online tests will be the gold standard for testing whether LFP decoding

can extend device lifespan.

The two key unknowns related to my study of neural responses to BMI perturbations are 1)

whether the same decoder-null to decoder-potent dynamics hold up as the decoder-potent subspace

grows with the number of controlled degrees-of-freedom (for instance, when controlling prosthetic

limbs), and 2) whether this phenomenon that I discovered in able-bodied monkeys is also present in

movement-impaired people. If we find that this beneficial arrangement does hold up, then not only

would this be a fortuitous turn of events, but we will also have the benefit of understanding why

error-related sensory signals are not interfering with BMI use. If it doesn’t, we are now equipped

with an analytical framework by which to identify these deleterious feedback signals. We can then


try to adapt our decoders, or encourage neural adaptation, in order to force the error-related activity

into decoder-null dimensions. Either way, the only way to find out what will happen is to forge ahead

with prosthetic limb control experiments while deliberately planning a careful study of perturbation

responses. The experiments and analyses described in this dissertation can serve as a model for how

to go about doing this.

My doctoral training in the Stanford Neurosciences Program, conceiving and carrying out the

research described in this document under the mentorship of Professor Krishna Shenoy, has prepared

me to try these approaches — as well as tackle new challenges and research directions — in the

service of providing increasing motor function to clinical trial participants during my postdoctoral

endeavors.

5.3 Additional graduate work not covered in this dissertation

My Ph.D. training afforded me with the opportunity to explore a variety of topics related to motor

control and motor neural prostheses. Two considerations determined my choice of which three

projects formed the core of this dissertation. First, they were each at a mature phase, being either

published (Chapter 3), in review (Chapter 2), or very near submission (Chapter 4). Second, they have

a thematic relationship: all three of these studies speak to how we can robustly decode movement

intent despite changes in, loss of, or sensory feedback effects upon, the observed neural signals. There

were a number of other projects to which I devoted considerable portions of my graduate work, but

which did not fit into the scope of this dissertation either because they were thematically less close,

or because they are in a less developed stage. In this section I enumerate and briefly describe the

remaining major projects which have a high likelihood of being developed into journal papers.

5.3.1 Drawing scientific conclusions about neural population activity without

sorting single neuron activity

During the course of my graduate work, I collected a large quantity of multielectrode array data

in which many electrodes recorded multiunit ‘threshold crossing’ spikes but not well-isolated single

neuron activity. Since this neural data had sufficient information to drive accurate BMI cursor

control, and since my analyses were providing interesting scientific insight, I wanted to be able to

report my discoveries without objections being raised that I was not using “spike sorted” single unit

activity. It would have been difficult and time-consuming for me to re-collect my data to record

single unit activity: this would require array re-implantation in one of my monkeys (or training a

new monkey), and it would have substantially reduced the number of neural information channels

available from the other monkey. In order to maximize the utility of each research animal, I was

compelled to think about whether I could justify performing neural population analyses on unsorted

threshold crossing data.


Thinking about the goals of my analyses, I made two key observations. First, I was attempting

to measure and describe the information present in a cortical ensemble, rather than describing the

behavior of individual neurons per se. Second, there is an emerging view in the field (e.g. Shenoy et al.,

2013) which argues that, at least in motor cortex and several other cortical areas, individual neurons’

activity is a distributed and redundant high dimensional “read out” of latent lower-dimensional

dynamics. In light of this, analyzing threshold crossings (which is the equivalent of summing together

the activity of some single neurons as well as adding the measurements of multiple more distant

neurons which otherwise would not have been detected) appears to be theoretically justified as long

as the latent neural state can still be estimated from this (different) high dimensional multiunit

neural projection.

I decided to empirically test whether one would come to the same scientific conclusions about

population neural activity with and without spike sorting, i.e. by analyzing ‘threshold crossings’. I

started this project by taking previous published scientific work from our laboratory (Ames et al.,

2014) and re-analyzing the data without spike sorting to see how this would affect the results. After

I began this project, my colleague Eric Trautmann joined the collaboration. Based on our respective

interests and project portfolio priorities, he is now taking primary charge of this work and will

see it through to publication. Eric re-analyzed data reported in two more prior papers from our

group (Churchland et al., 2012; Kaufman et al., 2014), and presented our results at the 2016 Cosyne

conference, titled “Sortfree: Using threshold crossings to evaluate scientific hypotheses in population

analyses” with authors Eric M. Trautmann, Sergey D. Stavisky, Mattew Kaufman, K. Cora Ames,

Stephen I. Ryu, and Krishna V. Shenoy (Trautmann et al., 2016).

We found that the conclusions drawn from these data were for the most part the same with and

without spike sorting. Although some electrodes’ peri-stimulus time histograms changed noticeably

compared to those of single units recorded from the same electrode, most electrodes showed similar

activity patterns. Many of the “leftover” threshold crossings that had not been spike sorted appeared

to be tuned for movement parameters, suggesting that our method worked not only because relatively

little information was being lost by potentially combining multiple units on a given electrode, but

also because we were now utilizing some previously missed information. More importantly, the

measured population neural dynamics and statistical hypothesis tests run on sorted single unit data

were very similar when run on threshold crossings. In light of this validation and the practical need

to more rapidly analyze growing data quantity created by new, higher density recording technologies,

I anticipate that analyzing threshold crossings in place of sorted spikes will become increasingly

common for drawing conclusions about population activity.


5.3.2 Observed BMI cursor position only weakly affects motor cortical

activity

Tracking the state of the motor ‘effector’ (also known as the ‘plant’) is a computation strongly

predicted to occur in the sensorimotor system under contemporary feedback control theories (e.g.

Wolpert et al., 1995; Scott, 2004). For example, the arm’s position is well represented in cortical

activity (Caminiti et al., 1990). An early project that emerged from my starting to think about

the neuroscience of BMI use was to look for motor cortical correlates of the BMI cursor’s position.

This information would be consistent with the BMI cursor taking on the role of the which the real

arm fills during natural movements. To investigate this, I analyzed the motor cortical correlates

of 1) hand position during a hand controlled target acquisition task, 2) cursor position during an

arms-restrained BMI task, and 3) hand and cursor position during an arm-free BMI task. I found

that although both hand and cursor position influenced firing rates, this effect was much stronger for

the hand’s position than for the cursor’s position, especially once the correlations between hand and

cursor were accounted for in the arm-free condition. I presented this work at the 2013 Society for

Neuroscience annual conference, in the form of a poster titled “Motor cortical activity tracks the

position of a brain-machine interface cursor”, with authors Sergey D. Stavisky, Jonathan C. Kao,

Paul Nuyujukian, Stephen I. Ryu, and Krishna V. Shenoy.

Given this weak effect of cursor position, and more importantly, the potential confound of the

monkey’s gaze following the cursor’s position, barring new analytical insight I plan to abandon

the scientific aim of this project. However, this work yielded an observation with practical neural

engineering value. The ReFIT decoder (Gilja et al., 2012) that has been the workhorse of our group’s

BMI efforts in recent years includes one decoder innovation called ‘position subtraction’. Inspired

by the hypothesis that cursor position effects neural activity, ReFIT treats this position effect as a

nuisance variable and subtracts its expected firing rate contribution before inferring velocity intent.

However, the utility of this design choice has not been previously tested via direct, back-to-back

closed-loop performance comparisons with and without this position subtraction operation. I did

exactly this, and found that although position subtraction did improve performance when the

monkey’s reaching arm was free, it did not help performance when the monkeys used the BMI without

moving their arms. This suggests that the apparent benefit of position subtract was because this

operation was indirectly accounting for the effect of the arm position (which is correlated with cursor

position), and that this operation is unlikely to be of use in paralyzed BMI users.

5.3.3 Single trial motor cortical correlates of a rapidly adapting visuomotor

internal model

Accurate motor control is mediated by internal models of how neural activity generates movement.

As part of a series of experiments showing that during BMI use the brain maintains an up-to-date


internal model of the BMI’s properties, I conducted matched experiments in which monkeys controlled

a cursor either with their hand (this project) or with a BMI (see next section) while the gain scaling

between the commanded velocity and the cursor’s movement was varied. The hand-controlled data

were initially intended to provide a point of comparison between BMI and natural arm control

visuomotor adaptation rates. However, I discovered an interesting effect in these data: neural activity

varied during the so-called “baseline” period in between reaches in a way that was evocative of a

neural correlate of the visuomotor internal model. I explored this effect further, which led me to write

up this result as its own manuscript which I anticipate submitting to a journal soon. The authors for

this manuscript are Sergey D. Stavisky, Jonathan C. Kao, Stephen I. Ryu, and Krishna V. Shenoy.

Previous studies of cortical changes during visuomotor adaptation focused on preparatory and

peri-movement epochs and analyzed trial-averaged neural data. In this project, we recorded

simultaneous neural population activity using multielectrode arrays and focused the analysis on

neural differences in the period before the target appeared. This let me ask whether there were

correlates of different internal models being “loaded” in motor cortex. I found that I could estimate

the monkey’s internal model of the gain using the neural population state during this pre-target

epoch. This neural correlate depended on the gain experienced on the previous trial, and it predicted

the speed of the subsequent reach. The ability to observe internal model changes with single-trial

resolution has utility for studying the interaction of internal models, movements, and errors on the

single-trial timescales at which these computations ultimately play out.

Furthermore, the ability to estimate the internal model before movement begins has potential

BMI applications. A brain-controlled robotic arm user will need an accurate internal model of how

her neural commands affect the arm; this is complicated by the arm’s own physical dynamics (which

could vary across joint conformations) and its interaction with objects (e.g., carrying something

heavy). My variable cursor velocity gain task can be viewed as a (much simplified) example of a user

under- and over-estimating the gain between their neural command and the effector. Therefore, to

explore the neural prosthesis utility of being able to estimate the internal model, I performed an

offline analysis of these data which showed that the neural activity can be used to compensate for

upcoming reach extent errors.

5.3.4 Rapid but limited adaptation to a change in BMI velocity gain

In the first part of this project, I used the matched hand- and BMI-controlled changing velocity gain

tasks introduced in the previous section to show that monkeys rapidly adapt to a changing BMI

cursor velocity gain similarly to how they adapt to hand-controlled cursor velocity gain changes. This

result adds to the nascent body of knowledge showing that, similarly to natural motor control, BMI

control employs internal models (Golub et al., 2015). I presented this work at the 2014 Society for

Neuroscience annual conference as a poster titled “Brain-machine interface performance is mediated


by an internal model of decoder velocity gain”, with authors Sergey D. Stavisky, Jonathan C. Kao,

Stephen I. Ryu, and Krishna V. Shenoy.

Despite rapid initial gain adaptation, I also observed that the monkeys were never able to fully

compensate for the changed BMI velocity gain. Although they increased their velocity command

when adapting to a slow gain and decreased the velocity command when adapting to a fast gain,

these changes were not sufficient to equalize their cursor speed profile to that of the gain 1 condition.

Consequently, their performance, as measured in achieved bitrate on a target grid task, was worse

when the gain deviated from ∼1. The cursor moved too slowly under low gain conditions, which

slowed down acquisition times, or too quickly under high gain conditions, which resulted in more

incorrect selections because it was harder for the monkey to precisely hold and acquire the correct

target. These first experiments were limited to short timescale adaptation in which the monkey

experienced a particular gain for only several hundred trials. However, in a follow-up experiment, we

held the low or high gain constant for an ‘medium’ time scale of more than an hour and 1,000+ trials.

I found that the additional time did not enable further adaptation or improvement. In contrast, when

we performed similar medium time scale experiments in which the manipulation was a visuomotor

rotation, rather than gain scaling, I did observed both short and medium time scale improvements,

consistent with (Jarosiewicz et al., 2008; Chase et al., 2012; Golub et al., 2015). This argues against

the lack of gain adaptation being due to low motivation or general learning inability on the part of

the monkeys.

Rather, my data suggest that there’s something about gain scaling that limits the motor cortical

network’s ability to adapt to and compensate for this perturbation. In theory, if the brain could

just double the firing rates of the neural population driving the BMI during under the gain 0.5

condition, or halve firing rates under the gain 2 condition, this would fully compensate for the changed

neural-to-kinematic gain mapping. The apparent inability of the network to do so implies that there

are constraints on what kind of adaptation it is capable of: in particular, there are limits to how

much firing rates of many neurons can be simultaneously gained up or down. One possible reason for

this is that the network has a limited dynamic range, perhaps optimized for the range of firing rates

useful for generating natural movements. If so, perhaps it is very difficult, if not impossible, for the

network to re-organize in order to operate outside this range — at least on short to medium time

scales. This study informs BMI designers to be aware of these constraints: for example, we should

build decoders that can output the full range of desired commands in response to neural inputs that

fall within the reachable dynamic range of the neural population being recorded from.

5.3.5 Do motor cortical dynamical rules constrain the activity that can

be generated for BMI control?

My inspiration for this study was a conversation I had with a BrainGate clinical trial participant

during my pre-doctoral research. This participant employed a BMI control strategy in which she


imagined moving a computer mouse with her hand. During one experiment in which the decoder

developed a bias that constantly pushed the cursor in one direction, the participant reported having

difficulty sustaining cursor velocity-generating mental imagery in the opposing direction because in

her mental imagery, her “hand had reached the limit of its range of motion”. In other words, she was

reporting the BMI equivalent of running one’s computer mouse off its mousepad. This anecdote made

me wonder whether motor cortex’s legacy of generating output for natural motor control imposed

constraints on what mental imagery — and by extension, patterns of neural activity — it could

generate. For instance, perhaps one cannot imagine a continual reaching movement, and hence one

cannot sustain the motor cortical correlates of a point-to-point reach.

These constraints are more formally described in an emerging view that motor cortex is a

dynamical system governed by a lawful set of dynamical “laws” (e.g. Churchland et al., 2010, 2012;

Shenoy et al., 2013). I wanted to probe how rigid these dynamics are in the context of BMIs: will

they impose constraints on the neural activity that can be generated to control the BMI? To test

this, I conducted a set of experiments in two monkeys testing whether motor cortex can generate

sustained reach-like activity for much longer than the duration of typical arm reaches. I did this

by dramatically reducing the gain between decoded velocity and cursor velocity; this had the effect

of asking the monkey to generate strong neural push for many seconds, rather than the several

hundred milliseconds that would normally be needed to reach the target. My preliminary analyses of

these data suggest that neural push can be sustained, which would be consistent with firing rates

being able to dwell in a reach-like state when the BMI task demands it. This would suggest that

motor cortex need not always adhere to the rapidly “unwinding” dynamical system rules described

by recent studies — that is, that the time-varying evolution imposed by neural dynamics can be

“paused” rather than rigidly obeyed. However, substantially more analysis is needed to flesh out this

story about whether neural dynamics constrain BMI control.

Bibliography

Aflalo, T., Kellis, S., Klaes, C., Lee, B., Shi, Y., Pejsa, K., Shanfield, K., Hayes-Jackson, S., Aisen,

M., Heck, C., Liu, C., & Andersen, R. A. (2015). Decoding motor imagery from the posterior

parietal cortex of a tetraplegic human. Science, 348, 906–910.

Ames, K. C., Ryu, S. I., & Shenoy, K. V. (2014). Neural Dynamics of Reaching following Incorrect

or Absent Motor Preparation. Neuron, 81, 438–451.

Andersen, R. A., Musallam, S., & Pesaran, B. (2004). Selecting the signals for a brain-machine

interface. Current opinion in neurobiology, 14, 720–6.

Anderson, K. D. (2004). Targeting recovery: priorities of the spinal cord-injured population. Journal

of neurotrauma, 21, 1371–1383.

Archambault, P. S., Ferrari-Toniolo, S., & Battaglia-Mayer, A. (2011). Online control of hand

trajectory and evolution of motor intention in the parietofrontal system. Journal of Neuroscience,

31, 742–52.

Asher, I., Stark, E., Abeles, M., & Prut, Y. (2007). Comparison of direction and object selectivity

of local field potentials and single units in macaque posterior parietal cortex during prehension.

Journal of Neurophysiology, 97, 3684–3695.

Bacher, D., Jarosiewicz, B., Masse, N. Y., Stavisky, S. D., Simeral, J. D., Newell, K., Oakley, E. M.,

Cash, S. S., Friehs, G., & Hochberg, L. R. (2015). Neural Point-and-Click Communication by a

Person With Incomplete Locked-In Syndrome. Neurorehabilitation and Neural Repair, 29, 462–471.

Bansal, A. K., Truccolo, W., Vargas-Irwin, C. E., & Donoghue, J. P. (2011a). Decoding 3D reach

and grasp from hybrid signals in motor and premotor cortices: spikes, multiunit activity, and local

field potentials. Journal of Neurophysiology, 107, 1337–55.

Bansal, A. K., Vargas-Irwin, C. E., Truccolo, W., & Donoghue, J. P. (2011b). Relationships

among low-frequency local field potentials, spiking activity, and three-dimensional reach and grasp

kinematics in primary motor and ventral premotor cortices. Journal of Neurophysiology, 105,

1603–19.

104

BIBLIOGRAPHY 105

Barrese, J. C., Rao, N., Paroo, K., Triebwasser, C., Vargas-Irwin, C., Franquemont, L., & Donoghue,

J. P. (2013). Failure mode analysis of silicon-based intracortical microelectrode arrays in non-human

primates. Journal of Neural Engineering, 10, 066014.

Bastian, A., Riehle, A., Erlhagen, W., & Schoner, G. (1998). Prior information preshapes the

population representation of movement direction in motor cortex. Neuroreport, 9, 315–9.

Belitski, A., Gretton, A., Magri, C., Murayama, Y., Montemurro, M. A., Logothetis, N. K., &

Panzeri, S. (2008). Low-frequency local field potentials and spikes in primary visual cortex convey

independent visual information. Journal of Neuroscience, 28, 5696–709.

Bensmaia, S. J. & Miller, L. E. (2014). Restoring sensorimotor function through intracortical

interfaces: progress and looming challenges. Nature Reviews Neuroscience, 15, 313–25.

Bishop, W., Chestek, C. C., Gilja, V., Nuyujukian, P., Foster, J. D., Ryu, S. I., Shenoy, K. V., & Yu,

B. M. (2014). Self-recalibrating classifiers for intracortical brain-computer interfaces. Journal of

Neural Engineering, 11, 026001.

Bokil, H. S., Pesaran, B., Andersen, R. A., & Mitra, P. P. (2006). A method for detection and

classification of events in neural activity. IEEE Transactions on Biomedical Engineering, 53,

1678–87.

Buzsaki, G., Anastassiou, C. A., & Koch, C. (2012). The origin of extracellular fields and currents

EEG, ECoG, LFP and spikes. Nature Reviews Neuroscience, 13, 407–420.

Cabel, D., Cisek, P., & Scott, S. (2001). Neural Activity in Primary Motor Cortex Related to

Mechanical Loads Applied to the Shoulder and Elbow During a Postural Task. Journal of

Neurophysiology, pp. 2102–2108.

Caminiti, R., Johnson, P., & Urbano, A. (1990). Making arm movements within different parts of

space: dynamic aspects in the primate motor cortex. Journal of Neuroscience, 10, 2039.

Carmena, J. M., Lebedev, M. A., Crist, R. E., O’Doherty, J. E., Santucci, D. M., Dimitrov, D. F.,

Patil, P. G., Henriquez, C. S., & Nicolelis, M. A. L. (2003). Learning to control a brain-machine

interface for reaching and grasping by primates. PLoS biology, 1, 193–208.

Castiello, U. (2005). The neuroscience of grasping. Nature Reviews Neuroscience, 6, 726–36.

Chase, S. M., Kass, R. E., & Schwartz, A. B. (2012). Behavioral and neural correlates of visuomotor

adaptation observed through a brain-computer interface in primary motor cortex. Journal of

Neurophysiology, 108, 624–44.

BIBLIOGRAPHY 106

Chase, S. M., Schwartz, A. B., & Kass, R. E. (2009). Bias, optimal linear estimation, and the differences

between open-loop simulation and closed-loop performance of spiking-based brain-computer

interface algorithms. Neural Networks, 22, 1203–13.

Chestek, C. A., Gilja, V., Nuyujukian, P., Foster, J. D., Fan, J. M., Kaufman, M. T., Churchland,

M. M., Rivera-Alvidrez, Z., Cunningham, J. P., Ryu, S. I., & Shenoy, K. V. (2011). Long-term

stability of neural prosthetic control signals from silicon cortical arrays in rhesus macaque motor

cortex. Journal of Neural Engineering, 8, 045005.

Churchland, M. M. (2015). Using the precision of the primate to study the origins of movement

variability. Neuroscience, pp. 1–9.

Churchland, M. M., Cunningham, J. P., Kaufman, M. T., Foster, J. D., Nuyujukian, P., Ryu, S. I.,

& Shenoy, K. V. (2012). Neural population dynamics during reaching. Nature.

Churchland, M. M., Cunningham, J. P., Kaufman, M. T., Ryu, S. I., & Shenoy, K. V. (2010). Cortical

Preparatory Activity: Representation ofMovement or First Cog in a Dynamical Machine? Neuron,

68, 387–400.

Churchland, M. M. & Shenoy, K. V. (2007). Delay of movement caused by disruption of cortical

preparatory activity. Journal of Neurophysiology, 97, 348–59.

Cisek, P. & Kalaska, J. F. (2010). Neural mechanisms for interacting with a world full of action

choices. Annual Review of Neuroscience, 33, 269–98.

Collinger, Boninger, M. L., Bruns, T. M., Curley, K., Wang, W., & Weber, D. J. (2013a). Functional

priorities, assistive technology, and brain-computer interfaces after spinal cord injury. The Journal

of Rehabilitation Research and Development, 50, 145.

Collinger, Wodlinger, B., Downey, J. E., Wang, W., Tyler-Kabara, E. C., Weber, D. J., McMorland, A.

J. C., Velliste, M., Boninger, M. L., & Schwartz, A. B. (2013b). High-performance neuroprosthetic

control by an individual with tetraplegia. Lancet, 381, 557–64.

Crammond, D. & Kalaska, J. (1996). Differential relation of discharge in primary motor cortex

and premotor cortex to movements versus actively maintained postures during a reaching task.

Experimental Brain Research, 108, 45–61.

Crammond, D. J. & Kalaska, J. F. (2000). Prior information in motor and premotor cortex: activity

during the delay period and effect on pre-movement activity. Journal of Neurophysiology, 84,

986–1005.

Cunningham, J. P., Nuyujukian, P., Gilja, V., Chestek, C. A., Ryu, S. I., & Shenoy, K. V. (2010). A

Closed-Loop Human Simulator for Investigating the Role of Feedback-Control in Brain-Machine

Interfaces. Journal of Neurophysiology.

BIBLIOGRAPHY 107

Cunningham, J. P., Nuyujukian, P., Gilja, V., Chestek, C. A., Ryu, S. I., & Shenoy, K. V. (2011).

A closed-loop human simulator for investigating the role of feedback control in brain-machine

interfaces. Journal of Neurophysiology, 105, 1932–1949.

Dadarlat, M. C., O’Doherty, J. E., & Sabes, P. N. (2014). A learning-based approach to artificial

sensory feedback leads to optimal integration. Nature Neuroscience.

Dangi, S., Moorman, H. G., Orsborn, A. L., & Carmena, J. M. (2014). Continuous Closed-Loop

Decoder Adaptation with a Recursive Maximum Likelihood Algorithm Allows for Rapid

Performance Acquisition in Brain-Machine Interfaces. Neural Computation, 1839, 1811–1839.

Dangi, S., Orsborn, A. L., Moorman, H. G., & Carmena, J. M. (2013). Design and Analysis of

Closed-Loop Decoder Adaptation Algorithms for Brain-Machine Interfaces. Neural Computation,

25, 1693–1731.

Dethier, J., Nuyujukian, P., Ryu, S. I., Shenoy, K. V., & Boahen, K. (2013). Design and validation

of a real-time spiking-neural-network decoder for brain-machine interfaces. Journal of Neural

Engineering, 10, 036008.

Douglas, S., Kirkpatrick, A., & MacKenzie, I. (1999). Testing pointing device performance and user

assessment with the ISO 9241, Part 9 standard. Proc. ACMSIGCHI Conf. Human Factors Comput.

Syst. (CHI’99), pp. 215–222.

Einevoll, G. T., Kayser, C., Logothetis, N. K., & Panzeri, S. (2013). Modelling and analysis of local

field potentials for studying the function of cortical circuits. Nature Reviews Neuroscience, 14,

770–785.

Ethier, C., Gallego, J., & Miller, L. (2015). Brain-controlled neuromuscular stimulation to drive

neural plasticity and functional recovery. Current Opinion in Neurobiology, 33, 95–102.

Ethier, C. & Miller, L. E. (2014). Neurobiology of Disease Brain-controlled muscle stimulation for

the restoration of motor function. Neurobiology of Disease.

Ethier, C., Oby, E. R., Bauman, M. J., & Miller, L. E. (2012). Restoration of grasp following paralysis

through brain-controlled stimulation of muscles. Nature, 485, 368–371.

Evarts, E., Fromm, C., Kroller, J., & Jennings, V. (1983). Motor Cortex control of finely graded

forces. Journal of Neurophysiology, 49, 1199–1215.

Evarts, E. V. (1968). Relation of pyramidal tract activity to force exerted during voluntary movement.

Journal of Neurophysiology, 31, 14–27.

BIBLIOGRAPHY 108

Even-Chen, N., Stavisky, S. D., Kao, J. C., Ryu, S. I., & Shenoy, K. V. (2015). An auto-correcting

brain machine interface: Error detection using spiking neuronal activity in the motor cortex. In

37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

Fan, J. M., Nuyujukian, P., Kao, J. C., Chestek, C. A., Ryu, S. I., & Shenoy, K. V. (2014). Intention

estimation in brainmachine interfaces. Journal of Neural Engineering, 11, 016004.

Fetz, E. E. (1969). Operant conditioning of cortical unit activity. Science, 163, 955–958.

Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude

of movement. Journal of Experimental Psychology, 47, 381–391.

Flint, R. D., Ethier, C., Oby, E. R., Miller, L. E., & Slutzky, M. W. (2012a). Local field potentials

allow accurate decoding of muscle activity. Journal of Neurophysiology, 108, 18–24.

Flint, R. D., Lindberg, E. W., Jordan, L. R., Miller, L. E., & Slutzky, M. W. (2012b). Accurate

decoding of reaching movements from field potentials in the absence of spikes. Journal of Neural


Flint, R. D., Wright, Z. A., Scheid, M. R., & Slutzky, M. W. (2013). Long term, stable brain

machine interface performance using local field potentials and multiunit spikes. Journal of Neural


Foundation, C. &. D. R. (2009). One Degree of Separation: Paralysis and Spinal Cord Injury in the

United States. Tech. rep.

Fraser, G. W., Chase, S. M., Whitford, A., & Schwartz, A. B. (2009). Control of a brain-computer

interface without spike sorting. Journal of Neural Engineering, 6, 055004.

Ganguly, K. & Carmena, J. M. (2009). Emergence of a stable cortical map for neuroprosthetic

control. PLoS biology, 7, e1000153.

Ganguly, K., Dimitrov, D. F., Wallis, J. D., & Carmena, J. M. (2011). Reversible large-scale

modification of cortical networks during neuroprosthetic control. Nature Neuroscience, 14, 662–667.

Georgopoulos, A., Kalaska, J., Caminiti, R., & Massey, J. (1983). Interruption of Motor Cortical

Discharges Subserving Aimed Arm Movements. Experimental Brain Research, 49, 327–340.

Georgopoulos, A., Kettner, R., & Schwartz, A. (1988). Primate motor cortex and free arm movements

to visual targets in three-dimensional space. II. Coding of the direction of movement by a neuronal

population. Journal of Neuroscience, 8, 2928–37.

Gilja, V., Chestek, C., Diester, I., Henderson, J., Deisseroth, K., & Shenoy, K. (2011). Challenges and

Opportunities for Next-Generation Intra-Cortically Based Neural Prostheses. IEEE transactions

on bio-medical engineering, XX, 1–10.

BIBLIOGRAPHY 109

Gilja, V., Nuyujukian, P., Chestek, C. A., Cunningham, J. P., Yu, B. M., Fan, J. M., Churchland,

M. M., Kaufman, M. T., Kao, J. C., Ryu, S. I., & Shenoy, K. V. (2012). A high-performance

neural prosthesis enabled by control algorithm design. Nature Neuroscience, 15, 1752–7.

Gilja, V., Pandarinath, C., Blabe, C. H., Nuyujukian, P., Simeral, J. D., Sarma, A. A., Sorice, B. L.,

Perge, J. A., Jarosiewicz, B., Hochberg, L. R., Shenoy, K. V., & Henderson, J. M. (2015). Clinical

translation of a high-performance neural prosthesis. Nature Medicine, 21, 1142–1145.

Gold, C., Henze, D. a., Koch, C., & Buzsaki, G. (2006). On the origin of the extracellular action

potential waveform: A modeling study. Journal of Neurophysiology, 95, 3113–3128.

Golub, M. D., Yu, B. M., & Chase, S. M. (2015). Internal models for interpreting neural population

activity during sensorimotor control. eLife, 4.

Graf, A. B. A. & Andersen, R. A. (2014). Brain-machine interface for eye movements. Proceedings

of the National Academy of Sciences of the United States of America, 111, 17630–5.

Green, A. M. & Kalaska, J. F. (2011). Learning to move machines with the mind. Trends in

Neurosciences, 34, 61–75.

Hatsopoulos, N. G. & Donoghue, J. P. (2009). The science of neural interface systems. Annual

Review of Neuroscience, 32, 249–66.

Heldman, D. A., Wang, W., Chan, S. S., & Moran, D. W. (2006). Local Field Potential Spectral

Tuning in Motor Cortex During Reaching. IEEE Transactions on Neural Systems and Rehabilitation

Engineering, 14, 180–183.

Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., Haddadin, S.,

Liu, J., Cash, S. S., van der Smagt, P., & Donoghue, J. P. (2012). Reach and grasp by people with

tetraplegia using a neurally controlled robotic arm. Nature, 485, 372–5.

Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., Branner,

A., Chen, D., Penn, R. D., & Donoghue, J. P. (2006). Neuronal ensemble control of prosthetic

devices by a human with tetraplegia. Nature, 442, 164–71.

Hochreiter, S. & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9,

1735–1780.

Homer, M. L., Nurmikko, A. V., Donoghue, J. P., & Hochberg, L. R. (2013). Sensors and decoding

for intracortical brain computer interfaces. Annual Review of Biomedical Engineering, 15, 383–405.

Huggins, J. E., Wren, P. A., & Gruis, K. L. (2011). What would brain-computer interface users

want? Opinions and priorities of potential users with amyotrophic lateral sclerosis. Amyotrophic

Lateral Sclerosis, 12, 318–324.

BIBLIOGRAPHY 110

Hwang, E. J. & Andersen, R. A. (2009). Brain control of movement execution onset using local field

potentials in posterior parietal cortex. Journal of Neuroscience, 29, 14363–14370.

Hwang, E. J. & Andersen, R. A. (2013). The utility of multichannel local field potentials for

brain-machine interfaces. Journal of Neural Engineering, 10, 046005.

Jackson, A. & Fetz, E. E. (2011). Interfacing With the Computational Brain. IEEE Transactions on

Neural Systems and Rehabilitation Engineering, 19, 534–541.

Jarosiewicz, B., Chase, S. M., Fraser, G. W., Velliste, M., Kass, R. E., & Schwartz, A. B.

(2008). Functional network reorganization during learning in a brain-computer interface paradigm.

Proceedings of the National Academy of Sciences of the United States of America, 105, 19486–91.

Jarosiewicz, B., Masse, N. Y., Bacher, D., Cash, S. S., Eskandar, E., Friehs, G., Donoghue, J. P.,

& Hochberg, L. R. (2013). Advantages of closed-loop calibration in intracortical brain-computer

interfaces for people with tetraplegia. Journal of Neural Engineering, 10, 046012.

Jarosiewicz, B., Sarma, A. A., Bacher, D., Masse, N. Y., Simeral, J. D., Sorice, B., Oakley, E. M.,

Blabe, C., Pandarinath, C., Gilja, V., Cash, S. S., Eskandar, E. N., Friehs, G., Henderson, J. M.,

Shenoy, K. V., Donoghue, J. P., & Hochberg, L. R. (2015). Virtual typing by people with tetraplegia

using a self-calibrating intracortical brain-computer interface. Science Translational Medicine, 7,

313ra179–313ra179.

Kalaska, J. & Drew, T. (1993). Motor cortex and visuomotor behavior. Exercise and Sport Sciences

Reviews.

Kalaska, J. F., Cohen, D. A., Hyde, M. L., & Prud’homme, M. (1989). A comparison of

movement direction-related versus load direction-related activity in primate motor cortex, using a

two-dimensional reaching task. Journal of Neuroscience, 9, 2080–2102.

Kandel, E., Schwartz, J., & Jessel, T. (2000). Principles of Neural Science. (New York: McGraw-Hill),

4th edn.

Kao, J. C., Nuyujukian, P., Stavisky, S., Ryu, S. I., Ganguli, S., & Shenoy, K. V. (2013). Investigating

the role of firing-rate normalization and dimensionality reduction in brain-machine interface

robustness. In 2013 35th Annual International Conference of the IEEE Engineering in Medicine

and Biology Society (EMBC), vol. 2013, pp. 293–298. (IEEE).

Kao, J. C., Stavisky, S. D., Sussillo, D., Nuyujukian, P., & Shenoy, K. V. (2014). Information Systems

Opportunities in Brain-Machine Interface Decoders. Proceedings of the IEEE, 102, 666–682.

Katzner, S., Nauhaus, I., Benucci, A., Bonin, V., Ringach, D. L., & Carandini, M. (2009). Local

origin of field potentials in visual cortex. Neuron, 61, 35–41.

BIBLIOGRAPHY 111

Kaufman, M. T., Churchland, M. M., Ryu, S. I., & Shenoy, K. V. (2014). Cortical activity in the

null space: permitting preparation without movement. Nature Neuroscience, 17, 440–8.

Kennedy, P. R., Kirby, M. T., Moore, M. M., King, B., & Mallory, A. (2004). Computer Control

Using Human Intracortical Local Field Potentials. IEEE Transactions on Neural Systems and

Rehabilitation Engineering, 12, 339–344.

Kim, S.-P., Simeral, J. D., Hochberg, L. R., Donoghue, J. P., & Black, M. J. (2008). Neural control

of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia.

Journal of Neural Engineering, 5, 455–476.

Knyazev, A. V. & Argentati, M. E. (2002). Principal Angles Between Subspaces in An A-Based

Scalar Product: Algorithms and Perturbation Estimates. SIAM Journal on Scientific Computing,

23, 2008–2040.

Koyama, S., Chase, S. M., Whitford, A. S., Velliste, M., Schwartz, A. B., & Kass, R. E. (2010).

Comparison of brain-computer interface decoding algorithms in open-loop and closed-loop control.

Journal of computational neuroscience, 29, 73–87.

Kruger, J., Caruana, F., Volta, R. D., & Rizzolatti, G. (2010). Seven years of recording from monkey

cortex with a chronically implanted multiple microelectrode. Frontiers in neuroengineering, 3, 6.

Kuo, B. C. (1987). Automatic Control Systems. (Englewood Cliffs: Prentice-Hall, Inc.), 5th edn.

Li, C.-S. R., Padoa-Schioppa, C., & Bizzi, E. (2001). Neuronal Correlates of Motor Performance and

Motor Learning in the Primary Motor Cortex of Monkeys Adapting to an External Force Field.

Neuron, 30, 593–607.

Lillicrap, T. P. & Scott, S. H. (2013). Preference Distributions of Primary Motor Cortex Neurons

Reflect Control Solutions Optimized for Limb Biomechanics. Neuron, 77, 168–179.

MacKenzie, I. S. & Buxton, W. (1992). Extending Fitts’ law to two-dimensional tasks. In Proceedings

of the SIGCHI conference on Human factors in computing systems - CHI ’92, pp. 219–226. (New

York, New York, USA: ACM Press).

MacKenzie, I. S. & Ware, C. (1993). Lag as a determinant of human performance in interactive

systems. In Proceedings of the SIGCHI conference on Human factors in computing systems - CHI

’93, pp. 488–493. (New York, New York, USA: ACM Press).

Malik, W. Q., Truccolo, W., Brown, E. N., & Hochberg, L. R. (2011). Efficient decoding with

steady-state Kalman filter in neural interface systems. IEEE Transactions on Neural Systems and

Rehabilitation Engineering, 19, 25–34.

BIBLIOGRAPHY 112

Markowitz, D. A., Wong, Y. T., Gray, C. M., & Pesaran, B. (2011). Optimizing the Decoding of

Movement Goals from Local Field Potentials in Macaque Cortex. Journal of Neuroscience, 31,

18412–18422.

Marr, D. C. & Poggio, T. (1976). From understanding computation to understanding neural circuitry.

Martens, J. (2010). Deep learning via Hessian-free optimization. 27th International Conference on

Machine Learning, 951, 735–742.

Martens, J. & Sutskever, I. (2011). Learning recurrent neural networks with Hessian-free optimization.

Proceedings of the 28th International Conference on Machine Learning, ICML 2011, pp. 1033–1040.

Mehring, C., Rickert, J., Vaadia, E., Cardoso de Oliveira, S., Aertsen, A., & Rotter, S. (2003). Inference

of hand movements from local field potentials in monkey motor cortex. Nature Neuroscience, 6,

1253–4.

Messier, J. & Kalaska, J. F. (2000). Covariation of primate dorsal premotor cell activity with direction

and amplitude during a memorized-delay reaching task. Journal of Neurophysiology, 84, 152–165.

Moran, D. (2010). Evolution of brain-computer interface: action potentials, local field potentials and

electrocorticograms. Current opinion in neurobiology, 20, 741–5.

Moran, D. W. & Schwartz, a. B. (1999). Motor cortical activity during drawing movements: population

representation during spiral tracing. Journal of Neurophysiology, 82, 2693–704.

Morrow, M. M. & Miller, L. E. (2003). Prediction of muscle activity by populations of sequentially

recorded primary motor cortex neurons. Journal of Neurophysiology, 89, 2279–88.

Musallam, S., Corneil, B. D., Greger, B., Scherberger, H., & Andersen, R. A. (2004). Cognitive

Control Signals for Neural Prosthetics. Science, 305, 258–262.

Nicolelis, M. A. L. & Lebedev, M. A. (2009). Principles of neural ensemble physiology underlying

the operation of brainmachine interfaces. Nature Reviews Neuroscience, 10, 530–540.

Nuyujukian, P., Fan, J. M., Gilja, V., Kalanithi, P. S., Chestek, C. A., & Shenoy, K. V. (2011).

Monkey Models for Brain-Machine Interfaces: The Need for Maintaining Diversity. In 33rd Annual

International Conference of the IEEE Engineering in Medicine and Biology Society. Boston, MA.

Nuyujukian, P., Fan, J. M., Kao, J. C., Ryu, S. I., & Shenoy, K. V. (2015). A High-Performance

Keyboard Neural Prosthesis Enabled by Task Optimization. IEEE Transactions on Biomedical

Engineering, 62, 21–29.

Nuyujukian, P., Kao, J. C., Fan, J. M., Stavisky, S. D., Ryu, S. I., & Shenoy, K. V. (2014).

Performance sustaining intracortical neural prostheses. Journal of Neural Engineering, 11, 066003.

BIBLIOGRAPHY 113

O’Doherty, J. E., Lebedev, M. A., Ifft, P. J., Zhuang, K. Z., Shokur, S., Bleuler, H., & Nicolelis,

M. A. L. (2011). Active tactile exploration using a brain-machine-brain interface. Nature, 479,

228–231.

O’Leary, J. G. & Hatsopoulos, N. G. (2006). Early Visuomotor Representations Revealed From

Evoked Local Field Potentials in Motor and Premotor Cortical Areas. Journal of Neurophysiology,

96, 1492–1506.

Omrani, M., Pruszynski, J. A., Murnaghan, C. D., & Scott, S. H. (2014). Perturbation-evoked

responses in primary motor cortex are modulated by behavioral context. Journal of Neurophysiology,

112, 2985–3000.

Orsborn, A. L., Moorman, H. G., Overduin, S. A., Shanechi, M. M., Dimitrov, D. F., & Carmena,

J. M. (2014). Closed-Loop Decoder Adaptation Shapes Neural Plasticity for Skillful Neuroprosthetic

Control. Neuron, 82, 1380–1393.

Pandarinath, C., Gilja, V., Blabe, C., Jarosiewicz, B., Perge, J. A., Hochberg, L. R., Shenoy, K. V.,

& Henderson, J. M. (2014). High-performance communication using neuronal ensemble recordings

from the motor cortex of a person with ALS. In American Society for Stereotactic and Functional

Neurosurgery. Washington, DC.

Pandarinath, C., Gilja, V., Blabe, C. H., Nuyujukian, P., Sarma, A. A., Sorice, B. L., Eskandar,

E. N., Hochberg, L. R., Henderson, J. M., & Shenoy, K. V. (2015). Neural population dynamics in

human motor cortex during movements in people with ALS. eLife, 4, 1–9.

Pasley, B., David, S., Mesgarani, N., & Flinker, A. (2012). Reconstructing Speech from Human

Auditory Cortex. PLoS biology, 10, e1001251.

Perge, J. A., Homer, M. L., Malik, W. Q., Cash, S., Eskandar, E., Friehs, G., Donoghue, J. P., &

Hochberg, L. R. (2013). Intra-day signal instabilities affect decoding performance in an intracortical

neural interface system. Journal of Neural Engineering, 10, 036004.

Perge, J. A., Zhang, S., Malik, W. Q., Homer, M. L., Cash, S., Friehs, G., Eskandar, E. N., Donoghue,

J. P., & Hochberg, L. R. (2014). Reliability of directional information in unsorted spikes and local

field potentials recorded in human motor cortex. Journal of Neural Engineering, 11, 046007.

Pesaran, B., Pezaris, J. S., Sahani, M., Mitra, P. P., & Andersen, R. A. (2002). Temporal structure

in neuronal activity during working memory in macaque parietal cortex. Nature Neuroscience, 5,

805–11.

Pohlmeyer, E. A., Mahmoudi, B., Geng, S., Prins, N. W., & Sanchez, J. C. (2014). Using reinforcement

learning to provide stable brain-machine interface control despite neural input reorganization. PloS

one, 9, e87253.

BIBLIOGRAPHY 114

Pohlmeyer, E. A., Perreault, E. J., Slutzky, M. W., Kilgore, K. L., Kirsch, R. F., Taylor, D. M.,

& Miller, L. E. (2007). Real-time control of the hand by intracortically controlled functional

neuromuscular stimulation. In Proceedings of the 2007 IEEE 10th International Conference on

Rehabilitation Robotics, pp. 454–458.

Pruszynski, J., Omrani, M., & Scott, S. (2014). Goal-Dependent Modulation of Fast Feedback

Responses in Primary Motor Cortex. Journal of Neuroscience, 34, 4608–4617.

Pruszynski, J. A., Kurtzer, I., Nashed, J. Y., Omrani, M., Brouwer, B., & Scott, S. H. (2011a).

Primary motor cortex underlies multi-joint integration for fast feedback control. Nature, 478,

387–90.

Pruszynski, J. A., Kurtzer, I., & Scott, S. H. (2011b). The long-latency reflex is composed of at least

two functionally independent processes. Journal of Neurophysiology, 106, 449–59.

Ramnani, N. (2006). The primate cortico-cerebellar system: anatomy and function. Nature Reviews

Neuroscience, 7, 511–22.

Rathelot, J. & Strick, P. (2009). Subdivisions of primary motor cortex based on cortico-motoneuronal

cells. Proceedings of the National Academy of Sciences, 106, 918.

Rickert, J., Oliveira, S., Vaadia, E., Aersten, A., Rotter S, & Mehring, C. (2005). Encoding of

Movement Direction in Different Frequency Ranges of Motor Cortical Local Field Potentials.

Journal of Neuroscience, 25, 8815–8824.

Riehle, A. & Requin, J. (1989). Monkey primary motor and premotor cortex: single-cell activity

related to prior information about direction and extent of an intended movement. Journal of

Neurophysiology, 61, 534–549.

Ryu, S. I. & Shenoy, K. V. (2009). Human cortical prostheses: lost in translation? Neurosurgical

Focus, 27, E5.

Sadtler, P. T., Quick, K. M., Golub, M. D., Chase, S. M., Ryu, S. I., Tyler-Kabara, E. C., Yu, B. M.,

& Batista, A. P. (2014). Neural constraints on learning. Nature, 512, 423–426.

Santhanam, G., Ryu, S. I., Yu, B. M., Afshar, A., & Shenoy, K. V. (2006). A high-performance

brain-computer interface. Nature, 442, 195–8.

Schearer, E. M., Liao, Y.-W., Perreault, E. J., Tresch, M. C., Memberg, W. D., Kirsch, R. F., &

Lynch, K. M. (2014). Multi-Muscle FES Force Control of the Human Arm for Arbitrary Goals.

IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22, 654–663.

Scherberger, H., Jarvis, M. R., & Andersen, R. A. (2005). Cortical local field potential encodes

movement intentions in the posterior parietal cortex. Neuron, 46, 347–54.

BIBLIOGRAPHY 115

Schwartz, A., Ketter, R., & Georgopoulos, A. (1988). Primate motor cortex and free arm movements

to visual targets in three-dimensional space. I. Relations between single cell discharge and direction

of movement. Journal of Neuroscience, 8, 2913–2927.

Schwartz, A. B. (2004). Cortical neural prosthetics. Annual Review of Neuroscience, 27, 487–507.

Scott, S. H. (2004). Optimal feedback control and the neural basis of volitional motor control. Nature

Reviews Neuroscience, 5, 532–46.

Scott, S. H. (2012). The computational and neural basis of voluntary motor control and planning.

Trends in cognitive sciences, 16, 541–9.

Seo, D., Carmena, J. M., Rabaey, J. M., Maharbiz, M. M., & Alon, E. (2014). Model validation of

untethered, ultrasonic neural dust motes for cortical recording. Journal of Neuroscience Methods,

244, 114–122.

Serruya, M., Hatsopoulos, N. G., Paninski, L., Fellows, M. R., & Donoghue, J. P. (2002). Instant

neural control of a movement signal. Nature, 416, 141–142.

Shadmehr, R. & Mussa-Ivaldi, F. A. (1994). Adaptive representation of dynamics during learning of

a motor task. Journal of Neuroscience, 14, 3208–3224.

Shadmehr, R., Smith, M. A., & Krakauer, J. W. (2010). Error correction, sensory prediction, and

adaptation in motor control. Annual Review of Neuroscience, 33, 89–108.

Shenoy, K. V. & Carmena, J. M. (2014). Combining Decoder Design and Neural Adaptation in

Brain-Machine Interfaces. Neuron, 84, 665–680.

Shenoy, K. V., Sahani, M., & Churchland, M. M. (2013). Cortical control of arm movements: a

dynamical systems perspective. Annual Review of Neuroscience, 36, 337–59.

Shmuelof, L. & Krakauer, J. (2011). Are we ready for a natural history of motor learning? Neuron,

72, 469–476.

Simeral, J. D., Kim, S.-P., Black, M. J., Donoghue, J. P., & Hochberg, L. R. (2011). Neural control of

cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical

microelectrode array. Journal of Neural Engineering, 8, 025027.

So, K., Dangi, S., Orsborn, A. L., Gastpar, M. C., & Carmena, J. M. (2014). Subject-specific

modulation of local field potential spectral power during brainmachine interface control in primates.

Journal of Neural Engineering, 11, 026002.

Stark, E. & Abeles, M. (2007). Predicting movement from multiunit activity. Journal of Neuroscience,

27, 8387–94.

BIBLIOGRAPHY 116

Stavisky, S. D., Kao, J. C., Nuyujukian, P., Ryu, S. I., & Shenoy, K. V. (2014). Hybrid Decoding

of Both Spikes and Low-Frequency Local Field Potentials for Brain-Machine Interfaces. In 36th

Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp.

3041–3044. Chicago, IL.

Stavisky, S. D., Kao, J. C., Nuyujukian, P., Ryu, S. I., & Shenoy, K. V. (2015a). A high performing

brainmachine interface driven by low-frequency local field potentials alone and together with spikes.

Journal of Neural Engineering, 12, 036009.

Stavisky, S. D., Kao, J. C., Sorokin, J. M., Ryu, S. I., & Shenoy, K. V. (2015b). System identification

of brain-machine interface control using a cursor jump perturbation. In 2015 7th International

IEEE/EMBS Conference on Neural Engineering (NER), pp. 643–647. (IEEE).

Suminski, A. J., Mardoum, P., Lillicrap, T. P., & Hatsopoulos, N. G. (2015). Temporal evolution of

both premotor and motor cortical tuning properties reflect changes in limb biomechanics. Journal

of Neurophysiology, p. jn.00486.2014.

Suminski, A. J., Tkach, D. C., Fagg, A. H., & Hatsopoulos, N. G. (2010). Incorporating Feedback from

Multiple Sensory Modalities Enhances Brain-Machine Interface Control. Journal of Neuroscience,

30, 16777–16787.

Sussillo, D., Nuyujukian, P., Fan, J. M., Kao, J. C., Stavisky, S. D., Ryu, S., & Shenoy, K. (2012). A

recurrent neural network for closed-loop intracortical brainmachine interface decoders. Journal of

Neural Engineering, 9, 026027.

Sutskever, I., Martens, J., & Hinton, G. (2011). Generating text with recurrent neural networks. In

Proceedings of the 28th International Conference on Machine Learning. Bellevue, WA.

Tabot, G. A., Dammann, J. F., Berg, J. A., Tenore, F. V., Boback, J. L., Vogelstein, R. J., &

Bensmaia, S. J. (2013). Restoring the sense of touch with a prosthetic hand through a brain

interface. Proceedings of the National Academy of Sciences of the United States of America, 110.

Tanji, J. U. N. & Evarts, V. (1976). Anticipatory Activity of Motor Cortex Neurons in Relation to

Direction of an Intended Movement. Journal of Neurophysiology, 39, 1062–1068.

Taylor, D. M., Tillery, S. I. H., & Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic

devices. Science, 296, 1829–32.

Todorov, E. (2000). Direct cortical control of muscle activation in voluntary arm movements: a

model. Nature Neuroscience, 3, 391–398.

Todorov, E. & Jordan, M. I. (2002). Optimal feedback control as a theory of motor coordination.

Nature Neuroscience, 5, 1226–35.

BIBLIOGRAPHY 117

Trautmann, E. M., Stavisky, S. D., Ames, K. C., Kaufman, M. T., Ryu, S. I., & Shenoy, K. V. (2016).

Sortfree: Using threshold crossings to evaluate scientific hypotheses in population analyses. In

Cosyne Abstracts, pp. Trautmann, E.M., Stavisky, S.D., Ames, K.C., Kaufm. Salt Lake City.

Tsu, A. P., Burish, M. J., GodLove, J., & Ganguly, K. (2015). Cortical neuroprosthetics from a

clinical perspective. Neurobiology of Disease.

Vargas-Irwin, C. E., Franquemont, L., Black, M. J., & Donoghue, J. P. (2015). Linking Objects to

Actions: Encoding of Target Object and Grasping Strategy in Primate Ventral Premotor Cortex.

Journal of Neuroscience, 35, 10888–10897.

Vargas-Irwin, C. E., Shakhnarovich, G., Yadollahpour, P., Mislow, J. M. K., Black, M. J., &

Donoghue, J. P. (2010). Decoding complete reach and grasp actions from local primary motor

cortex populations. Journal of Neuroscience, 30, 9659–9669.

Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., & Schwartz, A. B. (2008). Cortical control

of a prosthetic arm for self-feeding. Nature, 453, 1098–1101.

Vogel, J., Haddadin, S., Simeral, J. D., Stavisky, S. D., Bacher, D., Hochberg, L. R., Donoghue, J. P.,

& van der Smagt, P. (2010). Continuous Control of the DLR Light-weight Robot III by a human

with tetraplegia using the BrainGate2 Neural Interface System. In International Symposium on

Experimental Robotics (ISER2010), pp. 1–12. (New Delhi, India: Springer).

Waldert, S., Lemon, R. N., & Kraskov, A. (2013). Influence of spiking activity on cortical local field

potentials. The Journal of Physiology, 591, 5291–5303.

Wang, D., Zhang, Q., Li, Y., Wang, Y., Zhu, J., Zhang, S., & Zheng, X. (2014). Long-term decoding

stability of local field potentials from silicon arrays in primate motor cortex during a 2D center

out task. Journal of Neural Engineering, 11, 036009.

Willett, F. R., Suminski, A. J., Fagg, A. H., & Hatsopoulos, N. G. (2012). Compensating for delays

in brain-machine interfaces by decoding intended future movement. In 2012 Annual International

Conference of the IEEE Engineering in Medicine and Biology Society, pp. 4087–4090. (IEEE).

Wise, S. P., Boussaoud, D., Johnson, P. B., & Caminiti, R. (1997). Premotor and parietal cortex:

corticocortical connectivity and combinatorial computations. Annual Review of Neuroscience, 20,

25–42.

Wise, S. P., Moody, S. L., Blomstrom, K. J., & Mitz, A. R. (1998). Changes in motor cortical activity

during visuomotor adaptation. Experimental Brain Research., 121, 285–99.

Wodlinger, B., Downey, J. E., Tyler-Kabara, E. C., Schwartz, A. B., Boninger, M. L., & Collinger,

J. L. (2015). Ten-dimensional anthropomorphic arm control in a human brainmachine interface:

difficulties, solutions, and limitations. Journal of Neural Engineering, 12, 016011.

BIBLIOGRAPHY 118

Wolpert, D. M., Ghahramani, Z., & Jordan, M. I. (1995). An internal model for sensorimotor

integration. Science, 269, 1880–1882.

Wolpert, D. M. & Miall, R. C. (1996). Forward Models for Physiological Motor Control. Neural

Networks, 9, 1265–1279.

Wu, W., Black, M. J., Gao, Y., Bienenstock, E., Serruya, M., Shaikhouni, A., & Donoghue, J. P.

(2003). Neural Decoding of Cursor Motion Using a Kalman Filter. Advances in Neural Information

Processing Systems 15: Proceedings of the 2002 Conference, pp. 133–140.

Wu, W., Gao, Y., Bienenstock, E., Donoghue, J. P., & Black, M. J. (2006). Bayesian Population

Decoding of Motor Cortical Activity Using a Kalman Filter. Neural Computation, 118, 80–118.

Wu, W. & Hatsopoulos, N. G. (2008). Real-time decoding of nonstationary neural activity in motor

cortex. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 16, 213–222.

Zagha, E., Ge, X., & McCormick, D. A. (2015). Competing Neural Ensembles in Motor Cortex Gate

Goal-Directed Motor Output. Neuron, 88, 565–577.

Zhuang, J., Truccolo, W., Vargas-Irwin, C., & Donoghue, J. P. (2010). Decoding 3-D reach and

grasp kinematics from high-frequency local field potentials in primate primary motor cortex. IEEE

Transactions on Bio-Medical Engineering, 57, 1774–84.

Documents

ADVANCING MOTOR NEURAL PROSTHESIS ROBUSTNESS AND NEUROSCIENCEshenoy/Theses/Stavisky.pdf · 2018-05-26 · The frontier challenges that must be solved before brain-machine interfaces