Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
TOWARDS CLINICALLY RELEVANT NEURAL PROSTHESES
A DISSERTATION
SUBMITTED TO THE DEPARTMENT OF BIOENGINEERING
AND THE COMMITTEE ON GRADUATE STUDIES
OF STANFORD UNIVERSITY
IN PARTIAL FULFILLMENT OF THE REQUIREMENTS
FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY
Paul Nuyujukian
December 2012
http://creativecommons.org/licenses/by-nc/3.0/us/
This dissertation is online at: http://purl.stanford.edu/xm026gn9792
© 2012 by Paul Herag Nuyujukian. All Rights Reserved.
Re-distributed by Stanford University under license with the author.
This work is licensed under a Creative Commons Attribution-Noncommercial 3.0 United States License.
ii
I certify that I have read this dissertation and that, in my opinion, it is fully adequatein scope and quality as a dissertation for the degree of Doctor of Philosophy.
Krishna Shenoy, Primary Adviser
I certify that I have read this dissertation and that, in my opinion, it is fully adequatein scope and quality as a dissertation for the degree of Doctor of Philosophy.
Kwabena Boahen
I certify that I have read this dissertation and that, in my opinion, it is fully adequatein scope and quality as a dissertation for the degree of Doctor of Philosophy.
Jaimie Henderson
Approved for the Stanford University Committee on Graduate Studies.
Patricia J. Gumport, Vice Provost Graduate Education
This signature page was generated electronically upon submission of this dissertation in electronic format. An original signed hard copy of the signature page is on file inUniversity Archives.
iii
iv
Abstract
Neural prostheses translate signals from the brain into useful control signals, manipulating
end-effectors such as computer cursors or robotic arms. Their aim is to offer greater inter-
action with the world for patients suffering from limb dysfunction due to spinal cord injury,
neurodegenerative disease, and other conditions leading to limb paralysis. Prior intracor-
tical electrode neural prosthesis studies have demonstrated compelling proof-of-concept
systems, but barriers to successful clinical translation still remain, such as performance and
robustness. Measures of performance include the speed, accuracy, and bitrate of the sys-
tem. Robustness refers to the sustained performance of the system within a day and across
days. The work presented here demonstrates algorithms and advances for neural prostheses
that increase both performance and robustness. The recalibrated feedback intention trained
Kalman filter (ReFIT-KF) increased performance by at least twofold compared to previ-
ously reported decoders, approaching the speed of natural arm movements. It achieved
bitrates of up to 4.5 bits per second (bps) and communicationrates of up to 10 words per
minute (wpm) when used on a typing task. These results were reliable and repeatable for
hours at a time across 4 array-years between two subjects. Utilizing neural spike threshold
crossings as a signal source, the ReFIT-KF algorithm also demonstrated sustainable per-
formance without any changes to decoder weights for one yearwith a degradation rate of
0.05 bps per month.Performance further increased when the ReFIT-KF was combined with
an HMM state decoder for the detection of clicks, eliminating the need for hold periods.
This combined ReFIT-KF and HMM decoder achieved bitrates of up to 6.5 bps and 15
wpm. Taken together, these findings may help advance neural prostheses closer to clinical
viability.
v
Acknowledgements
For my parents, in recognition of their unyielding support.
vi
Contents
Abstract v
Acknowledgements vi
1 Introduction 1
1.0.1 Neural prostheses. . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.0.2 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2 Monkey Models 5
2.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 Rhesus monkey models. . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3.1 Arm restrained, no EMG modulation. . . . . . . . . . . . . . . . 7
2.3.2 Arm restrained, no EMG measurement. . . . . . . . . . . . . . . 8
2.3.3 Arm not restrained, not visible. . . . . . . . . . . . . . . . . . . . 8
2.3.4 No arm movement, nerve block. . . . . . . . . . . . . . . . . . . 9
2.3.5 Optogenetics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3.6 Freely moving . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.4 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.6 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3 A Closed-Loop Human Simulator 13
3.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
vii
3.2 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.3.1 Neural Prosthetic Experimental Hardware. . . . . . . . . . . . . . 20
3.3.2 Neural Prosthetic Experimental Task. . . . . . . . . . . . . . . . . 23
3.3.3 Human Experiments. . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3.4 Animal Experiments. . . . . . . . . . . . . . . . . . . . . . . . . 29
3.3.5 Generating Synthetic Neural Activity. . . . . . . . . . . . . . . . 30
3.3.6 Decoding Neural Activity . . . . . . . . . . . . . . . . . . . . . . 33
3.3.7 Performance Analysis. . . . . . . . . . . . . . . . . . . . . . . . 34
3.4 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4.1 Online Prosthetic Data. . . . . . . . . . . . . . . . . . . . . . . . 37
3.4.2 Comparing Online Analysis to Offline Analysis. . . . . . . . . . . 42
3.4.3 Comparing OPS to BMI. . . . . . . . . . . . . . . . . . . . . . . 46
3.5 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.5.1 The OPS for general system design questions. . . . . . . . . . . . 50
3.5.2 The OPS for specific algorithmic questions. . . . . . . . . . . . . 52
3.5.3 The OPS for neuroscientific questions. . . . . . . . . . . . . . . . 54
3.5.4 Cautionary notes regarding the OPS. . . . . . . . . . . . . . . . . 55
3.5.5 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.6 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4 ReFIT-KF Neural Prosthesis 58
4.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.2 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.3 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.3.1 Performance Overview. . . . . . . . . . . . . . . . . . . . . . . . 60
4.3.2 Generalization & Robustness. . . . . . . . . . . . . . . . . . . . . 63
4.3.3 Two Innovations . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.3.4 Innovation 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.3.5 Innovation 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.4 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
viii
4.5 Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.6 Acknowledgements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5 Task Optimization 76
5.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.2 Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.3 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
5.3.1 Grid Task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.3.2 Radial Task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.3.3 Sustained intraday performance. . . . . . . . . . . . . . . . . . . 86
5.4 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
6 Sustained performance & typing 87
6.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
6.2 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6.3 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6.4 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
6.5 Methods Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6.6 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
7 Hidden Markov model ReFIT-KF 97
7.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
7.2 Decoder Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
7.3 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
8 Conclusion 104
A Publications 106
Journal papers 107
Conference papers 109
ix
B Patents 111
Bibliography 112
x
List of Tables
2.1 Current and emerging monkey models. . . . . . . . . . . . . . . . . . . . 8
3.1 Categorization of the data sets collected and analyzed inthis paper. The
Results show offline and offline analyses (blue and red in this table and in
subsequent figures), human and monkey subjects (left and right columns
in this table), synthetic and real neural data (identical toOPS and BMI
modes, the top two and bottom two rows here), and finally the continuous
and interleaved task variants (grouped within each cell of this table). These
permutations allow us to perform a breadth of comparisons and controls to
answer the key questions of this study.. . . . . . . . . . . . . . . . . . . . 28
4.1 Performance of ReFIT-KF based control with observation based algorithm
training. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.1 Acceptance region size as a function of targets on the screen for the grid task84
xi
List of Figures
3.1 Concept figure for Online Prosthetic Simulator opportunity. The x-axis
shows four testing paradigms in terms of increasing realism. Offline data
analysis is perhaps the least reasonable proxy to eventual user mode, as it
entirely neglects the closed-loop control. On the other endof the spectrum
is the human clinical trial, which is precisely the eventualuser mode. Left
axis (blue) shows the difficulty associated with testing each algorithm or
algorithmic parameter setting. Right axis (red) shows the number of algo-
rithm and parameter choices that are reasonably testable, given costs and
other constraints.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Experimental rig which can be used both for Online Prosthesis Simulator
(OPS) and real neural Brain Machine Interface (BMI) experiments. A hu-
man (shown in gray) or monkey subject reaches (red trace) in a3D volume
obscured from view. An overhead position tracker tracks endpoint kine-
matics. Control PCs process those data and render the subject’s real reach
(in control trials) or a prosthetic decoded reach (in OPS or BMI trials, the
red trace). Two monitors (blue) project a stereo 3D image onto mirrors
(virtual 3D environment shown at right). Also displayed is athe subject’s
reach target (green sphere).. . . . . . . . . . . . . . . . . . . . . . . . . 21
xii
3.3 OPS and BMI schematic.Left Panel: OPS Mode - (a) A healthy hu-
man subject or a monkey (human shown) makes real arm reaches in a 3D
reaching environment (see Figure 3.2). (b) Recorded kinematics of that
real reach are used to generate synthetic neural activity, which is then input
to the (c) Kalman filter neural decode algorithm. The decode algorithm
decodes synthetic neural activity into physical reaching behavior, and (d)
the decoded reach is then rendered back to the subject in the 3D visual en-
vironment, which allows the subject to bring to bear all of his/her online,
closed-loop control strategies to drive a desired reach.Right Panel: BMI
Mode - (a) A monkey makes real arm reaches. (b) The real neural activ-
ity associated with that reach is recorded, which is then input to the (c)
Kalman filter decode algorithm. The decode algorithm decodes real neural
activity into physical reaching behavior, and (d) the decoded reach is then
rendered back to the monkey in the 3D visual environment, which allows
similar closed-loop control as in OPS mode. It is important to note that the
OPS mode of the left panel can be used both by a human and a monkey
(this will provide an important validation of the OPS).. . . . . . . . . . . 25
xiii
3.4 Performance metrics for Human Online Prosthetic Simulator (Human OPS)
decode trials.Left Panel: Continuous Task variant - (a) Failure Rate: the
percentage of trials where the subject did not successfullyacquire and hold
the reach target. (b) Time to Target: the time required to reach the target for
successful trials. In both panels, data from five subjects are shown in light
colors. The average of all trials is shown in dark blue. This average shows
in both cases a significant linear trend indicating that smaller bin widths
will lead to better performance. (c) To compare offline and online data, we
use a third metric: integrated distance to target (failure rate and time to tar-
get can not be calculated for offline data, as offline failure rate approaches
100% without online feedback, which is again telling of the inappropriate-
ness of offline analysis). This metric is normalized by triallength (total
trial time). Right Panel: Interleaved Task variant - These show the same
metrics as the left panel, reinforcing with another task variant these trends.
In all panels, 95% confidence intervals (A: binomial distribution, B, C:
Gaussian) are shown as error bars.. . . . . . . . . . . . . . . . . . . . . . 38
xiv
3.5 Performance metrics for Monkey Brain Machine Interface (Monkey BMI)
and Online Prosthetic Simulator (Human OPS) decode trials.Left Panel:
BMI Continuous Task variant - (a) Failure Rate: the percentage of trials
where the subject did not successfully acquire and hold the reach target.
(b) Time to Target: the time required to reach the target for successful
trials. In both panels, data from five subjects are shown in light colors.
The average of all trials is shown in dark blue. This average shows in
both cases a significant linear trend indicating that smaller bin widths will
lead to better performance. (c) To compare offline and onlinedata, we use a
third metric: integrated distance to target (failure rate and time to target can
not be calculated for offline data, as offline failure rate approaches 100%
without online feedback, which is again telling of the inappropriateness of
offline analysis). This metric is normalized by trial length(total trial time).
Middle Panel: BMI Interleaved Task variant - These show the same
metrics as the left panel, reinforcing with another task variant these trends.
Right Panel: OPS Interleaved Task variant- The monkey can also run
the OPS task with synthetic neural data. This panel shows thesame metrics
as other panels, and these metrics all show similar trends. Bycomparing
BMI and OPS within subject on the same task (middle and right panels), we
have an indication that the OPS is providing a valuable proxyto real neural
prosthetic systems. In all panels, 95% confidence intervals(A: binomial
distribution, B, C: Gaussian) are shown as error bars.. . . . . . . . . . . . 41
xv
3.6 Comparing offline analysis to online analysis in Human OPSmode. Left
Panel: Continuous Task variant- The blue error bars are replicated from
Figure 3.4, Panel C. The dark blue line is a linear fit of that data, show-
ing a significant performance trend indicating that shorterbin widths imply
better performance. A light blue quadratic fit (not visibly or statistically
different from a line) to the same data is obscured by the linear fit. In
red, we perform the same analysis offline, using the real reaching sets from
these users. We perform offline decodes at each bin width, allowing us to
generate the same mean integrated distance to target metric. These error
bars are the offline analogs to the blue error bars. Not surprisingly, offline
analysis has worse performance than online, since there wasno benefit of
feedback. More significantly, however, is the shape of the data. This offline
analysis suggests statistically significant performance optima of roughly
100-150ms. This can be seen in the significant quadratic fit tothe data
(dark red). The linear fit (light red) does a clearly poorer job of fitting
the data. Note that the characteristic “U-shape” of the offline data tells a
very different story than the online analysis, indicating the important dif-
ferences between these two testing paradigms.Right Panel: Interleaved
Task variant - The same analysis and implication holds true as in the left
panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
xvi
3.7 Comparing offline analysis to online analysis in Monkey BMIand Monkey
OPS modes.Left Panel: BMI Continuous Task variant - The blue error
bars are replicated from Figure 3.4, Panel C. The dark blue line is a linear fit
of that data, showing a significant performance trend indicating that shorter
bin widths imply better performance. A light blue quadraticfit (not visibly
or statistically different from a line) to the same data is obscured by the
linear fit. In red, we perform the same analysis offline, usingthe real reach-
ing sets from these users. We perform offline decodes at each bin width,
allowing us to generate the same mean integrated distance totarget metric.
These error bars are the offline analogs to the blue error bars. Not sur-
prisingly, offline analysis has worse performance than online, since there
was no benefit of feedback. More significantly, however, is the shape of
the data. This offline analysis suggests statistically significant performance
optima of roughly 150-200ms. This can be seen in the significant quadratic
fit to the data (dark red). The linear fit (light red) does a clearly poorer
job of fitting the data. Note that the characteristic “U-shape” of the offline
data tells a very different story than the online analysis, indicating the im-
portant differences between these two testing paradigms.Middle Panel:
BMI Interleaved Task variant - The same analysis and implication holds
true as in the left panel.Right Panel: OPS Interleaved Task variant-
Again, the same analysis and implication holds true as in theleft panel.
Taken together, these three panels show that both the BMI and OPS modes
tell the same story: shorter bin widths imply better performance, whereas
the offline analyses indicate an incorrect trend that leads to varying and
misleading performance optima.. . . . . . . . . . . . . . . . . . . . . . . 45
xvii
3.8 Summary of online/offline differences; summary of OPS/BMI similarities.
In the previous figures, we fit quadratic curves to both the offline and online
performance data. Here, we plot those quadratic regressioncoefficients
and their 95% confidence intervals. Any confidence interval overlapping
0 should be read as, ”this data is not significantly differentthan a linear
trend.” This figure shows that nearly all online analyses in monkey and hu-
man do not have significant quadratic terms, agreeing with the implication
from all previous data that shorter bin widths lead to betterperformance.
Comparing that to the red error bars, again the central point is reiterated:
online analysis and offline analysis (whether in real neuraldata or in syn-
thetic neural data) tell very different stories. Furthermore, in addition to
online and offline being very different, we see that OPS and BMIare in
fact very similar. The OPS gives a valuable means by which to sweep per-
formance based on this algorithmic parameter.. . . . . . . . . . . . . . . 47
4.1 Performance comparison.Native arm control shown in blue, ReFIT-KF
in red, and Velocity-KF in green. (a) Representative traces of cursor path
during center-out-and-back reaches. Dashed lines (not visible to the mon-
key) are the demand boxes for the eight peripheral targets and the central
target, shown as translucent green circles. (b) Bar graphs plotting maxi-
mum deviation from a straight-line path to the target on eachsuccessful
trial (mean± SE). (c,d) Histograms of time to target for successful trials
are shown as line graphs for monkeys J and L. The inset bar graphs plot the
time to target (mean± SE). (e,f) Line graphs plotting the mean distance to
the target as a function of time. The inset bar graph plots themean± SE
of the dial-in time, or the time required to finally settle on the demand box,
after first acquired, to successfully hold for 500 ms. The thickened portion
of the line graphs also indicate dial-in time, beginning at the mean time of
first target acquire, and ending at mean trial duration minus500ms. Data
from experiments J & L 20101027-29, 20101102.. . . . . . . . . . . . . . 61
xviii
4.2 Performance of ReFIT-KF control across 4 years.Performance is mea-
sured by the Fitts’ law metric. Data from monkey J and monkey Lare
shown as 94 orange circles and 182 cyan squares, respectively. Each point
plots the performance of the ReFIT-KF algorithm trained on that experi-
mental day. The eight filled data points (four for each monkey) are calcu-
lated from the same datasets used to generate Figure 4.1. Linear regression
lines for data from monkey J (orange) and monkey L (cyan) are shown. The
slope of the regression line for monkey J is not statistically significant from
zero (p>0.34). The slopes for monkey L is statistically significant from
zero (p<0.001). For all datasets shown, the trial success rate was>90%. . . 62
4.3 Performance comparison of native arm vs. ReFIT-KF for the pinball
task. Native arm control is in blue, ReFIT-KF in red. In this task, each tar-
get location is selected from a uniform distribution spanning the workspace.
(a) Each column shows data from 20-minute segments. The top rows are
randomly selected cursor traces for 4 subsequent target acquisitions. Target
demand boxes are shown as dashed lines and target sequence isindicated
from 0 to 4. The bottom row shows normalized histograms of time to target
for successful trials. Arrows below the plot indicate average time. (b) Tar-
get acquisition rate per minute throughout the sessions is shown. The sharp
rate drop indicates when the monkey lost interest in the task. A histogram
of acquisition rate across the sessions is inset. The nativearm and ReFIT-
KF sessions (L-2010-04-01 and L-2010-04-12) were on two separate days,
within 11 days of each other, when the monkey demonstrated a high degree
of motivation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
xix
4.4 Performance comparison of native arm vs. ReFIT-KF for the obstacle
avoidance task. Native arm control shown in blue, ReFIT-KF in red. In
this task the monkey had to move the cursor from the initial target (labelled
0) to the final target (labelled 1, demand box shown as dashed line) with-
out hitting the magenta-colored barrier. One representative cursor trace is
shown from each of the four principle observed movement types: curve un-
der, curve over, straight (no barrier), and collision into barrier. These data
are from experiment J-2010-03-09.. . . . . . . . . . . . . . . . . . . . . . 65
4.5 Illustrations of the online neural control paradigm and the ReFIT-KF
training methodology. (a) The input to the control algorithm at timet is a
vector of spike counts,yt , from implanted electrodes.yt is translated into
a velocity output,vt , to drive the cursor. (b) ReFIT-KF is trained in two-
stages. Initially, cursor kinematics and neural activity are collected during
arm control or during an observation phase in which cursor movement is
automated (i). These arm movement or observed cursor kinematics (ii) are
regressed against neural activity to generate an initial control algorithm.
Then, a new set of cursor kinematics and neural activity are collected using
the initial algorithm in closed loop (iii). The kinematics collected during
neural control (red vectors in (iv)) are used to estimate intention by rotat-
ing the velocities towards the goal (cyan vectors in (iv)). This estimate of
intended kinematics is regressed against neural activity to generate and run
ReFIT-KF (v). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.1 Experimental setup. Monkey was placed in front of a Wheatstone stereo-
graph [133] such that hand was not visible and presented witha virtual en-
vironment within which to interact. The monkey’s objectivewas to acquire
the target lit in green. Neural data was recorded simultaneously during the
task. Under neural prosthesis operation, the movement of the cursor was
determined by the decoder.. . . . . . . . . . . . . . . . . . . . . . . . . . 78
xx
5.2 Task layouts.a Grid task layout in a 25 target configuration. The accep-
tance windows are adjacent and non-overlapping.b Radial task layout for
8 targets. In the radial task, there were regions that were left as gaps in
the workspace to reduce inadvertent selection. In both tasks, the dotted
lines that are not shown to the monkey outline the acceptanceregion for a
particular target.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.3 Experimental data of grid task. Monkey J: a-c. Monkey L: d-f. a Acquire
time histogram of grid task with task parameters of 25 targets and 450 ms
hold time.b Bitrate heatmap of task parameters swept for grid task. Each
point in black represents a tested pair of task parameters.c Information
plot of sustained performance across a day for grid task. Thegray line
plots the average bitrate as a function of time and the black line represents
the cumulative information transmitted over the duration of the experiment.
d Acquire time histogram of grid task with 25 targets and 450 mshold time.
eBitrate heatmap as in b.f Information plot as in c.. . . . . . . . . . . . . 82
5.4 Experimental data of radial task. Monkey J: a-c. Monkey L: d-f. a Acquire
time histogram of radial task with task parameters of 8 targets and 9 cm
distance.b Bitrate heatmap of task parameters swept for radial task. Each
point in black represents a tested pair of task parameters.c Information plot
of sustained performance across a day for radial task. The gray line plots
the average bitrate as a function of time and the black line represents the
cumulative information transmitted over the duration of the experiment.d
Acquire time histogram of radial task with 8 targets and 7 cm distance.e
Bitrate heatmap as in b.f Information plot as in c.. . . . . . . . . . . . . . 83
5.5 Bitrate plots for the grid task when the number of targets was varied.a
Monkey Jb Monkey L . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.6 Bitrate plots for the grid task when the hold time was varied. a Monkey J
b Monkey L . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.7 Bitrate plots for the radial task when the number of targets was varied.a
Monkey Jb Monkey L . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
xxi
5.8 Bitrate plots for the radial task when the distance to the targets was varied.
a Monkey Jb Monkey L . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
6.1 Decoding without retraining.a Daily performance plot of 14 day long ex-
periment with Monkey J. The gray line plots the instantaneous, smoothed
information rate across time. The black line plots the cumulative informa-
tion transmitted across the experiment. The x-axis indicates the time the
experiment was run, each tick mark notes an hour in the day.b Similar
performance plot for Monkey L. Plots are from datasets J110926-J111014
& L110912-L110925.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
6.2 Typing task.a Plot of instantaneous, smoothed typing rate across time for
Monkey J while typing an article. Gaps in trace represent times where the
monkey paused and was not typing.b Histogram of words per minute for
Monkey J, same data as in a.c & d Same as in subpanels a & b, but for
Monkey L. Typing data from from datasets J111021 & L111103.. . . . . . 91
6.3 Historical models.a Longitudinal study showing performance of Mon-
key J with a static decoder over one year. The filled in circlesdenote
performance from the first month of experiments with a singledecoder
(Fig 6.1a). That decoder, held constant, was tested over a year. The solid
line was regressed against the performance across a year andis the equa-
tion: Information rate= −0.034· (# months)+3.78bpsb Cross-sectional
historical study of decoder performance from previously recorded data in
Monkey J. Each column of points represents a single model tested. Each
color/symbol represents an experimental day. The peformance at time 0
represents the performance of the decoder on that experimental day. The
black line shows the average performance for each model.c Same as
in subpanel b, but for Monkey L. Panels b & c from datasets J120117-
J120120, L120207-L120210, & L120214.. . . . . . . . . . . . . . . . . . 93
xxii
6.4 Tuning Analysisa Tuning curves of top 5 contributory channels for a 6
month-old decoder for Monkey J. The number to the left of the axis denotes
the channel. Each colored curve represents a different experimental day.
The black curve represents the tuning on the day the decoder was generated
(J110722). The straight lines represent the channel’s cosine fit preferred
direction. b Same tuning curves as in subpanel a, but for Monkey L. The
original decoder was from L110805.c Bar graph of contribution-scaled,
per-channel rotation in tuning for Monkey J. The first columnplots the
unsigned weighted summed rotation of all channels while thesecond plots
the signed net rotation.d Same plot as in subpanel e, but for Monkey L.e
Plot of individual contribution of top 30 channels for Monkey J (red) and L
(blue). f Plot of cumulative sorted per channel contribution to decoder. All
plots derived from same datasets as in Figure 6.3.. . . . . . . . . . . . . . 95
7.1 Two state transition graph. Graphical model of the transition flow for the
two state HMM used with Monkey J. The system starts in the movestate
and can stay in that state at the next timestep. It can likewise transition to
the stop state if there is sufficient information from the neural emissions
matrix to push it to that state. If the system stays in the stopstate for two
timesteps, the HMM notes a click and immediately transitions back to the
move state. The state is locked in the move state for a lockouttime (around
250 ms), before it is allowed to transition out of the move state again. . . . 99
xxiii
7.2 Four state transition graph. Graphical model of the transition flow for the
four state HMM used with Monkey L. The system starts in the slow state.
The system can stay in any state (except stop) for an arbitrary number of
timesteps and can transition to the idle state from any state. Transitions
from the movement based states (slow 1, fast, slow 2) are onlyallowed to
proceed in a stepwise fashion. From the slow state, if the kinematic profile
(magnitude velocity) is above the specified threshold, the system transitions
to the fast state. Once the kinematic profile falls below the specified thresh-
old, the state transitions to the second slow state. The system can transition
to the stop state Only from the second slow state. Upon transition to the
stop state, the system communicates a click signal and immediately returns
to the first slow state. As in the two state HMM, there is a lockout time
enforced before a second stop state can be entered.. . . . . . . . . . . . . 100
7.3 Bitrate for Monkey J with HMM ReFIT-KF decoder. Plot of bitrate
over time for two different days(a) and aggregate bitrate histogram(b) for
the two state HMM ReFIT-KF decoder with Monkey J on a randomized
target grid task. The task was a 7x7 grid with 49 targets and was taken
from datasets J120405 dn J120406.. . . . . . . . . . . . . . . . . . . . . . 101
7.4 Bitrate for Monkey L with HMM ReFIT-KF decoder. Plot of bitrate
over time for three different days(a) and aggregate bitrate histogram(b) for
the four state HMM ReFIT-KF decoder with Monkey L on a randomized
target grid task. The task was a 5x5 grid with 25 targets and was taken from
datasets L120426 L120427 and L120430.. . . . . . . . . . . . . . . . . . 101
7.5 Typing rate for Monkey J with HMM ReFIT-KF decoder. Plot of typing
rate over time(a) and typing rate histogram(b) for the two state HMM
ReFIT-KF decoder with Monkey J. The task was a 6x6 grid with 36 targets
and was taken from datasets J120409.. . . . . . . . . . . . . . . . . . . . 102
7.6 Typing rate for Monkey L with HMM ReFIT-KF decoder. Plot of typ-
ing rate over time(a) and typing rate histogram(b) for the four state HMM
ReFIT-KF decoder with Monkey L. The task was a 6x6 grid with 36 targets
and was taken from datasets L120501.. . . . . . . . . . . . . . . . . . . . 102
xxiv
Chapter 1
Introduction
1.0.1 Neural prostheses
Neural prostheses are systems that translate neural signals into control signals for pros-
thetic devices. These neural signals could be from electroencephalography, [5,72,82] elec-
trocorticography, [8, 78, 84, 105, 106] or the intracortical electrodes as presented in this
work. [12, 57, 58, 94, 102, 109, 124, 130, 132] The target prosthetic devices include com-
puter cursors, keyboards, robotic arms, or any other end effector that could be manipulated
by these control signals. There have been compelling proof-of-concept demonstrations of
these systems, but there are still significant barriers to clinical translation. [52, 63] These
include issues of limited performance, poor robustness, and the need for frequent system
retraining. Performance is measured by values such as trialsuccess rate, straightness of
movements, and information theoretic measures such as bitrate. Robustness refers to the
sustained performance of the system in two domains: sustainable performance within days
(intraday) and across days (interday). Neural prostheses to date have shortcomings in all of
these domains that may need to be improved to make the successful translation to the clinic.
Presented here is the development of an intracortical neural prosthesis system evaluated in
a rhesus monkey model that may address some of these issues. The recalibrated feedback
intention trained Kalman filter (ReFIT-KF) improves the performance of neural prostheses
in a two-dimensional cursor task by at least twofold over current systems, enabling faster,
straighter, and more accurate cursor movements. When evaluated on an optimized grid task
1
2 CHAPTER 1. INTRODUCTION
that enabled the measurement of throughput via achieved bitrate, the ReFIT-KF achieved
rates of up to 4.5 bits per second (bps). This high performance was sustainable for several
hours at a time. Further, when evaluated for performance without retraining, the ReFIT-KF
can sustain performance for a year without any changes to thedecoder with degradation
rate of 0.05 bps per month, demonstrating long-term robustness. The ReFIT-KF achieved
a typing speed of up to 10 words per minute (wpm) when used to simulate typing on a grid
keyboard layout. When combined with an HMM used to decode a click state, performance
further increased. This combined ReFIT-KF HMM achieved a throughput of 6.5 bps and
a typing speed of 15 wpm. Taken together, these advances may increase the performance
and robustness of neural prostheses and help increase theirclinical viability.
1.0.2 Outline
In compiling this dissertation, the intent was to present a coherent and meaningful story
about the development of high-performing neural prostheses developed in the Shenoy Lab.
Over the course of the past four years, the lab has developed novel techniques and al-
gorithms for neural prostheses that enabled higher performing systems. This outline will
detail the chapters of the dissertation, briefly introducing the main ideas and providing con-
text as how how concepts and ideas from the prior chapters were incorporated in subsequent
chapters.
Monkey models
This first chapter details the rationale behind the use of monkeys as an appropriate
animal model when developing clinical neural prostheses. It discusses the different
monkey animal models that have been used and their relative benefits and shortcom-
ings. It also details why the work presented in this dissertation has been based off of
the single arm-restrained monkey model, as it provides the least constraints on the
dimensions that the neural state can explore.
A closed-loop human simulator
This second chapter describes a series of offline simulations and online experiments
using both humans and monkey subjects to evaluate the behavior of neural prosthe-
ses. These experiments highlighted the discrepancy between offline predictions and
3
optimal online performance. Specifically, using this simulator system, it was deter-
mined that shorter neural bin times yield higher performingsystems. The findings of
this study helped lay the foundations for subsequent decoder development.
ReFIT-KF neural prosthesis
This chapter details the development and testing of the ReFIT-KF. This algorithm
performed at least twice as well as prior continuously controlled endpoint decoders.
The ReFIT-KF is comprised of two innovations that led to this increase in perfor-
mance. The first is the cursorGoal correction in which the training set is manipulated
based on assumed kinematic intentions before building the decoder. The second is a
modification to the decoder in which the position of the cursor on screen is accounted
for and removed from the uncertainty of the state estimationin the decoder. Taken
together, these two innovations lead to increased performance on a cursor task and
also enable repeatable performance over years that is sustainable for hours a day.
Task optimization
This chapter describes two new keyboard-like tasks that were used to evaluate the
achievable bitrate from the ReFIT-KF. The two tasks presented were a grid keyboard
task and a radial keyboard task. For each task, the two primary parameters that govern
the task layout were manipulated in an effort to find the optimal task parameters that
maximize bitrate. It was found that the overall bitrate of the decoder did not vary
largely between tasks, as the maximum bitrate for a given subject was similar in both
tasks, with a maximum bitrate of 4 bps. These optimal task parameters were then
used as the basis for future experiments.
Sustained performance and typing
This chapter describes the performance of the ReFIT-KF on a grid keyboard task
using the optimal parameters found from the prior chapter. Specifically, the exper-
iments looked at sustained performance without retrainingsessions. One subject
demonstrated sustained performance of a single set of decoder weights for a year,
achieving a maximum rate of 4.5 bps with a degradation rate of0.05 bps per month.
Further, when the ReFIT-KF was used to transmit text, it achieved rates of up to
10 wpm.
4 CHAPTER 1. INTRODUCTION
HMM-ReFIT-KF decoder
This chapter details the decoding of a click signal using a hidden Markov model state
decoder running in parallel to the ReFIT-KF that controls cursor velocity. Using these
two decoders simultaneously enables the elimination of thehold time requirement
present in all the work presented thus far. Eliminating holdtime speeds up the rate
of selection and could increase performance. On a grid task,bitrates of up to 6.5 bps
were achieved and typing rates of up to 15 wpm. These represent a 40-50% increase
in performance over the ReFIT-KF decoder when used alone.
Chapter 2
Monkey Models
This chapter was a conference paper at the 33rd annual IEEE EMBS Conference in 2011
titled “Monkey models for brain-machine interfaces: the need formaintaining diversity.”
It was authored by Paul Nuyujukian,* Joline Fan,* Vikash Gilja, Paul Kalanithi, Cynthia
Chestek, and Krishna Shenoy. The author’s contribution to this work was in drafting and
writing the manuscript.
2.1 Abstract
Brain-machine interfaces (BMIs) aim to help disabled patients by translating neural signals
from the brain into control signals for guiding prosthetic arms, computer cursors, and other
assistive devices. Animal models are central to the development of these systems and have
helped enable the successful translation of the first generation of BMIs. As we move toward
next-generation systems, we face the question of which animal models will aid broader
patient populations and achieve even higher performance, robustness, and functionality.
We review here four general types of rhesus monkey models employed in BMI research,
and describe two additional, complementary models. Given the physiological diversity
of neurological injury and disease, we suggest a need to maintain the current diversity of
animal models and to explore additional alternatives, as each mimic different aspects of
injury or disease.
5
6 CHAPTER 2. MONKEY MODELS
2.2 Introduction
Neurological injury and disease often result in the permanent loss of motor and sensory
function. In some cases the disability is so severe that it isnot possible to feed oneself or
even communicate. BMIs are a new class of electronic medical systems that aim to improve
the quality of life for disabled patients. These systems interface with the central nervous
system and use neural signals from the brain to control prosthetic devices (e.g., [52]). In
recent years, first-generation BMIs developed with rhesus monkey animal models have
translated from the laboratory into an FDA Phase-I clinicaltrial focused on tetraplegics
and ALS patients (BrainGate I & II, [31,56,57,69,70,117]).
Next-generation BMIs aim to further improve the quality of life of disabled patients
and expand to larger patient populations, such as amputees and paraplegics. These next-
generation systems may meet this goal if performance, robustness, and functionality can
be increased substantially [44, 98]. But what types of monkey models are best suited to
develop next-generation BMIs? It is currently unclear if a single animal model can capture
all aspects of injury and disease or mimic different physiological conditions observed in
patient populations. Thus, we suggest that maintaining andexpanding the diversity of
available monkey models is critical.
We briefly review four major existing rhesus models, and briefly describe two emerging
models. We note that there are several other models not covered here, that either exist or
are possible. These models may employ pharmacological, lesioning, or electrical micro-
stimulation methods. Together, this set of animal models should help provide a diverse
resource for laboratory research being conducted, which may prove essential for the suc-
cessful translation of next-generation BMIs.
2.3 Rhesus monkey models
There appear to be four widely used rhesus monkey models. These have been employed
in recent BMI laboratory experiments, and are listed in Table2.1. We note that the animal
model used while training these systems may be different from the animal model used
while evaluating BMI performance. However, for simplicity,we have categorized these
2.3. RHESUS MONKEY MODELS 7
references based only on the animal model used in evaluationof BMI performance.
2.3.1 Arm restrained, no EMG modulation
In the first model, the monkey’s contralateral arm is restrained and monitored for task-
modulated muscle activity1. The absence of this muscle activity does not require enforcing
electromyography (EMG) silence, but rather that EMG activity not change within or across
behavioral trials. This model has been employed by Schwartzand colleagues, where it was
used in a control experiment to see if neural cursor control could be maintained without arm
movements or EMG modulation [124]. This model has also been employed by Nicolelis,
Carmena, and colleagues [12, 38] as well as by Andersen, Shenoy, and colleagues, al-
beit measuring EMG in separate experiments to assure no EMG modulation during an
“instructed-delay” period [88,102].
This monkey model “looks like a paralyzed patient,” which may be beneficial to BMI
research insofar as there are no arm movements or muscle contractions that would lead
to sensory signals. These signals, including vision, somatosensation, and proprioception,
could feed back to the cortical area driving the BMI and could influence BMI control.
Such sensory input could be viewed as a potential confounding factor, as it is presumably
not present in paralyzed patients as a potential source of useful information, or it could be
viewed as an important opportunity for increasing performance [122]. There also appear
to be two open questions with this model. The first is whether the range and pattern of
possible neural activity is constrained, by virtue of the animal being restricted to not move
the arm or modulate EMG activity. Paralyzed patients would presumably not have this
neural constraint since the injury or disease prevents neural activity from reaching muscles,
regardless of its range or pattern. Second, we ask whether all relevant muscle groups can be
monitored to assure no task-relevant EMG modulation given that the homunculus is highly
fractured and individual neurons may respond with respect to multiple muscle groups. This
is a practical concern, but one that is brought into focus by recent studies highlighting how
individual neurons in the nominal arm area of primary motor cortex contain considerable
1The contralateral arm with respect to the electrode-array implant. The ipsilateral arm is also often re-strained since motor cortical activity is also related to ipsilateral arm movements (e.g., [39]) However theipsilateral arm’s EMG activity is seldom if ever measured.
8 CHAPTER 2. MONKEY MODELS
Table 2.1: Current and emerging monkey models
Current Monkey Models ReferencesA) Arm restrained, no EMG modulation [12,38,88,102,124]B) Arm restrained, no EMG measurement[36,116,130]C) Arm not restrained, not visible [46,47,87,91,99,109,116,124]D) No arm movement, nerve block [86,96]
Emerging Monkey Models ReferencesE) Optogenetics [9,28,44,53]F) Freely moving [16,34,45,61,83]
hand and finger movement related activity [129].
2.3.2 Arm restrained, no EMG measurement
The second model again restrains the monkey’s contralateral arm, but does not measure
EMG activity. Some finger, hand, and arm movement as well as force generation is
therefore allowed. This model has been employed by Schwartz, Vaadia, and colleagues
[36,116,130].
This monkey model “looks a bit less like a paralyzed patient”because there may be
some visible movement. This allowed movement may be beneficial, as described above,
because the neural population under observation may be lessconstrained. This second
model also recognizes, implicitly, that it is challenging to record from all relevant muscles
to confirm that there is no EMG modulation. Instead, small movements and small amounts
of force production are allowed. An open question with this model is whether muscle co-
contraction, isometric force production against the arm restraint, and/or small movements
are producing sensory signals that could feed back to the cortical area driving the BMI.
2.3.3 Arm not restrained, not visible
The third model does not restrain the monkey’s contralateral arm. It also does not allow
the arm to be viewed by the animal, as was the case for the first two monkey models. This
2.3. RHESUS MONKEY MODELS 9
model has been employed by Schwartz, Donoghue, Andersen, Vaadia, Batista, Yu, Shenoy,
and colleagues [46,47,87,91,99,109,116,124].
This monkey model “looks less like a paralyzed patient” because there is frank visi-
ble (to an observer, not the animal) movement. This movementmay again be beneficial
because the neural population under observation may be lessconstrained or altogether un-
constrained. There appear to be two open questions with thismodel. The first is how
to interpret the presumed presence of proprioceptive and somatosensory signals that feed
back to the cortical area driving the BMI, as a result of arm movements. Second, several
groups have observed that sometimes monkeys stop moving their arms and the BMI con-
tinues to operate. It would appear that monkeys can continueto operate what is in essence
the first or second animal model after voluntarily transitioning to keeping the arm motion-
less [12,87,109]. This may suggest that the difference between these three animal models
is not large from the perspective of BMI algorithm design and operation.
2.3.4 No arm movement, nerve block
The fourth monkey model employs a local anaesthetic to blockmuch, if not all, of the
efferent motor signals going to the arm as well as afferent sensory signals coming from the
arm. This is a newer model and has been employed by Fetz, Miller, and colleagues [86,96].
This monkey model “looks like a paralyzed patient” since thearm, hand, and fingers
are (temporarily, reversibly) paralyzed and sensory information is substantially attenuated.
As such, this model may closely mimic a spinal cord injury patient. This model may also
be beneficial, as described above, because the neural population under observation may be
less constrained or altogether unconstrained. There appear to be several open questions
with this model, largely due to its recent development. The first stems from the observation
that nerve blocks are peripheral nerve interventions, whereas spinal cord injury patients
have lesions in the central nervous system. Second, the degree of afferent activity block is
presumably difficult to verify (i.e., daily nerve conduction studies are not feasible). Third,
nerve block is a temporary, acute intervention, whereas spinal cord injury is a chronic
condition. As a result, adaptive or deteriorative changes in the cortex of a patient with
spinal cord injury are presumably not modeled. The daily novelty of paralysis may also
10 CHAPTER 2. MONKEY MODELS
be a distraction to the animal and could even present cue conflict (i.e., can see the arm,
but can’t move or feel it). These factors could lead to different cortical neural activity than
would be present in the chronic patient case. Fourth, the pharmacokinetics of the local
anaesthetic result in a gradual return of motor and sensory function over the course of a
long experiment. This is potentially complicated by the difficulty of verifying the degree
of sensory block and mitigating its day-to-day variability.
Finally, the technical complexities are non-trivial, including surgically implanting a
nerve cuff and drug reservoir, injection-filling the reservoir periodically, and animal hus-
bandry following experiments. While not necessarily a limitation, this additional technical
complexity is a consideration when selecting animal models. As discussed below, optoge-
netic and freely moving monkey models also have additional technical complexity.
2.3.5 Optogenetics
One emerging model that could potentially (temporarily, reversibly) emulate paralysis,
stroke, or other disorders is an “optogenetic monkey model.” Optogenetics provides a
method for exciting or inhibiting neural activity while simultaneously recording, and does
so with high spatial (individual neuron, specific neuron types, specific projection pattern)
and temporal (millisecond timescale) resolution. Severaloptogenetic methods have re-
cently been translated to rhesus monkeys [9,28,53]. With these methods it may be possible
to use light to reversibly mimic injury and disease: “synthetic cord injury” (e.g., inhibit
activity in descending motor fibers with NpHR), “synthetic stroke” (e.g., inhibit activity
within an illumination-defined volume of gray matter with NpHR), and “synthetic spastic-
ity” (e.g., excite activity at specific rhythms with ChR2, or elevate baseline activity with
SFO). It may also be possible to use these methods as part of the BMI system itself to, for
example, “write in” artificial sensory information coming from sensors built in to prosthetic
hands [44].
2.3.6 Freely moving
Another emerging model that could potentially be used to understand more naturalistic
and freeform movements, and is relevant for designing BMIs toassist amputees who live
2.4. DISCUSSION 11
active lives, is a “freely moving monkey model.” It is now possible to build miniature head-
mounted systems to record from multi-electrode arrays, transmit these data wirelessly to
a nearby receiver, and use modern computer vision and markerless motion tracking tech-
niques for high-precision behavioral measurement [16, 45, 83]. In combination, this en-
ables monkeys to move freely and perform a variety of naturalistic tasks while studying
how populations of neurons control movement. This could lead to a more comprehen-
sive understanding of cortical motor control across a wide range of naturalistic movements
and behavioral contexts, and thereby better mimic the lifestyle of arm amputees as well as
model healthy humans [34]. The freely moving monkey model also enables studies explor-
ing chronic neural stimulation, including stimulation contingent on specific neural activity,
and has been employed to demonstrate motor plasticity [61].
2.4 Discussion
A range of rhesus monkey models is currently being employed to help advance BMI de-
sign, and their translation to disabled patients. First-generation BMIs have shown substan-
tial promise and are currently part of a clinical trial. Withthe desire to provide even greater
benefit to the current patient populations, as well as to helpmore patients who suffer from a
wider range of disabilities, next-generation systems are coming into focus. A central ques-
tion to this new endeavor is what types of rhesus models are currently available, and what
new models might be needed. We have attempted to provide a brief review of the existing
models, categorized into four broad types, as well as an overview of two emerging monkey
models currently under development. Two additional pointsare worth highlighting.
First, animal models are essential and appear to be working fairly well. As such, this
diversity of models ought to be maintained. BMIs are still in their early days, and consider-
able additional basic science, basic engineering, and pre-clinical testing are essential. The
current animal models have already led to a clinical trial and, equally importantly, the basic
BMI system architecture appears to work with little design modification when switching
between animal models. This suggests that rhesus monkey models are both appropriate
to translational BMI efforts, and that such a diverse set of animal models may map well
across a range of physiological conditions. It could be thatthe different animal models
12 CHAPTER 2. MONKEY MODELS
each mimic a different neurological injury or disease, and having a set of BMIs that each
operate well with one or more of these models is prudent.
Second, we ought to not only maintain the diversity of rhesusmodels, but also continue
to investigate new ones. These new animal models may providethe field with useful plat-
forms for examining various pathological presentations. It is important to recognize that
BMIs aim to assist patients with a variety of different neurological injuries and disease. A
wide variety of pathology can result in debilitating loss ofmotor function while preserving
cortical areas. Although cervical spinal cord injury is theprototypical example, traumatic
injury can occur anywhere along the pathway, from cortex to subcortical structures to the
distal limb. Many other mechanisms for loss of function alsoexist, including neurodegen-
erative diseases, autoimmune conditions, neuropathies, and myopathies. This suggests that
different monkey models may be needed to mimic these varied pathologies. Even within
the sub-class termed “upper spinal cord injury” there is a whole spectrum of actual injuries
and associated dysfunctions that may leave a patient with varied impairments from moder-
ate paresis to full paralysis, with or without concomitant sensory deficits. Similarly, then,
a rich and growing spectrum of monkey models is presumably required to cover the range
of spinal cord injuries as, ultimately, it is unlikely that asingle monkey model will suffice.
2.5 Conclusion
A range of rhesus monkey models currently exists for BMI research, and they can be
broadly categorized into four types. Two additional types of rhesus models that are emerg-
ing were also reviewed briefly. We suggest that this diversity of models is important,
should be maintained, and expanded as part of the overall effort to design and translate
next-generation BMIs.
2.6 Acknowledgments
We thank the members of the Neural Prosthetics Systems Laboratory at Stanford University
for valuable scientific discussions and editorial feedback.
Chapter 3
A Closed-Loop Human Simulator
This chapter appeared as a paper in the Journal of Neurophysiology in 2010 titled “A
closed-loop human simulator for investigating the roles offeedback-control in brain-machine
interfaces.” It was authored by John Cunningham, Paul Nuyuyujukian, Vikash Gilja, Cyn-
thia Chestek, Stephen Ryu, and Krishna Shenoy. The author’s contribution to this work
was in experimental design, data collection and analysis, and assiting in writing the paper.
3.1 Abstract
Neural prosthetic systems seek to improve the lives of severely disabled people by decoding
neural activity into useful behavioral commands. These systems and their decoding algo-
rithms are typically developed “offline,” using neural activity previously gathered from a
healthy animal, and the decoded movement is then compared tothe true movement that
accompanied the recorded neural activity. However, this offline design and testing may
neglect important features of a real prosthesis, most notably the critical role of feedback-
control, which enables the user to adjust neural activity while using the prosthesis. We hy-
pothesize that understanding and optimally designing high-performance decoders require
an experimental platform where humans are in closed-loop with the various candidate de-
code systems and algorithms. It remains unexplored the extent to which the subject can,
for a particular decode system, algorithm, or parameter, engage feedback and other strate-
gies to improve decode performance. Closed-loop testing maysuggest different choices
13
14 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
than offline analyses. Here we ask if a healthy human subject,using a closed-loop neural
prosthesis driven by synthetic neural activity, can informsystem design. We use this On-
line Prosthesis Simulator (OPS) to optimize “online” decode performance based on a key
parameter of a current state-of-the-art decode algorithm -the bin width of a Kalman filter.
First, we show that offline and online analyses indeed suggest different parameter choices.
Previous literature and our offline analyses agree that neural activity should be analyzed
in bins of 100-300ms width. OPS analysis, which incorporates feedback-control, sug-
gests that much shorter bin widths (25-50ms) yield higher decode performance. Second,
we confirm this surprising finding using a closed-loop rhesusmonkey prosthetic system.
These findings illustrate the type of discovery made possible by the OPS, and so we hy-
pothesize that this novel testing approach will help in the design of prosthetic systems that
will translate well to human patients.
3.2 Introduction
Debilitating conditions like spinal cord injuries can leave a human without voluntary mo-
tor control. However, in many cases, the brain itself maintains normal function. Millions
of people worldwide suffer motor deficits due to these diseases and injuries that result in
significantly diminished ability to interact with the physical world. Indeed, tetrapalegic
humans list regaining “arm/hand function” as the top priority for improving their quality
of life, as restoring this function would allow significant independence [2]. To address this
huge medical need, brain-machine interfaces (BMI, also called neural prosthetic systems
or brain-computer interfaces) seek to access the information in the brain and use that in-
formation to control a prosthetic device such as a robotic arm or a computer cursor. Such
systems, if successful, would have a large quality of life impact for many people living with
these debilitating medical conditions.
In the last decade, advances in neural recording technologies have accelerated research
in neural prostheses. Technologies for neural recording include electroencephalography
(EEG), electrocorticography (ECoG), and penetrating electrode or microwire arrays (see
[76] for a review). To design a prosthetic arm that can be controlled continuously with high
precision, most work has focused on penetrating electrodesimplanted directly into motor
3.2. INTRODUCTION 15
cortical areas [107, 130]. Researchers use nonhuman primates (e.g., rhesus monkeys) or,
increasingly, human participants [57, 69]. There are many medical, scientific, and engi-
neering challenges in developing such a system [76,98,107,108], but all neural prostheses
share in common the need for a decode algorithm. Decode algorithms map neural activity
into physical commands such as kinematic parameters to control a robotic arm.
Much work has gone into this domain, and many experimental paradigms and decoding
approaches have been developed and used [3,6,7,11,12,14,17,30,38,41,42,54,57,67–69,
71,74,77,79,86–88,95,100,102,109,113–115,119–121,124,130–132,138–141]. Most re-
search in decode algorithms has been done with offline data analysis using simulated neural
data [6,67,119–121,131] and/or neural data that was previously recorded from a healthy
animal [3, 6, 7, 11, 17, 30, 41, 42, 54, 68, 77, 87, 95, 102, 109, 113–115, 124, 132, 138–141].
In these studies, the benchmark for success is often how wellthe decoded arm trajec-
tory matches the true arm movement that was recorded in conjunction with the (possi-
bly simulated) neural activity. A smaller number of papers have used an online, closed-
loop paradigm to illustrate that prostheses can be meaningfully controlled by humans or
monkeys [12,14,38,57,69,71,79,86–88,102,109,124,130], but only a few of these pa-
pers [14,71,79] compare closed-loop performance of different algorithmsin monkeys, and
only one of these papers [69] compares the closed-loop performance of two different de-
code algorithms in humans. This reality is at least in part driven by the substantial resources
and effort required for online studies in animals and humans, thereby prohibiting extensive
online algorithmic comparisons.
Despite this abundance of work, our ability to decode arm movements accurately re-
mains limited. To decode an arbitrary reach, one current state-of-the-art algorithm is
perhaps the Kalman filter (introduced nearly fifty years ago in [65], used in this context
in [69,139]), which is the only algorithm that has been vetted in onlinehuman experiments
as having better performance than some competing possibilities such as a linear filter [69]
(though the closed-loop monkey studies of [14, 71, 79] indicate that other algorithms are
also competitive). Current achievable performance is encouraging and we have exciting
proofs of concept [57, 102, 130], but we must advance considerably before these systems
are clinically viable, and further still before we achieve decoded movements with speed
and accuracy comparable to a healthy arm (e.g., near perfect, sub-second accuracy).
16 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
In response to this critical need, several groups have proposed more advanced mathe-
matical approaches to neural prosthetic decoding [3,6,7,30,41,67,68,74,79,87,95,100,
113, 115, 119–121, 131, 140, 141]. However, none of these methods has seen widespread
adoption (across research studies or in critical translational work), in part due to uncertainty
of how these methods translate to closed-loop decode performance.
Offline evaluation of algorithms may neglect potentially important features of a real
neural prosthesis, including the user’s ability to modify control strategies to improve pros-
thetic performance. Truly understanding decode performance requires the human learning
machine (the brain and motor plant) to be in closed-loop withthe decode algorithm. In
this online, closed-loop setting, as soon as a prosthesis user sees a decoded arm reach (the
action of a robotic arm or the path of a cursor on a computer screen), he/she will bring to
bear all of his/her modification strategies to drive a desirable reach.
As a specific example of offline vs. online evaluation (in anticipation of the experiments
done here), a previous offline study found that prosthetic decode error is minimized when
the time bin over which neural activity is integrated (a windowed spike count in the Kalman
filter) is 200-300ms [139]. This bin width also represents the time step at which the algo-
rithm updates its estimate of the decoded reach. However, itmay be that in a closed-loop
experiment, when reaches last only roughly 1000ms, the intermittent “hopping” behavior
of a decoded reach will frustrate the user. Perhaps better control could be gained with a
more frequent update, where feedback-control would compensate for the increased noise
in the decode. Indeed, in [69], shorter bin widths (50 and 100ms) were used in online
human experiments. It would appear that shorter bin widths were found to be better in
initial online testing, though perhaps not optimized online, which may be in part due to the
overall difficulty/challenges of testing with disabled human participants. Thus, it remains
unclear how this and other parameters should be set in futurestudies. This simple ques-
tion, motivated in part by the work of [69,139], can be answered with current algorithmic
technologies, but it requires closed-loop validation. Thefield should investigate the extent
to which the subject can, for a given decode algorithm, engage online control strategies
to improve decode performance. Closed-loop testing may suggest different priorities for
algorithmic development than offline analyses.
3.2. INTRODUCTION 17
101
102
103
104
100
101
102
103
104
100
Offline Analysis Human OPS Animal Model Human Trial
Cost per
tested
algorithm/
parameter
choice
(risk, time,
financial,...)
Number of
algorithm/
parameter
choices that
can be tested
Unexplored
Increasing Realism
Figure 3.1: Concept figure for Online Prosthetic Simulator opportunity. The x-axis showsfour testing paradigms in terms of increasing realism. Offline data analysis is perhapsthe least reasonable proxy to eventual user mode, as it entirely neglects the closed-loopcontrol. On the other end of the spectrum is the human clinical trial, which is preciselythe eventual user mode. Left axis (blue) shows the difficultyassociated with testing eachalgorithm or algorithmic parameter setting. Right axis (red) shows the number of algorithmand parameter choices that are reasonably testable, given costs and other constraints.
Addressing this problem is highly challenging, since fullydoing so would imply val-
idating every algorithmic choice, ideally, in a human clinical trial. Algorithmic choices
include both the structure of the algorithm itself and the parameter settings that should be
optimized, resulting in thousands of decode possibilities. Given the invasiveness and re-
source requirements of a full neural prosthetic clinical trial, this approach is infeasible. To
address this challenge, the field has employed an appropriate animal model such as a rhesus
monkey. However, given the large resource and temporal requirements of awake behaving
intracranial experiments, such an approach to widespread algorithm design is still imprac-
tical. Faced with this reality, most algorithmic work has been done in offline neural data
(simulated or real).
We ask here if a healthy human subject, using an entirely noninvasive prosthetic device
driven by synthetic neural activity, can meaningfully inform the design of prosthetic decode
algorithms. This system, which we call an Online ProsthesisSimulator (OPS), represents a
middle ground between simple (but perhaps less realistic) offline testing and more realistic
18 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
(but difficult and resource intensive) animal model and human clinical trials. We detail the
concept of this proposed system in Figure3.1. This figure shows in blue the dramatically
increasing complexity of testing each algorithmic choice (algorithm or parameter setting
within a single algorithm) as researchers move towards animal studies or human clinical
trials. This figure also shows (in red) the corresponding dramatic decrease in the number of
algorithmic choices that can be meaningfully tested. Here we ask two questions: first, how
different are offline and online analyses; and second, how similar are the OPS and closed-
loop animal model BMI? The OPS, by analogy to flight simulatorsor silicon integrated
circuit simulation software like “SPICE,” may allow more realistic evaluation of current
and future prosthetic decode approaches.
The creation of the OPS and the messages of this paper have important connections to
previous work. First, we note that previous literature has used signal sources other than
motor cortex to control an external BMI-like interface. In [97], the authors noted that EMG
may serve as a useful proxy to motor cortical signal for a BMI (apossible extension to
the OPS which we discuss at length in the Discussion). While connections to BMI design
were discussed, this study primarily investigated the ability of the motor system to learn
a non-intuitive control mapping (connecting to an important “fundamental neuroscience”
aspect of the OPS that we describe in the Discussion). From that study we draw the key
message that aspects of BMI design can be studied without the need for an invasive BMI.
More closely, [25] used a sensored glove to map hand movements to cursor control and
to study how subjects learned in the presence of an adaptive control mapping. They draw
connections to “co-adaptation” for BMI, which has been a question of interest since [124].
These two papers support the notion that a simulation systemlike the OPS can be used to
study non-trivial interface controllers. While these studies focus primarily on neuroscien-
tific aspects of human motor learning, the OPS is distinct by being designed specifically to
investigate neural prosthetic algorithm and system designchoices.
Second, two recent related works have investigated online vs. offline analysis in a spe-
cific BMI algorithm setting. First, [14] compared two BMI algorithms in standard offline
analysis and showed stark performance differences. However, when the same two algo-
rithms were then analyzed under closed-loop control in a monkey experiment, the perfor-
mance differences between these two algorithms became considerably less. One significant
3.2. INTRODUCTION 19
message of that study is that certain types of error (directional biases) are readily learnable
in an online context, and so algorithms should not necessarily be discarded because of bi-
ases that impact offline performance negatively. Their second work [71] then compared
more algorithmic features, such as directional biases (again finding a discrepancy between
offline and online performance implications) and trajectory smoothness (which seemed to
matter in both online and offline contexts). This work is important in supporting the dis-
tinction between offline and online, but the OPS is distinct in that it offers a non-invasive
simulation environment for rapidly testing such algorithmic features in true closed-loop.
One abstract [81] did use humans non-invasively in this offline vs. online context, with a
similar spirit to the OPS. Here subjects used a joystick or EMG to control a virtual arm, and
the authors revealed the negative impact of systems that accentuate low-frequency control
signals, insomuch as such signals slow online feedback-control.
The studies [14, 71] also included simulation of one difference between onlineand
offline control and supported the value of further simulations. However, as the authors
noted, this simulation model did not include online correction and the use of feedback-
control. They rightly noted that such a computer simulationwould be considerably more
involved and heavy with assumptions. Here, we introduce theOPS as a system to accurately
model the human feedback-control system - using an actual human in closed-loop with the
decoding algorithm. Furthermore, the OPS system allows verification of these findings and
extensions to new algorithmic questions without full animal model experiments.
The remainder of the paper is as follows: in Methods, we describe the experimental
hardware and software platform that allows us to test the OPSin humans, the OPS in mon-
keys, and a real neural prosthetic BMI in monkeys (we term these testing scenarios “human
OPS mode,” “monkey OPS mode,” and “monkey BMI mode,” respectively). We describe
two variants to a simple center out reaching task that we had both humans and monkey
perform, and we detail relevant data analysis methods. In Results, data from humans and
monkey demonstrate that the subjects using the OPS paradigmshow significant perfor-
mance differences when using different bin widths of a Kalman filter decode algorithm.
We then compare these online results to offline analysis to make the first major point of
this paper: offline analysis does not provide an accurate picture of online performance. As
20 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
a second major point, we then show using the monkey data that the OPS paradigm accu-
rately reflects these trends, indicating similar algorithmic choices as does real BMI mode.
Finally, in Discussion, we discuss the implications of these results.
3.3 Methods
In this work, we performed human and animal experiments, offline and online analyses, in
OPS and BMI modes, and with two different task variants. To manage the description, we
break up the methods below as follows. First, we describe therelevant experimental hard-
ware that was used for all experiments. Second, we describe the reaching task performed
by both the animals and humans. Third, we describe human-specific protocols. Fourth, we
describe animal-specific protocols (including surgery andneural data acquisition). Fifth, as
the ability to generate sensible synthetic neural data is a key aspect of the OPS, we describe
in detail our methods and choices for neural spiking models.Sixth, we describe the decode
algorithm that was used to generate prosthetic cursor control in both the online and offline
cases. Seventh and finally, we describe the methods used to analyze the performance of
these varied experimental conditions.
3.3.1 Neural Prosthetic Experimental Hardware
We first describe the relevant experimental hardware and system. This experimental rig is
diagrammed schematically in Figure3.2.
Importantly, this hardware was used in all experiments, both by the human and animal
subjects, in both OPS and BMI modes, so we describe it here without distinction to the user.
This choice is intentional to further emphasize our effort to make the OPS a close proxy
to real BMI mode. In the human experiments, the subject sat in achair and placed his/her
head comfortably on a chin rest. In the animal experiments, the subject sat with a fixed
head position in a custom chair. In both cases, the subject’snose was positioned directly
in front of a pair of mirrors (at 45 degree angles from the eye). Each mirror reflected an
image from a pair of LCD monitors on either side of the subject’s head. These monitors
displayed identical images, but with a slight (multi-pixel) offset to create a disparity cue
3.3. METHODS 21
control
PCs
position tracker
stereo monitors
(for 3D effect)
mirrors
human or
monkey
reach
target
decoded
arm reach
3D visual percept
3D real
arm reach
(recording, signal
processing, neural
or synthetic neural
decoding)
Figure 3.2: Experimental rig which can be used both for Online Prosthesis Simulator (OPS)and real neural Brain Machine Interface (BMI) experiments. A human (shown in gray) ormonkey subject reaches (red trace) in a 3D volume obscured from view. An overheadposition tracker tracks endpoint kinematics. Control PCs process those data and render thesubject’s real reach (in control trials) or a prosthetic decoded reach (in OPS or BMI trials,the red trace). Two monitors (blue) project a stereo 3D imageonto mirrors (virtual 3Denvironment shown at right). Also displayed is a the subject’s reach target (green sphere).
22 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
leading to a stereoscopic depth percept, creating the illusion of a 3D environment (a typical
Wheatstone stereo 3D display [133]).
The subject made arm reaches in the large 3D volume behind themirrors. We recorded
kinematic parameters of the reach endpoint using a small reflective bead on the subject’s
finger, which allowed overhead optical position tracking (Polaris, Northern Digital, Water-
loo, Ontario). Hand position was measured at 60Hz to a resolution of 0.35mm RMS. These
data were recorded at 1000Hz by a generic x86 “behavior” computer running an embedded
operating system (xPC, MathWorks, Natick, MA) executing custom software. This com-
puter processed the kinematic data (for example, in OPS mode, to generate synthetic neural
data and decode that data into a decoded reach, specifics described below) or neural data
(in BMI mode, specifics described below), and transmitted a cursor position via ethernet
to a “visualization” x86 computer running a stripped-down Linux operating system. This
machine executed visualization software (MSMS, USC MDDF, Los Angeles, CA) which
rendered images to the Wheatstone display in near real-time (measured latency and jitter
in our system: 7±4ms). This low-latency system critically allowed us to investigate short
timescale prosthetic design questions. For example, in this work we were interested in
optimizing the integration bin width of the Kalman filter. However, if our system (which
includes the end-to-end integration of behavioral control, neural data streaming and record-
ing, and decoding) had latency and jitter in excess of 25ms, 50ms, or more, we could not
have investigated any bin widths lower than that threshold.This carefully engineered sys-
tem was thus an important enabler of this study.
This virtual environment rendered, against a black background, a reach target (4cm
diameter green sphere) and the subject’s decoded hand/cursor position (as a 4cm diameter
gray sphere, though it is shown in Figure3.2as a red sphere for clarity of illustration). In
control reaching trials, we rendered the subject’s true hand position back to the display as
the gray cursor. In closed-loop prosthetic trials (either OPS or BMI mode), we rendered a
decoded prosthetic reach as the same gray cursor. This system, similar in spirit to the virtual
environment in [124], allowed online, closed-loop neural prosthesis trials appropriate for
the work proposed here.
3.3. METHODS 23
3.3.2 Neural Prosthetic Experimental Task
Having described the hardware in which the subjects made reaches, we here describe the
specific structure of the experimental task. To make close comparisons between the offline
and online analyses, and between the OPS and BMI modes, we heldconstant the task
design across all human and monkey subjects.
All subjects made center-out reaches in the virtual environment described above. Each
reach consisted of the subject moving the gray cursor (undersubject control) to the green
reach target. The green reach target alternated between a center point and a pseudo-
randomly chosen target at one of 8 target locations evenly spaced on a ring of radius 8cm.
Only one target appeared per trial. In all data, only the center-out reaches (those from the
center point to the periphery) were analyzed.
The specific trial timeline proceeded as follows: the trial began when the green reach
target (either center point or one of the peripheral targets) appeared on the screen. As this
was a direct-reach paradigm, the target appearance was the subject’s “go” cue, and after
a short reaction time, the subject moved the gray cursor to the green target. The reach
was successful if the gray cursor was held within a demand box4cm wide around the
reach target. The cursor had to remain within the acceptancewindow for a hold period of
500ms, after which a reward was given (a tone for human subjects, and a tone and drop
of juice for the monkey). If that success criterion was not satisfied within 3000ms, the
trial was considered a failure and timed out. After either a trial success or failure, an inter-
trial interval of 40ms was imposed before the beginning of a new trial. The cursor was
controlled by the subject using one of three modes: real reaching, OPS prosthetic reaching,
and BMI prosthetic reaching, as described in the next paragraph. During all of these control
modes, the subject (monkey and human) was allowed free movement of his/her limb (as has
been done previously in BMI literature, for example [12, 109, 124]). Because the subject
only saw the cursor (the real arm was hidden from view), the subject remained motivated to
complete the task by the reward structure, which rewarded successful control of the cursor.
Though there was a sensory discrepancy between the visual feedback and the real arm’s
proprioception (in prosthetic trials), previous motor neuroscience work [97] and previous
BMI literature [12, 109, 124] suggest that this confound is minor and does not seriously
24 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
affect task learning. Whether the true arm is restrained or allowed to reach, there is still a
proprioceptive confound, so this potential limitation is not specifically of the OPS or of this
study, but rather of all able-bodied animal or human BMI studies. We discuss this aspect
of the experimental design in more depth in the Discussion.
There are three modes by which the subject controlled the gray cursor: real reaching,
OPS prosthetic reaching, and BMI prosthetic reaching. In real reaching, the gray cursor re-
flected the kinematics of the true underlying arm reach. Thisdata is that which is typically
collected for offline prosthetic decode analysis, and it wasused for training the parameters
of the prosthetic decode algorithms. Though other trainingparadigms have been used [57],
we chose this simple training method to parallel existing offline and online literature. In
BMI prosthetic reaching (monkey only), neural activity was recorded from the motor cor-
tex, and that data was decoded into kinematics which controlled the gray cursor. In OPS
prosthetic reaching mode (monkey and human subjects), synthetic neural activity was gen-
erated from the kinematics of the subject’s arm, and that neural activity was then decoded
with a decoding algorithm into control commands for the graycursor. The specifics of
recording, generating synthetic activity, and decoding are detailed below.
The parallel between OPS and BMI control modes is detailed in Figure 3.3. In both
the left and right panels of Figure3.3, the subjects make (a) a real reach (red trace), and
then (b) corresponding neural activity is recorded (synthetic neural activity generated from
the real reach in OPS mode - left panel, and real neural activity corresponding to the real
reach in BMI mode - right panel). That data is then input to the (c) decode algorithm,
which decodes the (possibly synthetic) neural activity into kinematics to control the gray
cursor, which is (d) rendered back to the subject. The decoded reach trajectory will differ
from the subject’s intended arm reach, due to noise (from neural spiking) and algorithmic
model mismatch (e.g. the Kalman Filter only approximately models the dependencyof
the neural activity on kinematics). The subject modifies his/her behavioral strategy so that
the prosthetic movement will mimic, as closely as possible,the desired reach trajectory.
Thus, both OPS and BMI modes offer a means to study the relevance of feedback-control
in neural prosthetic system use.
We also tested two variants of the center-out prosthetic reaching task. In the first - the
“continuous” task variant, prosthetic reaches were made both for center-out reaches to the
3.3. METHODS 25
(a) True physical
behavior
(b) Synthetic neural activity
(generated based on
true physical behavior)
(d) Decoded behavior (rendered
in 3D visual display)
Decode Algorithm
(c)
OPS Mode (Human or Monkey) BMI Mode (Monkey Only)
(a) True physical
behavior
(b) Real neural activity
(d) Decoded behavior (rendered
in 3D visual display)
Decode Algorithm
(c)
Figure 3.3: OPS and BMI schematic.Left Panel: OPS Mode - (a) A healthy humansubject or a monkey (human shown) makes real arm reaches in a 3D reaching environment(see Figure3.2). (b) Recorded kinematics of that real reach are used to generate syntheticneural activity, which is then input to the (c) Kalman filter neural decode algorithm. Thedecode algorithm decodes synthetic neural activity into physical reaching behavior, and (d)the decoded reach is then rendered back to the subject in the 3D visual environment, whichallows the subject to bring to bear all of his/her online, closed-loop control strategies todrive a desired reach.Right Panel: BMI Mode - (a) A monkey makes real arm reaches.(b) The real neural activity associated with that reach is recorded, which is then input to the(c) Kalman filter decode algorithm. The decode algorithm decodes real neural activity intophysical reaching behavior, and (d) the decoded reach is then rendered back to the monkeyin the 3D visual environment, which allows similar closed-loop control as in OPS mode. Itis important to note that the OPS mode of the left panel can be used both by a human anda monkey (this will provide an important validation of the OPS).
26 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
peripheral targets and for the reaches returning to the center target. We collected full data
sets from this task in the human OPS, and there was a mildly frustrating “drift” effect that
could occur, where, over many reach trials, the cursor wouldbecome increasingly offset
from the true arm reach. Though this never created a problem for the data (performance
was robust to this effect), it was reported by some subjects to be frustrating. As such, we
developed a second task variant. In the “interleaved” task variant, the reaches returning
to the center target were controlled by the real arm kinematics. This interleaving removed
the potentially frustrating drift effect. The center-out reaches (which are the only reaches
analyzed) remained entirely under prosthetic control. Though in eventual neural prosthesis
system use the “return to center” may be controlled by the system itself, this interleaved
variant was a simple way to keep the subject engaged in the task and to avoid confusion.
We ran human subjects in both OPS continuous and OPS interleaved. We ran the monkey
on BMI continuous and BMI interleaved, which presented no difficulty, since the mon-
key had been highly trained using real neural activity in exactly this continuous paradigm
with this BMI controller. For the monkey in OPS mode, we only ran OPS interleaved, as
the moderately frustrating effects reported by human users(with OPS continuous) would
require more extensive training with the monkey, and that variant is of unclear additional
merit on top of the interleaved task.
For each of these variants, we ran 5 full experiments. Here wefirst describe the full
experiment structure and how it allowed fitting and testing of numerous decode models;
the particulars of task training are left to the following ”Human Experiments” and ”Ani-
mal Experiments” subsections. A full experiment is a block structure designed as follows.
First, the subject (human or monkey) performed one block of 200 trials in real reach mode.
These 200 real reaches and the corresponding neural recordings are used to train the de-
coding models (Kalman Filters with different temporal bin widths). The subject was given
a roughly two minute break, during which online prosthetic control mode (either OPS or
BMI) was switched on. The subject then performed 7 blocks of 100 prosthetic reaches
each, where there was a short 15 second break between blocks.These 7 blocks of 100
reaches were then repeated again in a different order. Thus,a “full experiment” included
roughly 1600 reaches - 200 real reaches followed by two runs of 7x100 reaches. Typically
this took subjects about an hour. This experimental structure allowed an algorithm choice
3.3. METHODS 27
or algorithmic parameter choice to be tried at several different values (here 7, but this could
be changed). As discussed, here we are interested in optimizing performance based on the
temporal bin width of the Kalman Filter decode algorithm. Accordingly, we chose 7 bin
widths - 25, 50, 100, 150, 200, 250, and 300ms - and we had the subjects do two blocks
with each bin width (one block of 100 reaches per 7-block run). We randomized the order
of the blocks (within the 7-block run) to control for any learning effects. Each of the 9
human subjects ran one full experiment (either OPS continuous or OPS interleaved, save
for subject PN, who ran each on different days), and thus we have 5 full experiments each
of human OPS continuous and human OPS interleaved. The monkey ran all fifteen full
experiments (5 each of monkey BMI continuous, monkey BMI interleaved, and monkey
OPS interleaved). From the real reaches that comprise the first block of the experiment, we
are able to run all offline decoders, so the offline analyses were built from each subject’s
real reaching blocks.
This general task structure allows us to study online vs. offline analyses, BMI vs. OPS
mode, continuous vs. interleaved, in humans vs. monkey. To clarify, the results section
will show ten different types of prosthetic decode analysis: offline human OPS continu-
ous, offline human OPS interleaved, offline monkey OPS interleaved, offline monkey BMI
continuous, offline monkey BMI interleaved, online human OPScontinuous, online human
OPS interleaved, online monkey OPS interleaved, online monkey BMI continuous, and on-
line monkey BMI interleaved. These data sets are categorizedexplicitly in Table3.1. These
permutations allow us to perform a breadth of comparisons and controls to answer the key
questions of this study. In the following two sections, we describe particular protocols used
for the monkey and human subjects only (not shared across monkeys and humans as the
above task and hardware details are).
3.3.3 Human Experiments
Human protocols were approved by the Stanford University Institutional Review Board.
Nine healthy adult human subjects performed the reaching tasks in both real reach and OPS
modes as described above. These subjects reached in the experimental apparatus previously
described. Five subjects performed a full experiment (200 real reaches plus 2 blocks of 100
28 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
Human Subjects Monkey Subjects
Online analysis with
synthetic neural data
Offline analysis with
synthetic neural data
Online analysis with
real neural data
Offline analysis with
real neural data
Online Human OPS Continuous
Online Human OPS Interleaved
(Figures 3.4, 3.6)
Online Monkey OPS Interleaved
(Figures 3.5, 3.7)
Offline Human OPS Continuous
Offline Human OPS Interleaved
(Figure 3.6)
Offline Monkey OPS Interleaved
(Figure 3.7)
No Data
(requires human clinical trial)
No Data
(requires human clinical trial)
Online Monkey BMI Continuous
Online Monkey BMI Interleaved
(Figures 3.5, 3.7)
Offline Monkey BMI Continuous
Offline Monkey BMI Interleaved
(Figure 3.7)
Table 3.1: Categorization of the data sets collected and analyzed in this paper. The Resultsshow offline and offline analyses (blue and red in this table and in subsequent figures),human and monkey subjects (left and right columns in this table), synthetic and real neuraldata (identical to OPS and BMI modes, the top two and bottom tworows here), and finallythe continuous and interleaved task variants (grouped within each cell of this table). Thesepermutations allow us to perform a breadth of comparisons and controls to answer the keyquestions of this study.
3.3. METHODS 29
reaches at each of 7 bin-widths, as described above) of the OPS continuous task, and five
subjects (four new subjects, and one repeated from the OPS continuous) performed a full
experiment of the OPS interleaved task.
Across all experiments, in OPS and real reach modes, human subjects required very
little training (only a few trials, which are not included inthe results and analysis) to un-
derstand the task and control the cursor well. During real reaching trials (200 reaches),
which are used both for training the decode algorithm and foroffline analysis, we asked
the subjects to reach to the targets at a comfortable, normalspeed. Since the virtual path
of the cursor matches perfectly the true kinematics of the arm, the subjects had no trouble
performing these reaches with 100% accuracy. We then pausedthe task and informed sub-
jects that they were being switched into prosthesis mode forseveral blocks of 100 reaches,
where the virtual reach would not perfectly match their trueunderlying reach. We asked the
subjects to try to maintain the same reach speed and accuracyas in their previous reaches,
and we enforced the time-to-target timeout if subjects tooktoo long to reach (3000ms, as
previously noted). Depending on the bin-width being used, subjects were more or less
able to acquire the targets successfully and quickly (theseperformance differences are the
results of the OPS).
These human experiments provide considerable data for testing the difference between
offline and online (OPS) analyses in terms of the subject’s ability to exploit feedback-
control to improve prosthetic reaching performance.
3.3.4 Animal Experiments
Animal protocols were approved by the Stanford University Institutional Animal Care and
Use Committee. We trained one adult male monkey (Macaca mulatta, monkey L) to per-
form center-out reaches as already described. Unlike humans, monkeys were motivated to
complete the reaches with a juice reward. During experiments, the monkey sat in a custom
chair (Crist Instruments, Hagerstown, MD) with the head braced. Hand position recording
and experimental control were as previously described.
To enable BMI trials, a 96-channel silicon electrode array (Blackrock Microsystems,
Salt Lake City, UT) was implanted straddling dorsal pre-motor (PMd) and motor (M1)
30 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
cortex (right hemisphere), as estimated visually from local landmarks, contralateral to the
reaching arm. Surgical procedures have been described previously [21,54,102]. Neural sig-
nals were monitored on each channel and high-pass filtered. Athreshold level of -4.5 times
RMS voltage was established for each channel, and all threshold crossings were recorded
as “spike times.” Spike sorting was not performed, as our previous experiments indicated
that doing so did not significantly alter performance to justify the additional computational
load [15] (see also [36,55,103]).
The monkey was trained over several months on the reaching task and BMI continu-
ous task variant [47]. For the interleaved task variant, in both the OPS and BMI modes,
the monkey quickly learned this task in a few days of training, as it was only a mi-
nor extension of his well trained continuous BMI behavior. Multiple data sets of each
task were collected. For the continuous BMI task, the monkey completed five full ex-
perimental sessions (wherein each 200-trial block with an individual bin width was pre-
sented twice in pseudo-random order, as described above) across three days, and those
sessions are denoted asL20091215.C3.1, L20091217.C3.1, L20091217.C3.2, L20091217.C3.3,
and L20091218.C3.1. For the interleaved BMI task, the monkey completed all five ex-
perimental sessions across two days, and those sessions aredenoted asL20091218.IRR3.1,
L20091220.IRR3.1, L20091220.IRR3.2, L20091220.IRR3.3, andL20091220.IRR3.4. For the inter-
leaved OPS task variant, the monkey completed all five full experimental sessions in a sin-
gle day, and those sessions are denoted asL20091221.IRR2.1, L20091221.IRR2.2, L20091221.IRR2.3,
L20091221.IRR2.4, andL20091221.IRR2.5. To preserve trials, real reach trials were only run
once per day for the monkey, not in each experiment.
These animal experiments provide considerable data for again testing the difference
between offline and online analyses (whether comparing to BMIonline or OPS online).
Further, this data allows us to test the extent to which the OPS paradigm accurately reflects
the trends of real BMI mode.
3.3.5 Generating Synthetic Neural Activity
There are many behavioral features correlated with motor cortical spiking activity, includ-
ing features of the arm (hand position and velocity, muscle activity, forces, joint angles,
3.3. METHODS 31
etc.), features of the reach (smoothness, etc.), and features of the eye (visual reference
frames, etc.); see [126] for extensive references and discussion. In this work, we recorded
endpoint kinematics of the subject’s arm, and we used these data to produce reasonable
simulated neural spike trains which were then decoded into kinematics for controlling the
prosthetic cursor.
Certainly generating synthetic neural data requires some uncomfortable assumptions
about neural tuning and spiking. While synthetic neural activity has been used in the past
in prosthesis studies [6,67,119–121,131], we are particularly interested here in creating a
simulation system that translates well into a real neural system. Much care is needed both
in construction and in interpretation (discussed more fully in the Discussion) to ensure the
legitimacy of the results. Accordingly, our approach was topick as simple a construction
as possible that resulted in decoded reaches (both online and offline) that were qualitatively
similar to our core experience in real neural prosthetic experiments [22,23,67,102,141].
To begin, we used cosine-tuning models [42, 85] to map kinematics to neural firing
rates. Under this tuning model, each neuron was defined by a tuning vector, a minimum
firing rate, and a maximum firing rate (these last two are equivalent to setting a mean rate
and a depth of modulation). First, we chose a population sizeof 96 neurons (equivalent
to the 96 electrode channels of real neural data that we recorded with the array). For each
of these synthetic neurons, we sampled tuning vectors uniformly from the unit 3D sphere,
for both position and velocity (so these tuning vectors corresponded to “preferred position”
and “preferred velocity” vectors). We sampled minimum firing rates uniformly between
0 and 20 spikes per second, and we sampled maximum firing ratesuniformly between
that neuron’s minimum rate and 100 spikes per second, in linewith motor cortical neuron
behavior we have previously observed [19,20,22]. These firing rates were then put through
a spiking process, which we chose to be a simple Poisson process with rate equal to the
firing rate in each time bint. This is also a standard choice in literature [26,107].
Mathematically, we say that neuronk had tuning vectorc(k), maximum firing rateλ(k)max,
and minimum firing rateλ(k)min. Then, we definedxt to be the kinematic parameters of the
arm in time bint, wherext ∈ R6, a vector of 3D position and velocity of the hand.xt was
suitably scaled such that the range of kinematics exhibitedby the subject produced firing
rates within the range of minimum and maximum firing rates. Wedefinedλ(k)t ∈ R
K and
32 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
yt ∈NK+ to be the neural firing rates and spiking activities at time bin t (each elementλ(k)
t of
λt andy(k)t of yt corresponding to the firing rate and number of spikes of each of theK = 96
synthetic neurons being generated). With these definitions, we write:
λ(k)t = (λ(k)
max−λ(k)min)c
(k) ·xt +λ(k)min (3.1)
yt | λt ∼ Poisson(h(λt)) (3.2)
wherec(k) ·xt represents the inner product between these two vectors, andthe functionh(λt)
clips the rateλt to 0 if the argument is negative and otherwise equals the argumentλt (as is
commonly done in literature [14], though given our parameter choices, this clipping rarely
occurred). In the first generation of the OPS system, we created this tuning to both 3D po-
sition and 3D velocity kinematics, as has been reported considerably in literature [42,85].
While this produced reasonable decoded behavior offline, online use was not qualitatively
similar to real prosthetic use. With strong position dependence, users immediately devel-
oped a strategy of moving the hand to the virtual location of the target, and then holding
that position. Because there was position tuning information, eventually the cursor would
acquire the target for a full hold period. This behavior was qualitatively very different from
the observed behavior in monkey BMI experiments, where subjects make several online
adjustments to the real reach and are rarely able to successfully acquire a target by holding
a hand position for a period of time. To correct this qualitatively inappropriate effect, we
removed the position dependence in the simulated neurons. Instead, firing rate was only
tuned to velocity (soxt and correspondingc(k) are now only three dimensional, including
3D velocity terms). With this model, the qualitative behavior of OPS users was highly
similar to that of a monkey using a real BMI. After substantialtesting to ensure that other
unrealistic strategies were not developed, we determined that this simple model was ade-
quate for testing the decode implications of the Kalman Filter bin width. We note that other
possibilities for noise generation could include adding noise sources to either the firing rate
or to the spike trains themselves (to model recording noise sources, for example), but we
leave those enhancements to future work.
Here we note two important facts about this choice of spikingmodel. First, this spiking
3.3. METHODS 33
model does not match the decode algorithm, in the sense that the Kalman Filter is not an
optimal decoder of neural activity generated in this way. Indeed, the Kalman Filter is also
not an optimal decoder of real neural activity, as the Kalmanfilter does not exactly model
the real neural system. Second, it is important to note that we do not claim that this is what
the neural system is actually doing, but instead it is a choice that allows us to study the
OPS framework and the relevance of feedback-control to the design of neural prosthetic
systems.
Finally, this method for generating synthetic neural activity can be extended in a number
of ways previously seen in literature (e.g., [119–121]), but the simple velocity-tuned model
was so qualitatively and quantitatively similar to real BMI mode that we were satisfied with
this straightforward choice.
3.3.6 Decoding Neural Activity
We decoded neural activity using the Kalman filter, as has been used for neural prosthetic
algorithms previously [69,137–139]. It is important to note that this algorithm can be used
both in online and offline contexts. In the online case, at each time point, the decoded
kinematics controlled the cursor, which the subject could see, use as feedback, and react to.
In the offline case, the neural (or synthetic neural) activity was collected during real reaches
(when the cursor matches the arm kinematics) and later decoded offline. By using identical
decode algorithms (and parameter settings, etc.), we can effectively study the difference
between online and offline control.
The Kalman filter (introduced in [65]) stipulates a linear dynamical system for arm
movements (often called the prior or the trajectory model),which says that kinematics at
timet should look something like kinematics at timet−1, i.e., smoothness. The model also
stipulates a linear observation model for neural activity,which says that observed neural
spiking is a noisy linear transformation of intended kinematics. As before, we definext to
be the kinematic parameters of the arm in time bint, wherext ∈R7, a vector of 3D position
and velocity of the hand (plus a bias/offset term). We also define yt ∈ RK to be the neural
activity at time bint (each element ofyt corresponding to the activity of each of theK = 96
neurons being recorded). The Kalman filter then assumes the following linear dynamical
34 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
system:
xt = Axt−1+wt (3.3)
yt = Cxt +qt (3.4)
whereA ∈ Rp×p andC ∈ R
K×p represent the state and observation matrices, andwt and
qt are additive, independent Gaussian noise (denotedwt ∼ N (0,W) andqt ∼ N (0,Q)).
Such a model is standard for Kalman filtering, and it allows very fast inference (decoding)
of kinematics from neural activity. Furthermore, the parameters{A,C,W,Q} can also be
quickly and exactly learned from training data (the standard details of which are left to the
references [65,69,137–139]).
Intuitively, the Kalman filter starts from kinematics at thebeginning of the trial and
proceeds iteratively through time, updating its estimatesof arm state and error covariance
at every time stept. These steps are entirely based on mathematical propertiesof the
Gaussian, and the algorithm is fast and stable. Again, this is precisely the context that has
been used for online and offline neural prosthetic algorithms previously [69,137–139].
One important feature to note within the above description is the time bint. Since the
above dynamical system is discrete-time, we must choose a window of time over which to
integrate neural activity for our prosthetic decode. This time, which we call the “decode
integration bin width” or just “bin width,” is a key parameter that can be chosen by the
experimenters. As noted in the introduction, previously published offline analysis has sug-
gested that a 200-300ms bin width is optimal [139], but this choice has not been validated
online. We will see that different settings of this parameter can have a large performance
effect both qualitatively and quantitatively. Thus, optimizing performance with respect to
this parameter is an important research question and a valuable test for the OPS, which we
present here.
3.3.7 Performance Analysis
Having decoded the neural activity as described in the section above, we must now in-
troduce metrics to quantify the performance of a particularprosthetic mode (task variant,
subject, bin width choice, etc.). Error metrics are numerous and a subject of study in their
3.3. METHODS 35
own right [29], but here we chose a few sensible metrics that have been seenpreviously in
literature. For consistency, we present all metrics aserror and notperformancemetrics;
i.e. lower is always better throughout the results.
First, for online analysis, failure-rate is an obvious and previously used choice (e.g.,
[102]). Failure-rate measures the fraction of trials in which the subject was unable to ac-
quire and hold the reach target within the allotted time. Second, time-to-target, as used
in [57] measures the amount of time the subject required to reach the target (for the last
time, since the subject could pass through the target a number of times before holding the
cursor at the target for the required hold period).
While these straightforward performance metrics offer a quantitative view of online per-
formance, they can not be meaningfully used for offline decode analysis. Offline decoded
reaches rarely reached the target and never successfully completed a trial (this makes sense
with a noisy decoder in the absence of feedback, which is yet another suggestion that online
analysis should perhaps be prioritized), so failure-rate is a vacuous metric in comparing al-
gorithms or algorithmic choices offline. Time-to-target issimilarly broken, and thus can not
be used for offline analyses. Standard metrics for offline analysis include mean-squared-
error (MSE) or similar, and correlation coefficient, and these have been used extensively
(see description in [22]). With these metrics, one compares the true, natural reachto the of-
fline decode of that same reach. Unfortunately, these sensible offline metrics are not useful
online, as there is no correct “true reach” that can be compared.
To meaningfully compare offline analysis with online analysis (the first major point of
this work), we require a metric that is suitable both in the offline and the online context.
As the subject was always motivated to move closer to the target during a reach (whether
in OPS, BMI, or real reaching mode), we can consider the distance to the target. We
define mean-integrated-distance-to-target as the averagedistance to the target during the
time course of the subject’s reach. On triali, if we call the target positionp(i)targ and the
position at timet during the trialp(i)t , the errorE(i) for trial i can be written mathematically
as:
E(i) =1
T(i)
T(i)
∑t=1
‖p(i)t − p(i)targ‖ (3.5)
36 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
whereT(i) is the length of triali. Using the mean distance instead of total distance is sensi-
ble because offline and online trials have characteristically different temporal lengths (since
offline trials are based on real reaches, these are generallyquicker than online, noisy pros-
thetic trials). It is important to note that, when comparingoffline and online performance
(and BMI vs. OPS mode performance), less will be made of absolute differences in perfor-
mance between offline and online (and BMI vs. OPS modes); rather, we are mostly inter-
ested in the optimal parameter settings that are suggested by different prosthetic modes.
Mean-integrated-distance-to-target is by no means the only metric possible for com-
paring offline and online trials, so we here give context for why we believe this metric to
be appropriate. First, since the actual task rewards movement of the cursor to the target
both in offline and online settings, having a metric that considers the target is sensible. One
might reasonably consider other metrics that compare the prosthetic reach to some true arm
kinematics such as an “ideal” real arm reach (perhaps gleaned from real reaching trials).
However, the subject is not rewarded for having a particularkinematic profile - the reward
is for task success,i.e. acquiring the target. Quantifying prosthetic reach error based on a
quantity that is unknown to the subject and only somewhat related to task success does not
seem appropriate. Put another way, a subject using a BMI mightdevelop a different suc-
cess strategy (changing his/her kinematics, for example),and “ideal reach” metrics could
arbitrarily penalize such a strategy. On the other hand, thedistance to target is sensible
because, by task design, we know that the subject is always trying to reduce this distance,
and thus motivation and performance are aligned.
Finally, we note that each of these metrics was calculated ona per trial basis, and thus
we could calculate average statistics and confidence intervals. Throughout the results we
use 95% confidence intervals, calculated via the binomial distribution for failure-rate anal-
yses and Gaussian distributions for time-to-target and mean-integrated-distance-to-target
analyses [142]. These performance analyses give us all the quantificationnecessary to
analyze the results gathered via previous methods.
3.4. RESULTS 37
3.4 Results
The results are composed of three pieces, which we describe here in brief to help the reader
navigate the section. We begin by showing the raw performance data across subjects. These
data will demonstrate that all subjects using the OPS and BMI modes show significant per-
formance differences when using different bin widths of a Kalman filter decode algorithm.
In other words, the raw data shows that the bin width does havemeaningful performance
implications and further that this prosthetic setup is a meaningful way to investigate those
implications. Next, we make the first major point of this study: offline analysis does not
provide an accurate picture of online performance. The datashow significant differences
in the shape of these performance curves based on offline or online (BMI/OPS) modes,
thereby suggesting very different algorithmic parameter choices. Finally, we make the sec-
ond point of this paper: the monkey BMI data, the human OPS data, and the monkey OPS
data all show similar performance trends. This importantlyindicates that, at least when it
comes to this one critical algorithm parameter, the OPS framework can stand as a reason-
able proxy to real BMI mode.
3.4.1 Online Prosthetic Data
We begin by showing the performance for both humans and monkey in the online prosthetic
modes, both BMI and OPS. In Figure3.4, we show the error for the human subjects; the
left column shows the OPS continuous task variant, and the right column shows the human
OPS interleaved variant. Within a given column, each panel shows the same data analyzed
with a different error metric: panel A - failure rate, panel B- time to target, and panel C -
mean integrated distance to target. Each of these metrics isdescribed above in Methods. As
a reminder, this third metric - mean integrated distance to target - is required to compare
offline and online analyses. In each panel, each faint line denotes the average error of a
particular human subject over the course of that subject’s full experiment. For example, in
the right column, panel A, the light green trace shows the average failure rate of subject
JH at each of the bin-widths. We see that, at a bin width of 200ms (subject JH did 200
reaches at this bin width, per the block structure protocol described above), JH failed to
acquire and hold the target on roughly 60% of trials. Since weare particularly interested
38 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
Human OPS Continuous
0 50 100 150 200 250 3000
20
40
60
80
100
Fa
ilure
Ra
te (
%)
Decode Integration Bin Width (ms)
JC20091213.C2
VG20091213.C2
MF20100318.C2
SK20100319.C2
PN20100321.C2
All Blocks
0 50 100 150 200 250 3001000
1200
1400
1600
1800
2000
Tim
e T
o T
arg
et (m
s)
Decode Integration Bin Width (ms)
A
0 50 100 150 200 250 30035
40
45
50
55
60
65
70
Me
an
In
teg
rate
d D
ista
nce
To
Ta
rge
t (m
m)
Decode Integration Bin Width (ms)
B
C
0 50 100 150 200 250 3000
20
40
60
80
100
Fa
ilure
Ra
te (
%)
Decode Integration Bin Width (ms)
PN20100321.IRR2
CC20100322.IRR2
JH20100322.IRR2
KS20100322.IRR2
RT20100323.IRR2
All Blocks
0 50 100 150 200 250 30035
40
45
50
55
60
65
70
Me
an
In
teg
rate
d D
ista
nce
To
Ta
rge
t (m
m)
Decode Integration Bin Width (ms)
Human OPS Interleaved
0 50 100 150 200 250 3001000
1200
1400
1600
1800
2000
Tim
e T
o T
arg
et (m
s)
Decode Integration Bin Width (ms)
A
B
C
Figure 3.4: Performance metrics for Human Online Prosthetic Simulator (Human OPS)decode trials.Left Panel: Continuous Task variant - (a) Failure Rate: the percentage oftrials where the subject did not successfully acquire and hold the reach target. (b) Time toTarget: the time required to reach the target for successfultrials. In both panels, data fromfive subjects are shown in light colors. The average of all trials is shown in dark blue. Thisaverage shows in both cases a significant linear trend indicating that smaller bin widths willlead to better performance. (c) To compare offline and onlinedata, we use a third metric:integrated distance to target (failure rate and time to target can not be calculated for offlinedata, as offline failure rate approaches 100% without onlinefeedback, which is again tellingof the inappropriateness of offline analysis). This metric is normalized by trial length (totaltrial time). Right Panel: Interleaved Task variant - These show the same metrics as theleft panel, reinforcing with another task variant these trends. In all panels, 95% confidenceintervals (A: binomial distribution, B, C: Gaussian) are shown as error bars.
3.4. RESULTS 39
in average error at a bin width (and less interested in inter-subject differences other than to
confirm that the trend is the same in all subjects), in dark blue we plot the average across
all subjects, along with 95% confidence intervals (as described in Methods). This color
scheme is consistent across all panels in Figure3.4.
First and foremost, it is immediately apparent across all panels that, in the human OPS,
shorter integration bin widths lead to lower error in the online setting. The raw data gives
confidence that there is meaningful performance differences across this algorithm parame-
ter. By performing a linear regression to the data in each panel (binomial noise model for
Failure Rates in panels A; gaussian noise model for other error metrics in panels B and C),
we can calculate the confidence level that each panel is indeed a positive sloping line (i.e.
shorter is better). For the human OPS continuous, those values are - panel A:p < 10−4;
panel B:p< 10−4; panel C:p< 10−4. For the human OPS interleaved, those values are -
panel A:p< 10−4; panel B:p< 10−4; panel C:p< 10−4. Thus, with high confidence we
can say that the human OPS indicates that shorter integration bin widths will lead to higher
online performance.
There are a few salient features to point out in this data. First, it is encouraging that all
error metrics suggest a similar trend in the data, giving us confidence that we are looking
at a real trend and not an artifact of the summary metric. Thisfact will also become useful
when we compare offline to online OPS, which we can only do withthe metric of panel C
(since failure rate and time to target are not meaningful in offline analysis, as previously
described). Second, it is encouraging that the data is rather insensitive to the human OPS
continuous or OPS interleaved task variants (left and rightcolumns). Recall that the in-
terleaved variant was introduced to prevent user frustration. The data’s robustness to that
frustration indicates that it was not critical to subject performance and did not invalidate
the data. Third, it is encouraging to see that subjects performed rather similarly compared
to one another. If there were substantial inter-subject differences, more investigation might
be required. However, in both OPS continuous and OPS interleaved, we had a few experi-
enced subjects (JC, PN, and VG, who had run several thousand trials in this task on days
prior to the data collection) and several naive subjects (MF, SK, CC, JH, KS, and RT, who
had not run the task before their full experiment). As no difference between these groups is
apparent, we are also given confidence that there are not significant learning/training effects
40 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
that need to be controlled in this data. Thus, in summary, Figure 3.4 tells us that shorter
bin widths will lead to lower error rates, and further that this is a highly robust effect in the
human OPS.
In Figure 3.5 we repeat the same presentation as in Figure3.4, but for the monkey
subject instead of the human subjects. As discussed in Methods, here there is only one
subject who performed many full experiments (as opposed to the human subjects, who
each did one full experiment). Accordingly, Figure3.5shows faint lines corresponding to
individual full experiments, and the dark blue correspondsto the average error across all
experiments. Note also that there are now three columns, including real neural prosthetic
mode - monkey BMI continuous and monkey BMI interleaved, and also monkey OPS
interleaved (these choices discussed in Methods). As in thehuman data, the findings are
largely the same. In online prosthetic mode, across two taskvariants, BMI/OPS mode, and
three error metrics, this data indicate that shorter integration bin widths will lead to higher
prosthetic performance. As previously noted in the Methods, we note that less will be made
of absolute differences in performance between offline vs. online and OPS vs. BMI; rather,
we are mostly interested in the optimal parameter settings that are suggested by different
prosthetic modes.
By performing a linear regression to the data in each panel (binomial noise model for
Failure Rates in panels A; gaussian noise model for other error metrics in panels B and C),
we can calculate the confidence level that each panel is indeed a positive sloping line (i.e.
shorter is better). For the monkey BMI continuous, those values are - panel A:p< 10−4;
panel B:p= 0.07; panel C:p< 10−4. For the monkey BMI interleaved, those values are -
panel A:p< 0.01; panel B:p< 10−3; panel C:p< 10−4. For the monkey OPS interleaved,
those values are - panel A:p< 10−4; panel B:p< 10−4; panel C:p< 10−4. Thus, with
high confidence we can say that the monkey BMI and OPS - like the human OPS - indicate
that shorter integration bin widths will lead to higher online performance.
3.4. RESULTS 41
0 50 100 150 200 250 3000
20
40
60
80
100
Fa
ilure
Ra
te (
%)
Decode Integration Bin Width (ms)
L20091215.C3.1
L20091217.C3.1
L20091217.C3.2
L20091217.C3.3
L20091218.C3.1
All Blocks
0 50 100 150 200 250 3000
20
40
60
80
100
Fa
ilure
Ra
te (
%)
Decode Integration Bin Width (ms)
L20091218.IRR3.1
L20091220.IRR3.1
L20091220.IRR3.2
L20091220.IRR3.3
L20091220.IRR3.4
All Blocks
0 50 100 150 200 250 3000
20
40
60
80
100
Fa
ilure
Ra
te (
%)
Decode Integration Bin Width (ms)
L20091221.IRR2.1
L20091221.IRR2.2
L20091221.IRR2.3
L20091221.IRR2.4
L20091221.IRR2.5
All Blocks
0 50 100 150 200 250 3001500
2000
2500
3000
3500
Tim
e T
o T
arg
et (m
s)
Decode Integration Bin Width (ms)0 50 100 150 200 250 300
1500
2000
2500
3000
3500T
ime
To
Ta
rge
t (m
s)
Decode Integration Bin Width (ms)
Monkey BMI Interleaved
A
0 50 100 150 200 250 300
40
50
60
70
80
90
100
Me
an
In
teg
rate
d D
ista
nce
To
Ta
rge
t (m
m)
Decode Integration Bin Width (ms)
B
C
Monkey OPS Interleaved
A
0 50 100 150 200 250 300
40
50
60
70
80
90
100
Me
an
In
teg
rate
d D
ista
nce
To
Ta
rge
t (m
m)
Decode Integration Bin Width (ms)
B
C
0 50 100 150 200 250 300
40
50
60
70
80
90
100
Me
an
In
teg
rate
d D
ista
nce
To
Ta
rge
t (m
m)
Decode Integration Bin Width (ms)
0 50 100 150 200 250 3001500
2000
2500
3000
3500
Tim
e T
o T
arg
et (m
s)
Decode Integration Bin Width (ms)
Monkey BMI Continuous
A
B
C
Figure 3.5: Performance metrics for Monkey Brain Machine Interface (Monkey BMI) andOnline Prosthetic Simulator (Human OPS) decode trials.Left Panel: BMI ContinuousTask variant - (a) Failure Rate: the percentage of trials where the subjectdid not suc-cessfully acquire and hold the reach target. (b) Time to Target: the time required to reachthe target for successful trials. In both panels, data from five subjects are shown in lightcolors. The average of all trials is shown in dark blue. This average shows in both cases asignificant linear trend indicating that smaller bin widthswill lead to better performance.(c) To compare offline and online data, we use a third metric: integrated distance to target(failure rate and time to target can not be calculated for offline data, as offline failure rateapproaches 100% without online feedback, which is again telling of the inappropriatenessof offline analysis). This metric is normalized by trial length (total trial time). MiddlePanel: BMI Interleaved Task variant - These show the same metrics as the left panel,reinforcing with another task variant these trends.Right Panel: OPS Interleaved Taskvariant - The monkey can also run the OPS task with synthetic neural data. This panelshows the same metrics as other panels, and these metrics allshow similar trends. By com-paring BMI and OPS within subject on the same task (middle and right panels), we have anindication that the OPS is providing a valuable proxy to realneural prosthetic systems. Inall panels, 95% confidence intervals (A: binomial distribution, B, C: Gaussian) are shownas error bars.
42 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
3.4.2 Comparing Online Analysis to Offline Analysis
With the several different variants of online prosthetic tasks (OPS vs. BMI, monkey vs.
human, continuous vs. interleaved), and with the real reachtrials that each subject per-
formed as part of the experiment, we now demonstrate the difference between offline and
online analysis of neural prosthetic systems. We noted in the previous section that the on-
line data indicate strongly that shorter bin widths will lead to lower error. As a reminder,
each panel C of Figures3.4 and3.5 shows an error metric that can be calculated for both
offline and online data. Accordingly, we took the real reaches from each subject during
these experiments, and we ran offline analysis as has been previously seen in much litera-
ture [3,6,7,11,17,30,41,54,68,77,87,95,109,113,115,124,132,138–141]. The offline
analysis consists of taking the neural data from a real reaching trial and running the Kalman
filter with a given bin width, thereby producing a decoded reach from the recorded neural
data alone. Because this data is offline, the same neural activity can be decoded at each
bin width; that is, only one block is needed for every seven that were needed for the online
analysis.
In Figure3.6 we show this offline analysis in red. Again, the left column isthe hu-
man OPS continuous task, and the right is the OPS interleaved. For clarity of illustration,
we only show the average data (not the individual experiments, which were shown as faint
traces in Figures3.4and3.5). The dark red points and confidence intervals (95%) represent
the distribution of the error at each bin width. In dark red, we fit a quadratic polynomial to
the data. In faint red, we show the linear regression to that same data. In blue, we show in
the same way the distribution of the online data analysis. Note that these points and con-
fidence intervals are exactly the same as the points and intervals from Figure3.4, Column
C (without the connecting line segments). In dark blue we show the linear regression to
the data, and in faint blue we show the quadratic fit. In both columns, the dark blue line
entirely obscures the quadratic fit (the quadratic term is nearly 0).
Again, a few salient features appear. First, online (blue) has consistently lower error
than offline (red). This is not surprising, as the subjects’ access to feedback information
in the online case should unambiguously improve performance. Nonetheless it is a sanity
check for the data and error metric. Second, and more importantly, we see that the offline
3.4. RESULTS 43
0 50 100 150 200 250 300
40
50
60
70
80
90
100
Me
an
In
teg
rate
d D
ista
nce
To
Ta
rge
t (m
m)
Decode Integration Bin Width (ms)
Human OPS Continuous Human OPS InterleavedPLACEHOLDER... TO COME
0 50 100 150 200 250 300
40
50
60
70
80
90
100
Me
an
In
teg
rate
d D
ista
nce
To
Ta
rge
t (m
m)
Decode Integration Bin Width (ms)
offline
online
Figure 3.6: Comparing offline analysis to online analysis in Human OPS mode.Left Panel:Continuous Task variant - The blue error bars are replicated from Figure3.4, Panel C.The dark blue line is a linear fit of that data, showing a significant performance trendindicating that shorter bin widths imply better performance. A light blue quadratic fit (notvisibly or statistically different from a line) to the same data is obscured by the linear fit. Inred, we perform the same analysis offline, using the real reaching sets from these users. Weperform offline decodes at each bin width, allowing us to generate the same mean integrateddistance to target metric. These error bars are the offline analogs to the blue error bars. Notsurprisingly, offline analysis has worse performance than online, since there was no benefitof feedback. More significantly, however, is the shape of thedata. This offline analysissuggests statistically significant performance optima of roughly 100-150ms. This can beseen in the significant quadratic fit to the data (dark red). The linear fit (light red) does aclearly poorer job of fitting the data. Note that the characteristic “U-shape” of the offlinedata tells a very different story than the online analysis, indicating the important differencesbetween these two testing paradigms.Right Panel: Interleaved Task variant - The sameanalysis and implication holds true as in the left panel.
44 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
error analysis has a characteristic “U-shape” indicating aperformance optimum at 100-
150ms bin widths, adding more evidence that offline and online analyses may not agree. We
note that the U-shape in offline errors makes intuitive sense(and has been shown elsewhere
[139]): integrate too little neural data (short bin widths), andthe estimate is swamped
by noise; integrate too much neural data (long bin widths), and the system updates the
decoded cursor too slowly. Some optimal tradeoff between these two extremes (i.e., a U-
shape) seems reasonable. Contrasting these U-shapes to the blue lines in Figure3.6(online,
where shorter bin widths give lower error), the online case may also make intuitive sense:
indeed shorter bin widths lead to a noisier estimate, but thecritical presence of feedback-
control allows the user to compensate for that noise, resulting in lower error. On the other
hand, longer bin widths still result in a slowly updating prosthesis, so this slow “hopping”
behavior makes feedback-control more difficult and resultsin relatively higher error.
In Figure3.7 we repeat the presentation of Figure3.6, but for the monkey instead of
humans. Offline analyses are shown in red, and online analyses are shown in blue. The raw
data are represented with points and confidence intervals, and linear and quadratic fits are
shown. In all online cases, the linear fit (dark blue) highly overlaps the quadratic fit, though
the quadratic can just be seen in the rightmost column. Again, the monkey results are con-
sistent with the human results. Online analysis in blue produces linear fits suggesting that
shorter bin widths produce lower error, whereas the offline analyses produce characteristic
U-shapes indicating a performance optimum around a bin width of 150-200ms. Note that
the monkey BMI continuous and BMI interleaved offline data (reddata in the left and mid-
dle columns) are identical: in real reaching trials, there is no difference between continuous
and interleaved cases, as all reaches are real. Accordingly, we used the same reaching data
for these two figures. Another interesting note is that, evendisregarding the U-shapes, the
linear fits to offline data are inconsistent: human offline linear analysis says that shorter
bin widths are better (red data in Figure3.6), but monkey offline linear analysis says that
performance either is insensitive to bin width or improves with larger bin widths (red data
in Figure3.7). Contrast that to the simple and consistent story that is seen across all on-
line (blue) data in Figures3.6and3.7 (and so too Figures3.4and3.5): shorter bin widths
will improve prosthetic performance. Thus, these data strongly suggest that offline analy-
sis gives an inconsistent and different answer than online analysis. Since the eventual user
3.4. RESULTS 45
0 50 100 150 200 250 300
60
80
100
120
140
160
Me
an
In
teg
rate
d D
ista
nce
To
Ta
rge
t (m
m)
Decode Integration Bin Width (ms)0 50 100 150 200 250 300
60
80
100
120
140
160
Me
an
In
teg
rate
d D
ista
nce
To
Ta
rge
t (m
m)
Decode Integration Bin Width (ms)
Monkey BMI Interleaved Monkey OPS InterleavedMonkey BMI Continuous
0 50 100 150 200 250 300
60
80
100
120
140
160
Me
an
In
teg
rate
d D
ista
nce
To
Ta
rge
t (m
m)
Decode Integration Bin Width (ms)
offline
online
Figure 3.7: Comparing offline analysis to online analysis in Monkey BMI and MonkeyOPS modes.Left Panel: BMI Continuous Task variant - The blue error bars are repli-cated from Figure3.4, Panel C. The dark blue line is a linear fit of that data, showingasignificant performance trend indicating that shorter bin widths imply better performance.A light blue quadratic fit (not visibly or statistically different from a line) to the same datais obscured by the linear fit. In red, we perform the same analysis offline, using the realreaching sets from these users. We perform offline decodes ateach bin width, allowing usto generate the same mean integrated distance to target metric. These error bars are theoffline analogs to the blue error bars. Not surprisingly, offline analysis has worse perfor-mance than online, since there was no benefit of feedback. More significantly, however,is the shape of the data. This offline analysis suggests statistically significant performanceoptima of roughly 150-200ms. This can be seen in the significant quadratic fit to the data(dark red). The linear fit (light red) does a clearly poorer job of fitting the data. Note thatthe characteristic “U-shape” of the offline data tells a verydifferent story than the onlineanalysis, indicating the important differences between these two testing paradigms.MiddlePanel: BMI Interleaved Task variant - The same analysis and implication holds true asin the left panel.Right Panel: OPS Interleaved Task variant- Again, the same analysisand implication holds true as in the left panel. Taken together, these three panels show thatboth the BMI and OPS modes tell the same story: shorter bin widths imply better perfor-mance, whereas the offline analyses indicate an incorrect trend that leads to varying andmisleading performance optima.
46 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
mode of neural prostheses is fundamentally online, this first finding calls into question the
validity of offline analysis in informing prosthetic designchoices.
3.4.3 Comparing OPS to BMI
The previous section showed that offline analysis may often be a poor proxy to online use of
a prosthetic system. This effect was apparent in both the OPSand BMI data across Figures
3.6and3.7. Thus the OPS, alongside BMI mode, has already made clear the importance of
feedback-control considerations in prosthetic design. While this first finding is of scientific
value, it does not clarify that the OPS is a strong proxy to BMI mode. We investigate that
question here.
Figure3.8 demonstrates the similarity between OPS and BMI modes in an online set-
ting. In previous analyses we fit both lines and parabolas to the collected online and offline
performance data. Some of those regressions appeared linear, others “U-shaped.” By in-
vestigating the quadratic fit term, we can see which data support a U-shape conclusion, and
which a linear. Again, this distinction is of critical importance because the shape of the per-
formance curve makes fundamentally different statements about how to design a prosthetic
system. We calculate the 95% confidence intervals on the quadratic regression coefficient
for each of the ten data sets, and we plot that data in Figure3.8. The blue points and inter-
vals again represent the online data, and the red represent offline data. Each row indicates
the prosthetic mode (human or monkey, OPS or BMI, continuous or interleaved). The gray
and white stripes serve only to visually distinguish the rows. The black line denoting 0
allows us to draw the key conclusion from this figure. If a confidence interval includes 0,
then we can not say with 95% confidence that the correspondingdata is U-shaped; rather
we conclude that shorter bin widths will indeed lead to better performance.
By considering only the blue data, we see importantly that allOPS and BMI modes
indicate that shorter bin widths are indeed better. The consistency of these findings indi-
cates that OPS and BMI modes are indeed valid proxies for one another. Only one case -
the monkey OPS interleaved - does not have a confidence interval overlapping 0, though it
does overlap the intervals of all other monkey data. In otherwords, that OPS quadratic fit
term is not statistically significantly different from the BMI regression terms. Furthermore,
3.4. RESULTS 47
0
Human OPS
Continuous
Human OPS
Interleaved
Monkey BMI
Continuous
Monkey BMI
Interleaved
Monkey OPS
Interleaved
Quadratic Fit Term (with 95% con!dence interval)
Online O"ine
0.002
Figure 3.8: Summary of online/offline differences; summaryof OPS/BMI similarities. Inthe previous figures, we fit quadratic curves to both the offline and online performance data.Here, we plot those quadratic regression coefficients and their 95% confidence intervals.Any confidence interval overlapping 0 should be read as, ”this data is not significantlydifferent than a linear trend.” This figure shows that nearlyall online analyses in monkeyand human do not have significant quadratic terms, agreeing with the implication from allprevious data that shorter bin widths lead to better performance. Comparing that to the rederror bars, again the central point is reiterated: online analysis and offline analysis (whetherin real neural data or in synthetic neural data) tell very different stories. Furthermore, inaddition to online and offline being very different, we see that OPS and BMI are in factvery similar. The OPS gives a valuable means by which to sweepperformance based onthis algorithmic parameter.
48 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
if we instead plotted 99% confidence intervals, the intervalof this monkey OPS interleaved
case would indeed include 0. Thus, this discrepancy is slight and of minor concern to the
interpretation of the data. Figure3.8 restates what was already visible in Figures3.6 and
3.7: blue online data across OPS and BMI modes agree.
In addition, this figure emphasizes the prior finding that offline analysis gives a different
answer than online analysis: the red offline data are all U-shaped with 95% (and indeed
99%) confidence, but the blue online data do not support that claim, and only show that
shorter bin widths are better. This figure also conveys another troubling aspect of offline
analysis, namely the inconsistency of the findings across different subjects and prosthetic
modes (i.e. the red intervals are quite varied amongst themselves).
This second finding suggests that the OPS may allow rapid and simpler testing of many
algorithmic choices and may be a better proxy to clinical usethan offline data analyses. In
this work, we carefully tested that finding by gathering realBMI experimental data also,
but in the future, careful experimental design with the OPS may allow many algorithms,
parameters, and indeed prosthetic systems to be optimized without the risk and difficulty
of implanted BMI subjects. Offering the OPS as an important part of the neural prosthesis
design toolkit is the broad aim of this study. In the following section, we discuss other uses
of the OPS, and we give caveats about ways in which the OPS may not be a valuable proxy
to BMI mode. These discussion points are critical to understanding the role of the OPS in
future neural prosthesis research.
3.5 Discussion
At a simple level, our results make three specific scientific points about optimizing the
performance of a neural prosthetic system by choosing the bin width of a Kalman Filter.
The first point, seen in the raw data of Figures3.4 and3.5, is that the bin width of the
Kalman filter does indeed have meaningful performance implications; that is, it is a critical
parameter that should be optimized in a neural prosthetic system. Second, the results show
that offline analysis is a poor representative of online, closed-loop performance. Figures3.6
and3.7show, across a variety of paradigms, that offline analysis indicates a very different
trend in the data than the actual online usage mode of a prosthetic system. Third, we see
3.5. DISCUSSION 49
that both the OPS and BMI suggest that shorter integration binwidths of 25-50ms will
lead to better performance, presumably reflecting the increased ability of the subject to
incorporate feedback-control strategies.
At a deeper level, our results imply two fundamental findingsabout the design of neural
prosthetic systems. First, we claim that feedback-controlis an essential consideration in
the design of these systems, as demonstrated by the fact thatoffline analysis of algorithms
is an inconsistent and problematic testing scenario. Across Figures3.6, 3.7, and3.8, we
see that offline analyses of various experimental paradigmsindicate parabolic performance
curves with error minima anywhere from 100-200ms (and previous literature has found
even higher optima, 200-300ms [139], though the experimental paradigm and evaluation
metrics were different in that study). Furthermore, the curvature of these parabolas - in
other words how costly it would be to have a suboptimal parameter setting - is inconsistent
across paradigms, as seen in Figure3.8. Most troubling, however, is that these varied
analyses all disagree fundamentally with online, closed-loop prosthetic use, which is in fact
the true usage mode of these systems. All online analyses suggest that performance curves
are linear, and that shorter parameter settings are better,which stands in stark contrast to
the parabolic offline implication. Thus, the first finding of this work is to say that offline
and online analyses are very different for neural prosthetic systems, and further that offline
is perhaps not adequately realistic to justify its continued use in the design of prostheses.
The second broad implication of this work is to present the OPS as a valuable proxy to
a real neural prosthetic system. By replacing recorded neural activity with synthetic neural
activity, the OPS enables analysis of human subjects using an online, closed-loop prosthetic
system. By reproducing the qualitative behavior of real prosthesis use, we hypothesized
that the OPS would produce similar performance curves as thereal BMI mode. We tested
this hypothesis by comparing humans and a monkey using the OPS to a monkey using
a real BMI, and we found that all reproduce similar performance trends in the bin-width
parameter sweep (Figure3.8summarizes this effect). Thus, in terms of this specific design
question for decoding algorithms like the Kalman filter, theOPS appears to be a high-
quality approximation to real online BMI use. However, a bigger question is the extent
to which the OPS can be relied upon to generalize to differentneural prosthetic design
questions in humans. As is portrayed in Figure3.1, having human results here (in addition
50 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
to monkey results) is important to present a full human BMI simulator that does not require
the resources of invasive human or monkey experiments. Human results are also important
for generalizing the findings of the OPS to areas that will notnecessarily be addressable
in monkeys. The following points discuss in depth a number ofways in which the OPS
will generalize well and a few ways in which it may not. These points of generalization are
natural steps for future work as well.
3.5.1 The OPS for general system design questions
With our specific finding that online and offline analyses are fundamentally different for
the Kalman filter bin width, we have demonstrated that feedback-control and subject in-
teraction are significant contributors to eventual system performance. This general fact is
a key principle of the field of human/computer interaction [1, 134]. However, the role of
humans in human/neural-prosthesis interaction remains largely unexplored (though see the
Introduction where we discuss the relevance of [25] to this question). The OPS framework
allows systematic study of this important question. Here wedescribe a few examples of
general design questions that can be studied with the OPS.
First, user interface design for neural prostheses will require systematic study. For ex-
ample, in this paper, the subject reached to one of eight targets on a single ring. In a clinical
application, where presumably the goal is to maximize information throughput or user ex-
perience, how many targets should be placed on the screen, and in what configuration? Our
previous work [23] addressed this point algorithmically, but that was in an offline setting.
Testing this online in the presence of feedback-control is important for user interface de-
sign, but it may also be overly time-consuming for a clinicaltrial subject. However, the
OPS, insomuch as it involves a human in closed-loop and has similar qualitative and quan-
titative performance to a real BMI, can readily be used to investigate this and similar user
interface questions.
Second, offline analyses do not explore the sensitivity of prosthesis users to noise (neu-
ral spiking variability, for example). Different systems and algorithms will cope with this
noise in different ways, and it is critical to understand howthese different algorithmic fea-
tures are managed by human subjects. In an extreme case, evenwith an optimal decoder
3.5. DISCUSSION 51
of neural activity, it is not clear at what level of noise a neural prosthetic device becomes
unusable. For example, an important clinical question is, “how many recorded neurons
are needed to control a prosthesis?” Such a question has major bearing on translational
efforts. While offline studies often perform “neuron-dropping” analyses (e.g., [12,79]) to
test the sensitivity of error to neural population size, these studies can not investigate the
neuron count at which a user can no longer control the prosthesis in a satisfactory way. To
our knowledge only one study [38] has done a similar analysis online, and it suggests that
dropping a meaningful percentage of a small number of stableneurons (that study recorded
from 10 and 15 neurons) can have a highly detrimental effect on relative performance. For
larger populations of neurons and different experimental contexts, however, this question
remains unanswered. The OPS allows this analysis: we can vary the number of synthetic
neurons used in the decode to find the point at which the human subject can no longer
successfully complete the task. This and the above featurescan not be tested in offline
data, when the user can not interact with the prosthesis. Furthermore, rigorously testing
this feature in an online, real BMI may be quite difficult giventhe rarity of human clinical
trials and the frustration that such a study would cause a monkey subject.
Third, in terms of the clinical utility of a BMI, the OPS allowscustomization of the
controller based on the specific user application. For example, BMI signals demonstrated
to date would have difficulty controlling a computer mouse ona normal operating system
without the use of specialized accessibility software (see[57]). This software could involve
non-linear mouse acceleration curves [62] or multiple-level mouse selections (selecting
once to zoom in and a second time to click [75]). Such software may be difficult to test
in an animal model due to having many free parameters to sweepand abstract selection
mechanisms that a monkey will not readily learn. Before usingsuch as system in a human
clinical trial, the OPS can serve as a test bench for developing these interfaces and testing
them at a wide range of potential signal quality.
More generally, again by analogy to human/computer interaction, the design of neural
prostheses will require systematic study. In addition, as the field moves towards clinically
viable prostheses, the role of a human in a prosthetic systemis increasingly important.
Indeed, recent years have seen increased importance of human studies for neural prostheses
52 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
[57,69,78]. The OPS should help systematize this design process with ahuman in closed-
loop control. We now discuss more detailed algorithmic features that the OPS can address.
3.5.2 The OPS for specific algorithmic questions
Generating synthetic neural activity is the key enabler of the OPS, as it allows many human
subjects to be tested without neural implants. However, generating synthetic neural data is
also the principal concern with the OPS, as this choice callsinto question the relevance of
OPS findings for real BMI systems. In the Results, we demonstrated that OPS and BMI
modes are consistent for optimizing the bin width of the Kalman Filter, showing that indeed
the OPS can be used as a valuable proxy. Nonetheless, for other algorithmic questions,
there remains the risk that the resulting OPS effects will not translate well into a real neural
system. To address this risk and to help ensure legitimacy ofthe result, we here identify
four areas in which the OPS will continue to allow meaningfulinvestigation of algorithmic
design.
First, algorithmic models for arm reaching can be meaningfully studied with the OPS.
The human motor plant imposes significant constraints on thefrequency content, speed,
and extent of a reach. For example, it is known that the motor system naturally produces
smooth movements [112]. This and other constraints have been well studied in humanbe-
havioral studies [43, 112, 135, 136], but they have been largely neglected in the design of
neural prosthetic decode algorithms. Steps in this direction have been taken to acknowl-
edge the point-to-point nature of reaches [74,87,119–121,140,141], but the field has not
produced a model for arm reaching that will generalize to theeventual prosthesis user
mode of unconstrained natural reaching. The Kalman filter used in this study stipulates a
linear prior model for arm reaching (Eq.3.3), but this model appears only in the decode
algorithm, not in the generation of synthetic neural activity. Critically, the generation of
synthetic neural data has no notion of a model of arm reaching. Accordingly, the OPS
can be legitimately used to study the online effect of different algorithmic models for arm
reaching, and performance improvements discovered here should port reliably to real BMI
mode.
Second, the OPS can be used to study the algorithmic mappingsfrom neural activity to
3.5. DISCUSSION 53
kinematics. Many decode algorithms like the Kalman filter stipulate a linear mapping from
neural activity to kinematic reach parameters (Eq.3.4), but there are also many approaches
that use nonlinear mappings. Understanding this algorithmic mapping is often considered
of critical importance, but it has not been exhaustively studied online (though see [79]). The
OPS of this manuscript generates synthetic neural data via alinear mapping (Eq.3.1), so
its current use for this question would be problematic, since the mapping from kinematics
to neural activity is linear by assumption. However, futurework can go further and use a
source of synthetic neural signal that is not directly and simply related to kinematics. We
think EMG offers a nice opportunity. EMG has been shown to have a linear relationship to
neural activity [104], so synthetic motor cortical neural activity can be reasonably generated
from EMG. However, the connection between kinematics and EMG is by no means simple
( [97], and particularly [125] discuss the difficulty of predicting kinematics from recorded
EMG). Thus, by using EMG to create reasonable synthetic neural activity, we can guarantee
a “non-trivial” relationship between the synthetic neuralactivity that we record and the
endpoint kinematics that we want to decode. This approach would allow us to use the OPS
to study the online implications of linear and non-linear algorithmic mappings between
kinematics and neural activity.
Third, we can evaluate very basic assumptions of particulardecode algorithms. In this
study we considered the decode integration bin width of the Kalman Filter. This time win-
dow specifies both the unit of time for output updates (changing the cursor position) and
the unit of time for input integration (integration of neural activity to use as the decode
signal). To remain specifically within the Kalman Filter framework as has been done in
the past, this agreement between update rate and integration rate is required. However, in
practice this assumption can be relaxed. It may be that the difficulty with the large inte-
gration windows has more to do with the hopping or stroboscopic behavior of the cursor
(the slow update rate) than with the long integration rate. Perhaps maintaining a long in-
tegration rate but making a quicker update rate (smoothly varying the cursor between two
decoded positions, for example) could improve performance. Our preliminary investigation
into this question suggests that further performance gainsmay be achievable by optimiz-
ing jointly the cursor-update-rate/neural-integration-rate pair. Since this question inves-
tigates human/neural-prosthesis interaction and not specific neural properties, the OPS is
54 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
well suited to address this optimization directly.
Fourth, another basic assumption of a decode algorithm is the time lags it assumes
between recorded neural activity and movement. This parameter is often optimized in BMI
applications (e.g., [139, 141]). However, much like the Kalman Filter bin width of this
study, time lags are rarely optimized online. Since time lags would be an assumption of the
synthetic neural activity in the OPS, discovering the “true” online optimal time-lag is not
likely to be an appropriate question for the OPS. However, understanding the sensitivity of
this choice is indeed readily testable with the OPS. One can choose a set of time lags for
the synthetic neural activity and then test performance sensitivity to different optimization
strategies (for example, studies have included no time lags, a single time lag across all
neurons, or individual lags per neuron). The OPS thus enables us to study how important
time lags are to closed-loop performance.
These four examples (by no means the only examples) show thatthe OPS paradigm
should continue to be valuable to validate algorithmic advances and translational needs.
3.5.3 The OPS for neuroscientific questions
While the OPS may be viewed primarily as a platform for designing closed-loop neural
prosthetic systems, it can also serve as a tool for investigating motor neurophysiology.
Findings in the applied BMI context can often offer insight into the normal-functioning
motor system. As a specific example, we found here that shorter temporal decode updates
(bin widths) lead to smaller decode error. However, as we note above (the third example in
“The OPS for specific algorithmic questions”), future work should aim to understand the
contribution to BMI decode errors between the stroboscopic effect of the cursor updates and
the integration of shorter amounts of neural data. This stroboscopic effect represents a form
of noise to the visual system. By altering that noise independently of other cursor control
parameters, one could study the effects of noise on learning[97], reflex adaptation in the
motor system [35], or multimodal sensory integration (separating visual and proprioceptive
contributions) [51,118,128].
More generally, the OPS can serve as an experimental system for motor control studies,
and it has interesting connections to that substantial literature (e.g., [112,135]). There has
3.5. DISCUSSION 55
been a great deal of research using variants of a robotic manipulandum to study perturba-
tions of normal motor control and to introduce visuomotor discrepancies (see [60] for a
review). These studies ask questions ranging from dynamic learning in the motor system,
to object manipulation, to limb stiffness measurements: a classic example is subjects mak-
ing reaches in the presence of an external force field [111]. These and other studies point
to the fact that the introduction of distortions between intended movement and perceived
movement has been a fundamental aspect of motor research formany years. The OPS is a
similar system that allows novel 3D visuomotor perturbations (in particular perturbations
that are relevant to the applied field of neural prostheses) and manipulations of the sys-
tem’s underlying dynamics at several levels (at the cursor directly, or more implicitly in the
neural encodings). Thus the OPS should be applicable to studies of motor control and 3D
motor adaptation. In addition to the novel perturbations this system may allow, the OPS
may also prove useful as motor control researchers move towards more freely behaving
experimental paradigms (as has been happening, for example, in motor electrophysiologi-
cal studies [101]). For example, due to electrode array shifts and head acceleration events,
recorded neural signals can change abruptly [101]). Studying 3D motor adaptation to such
signal changes in this less constrained context would be beneficial both scientifically in
increasing our understanding of motor learning, and in an applied setting by informing the
field which aspects of motor control are contextually relevant for BMI. Certainly much cau-
tion should be taken in interpreting the neuroscientific implications of a BMI result, but the
OPS should allow investigation of both applied neural prosthesis and fundamental motor
neurophysiology questions.
3.5.4 Cautionary notes regarding the OPS
The Results and our arguments for more systematic online analysis suggest that the OPS
should be used to address a variety of neural prosthetic design choices. However, a number
of questions can not readily be asked with the OPS, and much care should be taken with
any simulation environment to ensure the legitimacy of any findings. Here we describe
important precautions with the OPS.
56 CHAPTER 3. A CLOSED-LOOP HUMAN SIMULATOR
As previously noted, the biggest potential pitfall of this system is the generation of syn-
thetic neural data. Generating synthetic neural data or asking design questions carelessly
can lead to tautologies that are uninteresting and potentially misleading. In this work, we
specifically optimized the bin width of the Kalman filter. Since the synthetic neural activity
is generated instantaneously and according to a Poisson likelihood model, the bin-width of
the Kalman filter is not related to how the neural activity is generated. Thus, the bin width
optimization was more a question of how the subjects interacted with systems with dif-
ferent noise characteristics. This question could be and was legitimately investigated with
the OPS, as indeed the real BMI experiments validated. However, with the same setup,
one could propose another hypothesis that would prove problematic. Consider a hypothe-
sis to test the plasticity of the mapping between neural activity and kinematics. Certainly
this is an interesting and important question both for scientific reasons and for the design
of neural prosthetic systems (e.g., [38]). However, the answer here is trivial: because we
synthetically created a static mapping between kinematicsand neural activity, indeed this
relationship is by definition without plasticity. So, questions must be asked that do not
directly test an assumption of the generative model for synthetic neural data. Though this
point is perhaps fairly obvious, it is the overarching caveat with the extensibility of the OPS
framework.
Another possible problem with the closed-loop paradigm regards proprioceptive feed-
back. As previously noted, there was a sensory discrepancy between the visual feedback
and the real arm’s proprioception (in prosthetic trials). Whether the true arm is restrained
or allowed to reach (both are regularly done in the BMI literature), there is still a proprio-
ceptive confound. Hence, this potential limitation is not specific to the OPS or this study,
but rather to all able-bodied animal or human BMI studies. Indeed, previous BMI literature
such as [12,109,124,130] have not reported difficulty with this proprioceptive error signal,
and there is no reason to expect that it is any more prominent in the OPS than in those
other able-bodied studies. Furthermore, previous neuroscience research has suggested that
this confound is minor: [97] found that proprioception did not prevent task learning but did
somewhat increase task difficulty. Other work has found thatvision is a stronger sensory
signal than proprioception in certain contexts (e.g., [127]), but this question is still debated
in the neuroscience community. Thus, while this possible limitation is important to bear in
3.6. ACKNOWLEDGMENTS 57
mind for the OPS and all able-bodied BMI studies, the significant progress that has been
made in this field even with this proprioceptive confound suggests that the OPS framework
should continue to be useful for aiding in the design of clinically relevant prostheses.
3.5.5 Summary
The OPS paradigm provides a middle ground between simple (but low reality) offline test-
ing and more realistic (but very involved) clinical trials.We showed here how the OPS can
be effectively used to find a surprising result that stands incontrast to offline analysis and
previous literature: the bin width of the Kalman filter should be decreased in size for online
prosthetic studies. These results showed the substantial disagreement between offline and
online analyses, and the results showed the substantial similarity between OPS and BMI
modes by using nonhuman prosthetic experiments as validation. While these findings are
valuable in their own right, the broader message of the work is that offline analysis may
be a poor proxy to eventual system use, and thus the field should investigate simulation
opportunities to create a principled design engineering process around neural prosthetic
systems. In this example, considering human interaction with the system was of critical
importance, and we speculate that such feedback-control isindeed critical for many other
aspects of prosthetic system design. Ideally, the OPS or similar online simulators may be-
come a piece of the neural prosthesis researcher’s toolkit,to allow rigorous design before
moving to the gold standards of monkey BMI experiments and human clinical trials.
3.6 Acknowledgments
We thank D. Franklin for helpful conversations, M. Risch for veterinary care, D. Haven for
technical support, and S. Eisensee for administrative support.
Chapter 4
ReFIT-KF Neural Prosthesis
This chapter appeared in Nature Neuroscience in 2012 as a paper titled “A high-performance
neural prosthesis enabled by control algorithm design.” The paper was authored by Vikash
Gilja, Paul Nuyujukian, Cynthia Chestek, John Cunningham, Byron Yu, Joline Fan, Mark
Churchland, Matt Kaufman, Jonathan Kao, Stephen Ryu, and Krishna Shenoy. The au-
thor’s contribution was in experimental rig design and setup, assistance in decoder design,
data collection and analysis, and writing the manuscript.
4.1 Abstract
Neural prostheses, also referred to as brain-machine interfaces or brain-computer inter-
faces, translate neural activity from the brain into control signals for guiding prosthetic de-
vices, such as computer cursors and robotic limbs, and thus offer disabled patients greater
interaction with the world. However, relatively low performance remains a critical barrier
to successful clinical translation; current neural prostheses are considerably slower with
less accurate control than the native arm. Here we present a new control algorithm, the re-
calibrated feedback intention-trained Kalman filter (ReFIT-KF), that incorporates assump-
tions about the nature of closed loop neural prosthetic control. When tested with rhesus
monkeys implanted with motor cortical electrode arrays, the ReFIT-KF algorithm outper-
forms existing neural prostheses in all measured domains and halves acquisition time. This
control algorithm permits sustained uninterrupted use forhours and generalizes to more
58
4.2. INTRODUCTION 59
challenging tasks without retraining. Using this algorithm, we demonstrate repeatable high
performance for years after implantation across two monkeys, thereby increasing the clini-
cal viability of neural prostheses.
4.2 Introduction
Neural prostheses have recently demonstrated considerable promise through proof-of-concept
animal experiments [12, 38, 80, 87, 109, 116, 122, 124, 130] and in human clinical trials
[57, 58, 69, 70] for partially restoring motor output in paralyzed individuals. Studies in
this field primarily focus on adapting insight and methods from the basic neuroscience of
cortical motor control to this engineering context. A critical example of this is the use
of motor cortex tuning models, which describe the relationship between single unit firing
rates and arm movement kinematics, to define a mapping for neural control of a computer
cursor in closed loop (e.g. [12, 109, 124]). When such a neural prosthesis is introduced
to a monkey, performance can increase across days through learning [12]. In addition to
controlling computer cursors, these systems have successfully driven robotic end effec-
tors [130]. Additional concepts from motor neuroscience have been incorporated, demon-
strating the potential to augment system performance by modeling neural activity related
to movement preparation [87] and proprioceptive feedback [122]. Recent work also sug-
gests that when the recorded neural population and control algorithm are held constant,
neural prosthetic performance increases over time as a stable neural output map is formed
and multiple mappings, once learned, can be retained and retrieved across different control
contexts [38]. Despite these new insights and additional algorithmic advances (e.g. [70]),
system performance on simple cursor control tasks remains low relative to native arm con-
trol performance, presenting a critical barrier to clinical translation [63].
In an effort to improve the performance of neural prostheses, we chose to focus on a
systems engineering approach. Building on existing methodsin the field, we developed
two key innovations that alter the modeling assumptions made by these algorithms and the
methods by which these algorithms are trained. Additionally, signal conditioning methods,
which transform recorded neural signals into control algorithm input, were chosen in an
effort to improve system stability and performance [18, 24]. As demonstrated in closed
60 CHAPTER 4. REFIT-KF NEURAL PROSTHESIS
loop neural control experiments, these methods result in high performance across multiple
cursor control tasks.
4.3 Results
4.3.1 Performance Overview
We trained monkeys to acquire targets with a cursor controlled by either native arm
movement or neural activity. We developed a novel algorithm, ReFIT-KF, that led to sub-
stantially higher-performance neural prosthetic control. Figure4.1a shows representative
continuous, uninterrupted cursor movements for three different modalities: native arm con-
trol, ReFIT-KF, and a velocity Kalman filter (Velocity-KF), that is state-of-the-art for cur-
rent neural prostheses (e.g., [58, 69, 70]). Monkeys were required to move the computer
cursor to a visual target and hold the cursor within a demand box for 500 ms to successfully
complete a trial and receive a liquid reward. Targets alternated between central and eight
peripheral locations. During online neural control sessions, the contralateral arm was not
restrained, and movement continued. However, the physicalmovement was not stereotyped
and would often attenuate or even stop during some neural control sessions, while retain-
ing performance. In a set of additional control experiments, both arms were restrained, and
little or no arm movement was observed with similar performance (see Table4.1).
The ReFIT-KF algorithm outperformed the Velocity-KF by several measures. First,
cursor movements with ReFIT-KF control were straighter (Fig. 4.1a,b), producing less
movement away from a straight line to the target. Cursor movements produced using the
ReFIT-KF were qualitatively similar to native arm movements(Fig. 4.1a). Second, these
movements were also completed faster. ReFIT-KF cursor control performance, as mea-
sured by the time to successfully acquire the target (Fig.4.1c,d), was 75-85% of native
arm control performance and at least twice Velocity-KF performance. In addition to lower
mean time to target, the variance was substantially smaller, which is important as this sig-
nifies greater movement consistency and fewer potentially frustrating long trials. Critical
4.3. RESULTS 61
Figure 4.1:Performance comparison. Native arm control shown in blue, ReFIT-KF inred, and Velocity-KF in green. (a) Representative traces of cursor path during center-out-and-back reaches. Dashed lines (not visible to the monkey) are the demand boxes forthe eight peripheral targets and the central target, shown as translucent green circles. (b)Bar graphs plotting maximum deviation from a straight-line path to the target on eachsuccessful trial (mean± SE). (c,d) Histograms of time to target for successful trials areshown as line graphs for monkeys J and L. The inset bar graphs plot the time to target(mean± SE). (e,f) Line graphs plotting the mean distance to the target as a function oftime. The inset bar graph plots the mean± SE of the dial-in time, or the time requiredto finally settle on the demand box, after first acquired, to successfully hold for 500 ms.The thickened portion of the line graphs also indicate dial-in time, beginning at the meantime of first target acquire, and ending at mean trial duration minus 500ms. Data fromexperiments J & L 20101027-29, 20101102.
62 CHAPTER 4. REFIT-KF NEURAL PROSTHESIS
Figure 4.2:Performance of ReFIT-KF control across 4 years.Performance is measuredby the Fitts’ law metric. Data from monkey J and monkey L are shown as 94 orange circlesand 182 cyan squares, respectively. Each point plots the performance of the ReFIT-KFalgorithm trained on that experimental day. The eight filleddata points (four for each mon-key) are calculated from the same datasets used to generate Figure4.1. Linear regressionlines for data from monkey J (orange) and monkey L (cyan) are shown. The slope of theregression line for monkey J is not statistically significant from zero (p>0.34). The slopesfor monkey L is statistically significant from zero (p<0.001). For all datasets shown, thetrial success rate was>90%.
to achieving this lower time to target, ReFIT-KF controlled cursor movements stopped bet-
ter. The ability to stop is a critical differentiator between the three control modes. The
Velocity-KF took only modestly longer to first acquire the target than other control modes,
but often significantly overshot the target, requiring additional time and multiple passes to
stably acquire and hold the target. This overshoot-correction time dominates the overall
time to successful target acquisition for Velocity-KF control (Fig. 4.1c,d) and is captured
by “dial-in time” (Fig.4.1e,f, bar graphs and bold lines): the average time required tomake
the final target acquisition after having first reached the target. Both native arm and ReFIT-
KF control allowed more precise stopping as compared to Velocity-KF (Fig. 4.1e,f; final
distance to target). Across all trials in eight experimental sessions with two monkeys, when
given five seconds to acquire targets, ReFIT-KF achieved a success rate of>99%, while
Velocity-KF had a success rate of 95%. The task difficulty waschosen to achieve a high
success rate for all three control modalities on the first experimental day. When the task dif-
ficulty is increased, the success rate with Velocity-KF can drop relative to the success rate
with ReFIT-KF, and similarly ReFIT-KF success rates and performance can drop relative
to native arm control (see generalization tasks described below).
Experiments across days and years demonstrated consistenthigh performance. As
4.3. RESULTS 63
shown in Figure4.2, performance was stable as measured by throughput on 280 individ-
ual experimental days. These data were collected across 29 months for monkey L and 16
months monkey J, spanning the range of 0.4 to 4.4 years post-array implantation. To ex-
plore the possibility that performance changed with time, we computed least squares linear
fits on these performance data for each monkey. The slopes of both regression lines are
positive, suggesting that performance was stable over the time period of the study and pro-
viding evidence consistent with the hypothesis that intracortical microelectrode arrays may
permit years of high performance neural control [18].
4.3.2 Generalization & Robustness
We also tested additional behavioral tasks to assess generalization of the ReFIT-KF
control algorithm. We fit the ReFIT-KF algorithm with center-out-and-back reaches, as
before, and then tested on a pinball task in which targets could appear at any location in the
2D workspace. Monkeys were again required to move the cursorto the target and hold it
for 500 ms to successfully complete a trial. Figure4.3shows over 90 minutes of data from
monkey L during two continuous pinball reaching sessions, one with native arm control
and one with the ReFIT-KF control. Given two seconds to acquire a target on each trial,
both sessions had success rates>98%. Across the whole session, the mean time to target
for ReFIT-KF control was 72% as fast as native arm control (ReFIT-KF: 710± 317 ms;
native arm: 519± 196 ms). Comparable performance was found for monkey J. Figure 4.3
illustrates that the performance with ReFIT-KF was not only high (over two-thirds as fast
as the natural arm; comparable distribution widths in Fig.4.1 and Fig.4.3a), but was also
sustained without intervention (Fig.4.3b). Sustained performance was typical of ReFIT-
KF control sessions, whereas Velocity-KF control sessionswith the same task parameters
had much lower success rates (<40%), and the monkeys could not be motivated to acquire
targets for more than 30 minutes.
To further test ReFIT-KF control, we trained monkey J to avoidvisually defined obsta-
cles that appeared in the direct path of the target (maze task[19,66]). The monkey reached
from a central starting target to either a left or right peripheral target. As shown in Fig.4.4,
64 CHAPTER 4. REFIT-KF NEURAL PROSTHESIS
Figure 4.3:Performance comparison of native arm vs. ReFIT-KF for the pinball task.Native arm control is in blue, ReFIT-KF in red. In this task, each target location is selectedfrom a uniform distribution spanning the workspace. (a) Each column shows data from20-minute segments. The top rows are randomly selected cursor traces for 4 subsequenttarget acquisitions. Target demand boxes are shown as dashed lines and target sequenceis indicated from 0 to 4. The bottom row shows normalized histograms of time to targetfor successful trials. Arrows below the plot indicate average time. (b) Target acquisitionrate per minute throughout the sessions is shown. The sharp rate drop indicates when themonkey lost interest in the task. A histogram of acquisitionrate across the sessions is inset.The native arm and ReFIT-KF sessions (L-2010-04-01 and L-2010-04-12) were on twoseparate days, within 11 days of each other, when the monkey demonstrated a high degreeof motivation.
4.3. RESULTS 65
Figure 4.4: Performance comparison of native arm vs. ReFIT-KF for the obstacleavoidance task. Native arm control shown in blue, ReFIT-KF in red. In this taskthemonkey had to move the cursor from the initial target (labelled 0) to the final target (labelled1, demand box shown as dashed line) without hitting the magenta-colored barrier. Onerepresentative cursor trace is shown from each of the four principle observed movementtypes: curve under, curve over, straight (no barrier), and collision into barrier. These dataare from experiment J-2010-03-09.
66 CHAPTER 4. REFIT-KF NEURAL PROSTHESIS
Figure 4.5: Illustrations of the online neural control paradigm and the ReFIT-KFtraining methodology. (a) The input to the control algorithm at timet is a vector of spikecounts,yt , from implanted electrodes.yt is translated into a velocity output,vt , to drivethe cursor. (b) ReFIT-KF is trained in two-stages. Initially, cursor kinematics and neuralactivity are collected during arm control or during an observation phase in which cursormovement is automated (i). These arm movement or observed cursor kinematics (ii) areregressed against neural activity to generate an initial control algorithm. Then, a new setof cursor kinematics and neural activity are collected using the initial algorithm in closedloop (iii). The kinematics collected during neural control(red vectors in (iv)) are used toestimate intention by rotating the velocities towards the goal (cyan vectors in (iv)). Thisestimate of intended kinematics is regressed against neural activity to generate and runReFIT-KF (v).
on some trials a barrier appeared along with the peripheral target. To successfully com-
plete a trial, the monkey had to use the cursor to acquire and hold the peripheral target for
500 ms without hitting the barrier. This task was difficult, but the monkey successfully
acquired and held the target on 77% of trials with his native arm (Fig. 4.4a) and on 75%
of trials with ReFIT-KF control (Fig.4.4b). Under ReFIT-KF control, mean time to target
for this task was 74% as fast as with native arm control (ReFIT-KF: 1253± 588 ms; native
arm: 932± 709 ms; mean +/- standard deviation). With Velocity-KF control, the monkey
could not complete the task and quickly became frustrated and disengaged. As in the pre-
vious tasks, the ReFIT-KF was fit with center-out-and-back movements and used without
modification for the maze task, demonstrating robustness across behavioral contexts.
4.3. RESULTS 67
4.3.3 Two Innovations
The described cursor control performance was achieved by redesigning the Velocity-KF
algorithm from a feedback control perspective. As shown in Fig. 4.5a, the prosthetic de-
vice constitutes a new physical plant with different dynamical properties than the native
arm. The subject controls this new plant by modulating measured neural signals (yt in
Fig. 4.5a-i), which are then decoded (Fig.4.5a-ii) into a velocity by the control algorithm.
This velocity is used to update the cursor on screen (Fig.4.5a-iii), affecting neural signals
on subsequent time steps. This closed loop control perspective suggests two design innova-
tions. The first innovation is a modification of neural prosthetic model fitting methodology.
The second innovation is an alteration of the control algorithm. These ReFIT-KF innova-
tions produce the neural prosthetic results described above.
4.3.4 Innovation 1
The first design innovation is to fit the neural prosthesis against estimates of intended ve-
locity. Previous algorithms [12,57,87,109] implicitly assume that the subject uses the same
control strategy for moving the native arm and the prosthetic cursor. Since these control
strategies may be quite different, we, in the vein of past studies [36,37,70,80,116,124,130],
evaluated methods that attempt to better capture the subject’s strategy during prosthetic
control. Ideally, the control algorithm would be fit to the subject’s intended cursor velocity
during closed loop neural control. Since we lack explicit access to the monkey’s inten-
tions, we hypothesized that the monkey wished to move directly towards the target; this
resembles movements made by the native arm and is a good strategy for acquiring rewards.
A two-stage optimization procedure (Fig.4.5b) is used to fit the neural prosthetic model
to these estimates of intended velocity during online neural control. In stage 1, the monkey
controls the cursor using his arm (Fig.4.5b-i). An initial model is fit using arm trajectories
and simultaneously recorded neural signals (Fig.4.5b-ii). The monkey controls the neural
prosthesis with this initial model (Fig.4.5b-iii). In stage 2, neurally controlled cursor
kinematics and neural signals are recorded and used to fit a new model with an estimate of
intended cursor velocity. By starting with cursor velocities collected during the previous
online control session (shown in red), these estimates are calculated for model fitting using
68 CHAPTER 4. REFIT-KF NEURAL PROSTHESIS
two transforms. First, the velocities are rotated towards the target (blue vectors) to generate
the set of estimated intended velocities (Fig.4.5b-iv). Second, if the cursor is on target, the
monkey’s best strategy is to keep the cursor still to satisfythe hold time requirement. Thus,
in the training set, we assume that the monkey’s intention during these hold periods was to
maintain the cursor position by commanding zero velocity. This zero velocity assumption
applied to the fitting of model parameters improves online performance without changing
the control algorithm. These estimated intentions and corresponding neural data are used
to fit the ReFIT-KF control algorithm (Fig.4.5b-v). It is important to note that the intention
estimation is appliedonly to training data:during online control the neural prosthesis has
no knowledge of the task goal or placement of targets (unlikee.g. [87,120,141]).
The aforementioned training protocol utilizes hand-controlled reaches as training data.
In a paralyzed individual, it is not possible to record arm kinematics for this step. Instead,
this training step could rely on the individual imagining a set of instructed movements.
To test this possible strategy, we trained the initial algorithm based on visual cue observa-
tion [70, 122], removing the requirement for arm control in step one. During these trials
the monkey watched a computer controlled training cursor that automatically moves out
to targets. The initial model was fit using automated training cursor trajectories and si-
multaneously recorded neural activity, without using measured arm movement. Table4.1
summarizes ReFIT-KF performance for three experimental sessions from each monkey in
which stage 1 of ReFIT-KF model training is based on observation data instead of arm
movements. The performance, as measured by Fitts’ Law [10], for these sessions is similar
to that attained for the native arm control initiated sessions described in the results section
and summarized in Figures4.1and4.2.
4.3.5 Innovation 2
The second design innovation builds on the observation thatneural activity is correlated
with both the velocity and the position of the cursor. Most existing neural prostheses model
a relationship between neural activity and either velocity[124,130] or position [57,109].
A human clinical trial has shown that neural prostheses modeling velocity have higher per-
formance [69,70]. However, if the control algorithm models only the velocity relationship,
4.4. DISCUSSION 69
Experiment Targetcenterdistance(mm)
Windowsize(mm)
Acquiretime(sec)
Index ofdifficulty(bits)
Fittsbits/sec
SuccessRate
L-2010-08-10 80 40 0.89 1.32 1.49 94%L-2010-08-11 80 40 0.89 1.32 1.48 95%L-2010-08-12 80 40 0.83 1.32 1.59 94%
J-2010-08-10 80 40 0.76 1.32 1.74 97%J-2010-08-16 80 40 0.82 1.32 1.60 97%J-2010-08-17 80 40 0.76 1.32 1.73 98%
Table 4.1: Performance of ReFIT-KF based control with observation based algorithm train-ing.
then position-based changes in firing will confound decodedvelocities. To mitigate this
effect, we explicitly model velocity as the user’s intention and cursor position as. This
modification allows the user to control velocity with measured neural signals while ac-
counting for the influence of cursor position. We explicitlyassume that the current cursor
position, determined by integrating the previous velocityoutput, is encoded in the neural
activity along with the monkey’s current intended velocityoutput. Thus, the expected con-
tribution of position to neural activity is removed, enabling more accurate estimation of
intended velocity (Fig.4.1).
4.4 Discussion
Other studies have noted the potential change in plant and control strategy and have ad-
dressed it by iteratively refining parameters during neuralprosthetic experiments [36, 37,
70,80,116,124,130]. This approach recognizes that control strategies, and therefore model
parameters, are best measured and understood during closed-loop neural prosthetic exper-
iments. However, randomizing initial parameters [36, 124, 130] may create a control al-
gorithm that never attains the highest possible performance, just as optimization problems
can easily become trapped in local optima. Although, if the recorded neural population
and the neural control mapping are held constant, the consequences of the plant mismatch
70 CHAPTER 4. REFIT-KF NEURAL PROSTHESIS
can be overcome through learning. Such learning was demonstrated with neural control
mappings built to reconstruct arm kinematics, as well as with a neural control mapping in
which neuron identities were shuffled, so the decoder outputwas no longer predictive of
native arm kinematics [38].
The focus of the present study was to obtain high control performance within a single
session by improving the neural control algorithm and optimizing its parameters. Although
the neural prosthesis constitutes a new plant with different properties than the native arm,
the motor cortices are involved in native arm control (e.g.,[64]). Therefore, we hypoth-
esized that initializing a model with the relationship between neural activity and natural
arm movement would allow the second stage of our training method to achieve a higher
level of optimization. Previous studies [36, 130] have relied on manipulating the control
task to refine the neural decoder, such as by providing assisted control. In those studies, an
automated correct answer was mixed with the output of the neural prosthesis. Over suc-
cessive iterations the weight of automated control was decreased by experimenter intuition
until control was driven only by neural activity. Our approach is different, as the control
task remains constant throughout a neural prosthetic session and only the training data are
manipulated between the first and second neural prosthetic sessions.
In a study with quadriplegic humans [69,70], the neural prosthesis was initially trained
with visual cue observation, similar to the control experiments described above. The study
also uses a second neural prosthetic training session to account for differences during online
control. Unlike ReFIT-KF training, both the neural cursor and the automated training cursor
were on screen during the second session. The neural cursor was presented to provide
feedback so that the participant could attempt to alter their neural output to better follow
the training cursor. After this second session, the neural prosthesis was fit with the training
cursor kinematics. Thus, the underlying assumption is thatthe training cursor kinematics
capture the intended kinematics during online control, while ReFIT-KF fitting assumes
that intended kinematics are best inferred by the output of the neurally controlled cursor
and knowledge of the task goals.
In studies with adaptive decoders [80, 116], the kinematics of the neurally controlled
cursor are continuously used to refine neural prosthetic parameters, also allowing them to
4.4. DISCUSSION 71
account for differences when switching to online control. However, they take different ap-
proaches to estimating intended kinematics for retraining. One approach is to use decoder
parameters non-causually, via smoothing, to estimate intended kinematics for retaining
without task or target information [80]. Impressively, this method was shown to slow per-
formance declines in one monkey over 29 days when using static spike sorting. However,
unlike with ReFIT-KF model fitting, initial performance was not surpassed, perhaps be-
cause without incorporating task goals, the method is subject to inaccuracies present in the
initial model fit. In another adaptive study [116], target information was incorporated in
the kinematics used for retraining. Their algorithm was retrained with a weighted aver-
age of decoded trajectory positions and the target positionfor each trial as an estimate of
intended position. In contrast, ReFIT-KF estimates intended velocities based on intuitive
rules applied to cursor position, decoded velocity, and target position.
ReFIT-KF explicitly treats position and velocity differently. The resulting neural pros-
thesis assumes that the monkey is controlling velocity and not position. We structured the
model assuming that velocity intentions evolve smoothly and that the influence of position
is based on the monkey’s internal model of the cursor. Furthermore, we assume that the
control algorithm output and the monkey’s internal belief about cursor position agree. In
reality, there is some mismatch between the control algorithm’s position estimation and
the animal’s internal belief due to inaccuracies in assessing visual information. There are
likely spatial and temporal components to this inaccuracy that are not modeled. The spatial
aspect is an inexact assessment of the last seen location, and the temporal aspect is due to
visual latency. The spatial aspect could be modeled as fixed position uncertainty. To fully
account for the temporal aspect, one could attempt to algorithmically model the animal’s
internal model of cursor dynamics since the last known position of the cursor. In this work,
we chose to start with a simpler model, assuming that this estimation, which is local in
time, is exact. It is possible that augmenting the algorithmto account for the mismatch be-
tween the temporally local forward model and our dynamics model could further increase
control performance. It is important to note that there may be alternative explanations for
the presence of position information in neural output. For example, this information could
be intended cursor position instead of an internal model estimate of current cursor position.
In support of the internal model hypothesis, a recent study suggests that a forward model of
72 CHAPTER 4. REFIT-KF NEURAL PROSTHESIS
cursor position is used during closed loop control [50]. However, further study of the role
of position information in the neural activity during online control is necessary and could
aid in the development of future control algorithms.
In experimental sessions, ReFIT-KF performance was stable until the monkey appeared
to lose interest in the task (e.g. drop in target acquisitionrate, Figure4.3b). This rapid drop
off is presumably the analogue of when a hypothetical human user is finished using their
neural prosthesis. It is expected that performance will drift over time [63], and methods
for continuous adaptation of neural control algorithm parameters may be necessary. In a
previous study [80], information from the output of the control algorithm was used with a
Bayesian approach to adapt parameters throughout sessions to sustain performance. If task
goals were known throughout neural prosthesis use, the intention estimates defined in this
study could be used in conjunction with these parameter adaptation methods. It may be
possible to estimate these task goals based on features of the neural prosthetic output. For
example, if a click or target selection signal is simultaneously decoded [70], indicating user
intended target selection, intended cursor velocities could be estimated for moments prior to
target selection. Additionally, in future work, it will be important to assess how multi-day
learning [38] affects the performance and robustness when control algorithm parameters
are set as described in this work, based on estimated movement intention, versus existing
methods for parameter initialization. Adapting the methods of this work to enable multi-
day learning and plasticity, such as by providing a consistent controller day over day, may
well lead to even better performance over time.
Long-duration, continuous, high-performance operation is central to successful trans-
lation of neural prostheses to human patients [63]. The above performance depended upon
three specific design choices used by both Velocity-KF and ReFIT-KF, in addition to the
two key innovations defining ReFIT-KF. First, we did not employ spike sorting. The goal
of spike sorting is to separate single channels composed of action potentials from many
neurons into multiple channels of spiking activity from individual neurons. This standard
practice can yield high information per electrode, but requires tracking each sorted action
potential shape over days, which has recently been shown to be extremely difficult for many
electrode channels [15,38,101]. To reduce signal instabilities that can result from imper-
fect spike sorting and neuron tracking, we counted the number of threshold crossings per
4.4. DISCUSSION 73
electrode instead of spike sorting [18,36]. Second, the results reported here were acquired
from arrays 19-40 months (monkey L) and 4-21 months (monkey J) after neurosurgical
implantation [15, 18]. The number of highly distinguishable single neurons on anelec-
trode array tends to decrease over time. Yet, remaining multiunit activity often has neural
prosthesis relevant tuning. By employing these older array implants, which had relatively
few clearly distinguishable single units, we confirm that threshold-crossing-based activity,
together with the ReFIT-KF, provides high performance for months and years after array
implantation (shown in Fig.4.2). Finally, we used a single, relatively short 50 ms neural
integration time window with no additional temporal lag, unlike some neural prosthetic de-
signs that explicitly incorporate neural data with longer histories and additional lags (e.g.,
multiple 100 ms time bin [38]; multiple 50 ms time bins with history as far back as 1
sec [57,122]). This choice was based on experiments with humans using anonline pros-
thetic simulator [24] and on subsequent neural control experiments with monkeys. Both
indicated that shorter time bins are preferable due to reduced closed-loop feedback time.
The sustained performance and robustness of these ReFIT-KF neural prosthetic experi-
ments demonstrate the potential to provide functional restoration for patients with a limited
ability to move and act upon the world due to neurological injury and disease. Although
descending pathways are compromised, motor cortex may be largely intact, enabling this
class of technology [56,57,69,70]. In recent years, brain interface technologies employing
a variety of signal sources, such as intra-cortical arrays described here, electroencephalog-
raphy (EEG) [82], and electrocorticography (ECoG) [106], have been developed. The
neural prostheses research community continues to create options for disabled individuals
and to assess relative risk and benefit [44]. In this report, we have investigated the princi-
pled design of closed loop neural control algorithms, resulting in the development of the
ReFIT-KF and demonstrations of a significant advance in performance and robustness. This
algorithm, closed loop control perspective, and system design methodology may be applied
to other neural prosthetic domains with the potential to considerably increase benefit and
the clinical viability prostheses.
74 CHAPTER 4. REFIT-KF NEURAL PROSTHESIS
4.5 Methods
All procedures and experiments were approved by the Stanford University Institutional
Animal Care and Use Committee (IACUC). Experiments were conducted with adult male
rhesus macaques (L and J), implanted with 96 electrode Utah arrays (Blackrock Microsys-
tems Inc., Salt Lake City, UT) using standard neurosurgical techniques [102]. Monkeys L
and J were implanted 19-33 months and 4-14 months prior to theexperiments. Electrode
arrays were implanted in the dorsal aspect of premotor cortex (PMd) and primary motor
cortex (M1), as estimated visually from local anatomical landmarks.
The monkeys were trained to make point-to-point reaches in a2D plane with a virtual
cursor controlled by the contralateral arm or by a neural decoder [24]. The virtual cursor
and targets were presented in a 3D environment (MSMS, MDDF, USC, Los Angeles, CA).
Hand position data were measured with an infrared reflectivebead tracking system (Po-
laris, Northern Digital, Ontario, Canada). Spike counts were collected by applying a single
negative threshold, set to 4.5 x root mean square of the spikeband of each neural channel.
Behavioral control and neural decode were run on separate PCs using the Simulink/xPC
platform (Mathworks, Natick, MA) with communication latencies of 3 ms. This system
enabled millisecond-timing precision for all computations. Neural data were initially pro-
cessed by the Cerebus recording system (Blackrock Microsystems Inc., Salt Lake City, UT)
and were available to the behavioral control system within 5± 1 ms. Visual presentation
was provided via two LCD monitors with refresh rates at 120 Hz,yielding frame updates
within 7 ± 4 ms. Two mirrors visually fused the displays into a single 3Dpercept for the
user, creating a Wheatstone stereograph (see Fig.3.2).
Over 180 experimental sessions with monkey L, and over 100 with monkey J, were
conducted. Central results were replicated multiple days ineach monkey, employing a
within-day A-B-A block structure trial design to highlight algorithmic impact and thereby
quantify performance and robustness.
4.6. ACKNOWLEDGEMENTS 75
4.6 Acknowledgements
We thank S. Stavisky for data collection assistance, J. Aguayo, M. Risch, & S. Kang for
surgical assistance & veterinary care, D. Haven & B. Oskotskyfor IT support, S. Eisensee,
B. Davis, & E. Castaneda for administrative assistance, and P.Ortega for mathematical
insight.
Chapter 5
Task Optimization
This work is as of yet unpublished. The experiments were designed with and data were
collected with the assistance of Joline Fan.
Chapters3 and4 detailed the development leading to a high performance neural pros-
thesis. This chapter builds upon those results, using the ReFIT-KF decoder on keyboard-
like tasks with the aim to measure achievable bitrate. The findings in this chapter are
unpublished at the time of this writing. The data shown here was collected in conjunction
with Joline M Fan in 2010.
5.1 Introduction
Communication neural prostheses can be used to transmit meaningful information by meaus-
ing signals from the brain. These could be used to control a mouse-like cursor, as shown
in Chapter4, but could also be used to control a keyboard for typing. Prior work with
non-penetrating signal sources have demonstrated sustained bitrates of up to 1.2 bps. Elec-
troencephalography [90] (EEG) and electrocorticography [105] (ECoG) can be used as
the signal source for neural prosthetics. With EEG bitratesof 0.13 [72] to 1.02 bps [5]
have been achieved and 0.38 [73] to 1.15 bps [8] with ECoG. A study with intracortical
electrodes demonstrated a peak channel capcity of 6.5 bps [102] for short bursts of time,
however these were not actualized bitrates. This prior study achieved this peak channel
capcity by optimizing the task parameters to enable high data throughput. In a similar
76
5.2. METHODS 77
fashion, the task parameters for a keyboard-like task must also be optimized to maximize
the bitrate achievable. Maximizing bitrate is desirable because it will enable the highest
communication rate for a given level of neural prosthesis control. In this study, we opti-
mized the performance of two keyboard-like tasks, a grid task and a radial keyboard task
and exceeded 3 bps in both tasks.
5.2 Methods
Two keyboard-like tasks were designed and tested with two rhesus macaques in an exper-
imental apparatus (Fi5.1) that mapped neural threshold crossing activity (at−4.5× RMS
of channel [18]) into two-dimensional cursor velocity using the ReFIT-KF neural decoding
algorithm (Ch4). [47] The first task, called the grid task, uniformly divided the workspace
into square acquisition regions each containing a potential target in yellow as shown in
Fig 5.2a. On a given trial, a random target was prompted by changing its color to green.
The monkeys task was to move the cursor into the acquisition region and hold over it for a
specified time, receiving a liquid reward if successful. An incorrect selection occurred if the
cursor dwelled on any of the yellow non-prompted targets forthe specified hold time. After
a selection, the next trial started immediately and anotherrandom target was prompted.
The second task had targets arranged circularly, each equidistant from the center of
the workspace (Fig5.2b) in a design inspired from similar human-computer interaction
input systems such as pie menus. [59, 89] In this radial task, selections were made by
moving the cursor from the center of the screen into an acquisition region in the periphery
without a requisite hold time. Immediately after selection, the cursor was reset to the center,
beginning the next trial.
Both the grid and radial tasks are examples of symmetric communication channels.
They can be used to transmit one of many symbols (targets) to the receiver and relay in-
formation. As with all real-world symmetric communicationchannels, the symbols trans-
mitted are not known to be correct and have some probability of error. The maximum
information communicable over the channel is a function of this error rate. The error rate
(along with the symbol transmission rate) can be used to calculate a theoretical channel
capacity: the highest possible bitrate given multi-symbolchannel codes and infinite time.
78 CHAPTER 5. TASK OPTIMIZATION
Figure 5.1: Experimental setup. Monkey was placed in front of a Wheatstone stereograph[133] such that hand was not visible and presented with a virtual environment within whichto interact. The monkey’s objective was to acquire the target lit in green. Neural data wasrecorded simultaneously during the task. Under neural prosthesis operation, the movementof the cursor was determined by the decoder.
5.2. METHODS 79
a b
Figure 5.2: Task layouts.a Grid task layout in a 25 target configuration. The acceptancewindows are adjacent and non-overlapping.b Radial task layout for 8 targets. In theradial task, there were regions that were left as gaps in the workspace to reduce inadvertentselection. In both tasks, the dotted lines that are not shownto the monkey outline theacceptance region for a particular target.
Channel codes are a means of transmitting information using multiple symbols selected in
a manner appropriate to the error rate so that despite symboltransmission errors, the prob-
ability of message loss is small. Although channel capacityis a very useful measure, in this
study, we chose instead to focus on achieved bitrate. Practically, in a clinical setting, no
subject can take advantage of sophisticated multi-symbol channel codes to transmit infor-
mation in real time as a computer would. The length of channelcodes would be limited to
single symbols, akin to a traditional keyboard. We thus limited the length of channel codes
used in this study to a single symbol as well. This created a problem for error detection,
for unless the subject was prompted for the symbol to transmit, there would be no way to
determine if a received symbol was correct. Since the monkeys’ participation in the task
required prompting of targets, we could determine empirically whether the received sym-
bol was correct. The only way to signal an incorrect transmission in a symmetric channel
is to transmit yet another symbol (e.g. the delete key on the keyboard). Thus, all incorrect
selections were required to be corrected by subsequent successful trials and were not incor-
porated into the final bitrate. The formula for calculating bitrate in the grid and radial tasks
80 CHAPTER 5. TASK OPTIMIZATION
is shown in the below equation.
B =log2(N)(Sc−Si)
t(5.1)
whereN is the total number of targets on the screen,Sc is the number of correct selections,
Si is the number of incorrect selections,t is the time elapsed. Inter-trial time and voluntary
breaks by the monkey were counted as part of the elapsed time.If the quantity(Sc−Si)< 0
then the bitrate is set to a floor at 0 bps, since bitrate cannotbe negative. Note that trials
where a selection was not made (i.e. the cursor was out of the workspace) were not counted
in Si since no symbol was transmitted, but were included in the elapsed time and slowed
overall bitrate. Equation5.1 is a conservative measure of evaluating bitrate, as a neural
prosthesis achieving 50% success rate would have an information rate of 0 bps. Despite
this harsh penalty for errors, this approach more accurately measures the clinical utility of a
neural prosthesis, as transmitting text involves similar challenges and penalties when used
in the real-world. Trials in which the subject made no selection and timed out were not
counted as incorrect selections as no symbol was transmitted, however the time spent was
included in the elapsed time for that block.
In both tasks, each trial prompted a single target out of manypotential choices. There-
fore, each selection conveyed information equal to the binary logarithm of the number of
targets on the screen. Viewing the neural prosthesis as a communication channel, the infor-
mation transmitted over timea function of several task parametersrepresents the throughput
of the system. With the aim of maximizing throughput, the task parameters must be set
appropriately. A trial can convey more information by increasing the number of targets,
but this raises the tasks difficulty and lowers the success rate. Thus, the number of targets
was an adjustable parameter in both tasks. In addition, eachtask had a second, unique,
adjustable parameter. In the case of the grid task, this parameter was the hold time required
to select a target. A long hold time would slow the overall rate of target selection whereas a
short hold time would lead to more inadvertent selections while navigating to the prompted
target. In the radial task, the distance to the targets was the adjusted parameter: placing
targets far from the center would slow the selection rate, while placing them close to the
center would increase the error rate. For each task, we foundthe optimal task parameters
5.3. RESULTS 81
that yielded high information throughput, as measured in bits per second (bps), striking the
best balance between success and selection rates.
5.3 Results
As each subject may have had differing cursor control ability, this two-dimensional param-
eter optimization was evaluated empirically for both monkeys since it could not be easily
determined by offline analysis. On a given day for a specific task, different pairs of task
parameters were tested in randomly ordered blocks of continuous and uninterrupted trials.
Each block was approximately 200 trials except when the taskwas too difficult and ter-
minated early. The randomization was repeated for several days to account for possible
ordering biases. Example histograms of the time to acquire the target for a pair of task
parameters for the grid task are shown in Figs5.3a/5.3d and likewise for the radial task in
Figs5.4a/5.4d.
Adjusting the task parameters significantly impacted the success and selection ratesand
thus the resulting information throughput. Figs5.3b/5.3e and Figs5.4b/5.4e show bitrate
heat maps of the optimization sweep for both monkeys on the grid task and the radial task,
respectively. Each point represents a pair of task parameters. For both monkeys across
both tasks the average throughput using the optimal task parameters was at least 3 bps.
Both monkeys had similar optimal parameters of 25 targets and450 ms hold time for the
grid task. For the radial task, the optimal number of targetswas 8 for both monkeys, but the
optimal target distance differed (Monkey J: 9 cm, Monkey L: 7cm). This slight difference
for the radial task represented a decrease of about 1 bps fromeach monkeys maximum,
highlighting the importance of sweeping task parameters tofind high throughput conditions
for each subject. Additionally, since bitrate falls steeply with increasing error rate, these
optimal parameters corresponded to high success rate conditions. These tended to be less
frustrating for the subject, facilitating longer sessionsdue to ease of use.
82 CHAPTER 5. TASK OPTIMIZATION
0 1 20
5
10
2+
Target Acquire Time (s)
Per
cent
of T
rials
0 9 25 49 64
250
450
650
850
Number of Targets
Hol
d T
ime
(ms)
0 30 60 90 120
Time (min)
0
1
2
3
4
Info
rmat
ion
Rat
e (b
its/s
ec)
0
10
20
Tot
al In
form
atio
n (k
bits
)
0 1 20
5
10
2+
Target Acquire Time (s)
Per
cent
of T
rials
0 9 25 49 64
250
450
650
850
Number of Targets
Hol
d T
ime
(ms)
0 30 60 90 120
Time (min)
0
1
2
3
4
Info
rmat
ion
Rat
e (b
its/s
ec)
0
10
20
Tot
al In
form
atio
n (k
bits
)
Monkey J
a b c
Monkey L
d e f
0 3bps
Figure 5.3: Experimental data of grid task. Monkey J: a-c. Monkey L: d-f. a Acquire timehistogram of grid task with task parameters of 25 targets and450 ms hold time.b Bitrateheatmap of task parameters swept for grid task. Each point inblack represents a tested pairof task parameters.c Information plot of sustained performance across a day for grid task.The gray line plots the average bitrate as a function of time and the black line representsthe cumulative information transmitted over the duration of the experiment.d Acquire timehistogram of grid task with 25 targets and 450 ms hold time.e Bitrate heatmap as in b.fInformation plot as in c.
5.3. RESULTS 83
0 1 20
5
10
15
2+
Target Acquire Time (s)
Per
cent
of T
rials
0 2 4 8 12 16
5
7
9
11
13
Number of Targets
Dis
tanc
e (c
m)
0 30 60 90 120
Time (min)
0
1
2
3
4
Info
rmat
ion
Rat
e (b
its/s
ec)
0
10
20
Tot
al In
form
atio
n (k
bits
)
0 1 20
5
10
15
2+
Target Acquire Time (s)
Per
cent
of T
rials
0 2 4 8 12 16
5
7
9
11
Number of Targets
Dis
tanc
e (c
m)
0 30 60 90 120
Time (min)
0
1
2
3
4
Info
rmat
ion
Rat
e (b
its/s
ec)
0
10
20
Tot
al In
form
atio
n (k
bits
)
Monkey J
a b c
Monkey L
d e f0 3bps
Figure 5.4: Experimental data of radial task. Monkey J: a-c.Monkey L: d-f. a Acquiretime histogram of radial task with task parameters of 8 targets and 9 cm distance.b Bitrateheatmap of task parameters swept for radial task. Each pointin black represents a testedpair of task parameters.c Information plot of sustained performance across a day forradial task. The gray line plots the average bitrate as a function of time and the blackline represents the cumulative information transmitted over the duration of the experiment.d Acquire time histogram of radial task with 8 targets and 7 cm distance.eBitrate heatmapas in b.f Information plot as in c.
84 CHAPTER 5. TASK OPTIMIZATION
Table 5.1: Acceptance region size as a function of targets onthe screen for the grid task
# targets Region (cm)9 825 4.836 449 3.42864 3
300 400 500 600 700 800 9000
0.5
1
1.5
2
2.5
3
3.5
4varying target number
hold time (ms)
stre
amB
itrat
e
9 targets25 targets49 targets64 targets
300 400 500 600 700 800 900-0.5
0
0.5
1
1.5
2
2.5
3
3.5varying target number
hold time (ms)
stre
amB
itrat
e
9 targets25 targets49 targets64 targets
a b
Figure 5.5: Bitrate plots for the grid task when the number of targets was varied.a MonkeyJ b Monkey L
5.3.1 Grid Task
The workspace for the grid task (Fig5.2a) was approximately a 20 cm square. The accep-
tance regions were set such that there were no gaps between targets. The more targets on
the screen, the smaller the acceptance region per target. The table below lists the acceptance
window size for varying number of targets.
Figs 5.5 and5.6 are the 2D traces of the bitrate data shown in the bitrate heatmap in
Figs5.3b/5.3e. In Fig5.5 the number of targets was varied, and in Fig5.6, the hold time
was varied. The error bars are bootstrapped standard deviations.
The data for the grid task consisted of four full randomized blocks of trials of all task pa-
rameter pairs, totalling 20716 trials for Monkey J. Blocks 1-3 were from datasets J100913-
J100917 and block 4 is from datasets J100928-J100930. For Monkey L, 17677 trials were
collected. Blocks 1-2 are from datsets L100913-L100917, andblocks 3-4 from L100920,
L100929, and L100930.
5.3. RESULTS 85
10 20 30 40 50 60 700
0.5
1
1.5
2
2.5
3
3.5
4varying hold times
number of Targets
stre
amB
itrat
e
250mm450mm650mm850mm
10 20 30 40 50 60 70-0.5
0
0.5
1
1.5
2
2.5
3
3.5varying hold times
number of Targets
stre
amB
itrat
e
250mm450mm650mm850mm
a b
Figure 5.6: Bitrate plots for the grid task when the hold time was varied.a Monkey JbMonkey L
50 60 70 80 90 100 110 120 130-0.5
0
0.5
1
1.5
2
2.5
3
3.5varying target number
radial Distance (mm)
stre
amB
itrat
e
2 targets4 targets8 targets12 targets16 targets
50 60 70 80 90 100 110-1
-0.5
0
0.5
1
1.5
2
2.5
3
3.5varying target number
radial Distance (mm)
stre
amB
itrat
e
2 targets4 targets8 targets12 targets16 targets
a b
Figure 5.7: Bitrate plots for the radial task when the number of targets was varied.aMonkey Jb Monkey L
5.3.2 Radial Task
In the radial task, there were fixed gaps in the acquisition region as depicted in Fig5.2b.
These gaps reduced the error rate of the task by providing space where no selections would
be made.
Figs5.7and5.8are the 2D traces of the bitrate data shown in the bitrate heatmap in Fig
5.4b/5.4e. In Fig5.7plot the number of targets was varied, and in Fig5.8, the distance to
the targets was varied. The error bars are bootstrapped standard deviations.
The data for the radial task were collected from datasets J100831-J100903 and J100906
for Monkey J, totalling 16754 trials. For Monkey L, the bitrate sweep are from datasets
L100901-L100903 and L100906-100907, totalling 14684 trials.
86 CHAPTER 5. TASK OPTIMIZATION
2 4 6 8 10 12 14 16 18 20-0.5
0
0.5
1
1.5
2
2.5
3
3.5varying radius
number of Targets
stre
amB
itrat
e
50mm70mm90mm110mm130mm
2 4 6 8 10 12 14 16 18 20-1
-0.5
0
0.5
1
1.5
2
2.5
3
3.5varying radius
number of Targets
stre
amB
itrat
e
50mm70mm90mm110mm
a b
Figure 5.8: Bitrate plots for the radial task when the distance to the targets was varied.aMonkey Jb Monkey L
5.3.3 Sustained intraday performance
Once the optimal parameters were found, we then evaluated the sustained performance of
the neural prosthesis by testing continuous, uninterrupted sessions. These sessions were
started at the beginning of an experimental day and run cuntil the subject was satiated. Figs
5.3c/5.3f plot the performance of the grid task for both monkeys and Figures 3c/3f plot the
session performance of the optimal parameters for each tasktype. Note that performance
is locally constant, at or above 3 bps, but trends downward due to decreasing motivation.
The monkeys accumulated more reward over the course of the experimental session until
they lost interest in the task and abruptly stopped.
5.4 Discussion
The experiments with these two keyboard-like tasks demonstrate that the ReFIT-KF de-
coder can be used to communicate information at higher ratesthan previously reported
with any neural prosthesis signal source. Rates of 3-4 bps were achieved, but were only
attainable after the task parameters were properly tuned for a specific subject’s level of
control. This suggests that an important aspect in the clinical deployment of neural pros-
theses is the proper optimization of each patient’s task to maximize their performance. It is
assumed, as in the case with the monkeys, that an easier to usetask leads to less frustration
and greater willingness to use the system.
Chapter 6
Sustained performance & typing
This chapter is unpublished. The data was collected with theassistance of Jonathan Kao,
Sergey Stavisky, and Joline Fan.
6.1 Abstract
Neural prostheses translate neural activity into control signals for guiding assistive devices
such as robotic arms and computer cursors, and strive to improve quality of life by pro-
viding greater interaction with the world for people with movement disabilities. Despite
encouraging proof-of-concept demonstrations, [12,32,40,57,58,88,94,102,124,130] sig-
nificant barriers to clinical translation remain. Several reports have described limitations to
adoption of intracortical neural prostheses, highlighting issues of performance [44,52,98]
and robustness [13, 63, 84] as well as the need for daily retraining. [52] Recent studies
from an ongoing clinical trial have shown that intracortical electrodes are viable for at least
five years, although signal quality, and resulting neural prosthesis performance, degrades
over time. [57,58,117] Here we present the results from monkeys using two and four year
old multielectrode arrays. We demonstrate sustained high performance with communica-
tion rates up to 4.5 bits per second during hours-long experiments. This was demonstrated
across weeks, and up to a year in one monkey, without retraining using a keyboard-like,
continuously controlled cursor task with the ReFIT algorithm. [48] This is the first demon-
stration of sustained online control for a year. Additionally, when used to type text, we
87
88 CHAPTER 6. SUSTAINED PERFORMANCE & TYPING
show information throughput exceeding 10 words per minute,the highest reported to date.
Finally, we show that the reason this neural prosthesis can transmit meaningful informa-
tion for many months without retraining is due to the relative stability of the neural signals
recorded and the consistency of the animal’s control strategy with the decoder. These
results suggest that long-term chronically implanted neural prostheses may help disabled
individuals successfully communicate text for hours a day,and do so for weeks and months,
without retraining.
6.2 Introduction
Robust performance is an important factor of neural prostheses and stems from having
stable neural signals. Prior studies have demonstrated thestability of single units over
time [17, 27, 123] and reports of stable decoding both offline [4, 18, 110] and online [38]
have been published, in addition to recent preliminary findings. [33, 93] However, there
have been no reports of robust online control consistent with clinically viable performance.
Robustness is important on at least two timescales: within a day (intraday) and across days
(interday). Neural prosthesis performance should be sustainable on real-world tasks for
hours at a time during a day and minimize or eliminate training time across days. This is
challenging because the number of single units recorded from intracortical arrays tend to
decrease over time. [57,117]
6.3 Results
To test our neural prosthesis across various ages of array conditions, we selected two rhe-
sus macaques implanted with multielectrode arrays two years (Monkey J) and four years
(Monkey L) prior to experiments. It was important to use aging multielectrode arrays in an
attempt to emulate the clinical scenario of chronically implanted patients because there is
currently no known method of predicting, or accelerating, neural signal decline. Monkey
J’s two arrays in M1 and PMd had many arm-movement modulated channels with substan-
tial tuning in all directions of the 2D task. Monkey L’s single array in M1/PMd had fewer
6.3. RESULTS 89
0
4In
form
atio
n R
ate
(b
its/s
ec)
0
500
To
tal In
form
atio
n (
kb
its)
1PM
1
1PM
2
12PM
3 4
1PM
5 6
12PM
7
12PM
8
12PM
9
1PM
10 11
2PM
12
2PM
13
7PM
14Day
0
4
Info
rma
tio
n R
ate
(b
its/s
ec)
0
300
To
tal In
form
atio
n (
kb
its)
4PM
1
3PM
2
3PM
3
2PM
4
2PM
5
3PM
6
3PM
7
4PM
8
3PM
9
3PM
10
3PM
11
2PM
12
6PM
13
7PM
14Day
a
b
Figure 6.1: Decoding without retraining.a Daily performance plot of 14 day long exper-iment with Monkey J. The gray line plots the instantaneous, smoothed information rateacross time. The black line plots the cumulative information transmitted across the experi-ment. The x-axis indicates the time the experiment was run, each tick mark notes an hourin the day. b Similar performance plot for Monkey L. Plots are from datasets J110926-J111014 & L110912-L110925.
arm-movement modulated channels and may mimic a patient with deteriorating neural sig-
nal quality. We first confirmed that the neural prosthesis wasstable within a day for a week
with decoder retraining but without additional hand-controlled sessions. We then evaluated
stability without decoder retraining sessions. Fig6.1 plots the results of this 14 day ex-
periment with both monkeys performing a keyboard-like neural prosthesis task. Decoders
were built using threshold crossings from all available channels under the ReFIT-KF [48]
algorithm. Using this task, an information rate was calculated based on the rate of target
selection and net successful trials. By conservative information-transmission measures,
Monkey J achieved an average rate of 3.4 bits per second (bps)and Monkey L averaged
2.6 bps over the course of the two consecutive weeks. For several minutes during the ex-
periments, Monkey J achieved up to 4.5 bps and Monkey L up to 4 bps. This performance
represents the highest achieved and sustained bitrate witha neural prosthesis.
Throughout this two week period, no additional training time was required other than
90 CHAPTER 6. SUSTAINED PERFORMANCE & TYPING
the initial decoder training on the first day. Monkey J’s signal quality was sufficiently
high that a single decoder was held constant daily for over a month during days 2-32.
This is the first report of daily, sustained, high performance without any intervention for a
month. Because of his poorer signal quality, a single decodercould not be held constant
for Monkey L, but instead he required seven different decoders over the two week period.
However, none of the six subsequently built decoders required any subject training time.
Additional functional decoders were automatically generated using the last hour of data of
each experimental day and were available if necessary, resulting in a growing collection
of available decoders. On a given experimental day, one of these, usually the one used
successfully the previous day, was selected within the first5 minutes of the experiment
and held for the duration of that day. A clinical system mightfunction in a similar way,
where a number of decoders could be quickly evaluated beforesettling on one for use in
that session. The results from Monkey J demonstrate that a single decoder can sustain
performance for at least a month, and the results from MonkeyL demonstrate that despite
diminished signal quality, high performance can be sustained for weeks without retraining
sessions.
Although bitrate is an accurate measure of performance, it can be difficult to appreci-
ate its meaning without a more practical context. For communication neural prostheses, a
nominally clinically relevant measure is typing speed in words per minute (wpm) or char-
acters per minute (cpm). To evaluate typing speed, we modified the grid task to simulate
the transmission of words and sentences. The monkeys were tasked to select targets in a
specific order that corresponded to the spelling of words as if the grid were a keyboard. The
task was interactive such that incorrect character selections required the subsequent selec-
tion of the delete key before resuming the text. In this manner, an approximation of the
real-world performance of the system could be measured. Monkey J achieved a peak speed
of 13 wpm and averaged 50 cpm over three hours in a single session (Fig. 6.2a & 6.2b),
completing an 8000 character passage during that time. On the same passage, Monkey
L achieved a peak speed of 10 wpm and averaged 36 cpm (Figs6.2c & 6.2d), completing
two-thirds of the passage in about two hours. This is the highest sustained neural prosthesis
performance on a real-world task. The monkeys are not cognizant of the words they are
constructing; instead, success here is the simulation of the pattern of key selection and task
6.3. RESULTS 91
0 60 120 180Time (min)
0
20
40
60
Typ
ing
ra
te(C
ha
racte
rs p
er
min
)0 5 10
0
10
20
Words per min
Pe
rce
nt o
f T
ria
ls
0 60 120 180Time (min)
0
20
40
60
Typ
ing
ra
te(C
ha
racte
rs p
er
min
)
0 5 100
10
20
Words per min
Pe
rce
nt o
f T
ria
ls
a b
c d
Figure 6.2: Typing task.a Plot of instantaneous, smoothed typing rate across time forMonkey J while typing an article. Gaps in trace represent times where the monkey pausedand was not typing.b Histogram of words per minute for Monkey J, same data as in a.c &d Same as in subpanels a & b, but for Monkey L. Typing data from from datasets J111021& L111103.
difficulty which a human subject would face. Moreover, the extent to which the cognitive
load of concurrent sentence formation and letter selectionwould affect performance cannot
be measured here. By combining the findings from Figs6.1& 6.2, on day 32 of the month-
long experiment, Monkey J achieved 10 wpm on a typing task using the same decoder held
constant in Fig6.1a. These findings together demonstrate that a neural prosthesis can be
both robust over weeks and convey meaningful information ata high rate.
These results encouraged us to explore the stability of decoders over even longer peri-
ods of time. Investigating the limits to sustained, robust performance may have implications
regarding the long-term utility of neural prostheses. Since the daily experimental testing
shown in Fig6.1were untenable for months on end, we evaluated long-term robustness in
two fashions: a longitudinal study and a historical cross-sectional analysis. First, in the
longitudinal study, performance of the same decoder used with Monkey J in Fig6.1a was
evaluated over a period of a year, shown in Fig6.3a. This study, performed exclusively in
92 CHAPTER 6. SUSTAINED PERFORMANCE & TYPING
Monkey J since only he could sustain high performance of a single decoder for multiple
months, is the first reported demonstration of high performance using a static decoder sus-
tained for a year. The regressed line for this model finds thatperformance declines at a rate
of half a bps over the course of a year, suggesting that the model may be stable across time.
Second, in the historical cross-sectional analysis, the performance of several decoders cre-
ated between 3-14 months prior were evaluated over contiguous experimental days. This
was done with both monkeys, summarized by Figures6.3b & 6.3c. The results of the
historical cross-sectional analysis support the findings of the longitudinal study, demon-
strating that performance can be sustained over several months but trends down as the age
of the model grows. Performance also becomes more subject todaily changes in neural
tuning, as not every prior decoder or experimental day worksconsistently across a year.
This experiment also suggests, based on the relative steepness of decline of each monkey’s
performance, that older arrays are less robust across months than newer ones, mirroring
the findings of the distribution of relative channel weightsbetween monkeys. Other factors
besides array age may contribute to signal quality such as surgical conditions, subject age,
health, immune status, and activity level; but these cannotbe easily evaluated due to the
limited number of subjects studied. Additionally, the use of two arrays in Monkey J may
have kept his signal quality high across time relative to Monkey L. Taken together, the lon-
gitudinal study and the historical cross-sectional analysis demonstrate for the first time that
a single decoder can maintain usable performance for several months up to a year.
6.4 Discussion
How is such stability is possible, particularly when it has been assumed that decoder perfor-
mance was very short-lived and required daily retraining? [52] To investigate, we analyzed
the tuning directions of highly contributory channels in the decoder. Figs6.4a & 6.4b plot
the tuning curves of the top 5 electrode channels that originally made the greatest contri-
bution to the 6-month-old decoders in both monkeys. Each color represents the tuning of
that channel on various experimental days, with black linesplotting the tuning on the day
the decoder was built. Surprisingly, among the most contributory channels, neural tuning
6.4. DISCUSSION 93
0 4 8 12Time (months)
0
2
4In
form
atio
n r
ate
(b
its/s
ec)
0000 4444 6666 8888 10101010Time (months)
0
2
4
Info
rma
tio
n r
ate
(b
its/s
ec)
00000 33333 66666 1414141414Time (months)
0
2
4
Info
rma
tio
n r
ate
(b
its/s
ec)a b c
Figure 6.3: Historical models.a Longitudinal study showing performance of Monkey Jwith a static decoder over one year. The filled in circles denote performance from the firstmonth of experiments with a single decoder (Fig6.1a). That decoder, held constant, wastested over a year. The solid line was regressed against the performance across a year and isthe equation: Information rate=−0.034·(# months)+3.78bpsb Cross-sectional historicalstudy of decoder performance from previously recorded datain Monkey J. Each column ofpoints represents a single model tested. Each color/symbolrepresents an experimental day.The peformance at time 0 represents the performance of the decoder on that experimentalday. The black line shows the average performance for each model. c Same as in subpanelb, but for Monkey L. Panels b & c from datasets J120117-J120120, L120207-L120210, &L120214.
across six months is relatively stable. The tuning stability of the most contributory chan-
nels is the enabling mechanism by which the decoders shown inFigs6.1& 6.3maintained
their performance over extended periods of time. However, this is not to say there is no
change in tuning across arrays over time. As shown in Figs6.4c & 6.4d, there is a many-
fold difference between the per-channel, contribution-scaled, absolute-versus-net rotation
of tuning. Thus, although there may be shifts in tuning in channels over time, they appear to
be random and tend to largely cancel one another out for the purposes of two-dimensional
decoding. In this fashion, having many channels for the decoder to draw information from
is beneficial, so that the change in tuning of a single channelwould not significantly af-
fect performance. Further, the smaller net rotation observed in Monkey J may explain why
he, with two newer arrays, could sustain performance with a single decoder for an entire
month, while Monkey L, with a single much older array, required several decoders across
two weeks. In support of this idea, Fig6.4e plots the individual contributions of the top
30 channels to the decoder for both monkeys. Because Monkey L’s array has poorer signal
quality and fewer channels to draw from, the decoder is forced to assign greater weights to
94 CHAPTER 6. SUSTAINED PERFORMANCE & TYPING
the most informative channels in Monkey L, whereas no channel in Monkey J is assigned
greater than 3% weight. On average, Monkey L’s decoder draws50% of its information
from about a dozen channels, while Monkey J’s decoder distributes that same weight across
more than twice that many channels (around 30). These relationships are further detailed in
Fig 6.4f, a plot of cumulative contribution as a function of sorted channels which details a
steeper rise of contribution per channel for Monkey L vs Monkey J. It is thus less surprising
that Monkey L’s decoder is less stable because it is more sensitive to fluctuations in tun-
ing of a single important channel. As evidenced with Monkey J, the more tuned channels
available for the decoder to draw from, the more robust its performance.
The robustness of the neural prosthesis for extended periods of time and its high perfor-
mance with regard to both bitrate and real-world metrics like words per minute illustrate the
potential for this class of neural prosthesis to provide a functional communication system
for people with movement disabilities. Although prior studies [12,38] had demonstrated de-
coder stability for a few weeks while tracking hand-selected neurons, this is the first study
to demonstrate that high-performance can be sustained for up to year without interven-
tion or monitoring. Recent years have seen the development ofneural prosthesis systems
employing various interfaces including electroencephalography, electrocorticography, and
the intracortical arrays used here. The neural prosthesis research community continues to
explore these methodologies while assessing their relative risk and benefit. While intracor-
tical arrays have been questioned for their assumed short lifetime and limited performance
advantage, here we have demonstrated their sustained, robust, and high performance for
weeks without retraining using 2-4 year old multielectrodearrays. Constructing reliable
neural prosthesis systems that minimize training time while offering high performance on
real world tasks is crucial to successful translation. The results shown here may help bring
these systems closer to clinical viability.
6.5 Methods Summary
All procedures and experiments were approved by the Stanford University Institutional
Animal Care and Use Committee (IACUC). Experiments were conducted with adult male
6.5. METHODS SUMMARY 95
123 61 110 119 129
75 53 89 76 82
Sum Net0
0.5
Rad
ians
Sum Net0
0.5
Rad
ians
0 30Channel0
0.1
Indi
vidu
alC
ontr
ibut
ion
(%)
0 200Number of channels0
1
Cum
ulat
ive
Con
trib
utio
n (%
)
a
b
c d e f
Figure 6.4: Tuning Analysisa Tuning curves of top 5 contributory channels for a 6 month-old decoder for Monkey J. The number to the left of the axis denotes the channel. Eachcolored curve represents a different experimental day. Theblack curve represents the tuningon the day the decoder was generated (J110722). The straightlines represent the channel’scosine fit preferred direction.b Same tuning curves as in subpanel a, but for Monkey L.The original decoder was from L110805.c Bar graph of contribution-scaled, per-channelrotation in tuning for Monkey J. The first column plots the unsigned weighted summedrotation of all channels while the second plots the signed net rotation. d Same plot as insubpanel e, but for Monkey L.ePlot of individual contribution of top 30 channels for Mon-key J (red) and L (blue).f Plot of cumulative sorted per channel contribution to decoder.All plots derived from same datasets as in Figure6.3.
96 CHAPTER 6. SUSTAINED PERFORMANCE & TYPING
rhesus macaques (L and J), implanted with 96 electrode Utah arrays (Blackrock Microsys-
tems Inc., Salt Lake City, UT) using standard neurosurgical techniques. Electrode arrays
were implanted in the dorsal aspect of premotor cortex (PMd)and primary motor cortex
(M1), as estimated visually from local anatomical landmarks. Monkey L had a single array
implanted, whereas Monkey J had two arrays implanted, one adjacent to the other. Aside
from the intraday stability experiments which used only theM1 array, all data presented
for Monkey J used both arrays. The monkeys were trained to make point-to-point reaches
in a 2D plane with a virtual cursor controlled by the contralateral arm or by a neural de-
coder. The virtual cursor and targets were presented in a 3D environment (MSMS, MDDF,
USC, Los Angeles, CA). Hand position data were measured with aninfrared reflective bead
tracking system (Polaris, Northern Digital, Ontario, Canada). Spike counts were collected
by applying a single negative threshold, set to 4.5× root mean square of the spike band of
each neural channel. [18] Behavioral control and neural decode were run on separate PCs
using the Simulink/xPC platform (Mathworks, Natick, MA) with communication latencies
of 3 ms. This system enabled millisecond-timing precision for all computations. Neural
data were initially processed by the Cerebus recording system (Blackrock Microsystems
Inc., Salt Lake City, UT) and were available to the behavioralcontrol system within 5± 1
ms. Visual presentation was provided via two LCD monitors with refresh rates at 120 Hz,
yielding frame updates within 7± 4 ms. Two mirrors visually fused the displays into a
single 3D percept for the user, creating a Wheatstone stereograph. This animal model was
selected because we felt it most closely mimics the neural state of a patient that would be
employing a BMI in a clinical study. [92]
6.6 Acknowledgments
We thank Mackenzie Mazariegos, John Aguayo, Wendy Kalkus, Erica Morgan, and Clare
Sherman for veterinary care. We thank Sandra Eisensee, Evelyn Castaneda, and Beverly
Davis for administrative support. We thank Boris Oskotsky for information technology
support.
Chapter 7
Hidden Markov model ReFIT-KF
This work is yet unpublished. The systems presented here andthe data shown were created
and gathered in conjunction with Jonathan Kao.
7.1 Introduction
Several neural prostheses have relied solely on the movement of the cursor as the single
control modality. In these tasks, the cursor would move overthe desired target and dwell
over it for a prescribed hold period. Target selection is conveyed at the end of this hold
period. However, this hold time can be undesirable because the duration of hold time can
make up a significant portion of the trial length. This can be up to nearly 50% of the trial
length in some cases. If hold time could be eliminated, the overall throughput of the system
would increase, transmitting more information faster (bitrate) and enabling faster typing
speeds. One common approach to eliminating hold time is to signal target selection using
an alternative decoding scheme, and thus implementing a click signal. Prior reports have
implemented click signals including using naive Bayes (NB) [102] classifiers and linear
discriminant analysis (LDA) [70,117]. These modalities work reasonably well, but the false
positive rate of those decoders was high under certain circumstances. Further, decoding a
click signal is akin to decoding states, and both NB and LDA can decode multiple states,
but provide no structure between states. In both these decoders, state transitions can and
do occur instantly and can be from any one state to another. This can be undesirable when
97
98 CHAPTER 7. HIDDEN MARKOV MODEL REFIT-KF
deployed in the setting of a neural prosthesis because task states often do not switch rapidly
and arbitrarily.
For these reasons, this work utilized an alternative state decoder: a hidden Markov
model (HMM). The advantages of an HMM are that, unlike a NB or LDA, an HMM models
noise of the underlying observations. Modeling noise improves the overall performance
of the system. In addition, it provides flexibility in modeling the number of states and
their transition characteristics. Both of these features were utilized in developing an HMM
state decoder that was run in parallel to the ReFIT-KF. In thisfashion, the ReFIT-KF [49]
controlled the velocity of the cursor while the HMM controlled the state of the click. Using
these two decoders together improved the performance of system as measured by bitrate,
up to a peak rate of 6 bits per second (bps) and a typing rate of 15 words per minute (wpm).
These findings demonstrate that the performance of cursor-controlled neural prosthetics
can be improved by running an HMM state decoder in parallel tothe cursor movement
decoder.
7.2 Decoder Design
All experimental methods were carried out as per Chapter4. The prosthesis’s function was
broken up into two decoders run in parallel. The movement of the cursor was controlled by
the ReFIT-KF as before, however selections were no longer communicated by holding over
the desired target. Instead, the HMM state decoder was run inparallel with the ReFIT-KF,
and handled the task of target selection. As implemented in this study, the HMM’s obser-
vations of the neural data were assumed to follow a multivariate Gaussian distribution, just
as the Kalman filter does. These multivariate Gaussian observations informed the updating
of the hidden state (click vs no click) via an emissions matrix. The hidden state itself was
also given dynamics and had a static transition matrix that was trained off kinematic data.
Tweaking the transition matrix can allow for shaping of the transition states, if desired.
For Monkey J, a two state HMM performed sufficiently well thatthe simple model shown
in Fig 7.2 performed well. However, for Monkey L, this simple model didnot perform
well. The two state model used with Monkey J resulted in too many false positive errors
in Monkey L and would additionally click when Monkey L was taking volitional breaks
7.3. RESULTS 99
move stop
Figure 7.1:Two state transition graph. Graphical model of the transition flow for the twostate HMM used with Monkey J. The system starts in the move state and can stay in thatstate at the next timestep. It can likewise transition to thestop state if there is sufficientinformation from the neural emissions matrix to push it to that state. If the system stays inthe stop state for two timesteps, the HMM notes a click and immediately transitions backto the move state. The state is locked in the move state for a lockout time (around 250 ms),before it is allowed to transition out of the move state again.
from the task. To address these issues, two changes were made. The first was the inclusion
of an idle state, which was trained against neural data collected while the monkey was not
engaged in the task. The second was the break up of the move state into three kinematic
states which were transitioned between based on the magnitude of the velocity of the cursor
in the ReFIT-KF. These two changes together resulted in a system that was controllable for
Monkey L and shown in Fig7.2.
7.3 Results
The HMM was tested in two fashions. The first was by measuring bitrate on a grid task as
in Chapters5 and6. The results of these experiments are shown in Fig7.3 for Monkey J
and Fig7.3 for Monkey L. Note that Monkey J can achieve a maximum bitrateof 6.5 bps
for minutes at a time.
Second, the performance of the system was also evaluated themore real-world typing
task presented in Chapter6. Here again, Monkey J achieved a peak typing rate of 15 wpm.
100 CHAPTER 7. HIDDEN MARKOV MODEL REFIT-KF
idle
slow fast slow stop
Figure 7.2:Four state transition graph. Graphical model of the transition flow for thefour state HMM used with Monkey L. The system starts in the slow state. The system canstay in any state (except stop) for an arbitrary number of timesteps and can transition to theidle state from any state. Transitions from the movement based states (slow 1, fast, slow2) are only allowed to proceed in a stepwise fashion. From theslow state, if the kinematicprofile (magnitude velocity) is above the specified threshold, the system transitions to thefast state. Once the kinematic profile falls below the specified threshold, the state transitionsto the second slow state. The system can transition to the stop state Only from the secondslow state. Upon transition to the stop state, the system communicates a click signal andimmediately returns to the first slow state. As in the two state HMM, there is a lockout timeenforced before a second stop state can be entered.
7.3. RESULTS 101
0 30 60Time (min)
0
1
2
3
4
5
6
7
Info
rmat
ion
Rat
e (b
its/s
ec)
0 1 2 3 4 5 6 70
5
10
Bitrate (bps)P
erce
nt o
f Tim
e
Figure 7.3:Bitrate for Monkey J with HMM ReFIT-KF decoder. Plot of bitrate overtime for two different days(a) and aggregate bitrate histogram(b) for the two state HMMReFIT-KF decoder with Monkey J on a randomized target grid task. The task was a 7x7grid with 49 targets and was taken from datasets J120405 dn J120406.
0 30 60 90 120 150
Time (min)
0
1
2
3
4
Info
rmat
ion
Rat
e (b
its/s
ec)
0 1 2 3 4 5 6 70
5
10
Bitrate (bps)
Per
cent
of T
ime
Figure 7.4:Bitrate for Monkey L with HMM ReFIT-KF decoder. Plot of bitrate overtime for three different days(a) and aggregate bitrate histogram(b) for the four state HMMReFIT-KF decoder with Monkey L on a randomized target grid task. The task was a 5x5grid with 25 targets and was taken from datasets L120426 L120427 and L120430.
102 CHAPTER 7. HIDDEN MARKOV MODEL REFIT-KF
0 30 60 90 120Time (min)
0
20
40
60
80
Typ
ing
rate
(Cha
ract
ers
per
min
)
0 5 10 150
10
20
30
Words per min
Per
cent
of T
ime
Figure 7.5:Typing rate for Monkey J with HMM ReFIT-KF decoder. Plot of typing rateover time(a) and typing rate histogram(b) for the two state HMM ReFIT-KF decoder withMonkey J. The task was a 6x6 grid with 36 targets and was taken from datasets J120409.
0 30 60 90 120Time (min)
0
20
40
60
80
Typ
ing
rate
(Cha
ract
ers
per
min
)
0 5 10 150
10
20
30
Words per min
Per
cent
of T
ime
Figure 7.6:Typing rate for Monkey L with HMM ReFIT-KF decoder. Plot of typingrate over time(a) and typing rate histogram(b) for the four state HMM ReFIT-KF decoderwith Monkey L. The task was a 6x6 grid with 36 targets and was taken from datasetsL120501.
7.4. CONCLUSION 103
7.4 Conclusion
The results from the HMM based decoder show that using a clicksignal can increase the
performance of the system. The combination of the ReFIT-KF controlling the cursor and
the HMM decoding state was effective in distributing roles to the various decoders despite
one input signal. This suggests that additional information may be encoded in the regions
of cortex being recorded from, and that an appropriate decoder can extract this information.
Further, the success of Monkey L’s mixed kinematic-based HMM suggests that decoders
can be used as signal sources for each other, yielding a layered decoding system that ex-
tract information from multiple levels. Together, these findings should provide a higher
performing and more flexible framework for real-time decoding, helping to increase the
clinical viability of neural prostheses.
Chapter 8
Conclusion
The work presented in this dissertation constitutes a systematic development of neural pros-
thetics. The theoretical underpinnings and applicabilityof the animal model are detailed
in the first chapter, setting the stage for understanding howearly developmental work oc-
curs before testing in human subjects. Such a foundation is necessary to demonstrate why
animal work is believed to hold relevance to future human work. The advances made us-
ing animal models is anticipated to also contribute similarly when used in patients. Thus,
using this animal model, an analysis of how online findings further motivates the need
for experiments of this class. This was the subject of the second chapter. From this, it
is understood that online, closed-loop animal experimentation advances are important be-
fore exposing humans to the risks associated with experimentation. Additionally, the same
chapter discussed how the results from offline testing cannot be trusted as they can yield
measurably incorrect findings. From here, the animal model and online, closed-loop per-
spective were used to develop an algorithm which significantly improved the performance
of a two-dimensional cursor under neural prosthetic control. This new algorithm, ReFIT-
KF, was only possible by inspecting and accounting for the system as a whole, subject
and decoder, and adjusting the decoder with an understanding of how the components of
the system interacted. The ReFIT-KF would not be possible without the attention to the
online, closed-loop nature of decoding, and the results area consequence of the perspec-
tive taken as laid out in the foundational work in the prior chapters. Subsequent chapters
then discussed the optimization and applicability of the ReFIT-KF under real-world tasks,
104
105
with an eye for maximizing meaningful and clinically relevant performance. The issue
of robustness was also addressed, demonstrating decoder stability for weeks and up to a
year. Finally, a hybrid decoder was presented using a combined system with the ReFIT-
KF controlling the cursor velocity and an HMM managing the click state. Taken together,
this development represents the progression from concept and theory to implementation,
advancement, evaluation, and refinement.
These results, however, have been limited exclusively to animal studies and need fur-
ther validation. The true test of these algorithms and advancements lies in the clinical
translation when tested in human subjects. Armed with thesefindings, the performance
and robustness of the neural prosthesis developed here appears high enough that the po-
tential benefit to patients may justify the risk of experimentation. The Shenoy lab fully
believes this and has joined part in the BrainGate clinical trial to take that next step. We
are optimistic that the same advances presented in this workwill carry over to the clinical
realm, where the need lies. At the same time, we recognize thechallenges in translation
and aim to iterate between the human and animal models rapidly. Just as Chapter3 found
that offline findings are a poor predictor of online results, so too will there be instances
where the animal model predictions are not in perfect alignment with the observations in
the clinic. However, keeping the animal model on hand as an experimental site for di-
agnosing, evaluating, and correcting the challenges facedduring translation will likely be
of great benefit in addressing the issues faced. It is an open question whether the human
subjects can achieve comparable performance to that demonstrated by the animal subjects.
Although the initial clinical tests will involve two-dimensional cursor control, additional
end effectors such as robotic arms can be slotted in for evaluation. The aim is to leverage
the engineering foundation and advances presented in this dissertation to the clinic, hope-
fully making neural prostheses more viable and improving the quality of life of the millions
of patients living with paralysis.
Appendix A
Publications
The following pages are listed all peer-reviewed publications published during graduate
training.
106
Journal papers
[1] CA Chestek, V Gilja, P Nuyujukian, JD Foster, JM Fan, MT Kaufman, MM Church-
land, Z Rivera-Alvidrez, JP Cunningham, SI Ryu, and KV Shenoy. Long-term stability
of neural prosthetic control signals from silicon corticalarrays in rhesus macaque mo-
tor cortex.Journal of Neural Engineering, 8(4):045005, Aug 2011.
[2] CA Chestek, V Gilja, P Nuyujukian, RJ Kier, F Solzbacher, SI Ryu, RR Harrison, and
KV Shenoy. Hermesc: low-power wireless neural recording system for freely mov-
ing primates.IEEE Transactions on Neural Systems and Rehabilitation Engineering,
17(4):330–338, Aug 2009.
[3] MM Churchland, JP Cunningham, MT Kaufman, JD Foster, P Nuyujukian, SI Ryu,
and KV Shenoy. Neural population dynamics during reaching.Nature, 487:51–56,
Feb 2012.
[4] JP Cunningham, P Nuyujukian, V Gilja, CA Chestek, SI Ryu, and KV Shenoy. A
closed-loop human simulator for investigating the role of feedback-control in brain-
machine interfaces.Journal of Neurophysiolgy, 105:1932–1949, Oct 2010.
[5] S Farshchi, P Nuyujukian, A Pesterev, I Mody, and JW Judy.A tinyos-enabled
mica2-based wireless neural interface.IEEE Transactions on Biomedical Engineer-
ing, 53(7):1416–1424, Jul 2006.
[6] S Farshchi, A Pesterev, P Nuyujukian, E Guenterberg, I Mody, and JW Judy. Em-
bedded neural recording with tinyos-based wireless-enabled processor modules.IEEE
Transactions on Neural Systems and Rehabilitation Engineering, 18(2):134–141, Apr
2010.
107
108 JOURNAL PAPERS
[7] S Farshchi, A Pesterev, P Nuyujukian, I Mody, and JW Judy.Bi-fi: an embedded
sensor/system architecture for remote biological monitoring. IEEE Transactions on
Information Technology in Biomedicine, 11(6):611–618, Nov 2007.
[8] H Gao, RM Walker, P Nuyujukian, KAA Makinwa, KV Shenoy, B Murmann, and
TH Meng. Hermese: A 96-channel full data rate direct neural interface in 0.13 um
cmos.IEEE Journal of Solid-State Circuits, PP(99):1–13, Feb 2012.
[9] V Gilja, CA Chestek, P Nuyujukian, JD Foster, and KV Shenoy.Autonomous head-
mounted electrophysiology systems for freely behaving primates. Current Opinion
Neurobiology, 20(5):676–686, Oct 2010.
[10] V Gilja, P Nuyujukian, C Chestek, JP Cunningham, BM Yu, JM Fan, JC Kao, SI Ryu,
and KV Shenoy. A high-performance neural prosthesis enabled by control algorithm
design.Nature Neuroscience, 15:1752–1757, 2012.
[11] RR Harrison, RJ Kier, CA Chestek, V Gilja, P Nuyujukian, SI Ryu, B Greger,
F Solzbacher, and KV Shenoy. Wireless neural recording withsingle low-power inte-
grated circuit.IEEE Transactions on Neural Systems and Rehabilitation Engineering,
17(4):322–329, Aug 2009.
[12] D Sussillo, P Nuyujukian, JM Fan, JC Kao, SD Stavisky, SIRyu, and KV Shenoy.
A recurrent neural network for closed-loop intracortical brain-machine interface de-
coders.Journal of Neural Engineering, 9(2):026027, Mar 2012.
Conference papers
[1] W Bishop, CA Chestek, V Gilja, P Nuyujukian, SI Ryu, KV Shenoy,and B Yu. Long-
term decoding stability without retraining for intracortical brain computer interface,
Feb 2012.
[2] CA Chestek, JP Cunningham, V Gilja, P Nuyujukian, SI Ryu, and KV Shenoy. Neural
prosthetic systems: current problems and future directions. In IEEE EMBS, pages
3369–3375, Sep 2009.
[3] CA Chestek, V Gilja, P Nuyujukian, SI Ryu, RJ Kier, F Solzbacher, RR Harrison,
and KV Shenoy. Hermesc: Rf wireless low-power neural recording system for freely
behaving primates. InProc. of IEEE ISCAS, pages 1752–1755, May 2008.
[4] J Dethier, V Gilja, P Nuyujukian, S Elassaad, KV Shenoy, and K Boahen. Spiking
neural network decoder for brain-machine interfaces. InProc. of the 5th International
IEEE EMBS Conference on Neural Engineering, pages 396–399, Jun 2011.
[5] J Dethier, P Nuyujukian, C Eliasmith, TC Stewart, SA Elasaad, KV Shenoy, and
KA Boahen. A brain-machine interface operating with a real-time spiking neural
network control algorithm. In J. Shawe-Taylor, R.S. Zemel, P. Bartlett, F.C.N. Pereira,
and K.Q. Weinberger, editors,Advances in Neural Information Processing Systems
24, pages 2213–2221. Dec 2011.
[6] S Farshchi, P Nuyujukian, A Pesterev, I Mody, and JW Judy.A tinyos-based wireless
neural sensing, archiving, and hosting system. InProc. of 2nd IEEE EMBS Neural
Engineering Conference, pages 671–674, Mar 2005.
109
110 CONFERENCE PAPERS
[7] JD Foster, O Freifeld, P Nuyujukian, SI Ryu, MJ Black, and KVShenoy. Combining
wireless neural recording and video capture for the analysis of natural gait. InProc.
of the 5th International IEEE EMBS Conference on Neural Engineering, pages 613–
616, Jun 2011.
[8] JD Foster, P Nuyujukian, O Friefeld, SI Ryu, MJ Black, and KVShenoy. A frame-
work for relating neural activity to freely moving behavior, 2012.
[9] V Gilja, P Nuyujukian, CA Chestek, JP Cunningham, BM Yu, JM Fan, SI Ryu, and
KV Shenoy. A brain machine interface control algorithm designed from a feedback
control perspective, 2012. in press.
[10] V Gilja, P Nuyujukian, CA Chestek, JP Cunningham, BM Yu, SI Ryu, and
KV Shenoy. High-performance continuous neural cursor control enabled by a feed-
back control perspective. InComputational and Systems Neuroscience, Feb 2010.
[11] RR Harrison, RJ Kier, B Greger, F Solzbacher, CA Chestek, V Gilja, P Nuyujukian,
SI Ryu, and KV Shenoy. Wireless neural signal acquisition with single low-power
integrated circuit. InProc. of IEEE ISCAS, pages 1748–1751, May 2008.
[12] P Nuyujukian, Fan JM, V Gilja, PS Kalanithi, CA Chestek, and KV Shenoy. Monkey
models for brain-machine interfaces: the need for maintaining diversity. InProc. of
the 33rd Annual IEEE EMBS Conference, Aug 2011.
[13] P Nuyujukian, JC Kao, Fan JM, SD Stavisky, SI Ryu, and KV Shenoy. A high-
performance, robust brain-machine interface without retraining, Feb 2012.
[14] RM Walker, H Gao, P Nuyujukian, K Mackinwa, KV Shenoy, TH Meng, and B Mur-
mann. A 96-channel full data rate direct neural interface in0.13um cmos. InProc. of
the 2011 Symposium on VLSI Circuits (VLSIC), Jun 2011.
Appendix B
Patents
Two patent filings were made during graduate training. The first is patent US 2011/0224572
A1, application number 12/932,070, filed on Feb 17, 2011, titled “Brain machine interface”
based on the ReFIT-KF decoder presented in Chapter4. The second is a provisional patent
application number 61/701974, filed 9/17/2012, titled “A Brain Machine Interface Utilizing
a Hidden Markov Model State Detector” based on the ReFIT-KF and HMM combination
decoder presented in Chapter7.
111
Bibliography
[1] P. Adler and T. Winograd.Usability: Turning Technologies into Tools. Oxford Press,
New York, 1992.
[2] K. D. Anderson. Targeting recovery: Priorities of the spinal cord injured population.
J. Neurotrauma, 21:1371–1383, 2004.
[3] P. Artemiadis, G. Shakhnarovich, C. Vargas-Irwin, J. Donoghue, and M. J. Black.
Decoding grasp aperture from motor-cortical population activity. In Proc. of the IEEE
EMBS, pages 518–521, 2007.
[4] W Bishop, Cynthia A Chestek, V Gilja, P Nuyujukian, Stephen IRyu, Krishna V
Shenoy, and Byron M Yu. Long-term decoding stability withoutretraining for in-
tracortical brain computer interface. InComputational and Systems Neuroscience,
2012.
[5] B Blankertz, G Dornhege, C Schafer, R Krepki, J Kohlmorgen, K R Muller, V Kunz-
mann, F Losch, and G Curio. Boosting bit rates and error detection for the classifica-
tion of fast-paced motor commands based on single-trial eeganalysis.Neural Systems
and Rehabilitation Engineering, IEEE Transactions on, 11(2):127–131, 2003.
[6] A.E. Brockwell, A.L. Rojas, and R.E. Kass. Recursive Bayesiandecoding of motor
cortical signals by particle filtering.J Neurophysiol, 91(4):1899–1907, 2004.
[7] E.N. Brown, L.M. Frank, D. Tang, M.C. Quirk, and M.A. Wilson. A statistical
paradigm for neural spike train decoding applied to position prediction from the en-
semble firing patterns of rat hippocampal place cells.J Neurosci, 18(18):7411–7425,
1998.
112
BIBLIOGRAPHY 113
[8] P Brunner, A L Ritaccio, J F Emrich, H Bischof, and G Schalk. Rapid communication
with a p300 matrix speller using electrocorticographic signals (ecog).Frontiers in
Neuroscience, 5(00005), 2011.
[9] L. Buchen. Neuroscience: Illuminating the brain.Nature, 465:26–28, 2010.
[10] S K Card, W K English, and B J Burr. Evaluation of Mouse, Rate-Controlled Iso-
metric Joystick, Step Keys, and Text Keys for Text Selectionon a CRT.Ergonomics,
21(8):601–613, 1978.
[11] J. M. Carmena, M. A. Lebedev, C. S. Henriquez, and M. A. Nicolelis. Stable ensemble
performance with single-neuron variability during reaching movements in primates.
J Neurosci, 25(46):10712–10716, 2005.
[12] J.M. Carmena, M.A. Lebedev, R.E. Crist, J.E. O’Doherty, D.M. Santucci, D.F. Dim-
itrov, P.G. Patil, C.S. Henriquez, and M.A.L. Nicolelis. Learning to control a brain-
machine interface for reaching and grasping by primates.PLoS Biology, 1(2):193–
208, Oct 2003.
[13] Jose M Carmena. Becoming bionic.IEEE Spectrum, 49:24–29, March 2012.
[14] S. M. Chase, A. B. Schwartz, and R. E. Kass. Bias, optimal linear estimation, and the
differences between open-loop simulation and closed-loopperformance of spiking-
based brain-computer interface algorithms.Neural Networks, 22:1203–1213, 2009.
[15] C. A. Chestek, J. P. Cunningham, V. Gilja, P. Nuyujukian, S.I. Ryu, and K. V. Shenoy.
Neural prosthetic systems: Current problems and future directions. InProc. of the
Annual Conference of the IEEE (EMBS), 2009.
[16] C. A. Chestek, V. Gilja, P. Nuyujukian, R. Kier, F. Solzbacher, S. I. Ryu, R. R. Har-
rison, and K. V. Shenoy. HermesC: Low-power wireless neural recording system for
freely moving primates.IEEE Trans. in Neural Systems and Rehab. Eng., 17:330–
338, 2009.
[17] C.A Chestek, A.P. Batista, G. Santhanam, B.M. Yu, A. Afshar,J.P. Cunningham,
V. Gilja, S.I. Ryu, M.M. Churchland, and K.V. Shenoy. Single-neuron stability during
114 BIBLIOGRAPHY
repeated reaching in macaque premotor cortex.J Neurosci, 27(40):10742–10750, Oct
2007.
[18] Cynthia A Chestek, Vikash Gilja, Paul Nuyujukian, JustinD Foster, Joline M Fan,
Matthew T Kaufman, Mark M Churchland, Zuley Rivera-Alvidrez,John P Cunning-
ham, Stephen I Ryu, and Krishna V Shenoy. Long-term stabilityof neural prosthetic
control signals from silicon cortical arrays in rhesus macaque motor cortex.J Neural
Eng, 8(4):045005, Aug 2011.
[19] M. M. Churchland, J. P. Cunningham, M. T. Kaufman, S. I. Ryu,and K. V. Shenoy.
Cortical preparatory activity: Representation of movement or first cog in a dynamical
machine?Neuron, 2010.
[20] M. M. Churchland and K. V. Shenoy. Temporal complexity and heterogeneity of
single-neuron activity in premotor and motor cortex.J Neurophysiol, 97:4235–4257,
2007.
[21] M. M. Churchland, B. M. Yu, S. I. Ryu, G. Santhanam, and K. V. Shenoy. Neural
variability in premotor cortex provides a signature of motor preparation.J Neurosci,
26(14):3697–3712, Apr. 2006.
[22] J. P. Cunningham, V. Gilja, S. I. Ryu, and K. V. Shenoy. Methods for estimating neural
firing rates, and their application to brain-machine interfaces.Neural Networks, 2009.
doi:10.1016/j.neunet.2009.02.004. In Press.
[23] J. P. Cunningham, B. M. Yu, V. Gilja, S. I. Ryu, and K. V. Shenoy. Toward optimal
target placement for neural prosthetic devices.J Neurophysiol, 100(6):3445–3457,
2008.
[24] John P Cunningham, Paul Nuyujukian, Vikash Gilja, Cindy AChestek, Stephen I
Ryu, and Krishna V Shenoy. A closed-loop human simulator for investigating the
role of feedback control in brain-machine interfaces.J Neurophysiol, 105(4):1932–
49, Apr 2011.
BIBLIOGRAPHY 115
[25] Z. Danziger, A. Fishbach, and F.A. Mussa-Ivaldi. Learning algorithms for human-
machine interfaces.Biomedical Engineering, IEEE Transactions on, 56(5):1502 –
1511, 2009.
[26] P. Dayan and L.F. Abbott.Theoretical Neuroscience. MIT Press, Cambridge, 2001.
[27] Adam S Dickey, Aaron Suminski, Yali Amit, and Nicholas GHatsopoulos. Single-
unit stability using chronically implanted multielectrode arrays. Journal of Neuro-
physiology, 102(2):1331–1339, August 2009.
[28] I. Diester, M. T. Kaufman, M. Mogri, R. Pashaie, W. Goo, O.Yizhar, C. Ramakrish-
nan, K. Deisseroth, and K. V. Shenoy. An optogenetic toolboxdesigned for primates.
Nature Neurosci, 14:387–397, 2011.
[29] S. A. Douglas, A. E. Kirkpatrick, and I. S. MacKenzie. Testing pointing device per-
formance and user assessment with the iso9241, part 9 standard. In Proc. of the ACM
Conf. on Human Factors in Computing Systems (CHI 99), pages 215–222. ACM, New
York, 1999.
[30] U.T. Eden, L.M. Frank, R. Barbieri, V. Solo, and E. N. Brown.Dynamic analysis
of neural encoding by point process adaptive filtering.Neural Computation, 16:971–
998, 2004.
[31] Editorial. Is this the bionic man?Nature, 442:109, 2006.
[32] C Ethier, E R Oby, M J Bauman, and L E Miller. Restoration of grasp following
paralysis through brain-controlled stimulation of muscles. Nature, 485(7398):368–
71, May 2012.
[33] R Flint, W Zachary, and M Slutzky. Control of a biomimeticbrain machine interface
with local field potentials: Performance and stability of a static decoder over 200
days. InProc. of the 34th Annual IEEE EMBS Conference, page In press, 2012. in
press.
[34] J. D. Foster, O. Freifeld, P. Nuyujukian, S. I. Ryu, M. J. Black, and K. V. Shenoy.
Combining wireless neural recording and video capture for the analysis of natural
116 BIBLIOGRAPHY
gait.Proc. of the 5th International IEEE EMBS Conf on Neural Eng, Cancun, Mexico,
pages 613–616, 2011.
[35] D. W. Franklin and D. M. Wolpert. Specificity of reflex adaptation for task-relevant
variability. J. Neurosci., 28(52):14165–14175, 2008.
[36] G. W. Fraser, S. M. Chase, A. Whitford, and A. B. Schwartz. Control of a brain-
computer interface without spike sorting.J Neural Engineering, 6(5):1–8, Oct 2009.
[37] Gregory J Gage, Kip A Ludwig, Kevin J Otto, Edward L Ionides, and Daryl R Kipke.
Naive coadaptive cortical control.Journal of Neural Engineering, 2(2):52, 2005.
[38] K. Ganguly and J. M. Carmena. Emergence of a stable cortical map for neuropros-
thetic control.PLoS Biology, 7(7), Jul 2009.
[39] K. Ganguly, L. Secundo, G. Ranade, A. Orsborn, E. F. Chang,D. F. Dimitrov, J. D.
Wallis, N. M. Barbaro, R. T. Knight, and J. M. Carmena. Cortical representation of
ipsilateral arm movements in monkey and man.J Neurosci, 29:12948–12956, 2009.
[40] Karunesh Ganguly, Dragan F Dimitrov, Jonathan D Wallis, and Jose M Carmena. Re-
versible large-scale modification of cortical networks during neuroprosthetic control.
Nat Neurosci, 14(5):662–7, May 2011.
[41] Y. Gao, M.J. Black, E. Bienenstock, S. Shoham, and J.P. Donoghue. Probabilistic
inference of hand motion from neural activity in motor cortex. Advances in NIPS, 14,
2002.
[42] A.P. Georgopoulos, A.B. Schwartz, and R.E. Kettner. Neuronal population coding of
movement direction.Science, 233:1416–1419, 1986.
[43] Z. Ghahramani and D. Wolpert. Modular decomposition invisuoumotor learning.
Nature, 386:392–395, 1997.
[44] V. Gilja, C. Chestek, I. Diester, J. M. Henderson, K. Deisseroth, and K. V. Shenoy.
Challenges and opportunities for next-generation intra-cortically based neural pros-
theses.IEEE Trans Biomed Eng, 58(7):1891–9, Jan 2011.
BIBLIOGRAPHY 117
[45] V. Gilja, C. A. Chestek, P. Nuyujukian, J. D. Foster, and K.V. Shenoy. Autonomous
head-mounted electrophysiology systems for freely-behaving primates. Curr Opin
Neurobio, 20:676–686, 2010.
[46] V. Gilja, P. Nuyujukian, C. A. Chestek, J. P. Cunningham, B. M. Yu, S. I. Ryu, and
K. V. Shenoy. A high-performance continuous cortically-controlled prosthesis en-
abled by feedback control design.Program No. 20.6. Neuroscience Meeting Planner.
San Diego, CA: Society for Neuroscience, Online., 2010.
[47] V. Gilja, P. Nuyujukian, C. A. Chestek, J. P. Cunningham, B. M. Yu, S. I. Ryu, and
K. V. Shenoy. High-performance continuous neural control enabled by a feedback
control perspective.Fronteirs in Systems Neuroscience. Conference Abstract: Com-
putational and systems neuroscience (COSYNE), 2010.
[48] V Gilja, P Nuyujukian, Cynthia A Chestek, John P Cunningham, Byron M Yu, Jo-
line M Fan, Stephen I Ryu, and Krishna V Shenoy. A brain machineinterface control
algorithm designed from a feedback control perspective. InProc. of the 34th Annual
IEEE EMBS Conference, page In press, 2012. in press.
[49] Vikash Gilja, Paul Nuyujukian, Cynthia A Chestek, John P Cunningham, Byon M
Yu, Joline M Fan, Jonathan C Kao, Stephen I Ryu, and Krishna V Shenoy. A high-
performance neural prosthesis enabled by control algorithm design. Nature Neuro-
science, in press, 2012.
[50] Matthew D Golub, Byron M Yu, and Steven M Chase. Internal Models Engaged by
Brain-computer Interface Control. In34th Annual International Conference of the
IEEE EMBS, pages 1327–1330, San Diego, CA, 2012.
[51] M. A. Graziano. Where is my arm? the relative role of vision and proprioception in
the neuronal representation of limb position.PNAS, 96:10418–10421, 1999.
[52] A M Green and J F Kalaska. Learning to move machines with the mind. Trends
Neurosci, 34(2):61–75, Feb 2011.
118 BIBLIOGRAPHY
[53] X. Han, X. Qian, J. G. Bernstein, H.-H. Zhou, G. T. Franzesi, P. Stern, R. T. Bron-
son, A. M. Graybiel, R. Desimone, and E. S. Boyden. Millisecond-timescale optical
control of neural dynamics in the nonhuman primate brain.Neuron, 62:191–198,
2009.
[54] N. Hatsopoulos, J. Joshi, and J. G. O’Leary. Decoding continuous and discrete motor
behaviors using motor and premotor cortical ensembles.J Neurophysiol, 92:1165–
1174, 2004.
[55] D. J. Herzfeld and S. A. Beardsley. Improved multi-unit decoding at the brain-
machine interface using population temporal linear filtering. J Neural Engineering,
7(4), 2010.
[56] L. R. Hochberg. Turning thought into action.N Engl J Med, 359(11):1175–1177,
2008.
[57] L. R. Hochberg, M. D. Serruya, G. M. Friehs, J. A. Mukand, M. Saleh, A. H. Caplan,
A. Branner, D. Chen, R. D. Penn, and J. P. Donoghue. Neuronal ensemble control of
prosthetic devices by a human with tetraplegia.Nature, 442:164–171, Jul 2006.
[58] Leigh R Hochberg, Daniel Bacher, Beata Jarosiewicz, Nicolas Y Masse, John D
Simeral, Joern Vogel, Sami Haddadin, Jie Liu, Sydney S Cash, Patrick van der Smagt,
and John P Donoghue. Reach and grasp by people with tetraplegia using a neurally
controlled robotic arm.Nature, 485(7398):372–5, May 2012.
[59] Don Hopkins. The design and implementation of pie menus. Dr. Dobb’s Journal of
Software Tools, 16(12):16–26,94, Dec 1991.
[60] I. S. Howard, J. N. Ingram, and D. M. Wolpert. A modular planar robotic manipu-
landum with end-point torque control.J. Neuroscience Methods, 181(2):199 – 211,
2009.
[61] A. Jackson, J. Mavoori, and E. E. Fetz. Long-term motor cortex plasticity induced by
an electronic neural implant.Nature, 444:56–60, 2006.
BIBLIOGRAPHY 119
[62] H. D. Jellinek and S. K. Card. Powermice and user performance. Proc of SIGCHI
Conf on Human Factors in Computing Systems, pages 213–220, 1990.
[63] Jack W Judy. Neural interfaces for upper-limb prosthesis control: opportunities to
improve long-term reliability.IEEE Pulse, 3(2):57–60, Mar 2012.
[64] J F Kalaska. From intention to action: motor cortex and the control of reaching
movements.Adv Exp Med Biol, 629:139–178, 2009.
[65] R. E. Kalman. A new approach to linear filtering and prediction problems.J. Basic
Engineering, 82:35–45, 1960.
[66] M T Kaufman, M M Churchland, G Santhanam, B M Yu, A Afshar, SI Ryu, and
K V Shenoy. Roles of monkey premotor neuron classes in movement preparation and
execution.J Neurophysiol, 104(2):799–810, Aug 2010.
[67] C.T. Kemere, K.V. Shenoy, and T.H. Meng. Model-based neural decoding of reaching
movements: a maximum likelihood approach.IEEE Transactions on Biomedical
Engineering – Special issue on Brain-Machine Interfaces, 51:925–932, 2004.
[68] S. Kim, J. C. Sanchez, Y. N. Rao, D. Erdogmus, J. M. Carmena, M. A. Lebedev,
M. A. L. Nicolelis, and J. C. Principe. A comparison of optimalmimo linear and
nonlinear models for brain-machine interfaces.J Neural Engineering, 3:145–161,
2006.
[69] S. Kim, J. D. Simeral, L. R. Hochberg, J. P. Donoghue, and M. J. Black. Neural
control of computer cursor velocity by decoding motor cortical spiking activity in
humans with tetraplegia.J Neural Engineering, 5:455–476, 2008.
[70] S. P. Kim, J. D. Simeral, L. R. Hochberg, J. P. Donoghue, G.Friehs, and M. J. Black.
Point-and-click cursor control with an intracortical neural interface system in humans
with tetraplegia.IEEE Trans Neural Syst Rehabil Eng, 19(2):193–203, Apr 2011.
[71] S. Koyama, S. M. Chase, A. S. Whitford, M. Velliste, A. B. Schwartz, and R. E.
Kass. Comparison of brain-computer interface decoding algorithms in open-loop and
closed-loop control.J Computational Neuroscience, 29(1-2):73–87, 2009.
120 BIBLIOGRAPHY
[72] D J Krusienski, E W Sellers, D J McFarland, Vaughan T M, and J R Wolpaw. Toward
enhanced p300 speller performance.Journal of Neuroscience Methods, 167(1):15–
21, 2008.
[73] D J Krusienski and J J Shih. Control of a visual keyboard using an electrocortico-
graphic brain-computer interface.Neurorehabilitation and Neural Repair, 25(4):323–
331, 2011.
[74] J. E. Kulkarni and L. Paninski. State-space decoding ofgoal-directed movements.
IEEE Sig Proc Magazine, 25(1):78–86, 2008.
[75] M. Kumar, A. Paepcke, and T. Winograd. Eyepoint: Practical pointing and selection
using gaze and keyboard.Proc of SIGCHI Conf on Human Factors in Computing
Systems, pages 421–430, 2007.
[76] M. A. Lebedev and M. A. L. Nicolelis. Brain-machine interfaces: past, present, and
future. Trends in Neurosci, 29, 2006.
[77] M.A. Lebedev, J.M. Carmena, J.E. O’Doherty, M. Zacksenhouse, C.S. Henriquez,
J.C. Principe, and M.A.L. Nicolelis. Cortical ensemble adaptation to represent ve-
locity of an artificial actuator controlled by a brain-machine interface.J Neurosci,
25(19):4681–4693, 2005.
[78] E.C. Leuthardt, G. Schalk, J.R. Wolpaw, J.G. Ojemann, andD.W. Moran. A brain-
computer interface using electrocorticographic signals in humans.Journal of Neural
Engineering, 1:63–71, 2004.
[79] Z. Li, J. E. O’Doherty, T. L. Hanson, M. A. Lebedev, C. S. Henriquez, and M. A. L.
Nicolelis. Unscented kalman filter for brain-machine interfaces. PLoS One, 4(7),
2009.
[80] Z Li, J E O’Doherty, M A Lebedev, and M A Nicolelis. Adaptive decoding for
brain-machine interfaces through bayesian parameter updates. Neural Computation,
23(12):3162–3204, Dec 2011.
BIBLIOGRAPHY 121
[81] A. R. Marathe, K. E. Ward, M. Maryampillai, and D. M. Taylor. Are your move-
ment decoding methods appropriate for online control of neuroprosthetics?Soc. for
Neurosci. Abstr., 2009.
[82] D J McFarland, W A Sarnacki, and J R Wolpaw. Electroencephalographic (eeg)
control of three-dimensional movement.J Neural Eng, 7(3):036007–036007, Jun
2010.
[83] H. Miranda, V. Gilja, C. A. Chestek, K. V. Shenoy, and T. H. Meng. HermesD:
A high-rate long-range wireless transmission system for simultaneous multichannel
neural recording applications.IEEE Transactions on Biomedical Circuits and Sys-
tems, 4:181–191, 2010.
[84] Daniel Moran. Evolution of brain-computer interface:action potentials, local field
potentials and electrocorticograms.Curr Opin Neurobiol, 20(6):741–5, Dec 2010.
[85] D.W. Moran and A.B. Schwartz. Motor cortical representation of speed and direction
during reaching.J Neurophysiol., 82(5):2676–2692, 1999.
[86] C. T. Moritz, S. I. Perlmutter, and E. E. Fetz. Direct control of paralysed muscles by
cortical neurons.Nature, 456:639–642, 2008.
[87] G. H. Mulliken, S. Musallam, and R. A. Andersen. Decodingtrajectories from poste-
rior parietal cortex ensembles.J Neurosci, 28(48):12913–12926, Nov 2008.
[88] S. Musallam, B.D. Corneil, B. Greger, H. Scherberger, and R.A. Andersen. Cognitive
control signals for neural prosthetics.Science, 305:258–262, 2004.
[89] William Newman and Robert F Sproull.Principles of Interactive Computer Graphics.
McGraw-Hill, 2 edition, Jun 1979.
[90] Luis Fernando Nicolas-Alonso and Jaime Gomez-Gil. Brain computer interfaces, a
review. Sensors, 12(2):1211–1279, 2012.
122 BIBLIOGRAPHY
[91] P. Nuyujukian, V. Gilja, C. A. Chestek, J. P. Cunningham, J.M. Fan, B. M. Yu, S. I.
Ryu, and K. V. Shenoy. Generalization and robustness of a continuous cortically-
controlled prosthesis enabled by feedback control design.Program No. 20.7. Neuro-
science Meeting Planner. San Diego, CA: Society for Neuroscience, Online., 2010.
[92] Paul Nuyujukian, Joline M Fan, Vikash Gilja, Paul S Kalanithi, Cindy A Chestek,
and Krishna V Shenoy. Monkey models for brain-machine interfaces: the need for
maintaining diversity.Conf Proc IEEE Eng Med Biol Soc, 2011:1301–5, 2011.
[93] Paul Nuyujukian, Jonathan Kao, Joline M Fan, Sergey D Stavisky, Stephen I Ryu,
and Krishna V Shenoy. A high-performance robust brain-machine interface without
retraining. InComputational and Systems Neuroscience, 2012.
[94] Joseph E O’Doherty, Mikhail A Lebedev, Peter J Ifft, Katie Z Zhuang, Solaiman
Shokur, Hannes Bleuler, and Miguel A L Nicolelis. Active tactile exploration using a
brain-machine-brain interface.Nature, 479(7372):228–31, Nov 2011.
[95] L. Paninski, S. Shoham, M.R. Fellows, N.G. Hatsopoulos,and J.P. Donoghue. Su-
perlinear population encoding of dynamic hand trajectory in primary motor cortex.J
Neurosci., 24:8551–8561, 2004.
[96] E. A. Pohlmeyer, E. R. Oby, E. J. Perreault, S. A. Solla, K.L. Kilgore, R. F. Kirsch,
and L. E. Miller. Toward the restoration of hand use to a paralyzed monkey: brain-
controlled functional electrical stimulation of forearm muscles.PLoS One, 4:e5924,
2009.
[97] S. M. Radhakrishnan, S. N. Baker, and A. Jackson. Learninga Novel Myoelectric-
Controlled Interface Task.J Neurophysiol, 100(4):2397–2408, 2008.
[98] S. I. Ryu and K. V. Shenoy. Human cortical prostheses: Lost in translation?Neuro-
surgical Focus, 27, 2009.
[99] P. T. Sadtler, S. I. Ryu, B. M. Yu, and A. P. Batista. High-performance neural pros-
thetic control along instructed paths.Proc. of the 5th International IEEE EMBS Conf
on Neural Eng, Cancun, Mexico, pages 601–604, 2011.
BIBLIOGRAPHY 123
[100] J. C. Sanchez, J. C. Principe, T. Nishida, R. Bashirullah, J. G. Harris, and J. Fortes.
Technology and signal processing for brain-machine interfaces: The need for beyond
the state-of-the-art tools.IEEE Sig Proc Magazine, 25(1):29–40, 2008.
[101] G. Santhanam, M. D. Linderman, V. Gilja, A. Afshar, S. I. Ryu, T. H. Meng, and
K. V. Shenoy. Hermesb: A continuous neural recording systemfor freely behaving
primates.IEEE Transactions in Biomedical Engineering, 54:2037–2050, 2007.
[102] G. Santhanam, S. I. Ryu, B. M. Yu, A. Afshar, and K. V. Shenoy. A high-
performance brain-computer interface.Nature, 442, 2006.
[103] G. Santhanam, M. Sahani, S. I. Ryu, and K. V. Shenoy. An extensible infrastructure
for fully automated spike sorting during online experiments. In Proceedings of the
26th Annual International Conference of the IEEE EMBS, volume 2, pages 4380–
4384, 2004.
[104] D. M. Santucci, J. D. Kralik, M. A. Lebedev, and M. A. L. Nicolelis. Frontal and
parietal cortical ensembles predict single-trial muscle activity during reaching move-
ments in primates.Eur. J. Neurosci, 22:1529–1540, 2005.
[105] G Schalk and E C Leuthardt. Brain-computer interfaces using electrocorticographic
signals.Biomedical Engineering, IEEE Reviews in, 4:140–154, 2011.
[106] G Schalk, K J Miller, N R Anderson, J A Wilson, M D Smyth, JG Ojemann, D W
Moran, J R Wolpaw, and E C Leuthardt. Two-dimensional movement control using
electrocorticographic signals in humans.J Neural Eng, 5(1):75–84, Mar 2008.
[107] A. B. Schwartz. Cortical neural prosthetics.Ann. Rev. Neurosci, 27:487–507, 2004.
[108] A. B. Schwartz, X. T. Cui, D. J. Weber, and D. W. Moran. Brain-controlled inter-
faces: Movement restoration with neural prosthetics.Neuron, 52(1):205 – 220, 2006.
[109] M.D. Serruya, N.G. Hatsopoulos, L. Paninski, M.R. Fellows, and J.P. Donoghue.
Instant neural control of a movement signal.Nature, 416:141–142, 2002.
124 BIBLIOGRAPHY
[110] Mijail Serruya, Nicholas Hatsopoulos, Matthew Fellows, Liam Paninski, and John
Donoghue. Robustness of neuroprosthetic decoding algorithms. Biological Cyber-
netics, 88:219–228, 2003. 10.1007/s00422-002-0374-6.
[111] R Shadmehr and F. A. Mussa-Ivaldi. Adaptive representation of dynamics during
learning of a motor task.J. Neurosci., 14(5):3208–3224, 1994.
[112] R. Shadmehr and S. P. Wise.The Computational Neurobiology of Pointing and
Reaching. MIT Press, Cambridge, 2005.
[113] G. Shakhnarovich, S. P. Kim, and M. J. Black. Nonlinear physically-based models
for decoding motor-cortical population activity. InAdvances in NIPS 18. MIT Press,
Cambridge, MA, 2006.
[114] K.V. Shenoy, D. Meeker, S. Cao, S.A. Kureshi, B. Pesaran,P. Mitra, C.A. Buneo,
A.P. Batista, J.W. Burdick, and R.A. Andersen. Neural prosthetic control signals from
plan activity.NeuroReport, 14:591–596, 2003.
[115] S. Shoham, L.M. Paninski, M.R. Fellows, N.G. Hatsopoulos, J.P. Donoghue, and
R.A. Normann. Statistical encoding model for a primary motorcortical brain-machine
interface.IEEE Transactions on TBME, 52(7):1313–1322, Jul. 2005.
[116] L. Shpigelman, H. Lalazar, and E. Vaadia (2009). Kernel-arma for hand tracking and
brain-machine interfacing during 3D motor control.D. Koller, D. Schuurmans, Y. Ben-
gio, L. Bottou (eds.) Advances in Neural Information Processing Systems, 21:1489—
1496, 2009.
[117] J. D. Simeral, S.-P. Kim, M. J. Black, J. P. Donoghue, andL. R. Hochberg. Neural
control of cursor trajectory and click by a human with tetraplegia 1000 days after
implant of an intracortical microelectrode array.J Neural Eng, 8(2):025027, Apr
2011.
[118] S. J. Sober and P. N. Sabes. Multisensory Integration during Motor Planning.J.
Neurosci., 23(18):6982–6992, 2003.
BIBLIOGRAPHY 125
[119] L. Srinivasan and E. N. Brown. A state-space framework for movement control to
dynamic goals through brain-driven interfaces.IEEE Transactions on Biomedical
Engineering, 54(3):526–535, 2007.
[120] L. Srinivasan, U. T. Eden, S. J. Mitter, and E. N. Brown. General purpose filter
design for neural prosthetic systems.J Neurophysiol, 98(4):2456–75, Oct 2007.
[121] L. Srinivasan, U. T. Eden, A. S. Willsky, and E. N. Brown.A state-space analysis for
reconstruction of goal-directed movements using neural signals.Neural Computation,
18:2465–2494, 2006.
[122] A. J. Suminski, D. C. Tkach, A. H. Fagg, and N. G. Hatsopoulos. Incorporating
feedback from multiple sensory modalities enhances brain-machine interface control.
J Neurosci, 30:16777–16787, 2010.
[123] S Suner, M R Fellows, C Vargas-Irwin, G K Nakata, and J P Donoghue. Reliability
of signals from a chronically implanted, silicon-based electrode array in non-human
primate primary motor cortex.Neural Systems and Rehabilitation Engineering, IEEE
Transactions on, 13(4):524–541, Dec 2005.
[124] D.M. Taylor, S.I. Helms Tillery, and A.B. Schwartz. Direct cortical control of 3D
neuroprosthetic devices.Science, 296, 2002.
[125] J. Tian and J He. Can EMG machine interface be used to model brain machine
interface? InProceedings of the 25th Annual International Conference of the IEEE
EMBS, pages 1658–1661, 2003.
[126] E. Todorov. Direct cortical control of muscle activation in voluntary arm movements:
a model.Nature Neuroscience, 3:391–398, 2000.
[127] P. Touzalin-Chretien, S. Ehrler, and A. Dufour. Dominance of vision over proprio-
ception on motor programming: Evidence from erp.Cereb. Cortex, 2009.
[128] R. J. van Beers, A. C. Sittig, and J. J. Denier van der Gon. Integration of pro-
prioceptive and visual position-information: An experimentally supported model.J
Neurophysiol, 81(3):1355–1364, 1999.
126 BIBLIOGRAPHY
[129] C.E. Vargas-Irwin, G. Shakhnarovich, P. Yadollahpour, J.M.K. Mislow, M. J. Black,
and J. P. Donoghue. Decoding complete reach and grasp actions from local primary
motor cortex populations.J Neurosci, 30:9659–9669, 2010.
[130] M. Velliste, S. Perel, M. C. Spalding, A. S. Whitford, andA. B. Schwartz. Cortical
control of a prosthetic arm for self-feeding.Nature, 453:1098–1101, Jun 2008.
[131] V. Ventura. Spike train decoding without spike sorting. Neural Computation,
20(4):923–963, Apr 2008.
[132] J. Wessberg, C.R. Stambaugh, J.D. Kralik, P.D. Beck, M. Laubach, J.K. Chapin,
J. Kim, S.J. Biggs, M.A. Srinivasan, and M.A.L. Nicolelis. Real-time prediction of
hand trajectory by ensembles of cortical neurons in primates. Nature, 408:361–365,
2000.
[133] C Wheatstone. Contributions to the physiology of vision- part the first. on some
remarkable, and hitherto unobserved, phenomena of binocular vision. Philosophical
Transactions of the Royal Society of London, 128:371–394, 1838.
[134] T. Winograd and F. Flores.Understanding Computers and Cognition: A new foun-
dation for design. Addison-Wesley, Norwood, NJ, 1987.
[135] D. Wolpert and Z. Ghahramani. Computational principles of movement neuro-
science.Nature Neuroscience, 3:1212–1217, 2000.
[136] D. Wolpert, Z. Ghahramani, and M. Jordan. An internal model for sensorimotor
integration.Science, 269:1880–1882, 1995.
[137] W. Wu, M.J. Black, Y. Gao, E. Bienenstock, M. Serruya, andJ.P. Donoghue. In-
ferring hand motion from multi-cell recordings in motor cortex using a Kalman filter.
In SAB’02-Workshop on Motor Control in Humans and Robots: On theInterplay of
Real Brains and Artificial Devices, pages 66–73, 2002.
[138] W. Wu, M.J. Black, D. Mumford, Y. Gao, E. Bienenstock, andJ.P. Donoghue. Mod-
eling and decoding motor cortical activity using a switching Kalman filter. IEEE
Transactions on TBME, 51(6):933–942, Jun. 2004.
BIBLIOGRAPHY 127
[139] W. Wu, Y. Gao, E. Bienenstock, J.P. Donoghue, and M.J. Black. Bayesian popula-
tion decoding of motor cortical activity using a Kalman filter. Neural Computation,
18(1):80–118, Jan. 2006.
[140] W. Wu and N. G. Hatsopoulos. Target-included model andhybrid decoding of
stereotyped hand movement in the motor cortex. InIntl Conf on Biomed Robotics
and Biomech, 2008. (IEEE RAS/EMBS), pages 55–60, 2008.
[141] B. M. Yu, C. Kemere, G. Santhanam, A. Afshar, S. I. Ryu, T. H.Meng, M. Sahani,
and K. V. Shenoy. Mixture of trajectory models for neural decoding of goal-directed
movements.J Neurophysiol, 97:3763–3780, 2007.
[142] J. H. Zar.Biostatistical Analysis. Prentice Hall, New Jersey, 1999.