Embed Size (px)
Citation preview
A NOVEL ACCESS TECHNOLOGY BASED ON
INFRARED THERMOGRAPHY FOR PEOPLE WITH
SEVERE MOTOR IMPAIRMENTS
by
Negar Memarian
A thesis submitted in conformity with the requirements
for the degree of Doctor of Philosophy
Institute of Biomaterials and Biomedical Engineering
University of Toronto
© Copyright by Negar Memarian 2010
ii
A NOVEL ACCESS TECHNOLOGY BASED ON INFRARED
THERMOGRAPHY FOR PEOPLE WITH SEVERE MOTOR
IMPAIRMENTS
Negar Memarian
Doctor of Philosophy
Institute of Biomaterials and Biomedical Engineering
University of Toronto
2010
Abstract
Many individuals with severe motor impairments are cognitively capable, but because of their
physical impairments, unable to express their intention through conventional means of
communication. Access technologies are devices that attempt to translate the intention of these
individuals into functional activity by harnessing their residual physical or physiological
abilities. The primary objective of this thesis was to design and develop a novel non-invasive and
non-contact access technology based on infrared thermal imaging. This access technology
translates the local temperature change associated with voluntary mouth opening to activation of
a binary switch such as a mouse click or key press. To this end, an algorithm based on motion
and temperature analyses, and morphological and anthropometric filters was designed to detect
mouth opening activity in thermal video in real-time. The secondary objective of this thesis was
to introduce a mutual information measure for objective assessment of binary switch users’
performance. A model was suggested, in which combination of cognitive and physical abilities
of the human user of a binary access switch constitute a communication channel. The proposed
iii
mutual information measure estimates the rate of information transmission in the ‘human
communication channel’ during stimulus response tasks. Using this measure, in a study with ten
able-bodied participants, the infrared thermal switch was validated against a conventional chin
switch. Impairments in body functions and structures that may contraindicate the use of the
infrared thermal switch were explored in a study with seven clients, with severe disabilities.
Potential hard and soft technological solutions to mitigate the effect of these impairments on
infrared thermal switch use were recommended. Finally the infrared thermal switch was tailored
to meet the needs of a young man with severe spastic quadriplegic cerebral palsy, who had no
other means of physical access.
iv
Acknowledgments
In the name of God, the merciful, the compassionate.
I am indebted to my supervisor, Dr. Tom Chau, for allowing me to join his wonderful research
group, his valuable advice and mentorship throughout the last three years, and his remarkable
tolerance and sense of understanding. I have learned many things from Dr. Chau. Most
importantly I learned that goodness grows when it is appreciated.
I am also grateful to my co-supervisor, Professor Tas Venetsanopoulos for his insightful
comments and kind support. It has been an honour for me to be a student of Professor
Venetsanopoulos.
I am thankful to my respected committee members for their constructive feedbacks and words of
encouragement.
I would like to thank members of the PRISM lab and the Holland-Bloorview Kids Rehabilitation
Hospital staff for their friendliness and helpfulness. In particular, I would like to acknowledge
Eric Wan for being a role-model of determination and optimism, and Pat McKeever for
reassuring me that miracles really happen.
I extend my sincere gratitude to the clients and their families, who kindly participated in this
research. I earnestly hope that the outcome of this thesis will improve the quality of life of these
respectable individuals.
I would like to acknowledge the Natural Sciences and Research Council of Canada, Whipper
Watson Scholarship, Bloorview Children Hospital Foundation Scholarship, IBM, Ontario
Centres of Excellence, and University of Toronto for their funding and support of this research.
My deepest appreciation goes to my family for their endless love and support. My heartfelt
gratitude to my parents for being truly inspirational and for setting the bar high; to my brother for
his witty sense of humour which can make me laugh under the toughest of circumstances; to my
sister for being the joy in my heart; and to my grandmother for her patience, prayers and good
wishes that reach me every day despite the vast oceans between us.
v
Table of Contents
Acknowledgments.......................................................................................................................... iv
Table of Contents .............................................................................................................................v
List of Tables ................................................................................................................................. ix
List of Figures ..................................................................................................................................x
1 Introduction .................................................................................................................................1
1.1 People with Severe Motor Impairments and Access Technologies .....................................1
1.1.1 Mechanical Switches ...............................................................................................3
1.1.2 Electromyography-Based Switches .........................................................................3
1.1.3 Voice Activated Technologies .................................................................................4
1.1.4 Computer Vision-Based Technologies ....................................................................4
1.2 Motivation ............................................................................................................................4
1.2.1 Infrared Thermography ............................................................................................7
1.2.2 Infrared Thermography of the Human Face ............................................................8
1.2.3 Infrared Thermal Switch ..........................................................................................8
1.3 Objectives ............................................................................................................................9
1.4 Roadmap ............................................................................................................................10
2 Detection of Mouth Open Activity in Infrared Thermal Video ................................................13
2.1 Abstract ..............................................................................................................................13
2.2 Background ........................................................................................................................14
2.2.1 Alternative Access Pathways .................................................................................14
2.2.2 Biomedical Applications of Thermal Imaging ......................................................15
2.2.3 Thermal Imaging as an Access Pathway ...............................................................15
2.3 Methods..............................................................................................................................16
vi
2.3.1 Participants .............................................................................................................16
2.3.2 Instrumentation and Setup .....................................................................................17
2.3.3 Thermal Video Processing .....................................................................................18
2.3.4 Algorithm Evaluation .............................................................................................22
2.4 Results and Discussion ......................................................................................................23
2.5 Conclusion .........................................................................................................................27
3 A Mutual Information Measure for Objective Performance Assessment of Binary Switch Users ..........................................................................................................................................28
3.1 Abstract ..............................................................................................................................29
3.2 Introduction ........................................................................................................................29
3.2.1 Access Pathways ....................................................................................................29
3.2.2 The Role of Context ...............................................................................................30
3.2.3 Gauging Contextual Effects through Information Theory .....................................31
3.2.4 Mutual Information in Transmission of Binary Information .................................33
3.3 Methods..............................................................................................................................35
3.3.1 Proposed Framework for Evaluating Contextual Effects on Single Switch Use ...35
3.3.2 Estimating Mutual Information..............................................................................36
3.3.3 Participants .............................................................................................................38
3.3.4 Protocol ..................................................................................................................38
3.3.5 Data Collection and Analysis.................................................................................39
3.4 Results ................................................................................................................................39
3.5 Discussion ..........................................................................................................................44
3.5.1 Contextual Factor Role Characterization ...............................................................44
3.5.2 Ranking Contextual Factors Based on the Significance of their Effect .................45
3.5.3 Inter-Subject Comparison ......................................................................................46
3.5.4 Objective Method of Performance Assessment for Users with Disability ............46
vii
3.5.5 Limitations of Present Study ..................................................................................47
3.6 Conclusion .........................................................................................................................48
4 Validating an Infrared Thermography-based Access Switch (the Infrared Thermal Switch) ..49
4.1 Abstract ..............................................................................................................................49
4.2 Background ........................................................................................................................50
4.3 Methods..............................................................................................................................51
4.3.1 Infrared Thermal Switch ........................................................................................51
4.3.2 Validity Testing Experiment ..................................................................................56
4.4 Results ................................................................................................................................60
4.5 Discussion ..........................................................................................................................61
4.6 Conclusion .........................................................................................................................65
5 Body Functions and Structures Pertinent to Infrared Thermal Switch Use ..............................66
5.1 Abstract ..............................................................................................................................66
5.2 Introduction ........................................................................................................................67
5.3 Methods..............................................................................................................................69
5.3.1 The Access Technology .........................................................................................69
5.3.2 The Clients .............................................................................................................70
5.3.3 Study Protocol ........................................................................................................71
5.3.4 Data Analysis .........................................................................................................74
5.4 Results ................................................................................................................................75
5.5 Discussion ..........................................................................................................................78
5.5.1 Potential Solutions .................................................................................................80
5.5.2 Other Factors Pertinent to Infrared Thermal Access .............................................83
5.6 Conclusion .........................................................................................................................83
6 Customizing the Infrared Thermal Switch for an Individual with Severe Motor Impairments ..............................................................................................................................85
viii
6.1 Abstract ..............................................................................................................................85
6.2 Introduction ........................................................................................................................86
6.3 Client Profile ......................................................................................................................88
6.4 Rationale for an Infrared Thermographic Solution ............................................................90
6.5 Methods..............................................................................................................................91
6.5.1 The Infrared Thermal Switch .................................................................................91
6.5.2 Switch Testing .......................................................................................................93
6.6 Results ................................................................................................................................96
6.7 Discussion ..........................................................................................................................98
6.7.1 User Feedback and Maintenance Issues ..............................................................101
6.7.2 Future Prospects ...................................................................................................102
6.8 Conclusion .......................................................................................................................103
7 Summary of Contributions ......................................................................................................104
References ....................................................................................................................................108
Appendix A: Research Ethics Approval ......................................................................................127
Appendix B: Questionnaire ..........................................................................................................138
ix
List of Tables
Table 1.1: Summary of major access switches based on oral motor control – Part I ..................... 5
Table 2.1: Performance of the proposed mouth opening detection algorithm .............................. 25
Table 3.1: Summary of the contextual factors explored in the present study and the
corresponding experimental procedures. ...................................................................................... 40
Table 3.2: Effect of stimulus duration on mutual information of the participant with disability. 45
Table 4.1: Selected qualitative feedback (pros and cons) from participants. ............................... 64
Table 5.1: Client descriptions (Part I) ........................................................................................... 72
Table 5.2: Detection results for clients who completed the mouth open-close test ...................... 75
Table 5.3: Client feedback about the mouth open-close test ........................................................ 76
Table 5.4: Impairments in body function and structure and consequent limitations to infrared
thermal access ............................................................................................................................... 77
Table 6.1: Result of the stimulus-response switch test. Reproduced from [Memarian et al.
(2010) [43]]. ..................................................................................................................................... 97
Table 6.2: Results of the scanning test. The numbers in parentheses are the actual counts.
Reproduced from [Memarian et al. (2010) [43]]. .......................................................................... 99
x
List of Figures
Figure 1.1: Role of access technology as a conduit to translate a client’s functional intent to a
corresponding functional activity [2]. ............................................................................................. 2
Figure 1.2: Thermogram of a human face with (a) mouth closed and, (b) mouth opened. Darker
intensities correspond to lower temperatures and brighter intensities represent higher
temperatures. The inside of the human mouth is obviously warmer (brighter) than the
surrounding tissue. .......................................................................................................................... 9
Figure 1.3: The overall organization of the five main chapters of this thesis. .............................. 12
Figure 2.1: Components of the proposed mouth opening detection algorithm............................. 18
Figure 2.2: The action of the different modules of the mouth opening detection algorithm. (a)
Input thermal video frame, (b) Segmented face region, (c) Warm facial zones, (d) Moving facial
zones, (e) Intersection of warm and moving objects within the face region, (f) After
morphological, size variation, and anthropometric filtering, (g) Final output; detected mouth
open is highlighted on the original video with a hollow box. ....................................................... 23
Figure 2.3: Robustness of the proposed algorithm to motion artefacts and changes in the
background. (a) Robustness to motion artefacts. Top row from left to right shows input thermal
video of an able-bodied participant moving his arm to his head (frames 63, 66, 70, and 74).
Bottom row depicts face segmentation in the corresponding frames. (b) Robustness to changes in
the background. Top row from left to right is an input thermal video of a participant with
disability while a passerby traverses the scene in the background (frames 1759, 1765, 1779,
1790). The corresponding face segmentation results are presented in the bottom row. ............... 26
Figure 3.1: Information-theoretic paradigm for single-switch access. A subject may react to
visual, written or auditory stimuli by pressing a single mechanical switch. The computer, which
generates the cues, can be considered the transmitter (T); the mechanical switch (a means of
acknowledging the cues) can be thought of as the receiver (R); the user constitutes the
communication channel between the transmitter and the receiver (CC). ..................................... 36
xi
Figure 3.2: All possible sender-receiver cue combinations. ......................................................... 38
Figure 3.3: Estimated mutual information of the twelve participants for the five experiments: (a)
effect of presentation modality, (b) effect of prediction of choice, (c) effect of ambient noise, (d)
effect of colour, (e) effect of presence of people in the environment. Graphs (a), (b) and (e)
depict significant differences (paired t-test). ................................................................................ 41
Figure 3.4: Estimated probability densities of the mutual information for (a) presentation
modality and (b) presence of people in the environment. ............................................................. 43
Figure 3.5: Participants’ mutual information in response to pictorial (dark bars), written
(unshaded bars) and auditory stimuli (grey bars). ........................................................................ 47
Figure 3.6: Difference in participants’ mutual information with and without the following
contextual factors: presence of people (unshaded bars) and colour (shaded bars). ...................... 48
Figure 4.1: Visual representation of the infrared thermal switch video processing algorithm. (a)
Input greyscale thermal video frame, (b) Result of face localization, (c) Result of intensity
analysis (warm area mask), (d) Intersection of warm area mask and motion mask, (e) Open
mouth detected and marked on the video. Note that for ease of visualization, images (c)-(e) only
show the smaller region demarcated by the dashed box in (b). .................................................... 53
Figure 4.2: Infrared thermal switch validity testing experiment setup. ........................................ 59
Figure 4.3: Efficiency of infrared thermal and chin switches averaged over all eighteen trials, for
all ten participants. The vertical lines show standard deviation. .................................................. 62
Figure 5.1: Infrared thermal image of a client’s face. Brighter tones represent warmer regions.
The oral cavity is warmer than the surrounding facial tissue. ...................................................... 70
Figure 5.2: Sample postures in which the client’s mouth is obscured from the thermal camera’s
view. (a) Client with low muscle tone, unable to consistently hold his head up; (b) Client with
habit of frequently rubbing her eyes. ............................................................................................ 79
xii
Figure 6.1: Image acquired by an infrared thermal camera. Darker intensities represent colder
regions and brighter intensities represent warmer regions. Inside the mouth is clearly warmer
than the surrounding facial tissue. Reproduced from [Memarian et al. (2010) [43]]. ..................... 92
Figure 6.2: Infrared thermal switch testing setup. Reproduced from [Memarian et al. (2010) [43]].
....................................................................................................................................................... 96
Figure 6.3: Percentage of correct activations per session. Reproduced from [Memarian et al.
(2010) [43]]. ................................................................................................................................... 101
1
Chapter 1
1 Introduction
This thesis focuses on the design and development of a novel access technology for people with
severe motor impairments. Using infrared thermal imaging, local temperature changes associated
with mouth opening and closing is harnessed to operate a binary switch. The client with
disability can use this binary switch to perform various activities such as selecting characters
from a scanning keyboard (typing), or making his/her wheelchair move (mobility). This thesis is
organized as a compilation of papers. This opening chapter reviews the general background,
motivation, objectives, and roadmap of this thesis.
1.1 People with Severe Motor Impairments and Access
Technologies
Severe motor impairments are generally the consequence of low incidence conditions with high
cost implications. Example conditions include cerebral palsy (CP), spinal cord injury (SCI),
muscular atrophy, multiple sclerosis (MS), and amyotrophic lateral sclerosis (ALS) [1]. Most
individuals with these conditions have intact minds that are trapped in non-functioning bodies,
and are therefore unable to communicate with the outside world.
Significant research in rehabilitation engineering has focused on the development of access
technologies that translate the patient’s intent into a functional activity. As shown in Figure 1.1
(from [2]), an access technology comprises two components: 1) access pathway or the actual
sensors or input devices by which an expression of functional intent (e.g., a movement or
physiological change) is transduced into an electrical signal; and 2) a signal processing unit that
analyzes (e.g., filtering and pattern classification) the input signal and generates a corresponding
2
control signal. The control signal is used to drive a user interface (e.g., computer), which in turn,
triggers the execution of an appropriate functional activity (e.g., typing) within a specific user
environment [2].
Figure 1.1: Role of access technology as a conduit to translate a client’s functional intent to a
corresponding functional activity1 [2].
A wide range of access technologies have been developed from simple mechanical switches to
sophisticated physiological signal-based switches [2]. Recommendation of the appropriate access
modality depends on the nature and severity of the impairment, and the strength, reliability and
endurance of the client’s control over the potential access sites. In Cook & Hussey’s Human
Activity Assistive Technology (HAAT) model, the closest analogy to an access technology is
what they term the “control interface” [3]. They characterize control interfaces based on their
spatial (e.g., overall physical size, weight, available targets), activation-deactivation (e.g.,
1 Figure 1.1 is reproduced from Tai K, Blain S, Chau T. A review of emerging access technologies for individuals with severe motor impairments. Assist Technol. 2008; 20:204-219. Publisher: Taylor & Francis Ltd, http://www.informaworld.com, reprinted by permission of the publisher.
3
method of activation, effort, durability), and sensory (e.g., the auditory, somatosensory, and
visual feedback produced during the activation of the control interface) features [3].
Most access technologies operate as switches; the detection of a specific physical activity or
physiological change closes a normally open single-pole single-throw switch. These access
technologies are usually referred to as ‘access switches’. Some other access technologies are
more sophisticated and allow the user to do more than just switch activation (e.g., speech
recognition technologies [4], mouse navigation [5]). Operation of this latter group of access
technologies requires appreciable physical and cognitive abilities and therefore cannot be a
viable solution for many clients with severe motor impairments. Some representative examples
of access technologies are listed below.
1.1.1 Mechanical Switches
Mechanical switches are activated by some mechanical stimulus like force or displacement. They
include, but are not limited to push buttons, wobble switches, button cap switches, joysticks,
head switches, sip and puff (breath control), chin switches, or combinations of the above [6], [3].
1.1.2 Electromyography-Based Switches
Electromyography (EMG) switches are sensitive to the electrical activity accompanying muscle
contraction. This switch can be a viable alternative access pathway for users with severe motor
impairments who retain voluntary control of at least one muscle site. Examples include [7] and
[8].
4
1.1.3 Voice Activated Technologies
Voice switches are activated by changing the pitch, loudness, and vowel quality of the user’s
vocalizations, hums, or whistles [9]. Some example technologies are the vocal cord vibration
switch [9], Vocal Joystick [10], Phonetic Control [11], and Whistling User Interface [12].
1.1.4 Computer Vision-Based Technologies
Computer vision-based technologies track the position change of a particular facial landmark
(e.g., eye gaze [13], [14], [15], tongue protrusion [16], head [17]) or extremity (e.g., finger [18],
hand [19]) in a non-contact fashion by processing video acquired typically by cameras that work
with the light of the visible spectrum (e.g., digital video camera, webcam). The detected
movement is translated into binary switch activation or cursor movement on a computer screen.
1.2 Motivation
Many clients with severe motor impairments retain some residual fine motor control above the
neck. In particular jaw movement and oral motor control may remain intact. Simpson et al. [20]
report that jaw movements are typically preserved in all but the most severe cases of spinal cord
injury, multiple sclerosis, and cerebral palsy [21]. Hence, various access switches, from simple
no-tech mouthsticks to high-tech computer vision-based switches, have been designed to make
use of that residual oral motor ability. Table 1.1 gives a summary of such access switches.
Among these switches, computer vision-based systems are particularly appealing given their
non-contact nature. Oftentimes clients prefer access technologies that do not require attachment
of sensors or external objects to their bodies. However, as indicated in Table 1.1, a major
limitation of most of the computer vision-based systems is that they are dependent on colour and
lighting conditions and thus their performance may vary with different users or working
environments [22].
5
Rep
orte
d T
arge
t Po
pula
tion
Qua
drip
legi
cs a
nd
othe
rs w
ith p
artia
l or
com
plet
e ar
m a
nd
hand
par
alys
is [2
3]
Clie
nts w
ith
cere
brop
athy
, m
inim
al re
sidu
al
visi
on a
nd
inte
llect
ual d
isab
ility
[2
9], [
30]
Clie
nts w
ith
inco
mpl
ete
C1-
C4
SCI r
esul
ting
in
quad
riple
gia
[31]
, [3
3];
Susp
ecte
d to
be
usef
ul fo
r clie
nts
with
CP,
or s
cler
osis
(r
epor
ted
resu
lts a
re
base
d on
a h
ealth
y su
bjec
t onl
y) [3
4]
Clie
nts w
ith c
ervi
cal
leve
l SC
I [36
]
Lim
itatio
ns
Req
uire
s goo
d or
al-m
otor
con
trol a
s wel
l as
con
sist
ent h
ead
and
neck
mov
emen
t co
ntro
l im
prac
tical
for m
ost c
lient
s w
ith d
isab
ility
L
ow o
pera
tion
spee
d an
d lo
w a
ccur
acy
[26]
M
ay c
ause
tem
pora
l-man
dibu
lar j
oint
(T
MJ)
dys
func
tion
or d
enta
l mig
ratio
ns
[27]
Con
tact
s the
clie
nt’s
skin
and
may
cau
se
skin
irrit
atio
n/ ra
sh fo
r som
e cl
ient
s M
ay e
nter
clie
nt’s
mou
th a
nd p
ose
infe
ctio
n ris
ks
Ope
ratio
n is
sens
itive
to p
ositi
onin
g Pr
one
to fa
lse
and
mis
sed
activ
atio
ns
Req
uire
s con
stan
t ste
riliz
atio
n to
avo
id
infe
ctio
n
Ris
k of
clie
nt c
hoki
ng e
spec
ially
for
vers
ions
pos
ition
ed in
side
the
mou
th
Nor
mal
spee
ch a
nd sw
allo
win
g m
ay
gene
rate
fals
e po
sitiv
es fo
r tho
se
vers
ions
pos
ition
ed in
side
the
mou
th
w
ill c
ause
frus
tratio
n an
d fa
tigue
M
ount
ing
and
pos
ition
ing
is c
halle
ngin
gfo
r the
ver
sion
s pla
ced
outs
ide
of
clie
nt’s
mou
th
Req
uire
s con
stan
t ste
riliz
atio
n to
avo
id
infe
ctio
n
Som
e cl
ient
s may
not
be
able
to p
rodu
ce
suff
icie
nt a
irflo
w to
ope
rate
the
switc
h
Exa
mpl
es
Cus
tom
-fit
mou
thst
ick
[23]
, Ben
dabl
e te
lesc
opic
mou
thst
ick
[24]
, H-A
Mod
ular
M
outh
stic
k [2
5]
Mou
ntab
le c
hin
switc
h [2
8], w
eara
ble
chin
m
icro
switc
h [2
9], [
30]
Sing
le m
echa
nica
l sw
itch
[31]
, Pre
ssur
e se
nsiti
ve b
utto
ns [3
2],
[33]
, Ind
ucto
r and
m
agne
t [34
]
Mou
ntab
le si
p &
puf
f [3
5], [
36]
Des
crip
tion
and
Met
hod
of
Act
ivat
ion
A p
oint
er (s
tick)
atta
ched
to a
m
outh
piec
e th
at th
e cl
ient
grip
s be
twee
n hi
s/he
r tee
th to
per
form
su
ch a
ctio
ns a
s typ
ing,
pai
ntin
g,
and
liftin
g sm
all o
bjec
ts
A m
echa
nica
l sw
itch
plac
ed
slig
htly
bel
ow th
e cl
ient
’s c
hin.
It
is a
ctiv
ated
whe
n it
com
es in
to
cont
act w
ith th
e cl
ient
’s c
hin
due
to th
e cl
ient
ope
ning
his
/her
m
outh
A sw
itch
that
is p
lace
d in
side
or
outs
ide
clie
nt’s
mou
th a
nd is
ac
tivat
ed b
y m
otio
n of
the
clie
nt’s
tong
ue
A tu
be in
whi
ch th
e cl
ient
sips
or
puff
s to
activ
ate
one
of th
e du
al
mod
es o
f the
switc
h
Acc
ess S
witc
h
Mou
thst
ick
Chi
n sw
itch
Tong
ue sw
itch
Sip
and
Puff
Tabl
e 1.
1: S
umm
ary
of m
ajor
acc
ess s
witc
hes b
ased
on
oral
mot
or c
ontro
l – P
art I
6
Rep
orte
d T
arge
t Po
pula
tion
SCI c
lient
s with
m
inim
al h
and
func
tion
(als
o te
sted
on
clie
nts w
ith M
S,
CP,
and
po
liom
yelit
is,
how
ever
resu
lts w
ere
not r
epor
ted)
[20
]; Te
trapl
egic
s [37
]
Clie
nts w
ith a
taxi
a an
d qu
adrip
legi
a fr
om c
ord-
inju
ries
[38]
; C
lient
with
spas
tic
quad
riple
gic
CP
[16]
Clie
nts w
ith se
vere
sp
astic
qua
drip
legi
c C
P, a
thet
oid
CP,
C1-
C2
inco
mpl
ete
SCI
[42]
Lim
itatio
ns
Req
uire
s sen
sor a
ttach
men
t (no
t pre
ferr
ed b
y m
any
child
cl
ient
s)
Req
uire
s clie
nt’s
teet
h to
mak
e co
ntac
t cle
anly
eno
ugh
to
gene
rate
a ja
w v
ibra
tion
sign
al th
at c
ould
be
dist
ingu
ishe
d fr
om th
e ba
ckgr
ound
noi
se [2
0]
Req
uire
s pre
cise
pos
ition
ing
of th
e se
nsor
on
the
user
’s
ear,
such
that
it to
uche
s the
trag
us
Rel
iabl
e co
mm
unic
atio
n re
quire
s the
wire
less
rece
iver
to
be p
ositi
oned
with
in li
ne-o
f-si
ght o
f the
use
r M
ay m
alfu
nctio
n du
e to
dro
pout
in th
e w
irele
ss li
nk
betw
een
the
earp
iece
and
the
com
pute
r int
erfa
ce
Req
uire
s cus
tom
ized
har
dwar
e fit
for d
iffer
ent e
ar sh
apes
an
d si
zes
Poss
ibili
t y o
f rad
io in
terf
eren
ce
Dep
ende
nt o
n lig
htin
g co
nditi
ons
Som
e m
etho
ds a
re d
epen
dent
on
skin
col
our
May
requ
ire in
tens
ive
proc
essi
ng/c
ompu
ting
reso
urce
s So
me
gest
ures
may
hav
e ne
gativ
e so
cial
con
nota
tion
(e.g
. to
ngue
pro
trusi
on) [
16]
Mou
ntin
g m
ultip
le c
amer
as m
ay b
e cu
mbe
rsom
e
Cos
tly re
lativ
e to
oth
er o
ptio
ns in
this
tabl
e (~
$200
0 U
S)
Det
ectio
n se
nsiti
vity
may
redu
ce a
fter t
akin
g co
ld d
rink
or
food
(thi
s pro
blem
is a
llevi
ated
with
tim
e)
Det
ectio
n se
nsiti
vity
may
redu
ce if
exc
essi
ve sa
liva
accu
mul
ates
in
the
mou
th (t
his p
robl
em is
alle
viat
ed b
y sw
allo
win
g on
ce o
r for
cibl
y ex
pelli
ng s
aliv
a fr
om m
outh
) P
rolo
nged
per
iods
of s
witc
h op
erat
ion
may
cau
se fa
tigue
or
jaw
pai
n E
xces
sive
mov
emen
t (su
ch th
at c
lient
’s fa
ce le
aves
ca
mer
a’s f
ield
of v
iew
) may
hin
der s
witc
h us
e
Exa
mpl
es
[37]
, [20
]
Mou
th
open
ing
[38]
, To
ngue
pr
otru
sion
[1
6]
[39]
, [40
], [4
1], [
42],
[43]
Des
crip
tion
and
Met
hod
of
Act
ivat
ion
An
acce
lero
met
er p
ositi
oned
ag
ains
t the
ear
, whi
ch d
etec
ts
the
jaw
vib
ratio
ns a
ssoc
iate
d w
ith th
e up
per a
nd lo
wer
teet
h co
min
g in
to c
onta
ct.
Vib
ratio
ns a
re tr
ansl
ated
into
sw
itch
activ
atio
n us
ing
a m
icro
proc
esso
r and
a w
irele
ss
rem
ote
rece
iver
One
or m
ore
vide
o ca
mer
as
dete
ct th
e cl
ient
’s v
olun
tary
fa
cial
ges
ture
s and
tran
slat
e th
em in
to sw
itch
activ
atio
n us
ing
com
pute
r alg
orith
ms
Loca
l tem
pera
ture
cha
nge
asso
ciat
ed w
ith c
lient
’s
volu
ntar
y m
outh
ope
ning
is
dete
cted
usi
ng a
n in
frar
ed
ther
mal
cam
era
and
trans
late
d to
switc
h ac
tivat
ion
by a
co
mpu
ter a
lgor
ithm
Acc
ess S
witc
h
Toot
h C
lick
Vid
eo-b
ased
sw
itch
Infr
ared
ther
mal
sw
itch
(pro
pose
d in
this
thes
is)
Tabl
e 1.
1: S
umm
ary
of m
ajor
acc
ess s
witc
hes b
ased
on
oral
mot
or c
ontro
l – P
art I
I
7
The aforementioned shortfall of conventional computer vision-based access switches motivated
us to design a new access switch based on oral motor control that would exploit the non-invasive
and non-contact features of vision-based solutions while also being invariant to skin colour and
lighting conditions. I decided to utilize infrared thermography for this purpose. As will be
discussed in the next section, infrared thermography is both independent of skin pigmentation
and ambient lighting conditions [44], [45].
1.2.1 Infrared Thermography
The basis of infrared thermography (also known as infrared thermal imaging) is the detection of
natural thermal radiation emitted by the surface of objects. An infrared thermal imaging camera
produces an image, known as an infrared thermogram, representing the temperature distribution
of objects in its field of view. An infrared thermograph of the human body is a record of the
temperature distribution of the epidermal layer of human skin [46].
Infrared thermal imaging involves no external source of infrared illumination. The radiation
arises from natural thermal radiation generated by every object in the scene according to
Planck’s Law for black body radiation (equation 1.1), where W eλ is the radiant exitance in
Wm 3− , 1c and 2c are constants (3.74182 10 16− Wm 2 and 1.43876 10 2− mK, respectively), λ is
the radiation wavelength and T is temperature [45].
⎟⎟⎠
⎞⎜⎜⎝
⎛−⎟
⎠⎞
⎜⎝⎛
=1exp 25
1
Tc
cW e
λλ
λ
(1.1)
Planck’s radiation law represents the maximum thermal power that can be radiated by an object,
and most surfaces radiate a fraction ε of the ideal black body radiation; ε is known as the
emissivity. Human skin has an emissivity of about 0.98. Thermal radiation from the skin
originates in the epidermis and is independent of race; it depends therefore only on the surface
8
temperature [44]. Emissivity varies with wavelength, but at depths between 3 and 15 µm below
the skin surface, skin emissivity is fairly constant, with a value of 0.975 ± 0.05 [45], [46].
Infrared thermography has been used mainly for surveillance purposes in military [47], remote
sensing [48], and medical research, specifically in breast cancer [49], [50], [51], and arthritis
[52], [53], [54] research.
1.2.2 Infrared Thermography of the Human Face
Studies of human face infrared thermography have shown that the temperature is relatively high
around the eye regions, especially the periorbital regions, i.e., the small areas between each eye
and the bridge of the nose [55], [56]. The forehead and chin are other facial landmarks that are
generally warm, while tip of the nose usually has the lowest temperature [56]. It has been
reported that increased blood perfusion in the periorbital muscles is associated with stress [57],
anxiety [58], and arousal [55] in humans. Figure 1.2(a) portrays a sample thermogram of a
human face. In this image, the darkest intensity (black) represents the lowest temperature and the
brightest intensity (white) represents the highest temperature. Temperatures in between these two
extremes are depicted with the spectrum of intensities shown in the temperature sidebar to the
right of each image.
1.2.3 Infrared Thermal Switch
In light of the above rationale, I propose the ‘infrared thermal switch’, a non-invasive and non-
contact access technology for people with severe motor impairments by using infrared thermal
imaging. This access technology works like a binary switch, activated by voluntary mouth
opening. Mouth opening involves both motion and change in temperature, as the inside of the
human mouth is generally warmer than the surrounding tissue (Figure 1.2(b)). I designed and
developed a video processing algorithm to track mouth open-close activity in infrared thermal
9
videos and translate the detected motion into real-time switch activation by using simple
hardware.
(a) (b)
Figure 1.2: Thermogram of a human face with (a) mouth closed and, (b) mouth opened. Darker
intensities correspond to lower temperatures and brighter intensities represent higher
temperatures. The inside of the human mouth is obviously warmer (brighter) than the
surrounding tissue.
1.3 Objectives
The objective of this thesis was to implement and test a new infrared thermography-based access
technology for people with severe motor impairments. For the proposed system to qualify as a
reliable access technology for the client population, several technical and clinical questions had
to be investigated. In that respect, the specific objectives of this thesis were:
A. To design and develop an algorithm for the detection of mouth open-close activity in
infrared thermal video.
B. To develop a real-time binary switch based on infrared thermographic sensing of
voluntary mouth opening.
10
C. To validate the proposed infrared thermography-based binary switch.
D. To determine important motor, physiological, sensory and cognitive factors that affect the
viability of the proposed infrared thermography-based switch for clients.
E. To customize the proposed infrared thermography-based switch for an individual with
severe motor impairments who has never had a means of access.
F. To design a quantitative method for measuring the effect of contextual factors on the
performance of binary switch users.
G. To investigate the effect of stimulus presentation modality (a contextual factor) on the
use of the proposed infrared thermography-based binary switch.
1.4 Roadmap
The roadmap of this thesis is based on the above objectives. Chapters 2-6 are each structured as a
journal article that addresses one or more of the above objectives. The overall organization of the
five main chapters of this thesis is depicted in Figure 1.3. The introduction (or background)
section of some chapters may contain similar information. Also, parts of the methods section in
some chapters may describe the same instrumentation and signal acquisition setup. Following the
five main chapters, the thesis closes with a summary of the contributions of this thesis.
In chapter 2 I present the inaugural investigation of infrared thermography as a non-invasive and
non-contact access technology. I report the algorithm for detection of mouth open-close activity
in infrared thermal videos and show the potential of this algorithm by applying it to non-
radiometric thermal videos of several individuals with or without motor disability. This chapter
addresses objective A.
11
In chapter 3 I introduce a mutual information measure to objectively assess the performance of
binary switch users during stimulus-response tasks. I show that common performance measures
(e.g. sensitivity and specificity) relating to switch use can be quantitatively unified within a
mutual information measure. I also show that this measure can be used as an objective means of
assessing the impact of contextual factors on switch use. I exemplify this by focusing on a simple
binary switch (i.e., single mechanical switch) and five selected contextual factors (objective F).
In chapter 4, I use the mutual information measure introduced in chapter 3 to validate the
proposed infrared thermal switch. Validity in the present context refers to the correlation
between the switch activations of the infrared thermal switch and those of an established gold
standard during repeated trials of a stimulus-response task. In this validation study, I also
investigate the effect of stimulus presentation modality (i.e., visual, auditory, or audiovisual) on
the information-theoretic efficiency and response time of the infrared thermal switch users
(objectives B, C, and G).
In chapter 5, I conduct an exploratory study to derive motor, physiological, sensory and
cognitive factors that are important to infrared thermal switch use. Based on interviews with
seven clients and their primary caregivers I derive a list of impairments in body functions and
structures that may impede infrared thermal switch access for clients with severe motor
impairments. Potential solutions to compensate for the effect of those impairments will also be
recommended (objective D).
In chapter 6, I present a case study of the infrared thermal switch for a client with severe spastic
quadriplegic cerebral palsy. I report the process of adopting a client-centred approach to provide
this client with a means of access. I describe the client’s progress with the infrared thermal
switch from initial access site reliability testing to eventual functional application where he used
the switch to make selections from multiple choices through scanning (objective E).
12
Figure 1.3: The overall organization of the five main chapters of this thesis.
13
Chapter 2
2 Detection of Mouth Open Activity in Infrared Thermal
Video
In this chapter, I report the inaugural algorithm for detection of mouth open-close activity in
infrared thermal videos and show the potential of this algorithm by applying it to non-
radiometric thermal videos of several individuals with and without motor disability.
The entirety of this chapter is reproduced from the following journal article: Memarian N,
Venetsanopoulos AN, Chau T. Infrared thermography as an access pathway for individuals with
severe motor impairments. Journal of NeuroEngineering and Rehabilitation. 2009; 6:11 (8 pp).
Since the Journal of NeuroEngineering and Rehabilitation, published by BioMed Central, is open
access, no permission was required from the publisher to reproduce the article here. This article
can be found on the publisher's website at http://www.jneuroengrehab.com/content/6/1/11
2.1 Abstract
Background: People with severe motor impairments often require an alternative access pathway,
such as a binary switch, to communicate and to interact with their environment. A wide range of
access pathways have been developed from simple mechanical switches to sophisticated
physiological ones. In this manuscript we report the inaugural investigation of infrared
thermography as a non-invasive and non-contact access pathway by which individuals with
disabilities can interact and perhaps eventually communicate.
14
Methods: Our method exploits the local temperature changes associated with mouth opening/
closing to enable a highly sensitive and specific binary switch. Ten participants (two with severe
disabilities) provided examples of mouth opening and closing. Thermographic videos of each
participant were recorded with an infrared thermal camera and processed using a computerized
algorithm. The algorithm detected a mouth open-close pattern using a combination of adaptive
thermal intensity filtering, motion tracking and morphological analysis.
Results: High detection sensitivity and low error rate were achieved for the majority of the
participants (mean sensitivity of all participants: 88.5% ± 11.3; mean specificity of all
participants: 99.4% ± 0.7). The algorithm performance was robust against participant motion and
changes in the background scene.
Conclusion: Our findings suggest that further research on the infrared thermographic access
pathway is warranted. Flexible camera location, convenience of use and robustness to ambient
lighting levels, changes in background scene and extraneous body movements make this a
potential new access modality that can be used night or day in unconstrained environments.
2.2 Background
2.2.1 Alternative Access Pathways
Individuals with severe physical impairments who are unable to communicate through speech or
gestures require an alternative means to convey their intentions. In the rehabilitation engineering
context, these alternative channels are called access pathways and they constitute the critical
front end of an access solution [2]. Some recent efforts have set out to non-invasively translate
physiological signals such as the electrical [59], [60] and hemodynamic activity [61], [62], [63]
of the brain or the electrodermal response of the skin [64], [65] into functional communication.
A comprehensive review of emerging access technologies can be found in [2].
15
2.2.2 Biomedical Applications of Thermal Imaging
Infrared thermography refers to the measurement of the radiation emitted by the surface of an
object in the infrared range of the electromagnetic spectrum, i.e., between wavelengths of 0.8 μm
and 1.0 mm [46]. Infrared cameras use specialized lenses manufactured from materials such as
germanium to focus thermal radiation onto a focal plane array of infrared detectors [66]. Thermal
cameras yield an image that is a spatial, two-dimensional (2-D) map of the 3-D temperature
distribution of the object [45].
Infrared thermography has been widely applied in health research, including, for example, breast
cancer detection [67], [51], brain surgery [68], [69], heart surgery [70], diagnosis of vascular
disorders [71], arthritis [54], pain assessment [72] and post-surgical follow-up in ophthalmology
[73].
Recently, Murthy and Pavlidis non-invasively measured human breathing using infrared imaging
and a statistical methodology based on multinormal distributions, the method of moments, and
Jeffreys divergence measure [74]. Their study was based on the fact that exhaled gases have a
higher temperature than the typical background of indoor environments. They achieved high
detection accuracy on a small set of subjects and suggested potential applications in polygraphy,
sleep studies, sport training, and patient monitoring [74].
2.2.3 Thermal Imaging as an Access Pathway
The goal of this paper is to investigate the potential of thermal imaging as an access pathway. In
particular, we introduce a thermographic binary switch activated by voluntary mouth opening.
Expired air and the oral cavity are generally warmer than the surrounding tissue and environment
while cyclic jaw movements do not cause significant increases in facial temperatures over time
[75]. Therefore localized temperature changes due to mouth opening and closing may be
16
detectable using video and image processing of thermographic data. Examples of patient groups
that may benefit from this access pathway are high level spinal cord injuries resulting in
quadriplegia and individuals with spastic quadriplegic cerebral palsy or general hypotonia2.
Like computer vision-based access pathways [76], thermal imaging is non-invasive and does not
require any sensor attachment to the user. However, thermography overcomes some of the major
limitations of conventional computer vision-based access pathways. Firstly, thermography is skin
colour invariant since there is no difference in emissivity between black, white and burnt skin, in
vivo or in vitro [77]. Human skin has an emissivity of about 0.98. Thermal radiation from the
skin originates in the epidermis and is independent of race; it depends therefore only on the
surface temperature [45], [46]. Secondly, thermal image quality is independent of ambient
lighting conditions and can thus be effective both night and day. Conceivably, this non-contact,
non-invasive access pathway could be tailored to the user's unique motor capacity, whether that
be mouth opening, eye blinking or simply deep breathing. These are all motor activities that may
generate measurable, local temperature changes. Furthermore, given that the key information is
thermal variation, a frontal view of the user may not be necessary, facilitating more flexible and
unobtrusive placement of the camera.
2.3 Methods
2.3.1 Participants
Eight able-bodied participants and two individuals with quadriplegia (one with a C1-C2
incomplete spinal cord injury and the other with severe spastic quadriplegic cerebral palsy)
participated in this study. All participants provided written consent. The experimental protocol
was approved by the research ethics board of the university and affiliated hospital.
2 Clarification: individuals with spastic quadriplegic cerebral palsy or individuals with general hypotonia.
17
2.3.2 Instrumentation and Setup
A THERMAL-EYE 2000B thermal video camera by L-3 Communications with thermal
sensitivity ≤100 mK [78] was connected via an NTSC to USB TV convertor (Dazzle
Multimedia). Videos were recorded as 240 × 320 AVI files (30 fps) and processed offline in
MATLAB & Simulink (version R2007b).
Participants were comfortably seated within a laboratory environment. Those with disability
remained in their wheelchairs. The thermal camera was positioned anterior and lateral to the
participant at a 45° angle. This camera location was chosen over the often-used frontal view,
keeping in mind the eventual application as an access switch where the user's field of view ought
to be unobstructed. In the 45° angle condition, infrared thermograms only exhibit a small error in
recorded temperatures3 [46]. Each participant was cued to open his or her mouth and to hold it
ajar for one second before closing the mouth. Participants were given an auditory prompt upon
every open and close action. The end of each mouth closing was followed by a 3 second rest
before the onset of the next mouth opening. The participants were instructed to maintain a
constant head position, so that their mouth movement stayed within the camera's field of view.
The thermal sensitivity of the infrared camera we used was well beyond what was needed to
detect the temperature change due to mouth opening. We are looking at temperature difference
of about 1.5 to 3°C between when mouth is closed and when it is open, while the thermal
sensitivity of our infrared camera was ≤100 mK.
3 For angles of view up to 45°, the recorded temperature error is 5.0≤ °C. This is calculated from
( ) ( ){ }TnT in
i λφλ εε −=Δ 0 , where TΔ is temperature variation, λφε i is surface emissivity, n can be
considered a quasi constant and T is the true temperature [79] (this footnote has been added for clarification and does not appear in the related journal article).
18
2.3.3 Thermal Video Processing
Figure 2.1 shows a schematic of our algorithm for detecting mouth openings from the thermal
video data. The system consisted of three main components, namely face segmentation, thermal
intensity-motion filtering and false positive removal. Each component will be discussed below.
To begin, the boundary pixels of each video frame (the first and last pixels of every column and
every row) were set to zero to detach objects that may be connected to the borders.
Figure 2.1: Components of the proposed mouth opening detection algorithm.
Face Segmentation
In addition to the participant's head and facial region, other body parts such as the participant's
neck, thorax and upper limbs also appeared in the videos. For the participants with disability,
parts of their wheelchairs were also captured on thermal video. Objects in the background, and in
a couple of instances people moving around the participant were also recorded. It was thus
essential to segment the participant's face region from all other non-target body parts and objects.
Each frame of the video was binarized. Given that facial temperature distributions vary within
and among individuals [80], we adopted Otsu's method to determine an adaptive rather than
fixed intensity threshold which minimized, on a frame by frame basis, the intra-class variance of
the greyscale values of the pixels to be binarized [81].
19
The binarized frames were then morphologically opened with a disk structuring element of
radius 5 pixels to remove small objects, break thin connections, remove thin protrusions, and
smooth object contours [82]. In the resulting image, the object with maximum area (presumably
the face region) was retained and the object's interior holes were filled by morphological closing
with a disk structuring element of radius 20 pixels. The camera-user distance and the user's head
size affect the dimension of the above mentioned structuring elements. In a real life application,
the camera will be mounted on the user's wheelchair at a fixed distance from the user's face.
Hence, once the appropriate parameters are selected in the initial calibration, they do not need to
be changed for subsequent use. An example of a segmented face region is depicted in Figure
2.2(b).
Thermal Intensity-Motion Filtering
All subsequent processing was applied to the intensity image and confined to the identified face
region. The region of interest (ROI) was the participant's mouth and the task of interest was
mouth opening. A combination of temperature thresholding and motion tracking was used to
perceive mouth opening. Warm zones inside the facial region were extracted by thresholding the
segmented face with a scaled version of Otsu's threshold [81] to favour higher intensity (i.e.,
warmer) pixels. The scale factor was empirically derived as
Scale factor = 3 – (mean intensity in face region – 150)/50 (2.1)
and typically ranged from 2.5 to 3. This segmentation yielded a warm zone mask which served to
detect instances of mouth opening. However, there were occasions where nearby facial regions
had similar temperatures as those of the oral cavity. A corroborating cue was therefore required
to accurately pinpoint a mouth opening event.
Since mouth opening involves motion, optical flow was utilized to estimate the direction and
speed of motion from one video frame to the next using the Horn-Schunck method [83]. Motion
20
vectors in each frame of the video sequence were computed by solving the optical flow
constraint equation
0=++ tyx IvIuI (2.2)
where xI , yI and tI are the spatiotemporal image brightness derivatives, u is the horizontal
optical flow and v is the vertical optical flow. By assuming that the optical flow is smooth over
the entire image, the Horn-Schunck method computes an estimate of the velocity field, [u v ] T ,
that minimizes this equation:
( ) dxdyyv
xv
yu
xudxdyIvIuIE tyx
⎪⎭
⎪⎬⎫
⎟⎟⎠
⎞⎜⎜⎝
⎛∂∂
+⎟⎠⎞
⎜⎝⎛∂∂
+⎟⎟⎠
⎞⎜⎜⎝
⎛∂∂
+⎩⎨⎧
⎟⎠⎞
⎜⎝⎛∂∂
+++= ∫∫∫∫2222
2α
(2.3)
In this equation⎟⎠⎞
⎜⎝⎛∂∂
xu
and ⎟⎟⎠
⎞⎜⎜⎝
⎛∂∂
yu
are the spatial derivatives of the optical velocity component u ,
and α scales the global smoothness term [83]. Motion vectors with velocity magnitude
exceeding the mean velocity (i.e., the average of velocity magnitudes across the most recent five
frames) per frame across time were retained, yielding a motion mask. The intersection of this
motion mask and the warm zone mask, introduced above, yielded all the regions of the face that
were both warm and moving.
False Positive Removal
Despite the combination of motion and thermal cues, the processed frames occasionally
contained non-mouth objects (false positives) such as parts of the chin, forehead and the
periorbital regions. These non-mouth objects were also warm and moving and were therefore
retained subsequent to the thermal intensity and motion filters. An example is the forehead,
which according to the literature, is the warmest part of the human body with a temperature
(34.5°C) close to that inside the mouth [44]. Therefore motion of the forehead may result in a
false positive.
21
To deal with these false positives, we deployed a series of additional filters based on
morphology, size variation between frames, and facial anthropometry. Objects that did not meet
the following morphological conditions were deemed as false positives and removed.
30 pixels < Area < 150 pixels
Eccentricity ≤ 0.9.
5.0box bounding of Area
object of Area>
The first condition rejects objects which are either too small or too large to be candidate mouth
openings. Likewise, the second condition removes regions that are too elongated to qualify as
mouth regions while the third condition eliminates hollow regions as the mouth is expected to be
solid. The constants in these morphological filters were selected to resemble the shape of the
open mouth and were empirically defined. In addition, objects whose size varied less than 25%
between the current frame and the frame occurring ten frames earlier were considered static
warm facial regions (e.g., forehead, chin, around the eyes, neck) and were also discarded. This
constitutes the size variation filter in Figure 2.1.
Finally we exploited the fact that facial anatomy is static (i.e., unlikely to change over time).
Based on human face anthropometry, the mouth is located in the lower half of the menton-sellion
length [84], [85]. When we partitioned the facial ROI along its major axis into four strips, we
noticed that indeed the mouth was usually located in the second strip from the bottom. With this
anthropometric filter, we dismissed candidate ROIs outside of the second facial quarter. Figure
2.2(c)–(g) demonstrate the action of the different processing modules.
22
2.3.4 Algorithm Evaluation
To facilitate algorithm evaluation, a truth set was prepared manually for each recorded thermal
video4. The truth set contained the frame numbers corresponding to the beginning and ending of
each mouth opening, the end points of the line maximally spanning the width of the mouth at the
onset of opening and the end points of the line maximally spanning the height of the mouth when
fully ajar. This truth set served as the gold standard for automatic algorithm evaluation. A true
positive was defined as the detection of a ROI temporally within the range of frames
corresponding to a gold standard mouth opening, and spatially situated within the bounding box
defined by the endpoints extracted above. All other detected objects were considered false
positives. A mouth opening that was missed by the algorithm was counted as a false negative. A
true negative occurred when there was no mouth opening and the algorithm concluded the same.
Sensitivity and specificity values were estimated.
4 The truth sets were developed by two people. i.e., one individual prepared the truth sets and then the second individual checked those truth sets for accuracy (this footnote has been added for clarification and does not appear in the related journal article).
23
Figure 2.2: The action of the different modules of the mouth opening detection algorithm. (a)
Input thermal video frame, (b) Segmented face region, (c) Warm facial zones, (d) Moving facial
zones, (e) Intersection of warm and moving objects within the face region, (f) After
morphological, size variation, and anthropometric filtering, (g) Final output; detected mouth
open is highlighted on the original video with a hollow box.
2.4 Results and Discussion
The performance of the proposed algorithm on the thermal video of ten participants is
summarized in Table 2.1. Detection of mouth opening is generally achieved with very high
24
sensitivity and specificity. The exception is the poorer result for participant 10, which is mainly
due to participant's posture, frequent involuntary head rotation away from the camera, and
suboptimal camera placement. This participant had an awkward position in his wheelchair (See
Figure 2.3(b)) which forced us to position the thermal camera at an angle and distance from the
participant that was not consistent with the other participants. Several improvements can be
made to enhance the results in situations like this: (1) The algorithm can be updated to track and
focus on the region of interest (participant's face) more accurately; (2) Multiple cameras5 can be
used to capture participant's facial region from different angles, so that the problem of participant
mouth leaving the camera's field of view will be mitigated; and (3) The user can be trained.
Figures reported in the present paper are the result of just one test session. Training is expected
to have a positive effect on user performance.
Specificity is generally higher than sensitivity as the algorithm was tuned to minimize false
positives, again keeping in mind the alternative access application where inadvertent switch
activations are arguably more costly than missed activations. Most of the false positives were
repeated detections of the same non-mouth object in multiple frames. The chin was the source of
the majority of the false positives, which tended to occur during actual mouth openings. This is
perhaps not surprising given that the chin is proximal to the mouth and moves as the jaw
descends to open the mouth. Further, the chin is reportedly the warmest facial area after the
forehead [86] when measured by thermography.
The proposed algorithm is robust against participant motion and changes to the background
scene. Figure 2.3(a) demonstrates an example of one of the participants moving his arm towards
his face. Although the arm is both warm and moving, and even touches the participant's face in
5 As infrared thermal cameras are costly and mounting multiple cameras may be cumbersome, this option is probably not the most practical solution (this footnote has been added for clarification and does not appear in the related journal article).
25
some frames, it was correctly disregarded by the algorithm. Figure 2.3(b) depicts an example of a
person entering and leaving the background scene. The algorithm successfully rejected the
background activity and did not generate any false positives.
Table 2.1: Performance of the proposed mouth opening detection algorithm
Participant Video length
(sec)
Total Video
frames
Actual # of mouth
openings
Sensitivity Specificity
1 256 7662 50 88.% 100%
2 252 7546 50 96% 100%
3 254 7621 50 96% 100%
4 252 7481 50 98% 100%
5 244 7424 50 88% 99%
6 243 7594 50 92% 98%
7 245 7664 50 94% 99%
8 243 7613 50 80% 100%
9* 153 4592 30 93% 99%
10* 272 8160 15 60% 99%
*Participant with severe disability.
The proposed combination of filters is location and position invariant; regardless of where in the
frame the user moves his or her head within the camera's field of view and independent of the
user's position (sitting or semi-supine), mouth opening could generally be located relative to the
segmented face region.
26
Figure 2.3: Robustness of the proposed algorithm to motion artefacts and changes in the
background. (a) Robustness to motion artefacts. Top row from left to right shows input thermal
video of an able-bodied participant moving his arm to his head (frames 63, 66, 70, and 74).
Bottom row depicts face segmentation in the corresponding frames. (b) Robustness to changes in
the background. Top row from left to right is an input thermal video of a participant with
disability while a passerby traverses the scene in the background (frames 1759, 1765, 1779,
1790). The corresponding face segmentation results are presented in the bottom row.
If one can voluntarily control mouth open and close action, sip and puff technology, EMG based
switches, and computer vision based switches can also be used. The advantage of the proposed
thermography-based access pathway over sip and puff and EMG based switches is that it is non-
invasive and non-contact, i.e., does not require attachment of any sensor or external object to the
27
user. Hence it is more hygienic and safe, as the risk of choking is also eliminated. Its advantage
over visible light computer vision based access pathways is that it is independent of
lighting/colour and can thus be used both night and day, indoor and outdoor.
Despite these encouraging findings, thermal imaging does have its limitations. Infrared thermal
cameras are more expensive than conventional (visible light) cameras. However, recent
innovations in affordable, pocket sized, portable thermal cameras [87] may eventually eliminate
the cost issue. Thermal image quality is susceptible to fluctuations in ambient temperature,
humidity and regional air circulation [46]. A robust thermographic access pathway may need to
dynamically compensate for changes in these contextual factors. A final limitation of thermal
imaging is the relatively low resolution of infrared cameras and the inherent difficulty in
discriminating between fine facial features. These issues may be mitigated by fusing thermal
videos with simultaneously recorded visible spectrum imagery [88].
2.5 Conclusion
We have demonstrated that infrared thermography can be used as a non-contact and non-invasive
access pathway for individuals who retain voluntary mouth opening and closing. Our analyses
suggest that the thermographic access pathway may be robust to various lighting levels, different
body postures, extraneous user movements, and background variations.
28
Chapter 3
3 A Mutual Information Measure for Objective
Performance Assessment of Binary Switch Users
The infrared thermography-based access technology proposed in chapter 2 is an example of a
binary access switch. With the ultimate goal of making this binary switch a robust means of
access under various personal and environmental conditions, it is crucial to assess the effect of
contextual factors on user performance. In this chapter, I introduce a mutual information measure
for objective assessment of binary switch user performance under the effect of different
contextual factors. I exemplify this measure by focusing on a single mechanical switch and five
selected contextual factors.
The entirety of this chapter is reproduced from the following journal article: Memarian N,
Venetsanopoulos AN, Chau T. Mutual information as a measure of contextual effects on single
switch use. The Open Rehabilitation Journal. 2009; 2:1-10.
Since the Open Rehabilitation Journal, published by Bentham Science Publishers Ltd., is open
access, no permission was required from the publisher to reproduce the article here. This article
can be found on the publisher's website at
http://www.bentham-open.org/pages/content.php?torehj/2009/00000002/00000001/1torehj.SGM
29
3.1 Abstract
Users with disability interact with augmentative and alternative communication devices,
environmental control units, and computers via an access technology. While caregivers routinely
exploit contextual information to interact meaningfully with individuals who are nonverbal and
have severe motor impairments, access technologies to date have largely ignored context.
Contextual factors include the environmental and personal factors in the model of functioning
and disability introduced by The World Health Organization's International Classification of
Functioning, Disability and Health in 2001.
We propose the use of mutual information as an objective means of measuring the impact of
contextual factors on mechanical single switch usage. We show that common performance
measures (e.g. sensitivity, specificity and response time) relating to switch use can be
quantitatively unified within a mutual information measure. We exemplify the use of mutual
information in the assessment of switch use in the presence of selected contextual factors. This
information theoretic measure facilitates performance comparison amongst users and can
potentially help in classification of contextual stimuli in terms of their impact (i.e. facilitating,
barrier, neutral). Our examples with able-bodied participants and an individual with disability
indicate that mutual information can be sensitive to changes in contextual factors. Mutual
information may thus inform the design of individualized access technologies.
3.2 Introduction
3.2.1 Access Pathways
Individuals with severe and multiple disabilities often cannot employ conventional means of
physical access, such as speech and gestures. An alternate channel is often required for
communication and interaction with the environment. In rehabilitation terminology, that channel
is termed an access pathway and constitutes the critical front end of an access solution [2]. From
30
a system engineering perspective, an access solution is a system that receives a physical or
physiological expression of the individual’s intention as input. The system ultimately translates
this input into a functional activity. A wide range of access pathways have been developed, from
simple mechanical switches to sophisticated physiological ones. For a comprehensive review of
emerging access technologies please see [2], [89].
3.2.2 The Role of Context
The usability of an access pathway is affected by both the personal characteristics of the user and
the milieu in which the device is used [3]. In the World Health Organization’s International
Classification of Functioning, Disability and Health (ICF) [90], these personal and
environmental characteristics are formally encapsulated into the concept of context, which is the
collection of factors that define the physical, social, cultural and attitudinal environment within
which people live their lives. Personal factors include age, sex, and indigenous status, personal
resources (including physical and mental abilities), and personal perceptions [91] while
environmental factors consist of the built environment, ambient temperature, time of day, air
quality, and ambient noise, among other characteristics of the surrounding milieu.
The ICF recognizes that contextual factors and their interactions can influence the health
domains of activity and participation. It is therefore not surprising that participation is measured
in terms of performance in the individual’s typical environment [92]. Likewise, it is believed that
while the physical and cognitive ability to use an access pathway is important, primary emphasis
should be placed on arranging favourable circumstances to enable the most effective and
efficient device usage [93]. Indeed, a contextual factor may facilitate the process of working with
an access pathway, hinder the process, or have no significant effect. For example, the presence of
people may distract the user and hence become a hindering contextual factor. On the other hand,
prior knowledge of the task at hand may help the user to anticipate future interactions and hence
serve as a facilitating factor. Identifying the type and magnitude of impact of contextual factors
may eventually lead to the development of context-aware access pathways. Such systems can
31
potentially sense and compensate for the negative effects of certain hindering contextual factors
or exploit facilitating factors to improve the robustness of an access pathway. However, access
strategies developed to date do not account for personal and environmental factors and thus their
usability declines when applied in more than one environment or by different users.
The importance of contextual factors in the delivery of assistive technology [94], functional
assessments [95], measurement of participation [96], [97], [98], technology assessment [1], and
modelling disability [99] has been recognized in the literature.
3.2.3 Gauging Contextual Effects through Information Theory
Based on the discussion in the previous section it therefore appears worthwhile to quantify the
impact of contextual factors on the effectiveness of a given access pathway. In particular, an
information theoretic approach may be useful, since access pathways, like other human-machine
interfaces, involve the transmission of information. The combination of cognitive and physical
abilities of the human user constitutes the communication channel through which the information
is transmitted. The characterization of the human being as a communication channel is a
nontrivial challenge [100]. For many years, quantitative models of information transmission in
humans have been a subject of interest in fields such as psychology and human-computer
interaction (HCI). In the 1950s, soon after C.E. Shannon proposed information theory and the
idea of mutual information in his famous 1948 paper [101], many psychologists tried to
determine maximum information transmission rates in humans for various tasks, such as choice-
reaction [102], [103], perception and learning [104], speed-accuracy of motor responses [105],
vigilance [106] and recognition memory [107], [108], [109]. Studies show that the rate of
information transmission in humans is affected by factors such as the dimensionality of the
stimulus, the probability of stimulus occurrence and the context in which the stimulus occurs
[110], [111].
32
Several groups have made use of information theory principles to model human-computer
interaction. More than half a decade ago, Hick [102] and Hyman [103] published their findings
from several choice-reaction experiments. The underlying theme of the ensuing Hick-Hyman law
was that response time is not only a function of the number of stimulus alternatives but can also
be considered a linear function of stimulus information (entropy). This finding spawned a
number of attempts to design optimal control and display codes for human-computer interaction
[112], [113], [114] although others have remarked that there has been limited uptake of these
early concepts [115]. Chan & Childress crafted theoretical relationships between human-machine
noise and human-machine output velocity [116], formulating the channel capacity of a human-
machine system. They applied this formulation to estimate information transmission rates in
human pursuit tracking [117]. On a separate front, Ogawa evaluated computer usability with a
human-to-computer information transmission model [118] while Poock and Blackstone
quantified the effectiveness of an augmentative and alternative communication (AAC) display by
measuring input (the symbols requested by the clinician) and output (the responses given by the
AAC user) entropies from which they estimated the relative information transmitted [119].
Recently, Sanger and Henderson optimized the graphical layout of an AAC device using a model
of information rate and channel capacity that exploited the relationship between movement time
and the number of buttons per screen, the size of the buttons, and the number of sequential
button presses per word [120].
Building upon previous research, the objective of this paper is to demonstrate that contextual
effects on a single switch access pathway can be meaningfully ascertained by estimating the rate
of information transmission in the human communication channel. Using data from human
participants, we illustrate a number of ways in which mutual information could be used to
quantify switch usage within an experimental setting. This mutual information framework may
inform the design of access pathways. Before we present our proposed framework however, we
review the concept of mutual information in the case of transmitting binary information through
a communication channel.
33
3.2.4 Mutual Information in Transmission of Binary Information
From an information theory perspective, the amount of information conveyed by a message from
a source is measured by entropy. The more we know about the message from the source, the less
the uncertainty or entropy, and the less the amount of information [121]. From information
theory, we also know that in order to transmit information, there needs to be a transmitter
(sender) and a receiver. The medium used to convey information from the transmitter to the
receiver is called the communication channel.
We consider the formulation of mutual information in the case of transmitting binary information
through a communication channel, that is, when there are only two message choices at any given
time. An example of this communication scenario is a single switch, which is often used as the
access pathway to derive a user interface. The switch can be either on (closed) or off (open). Let
},{ 21 xxX = represent the messages that the transmitter sends, i.e., =1x ON, =2x OFF. Similarly,
let },{ 21 yyY = represent the messages that reach the receiver, i.e., =1y ON, =2y OFF. From
information theory [122], the entropy of the transmitter (that is the rate at which the message
source generates information) for the binary case is:
∑=
−=2
1
)(log)()(i
ii xpxpXHbits per message (3.1)
where ( )ip x is the probability of transmitting message i.
Similarly at the receiver end the entropy is:
∑=
−=2
1
)(log)()(j
jj ypypYH bits per message (3.2)
where ( )jp y is the probability of receiving message j.
34
In real communication channels, the transmitted and received messages may not coincide as a
result of channel noise. The uncertainty as to which message was transmitted when a given
message is received, is written as )( YXH and is a natural measure of the information lost in
transmission [122]. The quantity ( )YXH or the conditional entropy or equivocation of X about Y is estimated as
∑ ∑∑= = =
−===2
1
2
1
2
1
)|(log)()|()()|()|(j i j
jijjijj yxpypyxpypyYXHYXH bits per message (3.3)
where ( | )i jp x y is the probability that message i was transmitted given that message j was
received. Conditional entropy depends on how often X is transmitted, or how often Y is
received, as well as on the errors made in transmission.
If we take )( XH and ( )YXH as entropies in bits, then );( YXI , the mutual information of X and Y is defined by [121]:
)|()();( YXHXHYXI −= (3.4a)
Equation (3.4a) can be written in terms of joint entropies as follows:
)](),([)();( YHYXHXHYXI −−=
),()()( YXHYHXH −+= (3.4b)
The joint entropy ),( YXH is found from (3.5), in which ),( ji yxp is the probability of occurrence
of each pair of outcomes:
2 2
21 1
( , ) ( , ) log ( ( , ))i j i ji j
H X Y p x y p x y= =
= −∑∑ (3.5)
35
Mutual information is the amount of information that we learn about X , by virtue of knowingY ,
or put another way, it is the amount of information about X that is transmitted through the
channel [123]. The reader unfamiliar with information theory may refer to [121], [122], [124],
[125], [126] for further reading.
3.3 Methods
3.3.1 Proposed Framework for Evaluating Contextual Effects on Single
Switch Use
Here we introduce an information theoretic interpretation of the interface between a human user
and a single switch access pathway, as depicted in Figure 3.1. The computer which presents the
visual and auditory cues can be considered as the transmitter, and the single switch (a means of
acknowledging the cues) can be thought of as the receiver. The user constitutes the
communication channel between the transmitter and the receiver. The proposed arrangement
resonates closely with the human-machine communication models suggested by [103] and [127].
The user’s accuracy and response time, and hence the characteristics of the communication
channel, are affected by external factors such as information presentation modality, ambient
noise or time of day as well as the cognitive and physical resources invoked.
Consider that the transmitter (T) in Figure 3.1 presents an auditory or visual cue to the user. We
distinguish between actionable (transmit “1”) and non-actionable (transmit “0”) cues, the former
being those which ought to trigger a switch activation by the user, or equivalently produce a 1 at
the receiver (R). Likewise, non-actionable cues should not produce a response from the user and
hence the corresponding receiver data should be 0.
36
Figure 3.1: Information-theoretic paradigm for single-switch access. A subject may react to
visual, written or auditory stimuli by pressing a single mechanical switch. The computer, which
generates the cues, can be considered the transmitter (T); the mechanical switch (a means of
acknowledging the cues) can be thought of as the receiver (R); the user constitutes the
communication channel between the transmitter and the receiver (CC).
3.3.2 Estimating Mutual Information
With the above framework and equation (3.4b), mutual information can be estimated from the
entropies )(XH , )(YH and ),( YXH . Based on signal detection theory in psychology [111],
Figure 3.2 summarizes all possible sender-receiver cue combinations for the framework
described in Figure 3.1. Rows of the table correspond to cues presented to the user (transmitted
cues, analogous to ix in equation (3.1)), while the columns represent cues acknowledged by the
user (received cues, analogous to jy in equation (3.2)). True positives (TP) indicate the number
of transmitted actionable cues that the user correctly acknowledged. False negatives (FN)
indicate the number of transmitted actionable cues that were erroneously missed by the user.
False positives (FP) indicate the number of non-actionable cues that the user erroneously
acknowledged, and true negatives (TN) indicate the number of non-actionable cues that were
correctly rejected by the user. The sum of all the cells is equal to the total number of messages
37
displayed to the user, which we denote as N . The empirically estimated probabilities of the
aforementioned events are [119]:
p (sending actionable cue) = NFNTPxp +
=)( 1 (3.6)
p (sending non- actionable cue) = NTNFPxp +
=)( 2 (3.7)
p (receiving actionable cue) = NFPTPyp +
=)( 1 (3.8)
p (receiving non-actionable cue) = NTNFNyp +
=)( 2 (3.9)
p (sending actionable cue & receiving actionable cue) = NTP=)y,p(x 11 (3.10)
p (sending actionable cue & receiving non-actionable cue) = NFNyxp =),( 21 (3.11)
p (sending non-actionable cue & receiving actionable cue) = NFPyxp =),( 12 (3.12)
p (sending non-actionable cue & receiving non-actionable cue) = NTNyxp =),( 22 (3.13)
In the present study, we obtained numerical estimates of mutual information by inserting the
above probabilities into Equations (3.1) to (3.5). Mutual information represents the information
shared between input X and outputY , i.e. what the user is cued to do with the switch and what
he/she actually does. When maximized over input distributions, mutual information gives us
channel capacity.
38
Figure 3.2: All possible sender-receiver cue combinations.
3.3.3 Participants
Twelve able-bodied adults, aged 27.3± 9.3 years (six male) and a 29 year old male with C1-C2
incomplete spinal cord injury, participated in this study. Participants had no visual, auditory or
cognitive impairments. Written consent was obtained from all participants.
3.3.4 Protocol
The protocol was approved by the Research Ethics Boards of Bloorview Kids Rehab and the
University of Toronto. The protocol consisted of a repeated trial of five different single-switch
selection activities. Participants were presented with a series of visual or auditory cues (Table
3.1). In the first session, participants responded to the cues in a controlled environment. In the
second session participants responded to the same cues but in the presence of a modified
contextual factor6. In each experiment outlined in Table 3.1, various visual or auditory messages
were presented to the user, one at a time. Each message was presented for 600 ms. Only a subset
of messages were actionable cues, meaning that their presentation by the computer (transmitter)
should ideally trigger a switch press by the user (receiver). For example in experiment 1(a), the
6 Choice of factors was guided by clinical acumen (this footnote has been added for clarification and does not appear in the related journal article).
39
actionable cue was a picture of a cat. Able-bodied participants were required to respond to
actionable cues by pressing the spacebar on a computer keyboard. The participant with disability
completed the tests by using a sip and puff switch. He was asked to respond to actionable cues
with a puff, which was immediately translated to a left mouse click by means of a USB-based
switch-to-click converter. The times of all switch activations as well as the time of presentation
of actionable cues were automatically logged for subsequent off-line analysis.
It is important to note that the protocol was intended to exemplify different applications of the
mutual information measure in gauging switch use context and not to conclusively determine
specific contextual effects across populations.
3.3.5 Data Collection and Analysis
The logged stimulus presentation times, and switch activation times were used to calculate the
number of TPs, FNs, FPs, and TNs. Each cue (regardless of being actionable or non-actionable)
was presented to the user for 600 milliseconds. i.e. the cues changed every 0.6 second. In order
to generate a TP, the user should have made switch activation within 600 milliseconds after onset
of an actionable cue. No switch activation in this 600 millisecond interval translated to a miss or
FN. A switch activation that occurred after the presentation of a non-actionable cue was recorded
as a FP. A non-actionable cue correctly ignored by the user was considered a TN. With these
data, mutual information was estimated according to section 3.3.2. In addition, sensitivity,
specificity and average response time were calculated for each experiment. Average response
time was calculated as the average time it took the user to generate true positives.
3.4 Results
The estimated mutual information for the five experiments is summarized by the box plots in
Figure 3.3. By visual inspection of these plots, we notice that mutual information may be
40
Table 3.1: Summary of the contextual factors explored in the present study and the
corresponding experimental procedures.
Experiment Contextual Factor Procedure
1 Presentation modality The user is asked to respond by activating the switch upon observing a specific object (e.g., a cat) on the display in three different trials: (a) 100 pictures are presented to the user one at a time. The object of
interest (i.e. picture of cat) appears at some random points in the sequence
(b) 100 words are displayed to the user, one word at a time. The word of interest (i.e. cat) appears at some random points in the sequence
(c) 100 words are spoken, one at a time, by the computer to the user. The word of interest (i.e. cat) is announced at some random points in the sequence
2 Prior knowledge The user is asked to perform the following two trials: (a) A mixed series of 100 characters including letters (English
alphabet all in caps), numbers and symbols are displayed to the user, one at a time. The user is asked to activate the switch only when he/she observes a letter of the alphabet. The user has no a priori knowledge of the next character in the sequence.
(b) Letters of the English alphabet (all in caps) are displayed to the user in order (total of 100 letters). The user is asked to activate the switch when he/she observes a vowel (i.e., ‘A’, ‘E’, ‘I’, ‘O’ and ‘U’).
3 Background noise A sequence of 100 pictures is displayed to the user. The user is asked to activate the switch when he/she observes a specific object on the screen (e.g. a cat), while: (a) The environment is quiet (b) A source of noise is present in the environment (background
conversations or music)
4 Colour The user is asked to perform the following two trials: (a) 100 Uncoloured shape outlines (circle, square, triangle, and
rectangle) are displayed in random sequence to the user, one at a time. The user is asked to activate the switch upon observing a specific shape (e.g. square).
(b) 100 Shapes are displayed one at a time, in random sequence to the user. The user is asked to activate the switch upon observing a specific shape with a specific colour (e.g. purple square). All shapes are colourless except the purple square and a purple cross that appear randomly in the sequence.
5 Presence of people in the environment
The user is asked to activate the switch upon hearing an auditory instruction while (a) There is nobody except the examiner in the environment
(controlled environment) (b) There are people trespassing and chatting around the user (natural
environment)
41
influenced by certain contextual factors. A paired t-test revealed significant differences between
various contextual conditions in Figure 3.3(a), (b), (e).
Figure 3.3: Estimated mutual information of the twelve participants for the five experiments: (a)
effect of presentation modality, (b) effect of prediction of choice, (c) effect of ambient noise, (d)
effect of colour, (e) effect of presence of people in the environment. Graphs (a), (b) and (e)
depict significant differences (paired t-test).
42
To determine if these changes are statistically significant, one would need to systematically
quantify the natural fluctuation in mutual information due to the inter-trial variability of human
performance. We exemplify this statistical testing with one able-bodied participant. The
participant repeated the experimental trials 15 times over the course of several days. Having
confirmed that the 15 sets of mutual information were normally distributed (Kolmogorov-
Smirnov test for normality), we used a paired samples t-test to compare mutual information
between the baseline condition (no manipulation of contextual factors) and in the presence of a
modified contextual factor. Results indicate that mutual information changed significantly
between written and auditory presentation modalities ( 148.4−=t , 001.0=p ). The corresponding
mutual information probability density functions are shown in Figure 3.4(a). These densities
were estimated from the mutual information of the 15 repeated trials, using a Gaussian mixture
model. One can visually verify the difference between written (dashed line) and auditory (dark
solid line) presentation modalities. For the same individual, mutual information was also
significantly lower over the 15 trials in the natural environment, likely due to the presence of
other people ( =t 2.386, =p 0.032). This change in mutual information is depicted in Figure
3.4(b) by the shifted density function (dashed line) corresponding to the mutual information
measured in the natural environment. Other contextual factors did not significantly change the
mutual information of this particular participant.
Table 3.2 shows the mutual information for the participant with disability. Here, we exemplify
the use of mutual information to gauge the effect of stimulus duration. MI600 and MI1000 denote
the participant’s mutual information when the stimulus presentation period was 600 and 1000
milliseconds, respectively. The participant with disability repeated the baseline condition (i.e.,
600 ms stimulus duration) seven times to form baseline densities of mutual information. Using
these baseline data and a one-sided, one-sample t-test, we statistically tested whether or not the
MI1000 was statistically greater than the mean of the baseline distribution. In all cases, we found
that indeed MI1000 > MI600 (p<0.05), implying that the longer stimulus duration is a facilitating
factor for the individual with disability.
43
Figure 3.4: Estimated probability densities of the mutual information for (a) presentation
modality and (b) presence of people in the environment.
44
3.5 Discussion
The mutual information measure provides a platform for quantitative assessment of contextual
effects on single switch use. In particular it offers the following advantages over the
conventional performance measures such as sensitivity and specificity.
3.5.1 Contextual Factor Role Characterization
Through the mutual information paradigm one can specify the type of effect of a particular
contextual factor (i.e., facilitating, neutral, or hindering). From Figure 3.5, we see that in
response to various presentation modalities, 42% of the participants (participants 2, 3, 4, 6, and
11) exhibited the highest mutual information in response to written stimuli, while another 42% of
the participants (participants 1, 7, 8, 10, 12) had the highest mutual information in response to
pictorial stimuli. Therefore we can infer that presenting the information in the written modality
can be facilitating for the former group and information presentation in the pictorial modality can
be facilitating for the latter group. Participants 3, 6, 7, 8, 11, and 12 (50% of participants) had
their lowest mutual information in response to auditory stimuli, implying that the auditory
modality is the least preferable for these participants. Message presentation exclusively by the
auditory modality can thus be considered a hindering factor for these individuals. Participant 9
responded to pictorial and written modalities with comparable mutual information, implying that
for this particular subject, presentation modality is a neutral factor; neither the pictorial nor the
written modality had an advantage over the other. From the above examples, it is evident that
with mutual information, we can ascertain the role of each contextual factor (facilitating,
hindering or neutral) for each subject, according to ICF prescriptions.
While the mutual information score is a single number, it allows us to gauge the explicit role of a
contextual factor, even for those factors that have non-uniform effects across conventional
performance measures. An example is the colour factor, which improved sensitivity but lowered
specificity for the majority of participants. Comparing the participants’ mutual information
45
values under no colour and colour conditions indicates that 67% of participants (participants 1, 2,
3, 6, 9, 10, 11, 12) had better information transmission (higher mutual information) when the
shapes were displayed to them without colour. Hence, the colour factor had an overall hindering
effect on this population of participants.
Table 3.2: Effect of stimulus duration on mutual information of the participant with disability.
Experiments MI600 [bits] MI1000 [bits]
Presentation modality (a) Pictorial (b) Written (c) Auditory
0.1991 ± 0.0521
0.1974 ± 0.0591
0.4271 ± 0.0950
0.4690
0.4690
0.5436
No prior knowledge (random stimulus presentation) 0.2289 ± 0.0865 0.8366
No background noise (quiet environment) 0.1991 ± 0.0521 0.4690
Uncoloured cues 0.1687 ± 0.0374 0.5197
No people in the environment (other than examiner) 0.7560 ± 0.1256 0.9982
MI600 (mean and standard deviation) and MI1000 denote the mutual information corresponding to stimulus presentation periods of 600 and 1000 milliseconds. In all cases, MI1000>MI600, p<0.05 by a one-sided, one-sample t-
test.
3.5.2 Ranking Contextual Factors Based on the Significance of their
Effect
Standard statistical tests such as the analysis of variance (ANOVA) are not conducive to the
ranking of independent variables (contextual factors) based on the significance of their effect on
the dependent variables (performance measures). For example, both colour and the presence of
people in the environment had significant effects on switch activation specificity. It is unclear
which effect was more influential. The mutual information measure however allows us to rank
different contextual factors according to the strength of their effects. Returning to the example of
individual effects of colour and the presence of people in the environment we calculated the
46
difference between a subject’s mutual information with and without the presence of colour and
subsequently, the difference between a subject’s mutual information in the absence and presence
of people in the environment. Figure 3.6 presents these mutual information deltas for all twelve
participants. The figure suggests that the performance of 75% of the participants (participants 2,
4, 5, 6, 7, 8, 9, 10, 11) was affected more significantly by colour than by the presence of people.
3.5.3 Inter-Subject Comparison
Mutual information addresses the problem of inter-subject comparison. Assignment of a unifying
quantitative measure to each participant provides the convenience of ranking participants based
on the magnitude of impact they experienced from each factor.
3.5.4 Objective Method of Performance Assessment for Users with
Disability
Results in Table 3.2 represent an example of how an individual’s mutual information can be
increased by modifying the characteristics of stimulus source (in this case, increasing the
presentation period of each stimulus). This observation suggests another benefit of using the
mutual information measure. When changing the physical characteristics of the communication
channel is not possible (i.e. disability cannot be removed), an optimal information transmission
rate can still be found by adjusting source characteristics (e.g. AAC display colours), receiver
characteristics (e.g. changing the type of switch), or accounting for the effect of environmental
and personal context (e.g. blocking out environmental noise or altering the presentation
modality).
47
3.5.5 Limitations of Present Study
The goal of this study was to demonstrate the use of mutual information as a measure of
contextual effects on single switch use, rather than to definitively establish specific contextual
effects. In terms of the latter topic, several improvements can be made in subsequent studies. For
example, one could go beyond the univariate analyses presented here and explore possible
interaction effects between multiple contextual factors through additional data collection and
multivariate analyses. The number of participants can be increased to provide a stronger basis for
identifying the main role of each contextual factor. Mutual information reliability has only been
considered in one case and should be further investigated in future studies. In particular, the
effect of time, a central contextual factor, can also be analyzed.
Figure 3.5: Participants’ mutual information in response to pictorial (dark bars), written
(unshaded bars) and auditory stimuli (grey bars).
48
Figure 3.6: Difference in participants’ mutual information with and without the following
contextual factors: presence of people (unshaded bars) and colour (shaded bars).
3.6 Conclusion
In this paper, we proposed the use of mutual information as a quantitative method for measuring
the impact of context, as defined by the World Health Organization’s ICF model, on information
transmission within a single-switch paradigm. Using empirical data from a single mechanical
switch experiment, we demonstrated that mutual information can provide: (a) an objective way
to determine the ICF classification of contextual factors (i.e., facilitator, barrier or neutral), (b)
the ability to rank different contextual factors according to the strength of their individual effects
on the performance of a given user, (c) a means of consolidating non-uniform effects of a
particular contextual factor into a unique measure, and (d) the ability to rank different individuals
based on their mutual information for a given task. This measure may be particularly useful as an
objective means of establishing optimal information transmission rates in individuals with
disability.
49
Chapter 4
4 Validating an Infrared Thermography-based Access
Switch (the Infrared Thermal Switch)
In this chapter I use the mutual information measure discussed in chapter 3 to validate the
proposed infrared thermography-based access technology introduced in chapter 2. In this
validation study, I also investigate the effect of stimulus presentation modality (i.e., visual,
auditory, or audiovisual) on the efficiency and response time of infrared thermal switch users.
The entirety of this chapter is reproduced from the following journal article: Memarian N,
Venetsanopoulos AN, Chau T. Validating an infrared thermal switch as a novel access
technology. BioMedical Engineering OnLine. 2009; 9:38.
Since the BioMedical Engineering OnLine, published by BioMed Central, is open access, no
permission was required from the publisher to reproduce the article here. This article can be
found on the publisher's website at http://www.biomedical-engineering-
online.com/content/9/1/38
4.1 Abstract
Background: Recently, a novel single-switch access technology based on infrared thermography
was proposed. The technology exploits the temperature differences between the inside and
surrounding areas of the mouth as a switch trigger, thereby allowing voluntary switch activation
50
upon mouth opening. However, for this technology to be clinically viable, it must be validated
against a gold standard switch, such as a chin switch, that taps into the same voluntary motion.
Methods: In this study, we report an experiment designed to gauge the concurrent validity of the
infrared thermal switch. Ten able-bodied adults participated in a series of 3 test sessions where
they simultaneously used both an infrared thermal and conventional chin switch to perform
multiple trials of a number identification task with visual, auditory and audiovisual stimuli.
Participants also provided qualitative feedback about switch use. User performance with the two
switches was quantified using an efficiency measure based on mutual information.
Results: User performance (p = 0.16) and response time (p=0.25) with the infrared thermal
switch were comparable to those of the gold standard. Users reported preference for the infrared
thermal switch given its non-contact nature and robustness to changes in user posture.
Conclusions: Thermal infrared access technology appears to be a valid single switch alternative
for individuals with disabilities who retain voluntary mouth opening and closing.
4.2 Background
An access technology is a system that senses a physical movement or physiological change from
a person and uses this information to drive a user interface [2]. This technology is particularly
useful for individuals with severe motor impairments who lack a conventional means of access
and communication. Examples of access technologies developed for this population are sip and
puff switches [36], chin switches [29], computer vision-based systems [18], [128], or
electromyography-based systems [7], [8]. While attempts to design and develop new access
technologies are ongoing [16], [64], [39], [9] an important challenge is to ensure that the new
access technology is valid. Validity in the present context refers to the correlation between the
switch activations of the access technology of interest and those of an established gold standard.
This concept is similar to criterion validity [129]. In engineering design, validity testing is
usually performed with prototypes to verify system functionality [130]. If an access technology
fails to be valid, the user will find it frustrating to use, and hence the technology will quickly lose
51
its appeal as an assistive device. On the other hand, if the system proves to be adequately valid,
then poor performance of the access technology is most likely due to user error rather than
technology error. In other words, validity testing is crucial to distinguish between user-dependent
errors and algorithm-dependent errors.
In the present paper we describe and discuss empirical validity testing of an enhanced version of
the infrared thermography-based access technology proposed in [39]. We report an experiment
designed to gauge the validity of this access technology with respect to a conventional chin
switch. The chin switch was chosen as the benchmark for two reasons. Firstly, the chin switch is
guaranteed to activate upon the application of force, and secondly, the chin switch can be
activated by exactly the same motion that triggers the infrared thermal switch, i.e., mouth
opening.
In the following sections, first, we explain the infrared thermal switch and its underlying
computational algorithm. Next, we detail the experimental setup for the validation study and the
ensuing data analysis. The paper will be rounded out with a presentation and discussion of the
empirical results.
4.3 Methods
4.3.1 Infrared Thermal Switch
The infrared thermography-based access technology used in this study is an enhanced version of
the system proposed in [39]. In particular, we employ a different motion estimation method and
introduce a new anthropometric filter of non-mouth objects to reduce false positive activations.
Also, we invoke additional software and hardware to realize a real-time infrared thermographic
single switch. This system captures the facial temperature distribution of a user with an infrared
52
thermal camera and translates the user’s voluntary mouth opening activity into a switch
activation using a computerized algorithm.
Instrumentation
The infrared thermal video was acquired in real-time with a ThermaCAM SC640 by FLIR [131].
The camera had a thermal sensitivity of 60≤ mK and a resolution of 640×480 pixels. The
acquired video was non-radiometric and grey scale, with bright intensities corresponding to
warm regions and dark intensities corresponding to cold regions (Figure 4.1(a)). The video was
sent to a DELL Inspiron 1560 laptop (Intel® Core™2 CPU T7200 @ 2.00 GHz and 2.00 GB of
RAM) via a fire wire cable for real-time processing. Upon detection of a mouth open-close
sequence by the video processing algorithm, a mouse click or key press was generated using a
latching relay by DLP Design Inc. and a Swifty USB switch interface by Origin Instruments™.
Video Processing Algorithm
The video processing algorithm for the infrared thermal switch was implemented in SIMULINK
R2008a. The algorithm consisted of three main modules, namely user face localization, motion
and intensity analyses, and filtering of non-mouth objects. In the following, we describe an
enhanced and online version of the original algorithm proposed in [39], with a new motion
estimator and new anthropometric filters. We remind the reader that although enhancements to
the original algorithm are presented below, the main focus of this study was the concurrent
validity testing of the thermal switch, which is detailed in the subsequent section.
Face Localization
Given that the face is generally warmer than the surrounding environment, the face localization
module began with an adaptive thresholding of the frame’s grey scale intensities based on Otsu’s
method [81]. Thresholding was followed by the selection of the user’s face, i.e., the largest round
or semi-round object within the black and white thresholded image. Zero padding was used at all
53
four sides of the frame to account for user head motion which may have caused the segmented
facial region to overlap with the frame borders. Figure 4.1(a) and Figure 4.1(b) show the original
input video frame and the localized face within that frame, respectively.
(a) (b)
(c) (d) (e)
Figure 4.1: Visual representation of the infrared thermal switch video processing algorithm. (a)
Input greyscale thermal video frame, (b) Result of face localization, (c) Result of intensity
analysis (warm area mask), (d) Intersection of warm area mask and motion mask, (e) Open
mouth detected and marked on the video. Note that for ease of visualization, images (c)-(e) only
show the smaller region demarcated by the dashed box in (b).
54
Motion and Intensity Analyses
This phase of the algorithm is based on two key observations. First, the inside of the human
mouth is typically warmer than the surrounding areas of the face. Consequently, an open mouth
appears as a bright patch on the thermal video. Second, opening and closing of the mouth
involves motion.
In order to detect the warm regions within the face, a second Otsu thresholding [81] was
performed on the segmented face, this time with double the previous threshold value. This
thresholding yielded the warm region mask, as shown in Figure 4.1(c).
For motion tracking, the sum of absolute differences (SAD) was used. By performing a two-
dimensional SAD between the current and previous frames, we basically looked for similarity
between the two consecutive images. Denote the current video frame as matrix I with
dimensions ),( ii NM and the previous video frame to be matrix T with dimensions ),( tt NM .
Then the two-dimensional SAD matrix [132], [133] is given by:
∑ ∑−
=
−
=
−++=)1(
0
)1(
0)),(),((),(
t tM
m
N
nnmTknjmIabskjSAD
(4.1)
Where
10 +−<≤ ti MMj
and
10 +−<≤ ti NNk
The greater the similarity between the two matrices, the smaller the SAD values. In contrast,
motion results in dissimilarity and hence a bigger SAD value. The algorithm retained only those
pixels in the frame whose SAD values fell in the motion range of mouth open/close activity.
55
Each person opens his or her mouth at a different speed and so individualized SAD thresholds
were selected during an initial calibration period. The motion analysis phase yielded a motion
mask, which included all the pixels that have comparable motion to mouth opening and closing
activity. Of course, this mask did not only include the mouth pixels, but also other areas that
have similar motion (e.g., eyebrows, eye lids), hence, the need for a simultaneous warm, region
mask. Figure 4.1(d) depicts the warm facial regions that also exhibited motion.
Filtering Non-Mouth Objects
Motion and intensity analyses identified multiple objects as open mouth candidates. However not
all of those objects represent a true open mouth. For example, the regions around the eyes, chin
and forehead were commonly singled out. These objects passed the preceding intensity and
motion filters because they were both warm [44] and moving. Thus in this last stage of the
algorithm, non-mouth objects were filtered based on size, morphological, and anthropometric
features. Specifically, if the object departed from one or more of the following conditions, it was
flagged as a false positive by the system and subsequently removed. The first condition is
basically a size filter that eliminated objects that were too small or too large to be an open mouth.
The minimum and maximum size limits in this equation are an order of magnitude larger than
the values previously reported in [39], because in the current study we use a higher resolution
infrared thermal camera, and also place the camera closer to the participant to minimize
background artefacts. The second and third conditions are morphological filters, which removed
respectively, regions that were too long to qualify as a mouth and areas that were too hollow as
the mouth was expected to be solid. The fifth condition is a newly introduced anthropometric
filter. Based on human face anthropometry, the mouth is located in the lower half of the menton-
sellion length and in the middle third of the head breadth [84]. The menton-sellion length is the
distance in the midsagittal plane between the menton landmark at the bottom of the chin and the
sellion landmark at the deepest point of the nasal root depression. The head breadth is the
maximum horizontal breadth of the head above the level of the ears [84], [85].
200 pixels < Area < 1500 pixels
Eccentricity ≤ 0.9
56
(Area of object)/(Area of bounding box) > 0.4
21
(menton-sellion length) < Object’s centroid < 43
(menton-sellion length)
31
(head breadth) < Object’s centroid < 32
(head breadth)
Objects that did not satisfy the expected size, shape, and location of an open mouth within the
face were filtered out in this phase. Finally the only object that satisfied all the above conditions
was considered to be an open mouth. The bounding box of this object is marked on the video
frame as shown in Figure 4.1(e)7.
4.3.2 Validity Testing Experiment
Objective
The objective of this experiment was to validate the detection algorithm of the infrared thermal
switch introduced above. The accuracy of the infrared thermal switch may be impacted by user
error or algorithm error. An error is either a missed activation or a false alarm. For example a
user-dependent miss is when the user does not open his/her mouth, when he/she is cued to do so,
while an algorithm-dependent miss is when the user does voluntarily open his/her mouth upon
cue, but the system does not detect it and thus does not activate the switch. A false activation can
similarly be generated either because of the user opening his/her mouth at the wrong time or the
algorithm erroneously picking up non-mouth objects as mouth opening. The infrared thermal
switch validation study was carried out to focus on the validity of the presented computerized
algorithm by isolating the algorithm-dependent errors from user-dependent errors.
7 In this algorithm, the rate limiting process is the initial Otsu thresholding in the face localization module. The algorithm has time complexity of ( )NMO × , where M and N are the dimensions of the input thermal video frame (this footnote has been added for clarification and does not appear in the related journal article).
57
Setup
User performance with the infrared thermal switch was compared with a chin switch during a
stimulus-response task. The chin switch was selected as the benchmark because of its guaranteed
activation upon application of force and also because it can be activated by exactly the same
motion that triggers the infrared thermal switch, i.e., mouth opening, therefore allowing for the
study of concurrent validity of the two switches. The chin switch (Ablenet Flex Switch – product
#58550) was mounted on a table in front of the user with a Slim Armstrong mounting system
such that the switch sat slightly below the user’s chin when his/her mouth was closed, and it
came in contact with the user’s chin when he/she opened his/her mouth. The infrared thermal
camera (FLIR ThermaCam SC640) was positioned anterior and lateral to the participant at a 45°
angle. Figure 4.2 depicts the experiment setup.
Participants
The participants were ten able-bodied university students, (23.5 ± 2.12 years old, 6 females).
None had prior experience with either a chin or infrared thermal switch. The purpose of the study
was explained to the participants prior to the study and they provided written consent to
participate. The participants' knowledge of the purpose of the study was unlikely to bias their test
performance and their views on advantages and disadvantages of switches. Since the study
objective was algorithm validation, we needed to focus on algorithm-dependent errors. Hence,
able-bodied participants were recruited to minimize user-dependent errors.
Protocol
The protocol consisted of 3 test sessions on separate days, with 6 trials per session. Each trial
was 3 minutes in length. The task was to select a target number assigned by the experimenter
from a pseudo-random sequence of numbers by opening and closing the mouth. Each sequence
of numbers involved 60 cues out of which 10 were actionable cues, i.e., 10 target numbers for
58
which the participant was expected to activate the switches. The numbers were presented every
2500 milliseconds, and in three stimulus modalities, namely, visual (i.e., number displayed on a
computer screen), auditory (i.e., number articulated in a synthesized voice via the computer
speakers), and audiovisual (i.e., both modalities simultaneously). In each session, there were two
trials of each modality. The participants were asked to open and close their mouth upon
observing and/or hearing the target number. The presentation modality order was randomized
throughout the trials to minimize bias. To avoid learning or habituation effects, a different target
number was used for each session. Overall, each participant received 180 actionable cues (3
sessions×6 trials×10 actionable cues). During each trial, the following data were automatically
logged for both the chin and the infrared thermal switches: stimulus presentation time, response
time, number of hits (true positives), true negatives, misses (false negatives), and false alarms
(false positives). The protocol was approved by the Research Ethics Boards of Bloorview Kids
Rehab and University of Toronto.
At the end of every session, the participants were asked to provide feedback about their
experience with each switch. The participants were asked questions to solicit their subjective
feedback and the research personnel took written notes of their answers. Specifically, they were
asked to comment on their preference of switch type based on the postural requirements of
switch use, the post-session level of fatigue, and the perceived disadvantages of using each
switch over an extended period of time.
59
Figure 4.2: Infrared thermal switch validity testing experiment setup.
Data Analysis
A mutual information measure (MI) [134], previously reported for single switch assessment was
used to objectively gauge the performance of each user. MI encapsulates different performance
measures in a single number. Based on the data collected in each trial, MI was calculated for the
participant’s chin switch performance and separately for his/her performance with the infrared
thermal switch. MI was calculated from the number of true positives, true negatives, false
positives and false negatives as shown in equation (4.2), where )(XH is the entropy of the
stimulus, H(Y) is the entropy of the switch, and Y)H(X, is their joint entropy.
H(Y)]Y)[H(X,H(X)MI −−= (4.2)
60
),()()( YXHYHXH −+=
According to [134], the maximum MI is obtained when no false negatives (misses) or false
positives (false alarms) occur, in which case MI = H(X). Information transmission efficiency is
thus calculated as:
Efficiency = )(XHMI
(4.3)
To rigorously assess if user performance with the infrared thermal switch was significantly
different from his/her performance with the chin switch, a two-sample Kolmogorov-Smirnov
(KS) test with a 5% significance level was conducted. Two-way repeated measures analysis of
variance investigated the effects of switch type, stimulus modality and their interaction on
efficiency, and response time.
4.4 Results
A plot of the participants’ efficiency with the infrared thermal and chin switches averaged over
all eighteen trials is shown in Figure 4.3. From visual inspection, the chin switch seems to
exhibit higher efficiency than the infrared thermal switch in most cases. However, a two sample
KS test indicated no significant differences between the efficiency measure for the two switches,
across participants (p = 0.16).
The two-way repeated measures ANOVA overwhelmingly indicated that there were no main
effects of switch type (p = 0.98) or presentation modality (p = 0.78) on efficiency and no effect
of their interaction on efficiency (p = 0.998).
61
The two-way repeated measures ANOVA also revealed no main effects of switch type (p =
0.25), or stimulus presentation modality (p = 0.82) on the response time. Also, no interaction
effect of switch type and stimulus presentation modality on the response time was found (p =
0.997).
4.5 Discussion
All participants achieved comparably high efficiencies with both the chin and the infrared
thermal switches. From Figure 4.3, all participants except one achieved an information
transmission efficiency of greater that 80% with the infrared thermal switch. The two-sample KS
test confirmed the validity of the infrared thermal switch.
From qualitative observations of participant behaviour, we noticed that during the trials,
participant 4 gradually tended to lean back in his seat, such that his chin became misaligned with
the chin switch. As a result, his chin did not contact the switch when he opened his mouth. This
resulted in many chin switch misses and thus the chin switch efficiency dropped as shown in
Figure 4.3. Participant 10 reported difficulty in maintaining a posture that did not activate the
chin switch unintentionally. This participant only opened her mouth slightly during the task,
meaning that there was very limited chin excursion. As a consequence, she leaned forward to
keep her chin very close to the chin switch. This physical arrangement generated many false
positives as her chin contacted the chin switch even when her mouth was closed. Unsurprisingly,
chin switch efficiency declined. Remarkably, in both these instances, the infrared thermal switch
maintained greater than 80% efficiency. These examples demonstrate that the infrared thermal
switch is more robust than the conventional chin switch to user motion and changes in user
posture. Thus, the infrared thermal switch may be a more appropriate option for people with
severe and multiple motor disabilities who may have involuntary and spastic movements.
62
Figure 4.3: Efficiency of infrared thermal and chin switches averaged over all eighteen trials, for
all ten participants. The vertical lines show standard deviation.
The absence of switch type and stimulus modality effects on performance efficiency implies that
the infrared thermal switch can be as useful as a conventional chin switch, regardless of the
sensory modality in which the information is presented to the switch user.
Most of the participants reported that the audiovisual stimulus modality was the easiest to follow
and respond to, while the audio only stimulus was the most difficult. This agrees with the recent
literature contending that the response time to bi-modal stimuli (e.g., audiovisual) is shorter than
the response time to uni-modal stimuli (e.g., visual or auditory alone) [135]. Some participants
found the combination of chin switch and audio stimulus particularly challenging. Those
participants closed their eyes and intently listened to the audio stimuli. While having their eyes
closed did not affect their infrared thermal switch performance, it did cause participants to miss
63
several chin switch activations simply because they lost track of the switch’s physical location.
This finding highlights the convenience of the infrared thermal switch for patients who may have
vision impairments; they do not need to actively track the location of the switch relative to their
chin.
Table 4.1 summarizes a list of switch pros and cons as indicated by the participants in their
qualitative feedback. All participants preferred the flexibility of the infrared thermal switch over
the chin switch. They found it difficult to maintain a stationary position at all times, as required
for accurate chin switch use. Arguably, with some users with disabilities who have limited
movement, such as those with muscular dystrophy, maintaining a stationary position would not
be an issue. The non-contact nature of the infrared thermal switch was another major plus
reported by the participants in this study.
There are further advantages associated with the infrared thermal switch. Being a non-contact
access technology, it is hygienic and does not involve the risk of contact injury. Infrared
thermography is lighting and colour invariant [77]. Thus the infrared thermal switch can be used
by people of all ethnicities, night or day, regardless of ambient lighting conditions. These
benefits make the infrared thermal access technology an appealing access tool for people with
severe motor disabilities. Any individual with voluntary control of his or her mouth movement
may be a candidate for infrared thermal switch use. Indeed, like other binary switches, the
infrared thermal switch can facilitate typing, game playing and communication, among other
activities. Despite these benefits, there are limitations associated with this new technology, such
as those indicated in Table 4.1. The findings in the present study are based on a modest sample
of ten able-bodied adults. However, the small between-subject variation in efficiency values
suggests that the sample was quite homogenous in terms of switch usage.
64
Table 4.1: Selected qualitative feedback (pros and cons) from participants.
Infrared Thermal Switch Chin Switch
Pros’s Flexibility: switch works even in spite of user
body motion and head rotation (as long as user
face is in camera’s field of view)
Instantaneous activation
Non-contact More sensitive switch
Possible to activate the switch with closed eyes
Con’s Reduced detection sensitivity immediately after
taking cold drink or food (this problem is
alleviated with time)
Low detection sensitivity due to
considerable head rotation or change
of posture
Reduced detection sensitivity if too much saliva
in the mouth (this problem is alleviated with
one swallow)
Too many false positives due to
small head motion, talking or any
activity that makes the chin touch
the switch.
Prolonged periods of switch operation may
cause fatigue or jaw pain
May cause infection if it enters user
mouth
May cause skin irritation
The practical issue of cost is worthy of mention. While fully radiometric infrared thermal
imaging is considered a costly technology, the proposed infrared thermal switch can be
affordably implemented given that only a grayscale intensity representation rather than the
absolute temperature of the client’s face is required. Hence, a less costly (approximately 2000
USD) non-radiometric camera can be used. In addition to the camera, very simple and
inexpensive hardware (e.g. module latching relay and USB switch converter) have been used in
the design of the infrared thermal switch. Infrared thermal cameras are usually designed to
withstand harsh climates and rough handling [14]. Nonetheless, repairing an infrared thermal
camera that is beyond its warranty period can be a costly undertaking.
65
In addition to cost, the required in situ technical adjustments should also be considered. The
client’s orofacial temperature is likely to vary subsequent to the consumption of food or drink or
with changes in ambient temperature. As a consequence, recalibration of the client’s intensity
and motion thresholds may be required at the start of each session.
4.6 Conclusion
In this paper, we presented an experiment with 10 able-bodied individuals to validate a novel
infrared thermal switch algorithm. A previously proposed mutual information measure was used
to quantify user performance. Users completed multiple trials of a number identification task
using both infrared thermal and conventional chin switches concurrently. User performance was
comparable between the proposed thermal switch and the gold standard chin switch, establishing
the concurrent validity of the former. This is a clinically positive result given that the thermal
switch offers the additional benefits of being non-contact and robust to user posture. The infrared
thermal switch is valid with respect to a conventional chin switch and ought to be considered for
clients with severe disabilities who retain voluntary mouth opening/closing.
66
Chapter 5
5 Body Functions and Structures Pertinent to Infrared
Thermal Switch Use
The validity of the infrared thermal switch was demonstrated in chapter 4. While validating the
technology is crucial, it is also imperative to study the factors associated with the user that may
impede the use of this access switch. To this end, this chapter discusses physiological, motoric,
sensory and cognitive impairments that may negatively affect infrared thermal switch use. The
discussion revolves around seven clients with severe disabilities in their current environments.
Where applicable, I indicate how the access technology or the physical environment can be
tailored to mitigate the activity limitations due to these impairments.
The entirety of this chapter is reproduced from the following journal article: Memarian N,
Venetsanopoulos AN, Chau T. Body functions and structures pertinent to infrared thermography-
based access for clients with severe motor disabilities. Assistive Technology. –Accepted for
publication.
Reprinted by permission of the publisher Taylor & Francis Ltd, http://www.informaworld.com
5.1 Abstract
Infrared thermography has been recently proposed as an access technology for individuals with
disabilities but body functions and structures pertinent to its use have not been documented.
Seven clients (2 adults, 5 youth) with severe disabilities and their primary caregivers participated
67
in the study. All clients had a Gross Motor Functional Classification System (GMFCS) level of
5, but each possessed a unique set of extant physical movements. We tested the clients’ ability to
activate the infrared thermal access technology via a cued mouth open-close exercise. In
addition, the clients or their primary caregivers were interviewed for descriptive information
about the clients’ physical, cognitive and sensory function, communication skills, medical
background, and history of switch use. Several impairments were identified as contraindications
to infrared thermal access, spanning physiological (e.g., frequent fluctuations in body
temperature, seizures, pain), motor (e.g., poor trunk control, involuntary movements, atypical
mouth posture) and sensory/cognitive (e.g., inconsistent contingency awareness) sub-domains.
We identified key impairments in body functions and structures that limit infrared thermography-
based access. Potential changes to the access technology (e.g., software and hardware) and
physical environment to overcome those limitations are suggested.
5.2 Introduction
Many individuals with severe physical disabilities face severe mobility limitations. This is while
their condition may not be associated with cognitive limitations [136]. Severe physical
disabilities include but are not limited to spastic quadriplegic cerebral palsy (CP), high level
spinal cord injury (SCI), severe forms of muscular atrophy, progressive multiple sclerosis (MS),
and late stage amyotrophic lateral sclerosis (ALS) [1]. In rehabilitation engineering, an access
technology is one which translates the client’s intentions into a functional activity (e.g., [36], [9],
[31]). Following the formulation of Tai, Blain & Chau [2], an access technology comprises two
components: 1) the actual sensors or input devices by which an expression of functional intent
(e.g., a movement or physiological change) is transduced into an electrical signal; and 2) a signal
processing unit that analyzes the input signal and generates a corresponding control signal. The
control signal is used to drive a user interface (e.g., computer), through which an appropriate
functional activity (e.g., typing) is performed within a specific user environment. A wide range
of access technologies have been developed from simple push buttons to sophisticated brain-
computer interfaces (e.g., [3], [137]).
68
Recommendations of the appropriate access technology depend on the user’s abilities,
preferences and personal resources, the user’s physical and social context, as well as the
requirements and characteristics of the technology itself [138]. Several models for optimal
matching of access technologies (and more broadly, assistive technologies) to clients with
disabilities exist in the literature, with the Matching Person & Technology (MPT) [139], and the
Human Activity Assistive Technology (HAAT) [3] models being two prominent examples. A
common feature in these models is the focus on physical and mental abilities of the potential user
of the access technology; e.g., referred to as ‘functional capabilities’ in the MPT model [140],
and ‘intrinsic enablers’ in the HAAT model [3]. In this paper we focus on physical, motor,
cognitive and sensory functions pertaining to access technology use. In keeping with
terminology of the International Classification of Functioning, Disability and Health (ICF) [90],
we will refer to the above as body functions and structures and the associated difficulties as
impairments.
While testing new access technologies with able-bodied participants is often a cost-effective
means of iterative prototyping, results from those tests are unlikely to be representative of the
performance of participants with disability. It is therefore important to specifically study the
body functions and structures which may promote or forestall the use of emerging access
technologies for clients with disabilities.
Recently, a new access technology based on infrared thermal imaging was proposed and
demonstrated [39]. This access technology, hereafter referred to as the infrared thermal switch, is
triggered by the user’s voluntary mouth open activity and is completely non-invasive and non-
contact. Unlike other computer vision-based access technologies, the infrared thermal switch is
lighting and color invariant, and hence its performance is less likely to vary with different users
or working environments. The infrared thermal switch is more hygienic and safe compared to
technologies such as a tongue switch or sip and puff, as the risk of switch-borne infection or
69
airway obstruction is eliminated. While the infrared thermal switch was developed for clientele
with extant control of the orofacial muscles responsible for jaw motion, impairments of body
functions and structures that may impede infrared thermal access have not been previously
described in the literature. In the present paper, we attempt to elucidate some of these key
impairments and propose potential solutions to mitigate their hindering consequences.
5.3 Methods
5.3.1 The Access Technology
The infrared thermal switch functions as a binary switch, which is activated by voluntary mouth
opening. Mouth opening involves both motion and local temperature change in the thermogram,
as the inside of the human mouth is generally warmer than the surrounding tissue. We
implemented a video processing algorithm to robustly track mouth opening activity in infrared
thermal video [39] and translate that activity into real-time switch activation using simple
hardware.
The infrared thermal video was acquired with a ThermaCAM SC640 by FLIR [131]. The camera
had a thermal sensitivity of 60≤ mK and a resolution of 640×480 pixels. The acquired video
was non-radiometric and grey scale, with bright intensities corresponding to warm regions and
dark intensities corresponding to cold regions (Figure 5.1). The thermal camera was positioned
anterior and lateral to the client at a 45° angle. This camera location was chosen over the often-
used frontal view, keeping in mind the eventual application as an access switch where the user's
field of view ought to be unobstructed. For angles of view up to 45°, the recorded temperature
error is 5.0≤ C° [46]. The reader is referred to [39] for technical details of the video processing
algorithm and the associated calibration.
70
The temperature, motion, and mouth morphology thresholds were calibrated on a session by
session basis. Every time a client tries to use the system, his/her saved profile from the last
session is loaded. Similar to visible light video-based access technologies (e.g., [16], [14]), minor
recalibration is required at the start of a new session.
Figure 5.1: Infrared thermal image of a client’s face. Brighter tones represent warmer regions.
The oral cavity is warmer than the surrounding facial tissue.
5.3.2 The Clients
Since the purpose of the study was to develop a list of impairments in body functions and
structures that might impede infrared thermal switch use, the inclusion criterion was intentionally
broad, i.e., a client was eligible to participate in the study as long as he/she had a motor
impairment. We recruited seven clients with severe motor disabilities (Level 5 on the Gross
Motor Functional Classification System; 3 females, 17.3± 8 years old) and their primary
caregivers. The clients were either inpatients at the hospital or outpatients (current or former)
who were invited to the hospital to take part in the study. All participants provided informed
71
consent. The study was approved by institutional ethics review boards of the participating
hospital and academic institution. Detail descriptions of each client are provided in Table 5.1.
5.3.3 Study Protocol
Each client and his or her primary caregiver attended a 1.5 hour session at a paediatric
rehabilitation hospital8. The tests were carried out in hospital rooms with ambient temperature
23± 1 °C and humidity 50± 5%. The room temperature and air flow was maintained through the
hospital’s central air conditioning system. The session was divided into two parts. In the first
part, the client was asked to perform a sequence of 20 cued mouth openings and closings. The
client was seated comfortably in his or her wheelchair or on the hospital bed in a semi-supine
position and was cued to open his or her mouth and to hold it ajar for one second before closing
the mouth. Clients were given an auditory prompt upon every open and close action. The end of
each mouth closing was followed by a 3 second rest before the onset of the next mouth opening.
A mouth open was defined as any mouth posture where the lips were separated and a mouth
close was defined as a mouth posture with the lips touching each other. We asked those clients
who could open and close their mouth voluntarily to rate their perceived level of fatigue after the
activity as either being ‘not fatiguing’, ‘moderately fatiguing’, or ‘very fatiguing’. Where
applicable, we also asked participants to explain their preference between the infrared thermal
switch and their current mode of access.
In the second part of the session, the client (where feasible) and the client’s primary caregiver
were interviewed about the client’s physical, cognitive and sensory function, communication
skills, medical/health background, and history of using other switches (Please see Appendix B at
the end of this thesis for the list of questions). The questions were inspired from the background
information questionnaire suggested by [3] (chapter 4, appendix 4-1A), the Assistive Technology
8 The name of the hospital is not mentioned to comply with the blind review policy.
72
Table 5.1: Client descriptions (Part I) Client Description A Demographic: 26 year old male; spastic quadriplegic CP; lives with family; primary caregiver is mother; 1
year college education. Physical: unable to voluntarily move trunk or limbs; comfortable position: semi-supine in wheelchair with neck laterally flexed left or right; rarely maintains forward posture; retains limited control of neck; exhibits adequate control of facial and ocular muscles (open/close mouth reliably; blink on cue). Cognitive: contingently aware; follows cues diligently; very bright; fully intact. Sensory: no vision or hearing impairments. Communication: current communication: non-speech vocalizations interpretable by mother; able to answer yes/no questions; expressive facial/body gestures (e.g., laughs, tilts head back and looks up when happy; limbs tense up and move involuntarily when excited; frowns and whines when in discomfort); attempts to say “no more” repeatedly to express boredom. Medical: seizure free and independently ventilates; fed orally. Switch history: Mini-cup switch using a finger; abandoned due to excessive false alarms, inconsistent control and involuntary movements.
B Demographic: 10 year old male; mixed spastic quadriplegic CP and hypotonia; lives with parents and siblings; primary caregivers are parents; grade 5 segregated special needs school. Physical: minimal trunk and neck control; frequently adopts forward-leaning posture; can assume upright position with considerable effort; difficulty maintaining head midline and can only raise head for short periods of time; comfortable position: reclined and supine; tolerates wheelchair with extensible supporting harness around his shoulders; mouth is usually open; can smile, move tongue and gaze in different directions but cannot raise eyebrows or blink voluntarily. Cognitive: contingently aware; follows cues. Sensory: near-sighted with cortical vision impairment; has used corrective glasses for a year; hearing intact. Communication: hearing is primary receptive communication modality; capable of yes (smiling or nodding his head) and no (shaking his head) communication; can vocalize with effort (e.g., mama, yes, no); expressive body language (e.g., smiles and vocalizes when happy; cries when in pain or discomfort; whines and grinds teeth on wheelchair tray when bored). Medical: independently ventilating; history of minor seizures (treated with Depekene); fed through a gastric feeding tube. Switch history: jelly bean hand switch activated by right forearm; actively engaged in switch training; accuracy is modest, motor execution is generally slow but considered non-fatiguing.
C Demographic: 11 year old female; quadriplegic dystonic CP; lives with parents and siblings; primary caregivers are parents; grade 6 special education class within integrated school. Physical: no trunk, neck, or limb control; comfortable position: seated or lying down; tolerates wheelchair with extensible supporting harness around her shoulders; mouth is usually open; rarely closes the mouth (closing the mouth requires a lot of effort); can smile, protrude tongue, raise eyebrows, and control eye movement voluntarily. Cognitive: contingently aware; follows cues. Sensory: inconsistent visual function; poor visual fixation; hearing intact. Communication: current communication: smiling/blinking for yes; can vocalize names of siblings or parents occasionally; expressive facial and body language (e.g., smiles or laughs when happy; laughs and kicks when excited, grimace and crying when in pain or discomfort). Medical: seizure free and independently ventilates; fed orally as well as with a gastric tube. Switch history: fiber optic eyebrow switch; adequate accuracy, motor execution is slow and occasionally fatiguing.
73
Table 5.1: Client descriptions (Part II) Client Description D Demographic: 16 year old male; spastic quadriplegic CP; lives with family; primary caregivers are parents;
grade 11 integrated secondary school with a fulltime educational assistant. Physical: scoliosis and weak upper body control prior to spinal surgery; improved trunk and neck control post surgery; can assume upright position; comfortable position: seated; adequate control of facial and ocular muscles (opens mouth, and blinks on cue); poor lip closure Cognitive: contingently aware; follows cues. Sensory: no vision or hearing problem. Communication: current communication: facial gestures and yes/no questions (nod for ‘yes’ and head shaking for ‘no’); smiles, laughs and squeals when happy or excited; moans when in pain; makes full fists and extends both arms forward when angry or bored. Medical: currently seizure free and ventilates independently; fed with gastric feeding tube. Switch history: Good accuracy and little fatigue with head switch; chin switch to drive wheelchair.
E Demographic: 14 year old female; moderate to severe athetoid CP and dyskinetic movements; lives at home with guardian who is also primary caregiver; grade 9 special education class within integrated school Physical: scoliosis; very inconsistent upper body control; requires full support via head rest for neck control; comfortable position: seated or semi-supine; low ocular and facial muscle control (poor lip closure and saliva control); can blink, smile, and open/close mouth voluntarily. Cognitive: contingently aware; follows cues; very bright; fully intact. Sensory: vision and hearing intact; difficulty focusing on images because of dyskinetic movement. Communication: expressive communication disorder; faint and slow speech comprehensible to primary caregiver only; communication mainly through eye-gaze and head gestures (nod for ‘yes’ and head shake for ‘no’); uses a DynaVox for writing and homework; big smile when happy; quiet and tense when in pain; fidgets and yawns when bored; high tone and involuntary body movements when excited. Medical: ventilates independently; frequent nocturnal seizures; on anti-dyskinetic agent; gastric tube fed. Switch history: golf ball joystick controlled with her right hand and a jelly bean switch triggered with her left elbow.
F Demographic: 13 year old female; chromosome 2 deletion; lives with family; primary caregivers are parents; grade 7 special education class within an integrated school. Physical: scoliosis; very little upper body control; frequently adopts forward-leaning posture; comfortable posture: seated; no meaningful facial gestures. Cognitive: contingent awareness uncertain. Sensory: far-sighted; has difficulty wearing glasses because of frequent eye rubbing. Communication: has no established means of communication; very vivid and inconsistent facial gestures (can be detected only by mother); coos when happy or excited; moans when in discomfort; closes or rubs eyes when bored. Medical: ventilates independently but needs a mask when she aspirates; two to six seizures every day (accompanied with a lot of involuntary body movement, vocalizations, and coughing); lethargic; fed with gastric tube. Switch history: has never had an access switch.
G Demographic: 31 year old male; C1-C2 incomplete spinal cord injury; lives in government-subsidized supportive housing with 24 hour attendant; primary caregivers are the care attendants and his mother; university undergraduate student. Physical: completely paralyzed below shoulders; comfortable posture: seated; very good control of neck and facial muscles (open/close his mouth reliably, make facial gestures, blink on cue); experiences tremors frequently; involuntary movements in shoulders and limbs in reaction to changes in temperature or startle. Cognitive: on par with senior undergraduate students. Sensory: near-sighted; wears glasses; no hearing problems. Communication: verbal; clear facial gestures. Medical: nocturnally ventilated; rarely experiences seizures; fed orally. Switch History: sip and puff to drive wheelchair, make mouse clicks, and to use environmental control unit; infrared tracker device to control the mouse cursor on the computer screen; Dragon Naturally Speaking speech recognition software for typing.
74
Device Predisposition Assessment (ATD PA), section A: consumer ratings of functional
capabilities [141], and previous literature pertaining to assistive technology assessment [142],
[143]. Involvement of the client and primary caregivers has been identified as a vital step to
assistive technology adoption [144] and tools such as questionnaires or interviews are often
recommended for this purpose [145]. The client/caregiver responses are summarized in Table
5.1. In many cases, the clients were nonverbal and could not practically respond to the entire
questionnaire due the lack of an established access technology. For this reason, we had to rely on
caregiver input in those cases.
We acknowledge that more test trials with each client will be useful in the assessment of the
client’s infrared thermal switch performance. However, it should also be noted that impairments
in body function and structures considered here are largely static conditions that are unlikely to
change drastically with time. Therefore it can be argued that the observations from one session
provide a representative snapshot of the client’s abilities at the present time.
5.3.4 Data Analysis
In order to assess the thermal switch algorithm’s performance for those clients who could open
and close their mouth when cued, truth sets were prepared manually, i.e., mouth open-close was
prompted by the researcher. For each client, the truth set contained ‘respond’ and ‘ignore’
blocks. The ‘respond’ blocks were the video frame numbers corresponding to the beginning and
ending of each mouth opening. The ‘ignore’ blocks were the video frames corresponding to the
rest (no cue) periods. Within each ‘respond’ block, a true positive (TP) was defined as the
detection of a candidate mouth open blob within the thermal image that was spatially situated
within the valid mouth zone. This zone was defined as the bounding box defined by the end
points of the line maximally spanning the width of the mouth at the onset of opening and the end
points of the line maximally spanning the height of the mouth when fully ajar. Blobs erroneously
detected as mouth open in ‘ignore’ blocks were counted as false positives (FP). A mouth open
that was missed by the algorithm (i.e., no detected object during a ‘respond’ block) was counted
75
as a false negative (FN). A true negative (TN) occurred when there was no mouth opening and
the algorithm concluded the same (i.e., no detected mouth open blob during an ‘ignore’ block).
Based on these numbers, sensitivity and specificity were calculated (Equations (5.1) and (5.2)). It
should be noted that the clients who completed the cued open-close test did comply fully with
the experimental protocol, i.e., they opened and closed only on cue. Hence any drop in sensitivity
or specificity is due to the infrared thermal switch and not the client.
100×+
=FNTP
TPySensitivit (5.1)
100×+
=FPTN
TNySpecificit (5.2)
5.4 Results
Four clients were able to complete the mouth open-close test (i.e., clients A, D, E, and G) as
shown in Table 5.2. Among these clients, the infrared thermal switch had the highest sensitivity
(90%) for client G and the lowest sensitivity (65%) for client D. Nevertheless, the switch
achieved very good specificity (95% or higher) for all four clients, indicating that the switch
experienced very few false alarms.
Table 5.2: Detection results for clients who completed the mouth open-close test
Client Sensitivity Specificity
A 90% 98.4%
D 65% 95.9%
E 85% 97.2%
G 90% 100%
76
The perceived levels of fatigue, means of access, and switch preference for these four clients are
summarized in Table 5.3. Aside from client G, who was verbal, responses from all other clients
were solicited through yes/no questions and interpreted by the caregiver. These responses were
complemented with the caregiver’s knowledge of the client’s physical ability and access switch
preference. Three of the four clients preferred the infrared thermal switch over their current
mode of access because the former is non-contact, more hygienic, and allows for more postural
flexibility.
Table 5.3: Client feedback about the mouth open-close test
Client Fatigue level Current mode of access Preference Reason
A Not fatiguing None (relies on vocalizations and gestures)
Infrared thermal switch
The switch will allow me to communicate my intention on my own. I will not need to have a caregiver to facilitate what I say. I can be more independent.
D Moderately fatiguing
Chin switch and head switch Head switch Head movement is less fatiguing than mouth opening for me. I am used to my head switch.
E Not fatiguing Elbow switch Infrared thermal switch
I can activate the elbow switch only if it is placed exactly under my left elbow. I like to be able to have a switch that allows me to make a selection even if I have to move.
G Not fatiguing Sip and puff Infrared thermal switch
The sip and puff switch goes inside my mouth. I have to be very careful about the hygiene of the sip and puff switch in order not to get infections or other diseases. The infrared thermal switch is non-invasive and non-contact. I will not have to worry about the hygiene issue with the infrared thermal switch.
As result of our structured interviews with the clients and their caregivers, we have assembled a
collection of impairments in body function and structure that may limit the activity of infrared
77
thermal switch access, as shown in Table 5.4. The impairments are divided into three categories,
namely physiological, motoric, and cognitive and sensory impairments.
Table 5.4: Impairments in body function and structure and consequent limitations to infrared
thermal access
Group Impairment Activity Limitation
Physiological Frequent body temperature fluctuations
Difficulty activating the infrared thermal switch at different times
Seizures Difficulty maintaining a stable posture; Difficulty breathing independently; Difficulty focusing on a task
Pain/discomfort Difficulty maintaining a stable posture; Difficulty focusing on a task
Motor Lack of upper body control Difficulty maintaining a stable posture; Difficulty holding head up; Difficulty facing the infrared thermal camera
Involuntary movements Difficulty maintaining a stable posture
Muscle spasms Difficulty facing the infrared thermal camera
Atypical mouth posture Difficulty with fully opening the mouth; Difficulty maintaining complete lip closure
Uncontrolled drooling Difficulty maintaining a closed mouth; Difficulty maintaining a stable posture (tendency to lean forward to discharge saliva)
Hypertonic or hypotonic muscles leading to premature fatigue
Difficulty performing mouth open-close activity; Difficulty focusing on a task
Sensory and cognitive
Impaired vision Difficulty perceiving visual cues and visual feedback from switch activation
Impaired hearing Difficulty perceiving auditory cues and aural feedback from switch activation
Inconsistent contingent awareness
Difficulty understanding the action required to trigger the infrared thermal switch (difficulty understanding cause and effect)
Difficulty following cues Difficulty performing timed exercises (e.g., scanning methods)
78
5.5 Discussion
Out of the four clients who completed the mouth open-close test, clients A and D both had
spastic quadriplegic CP. Yet, the results in Table 5.2 show that their sensitivity and specificity
scores were quite different. Client D’s lower sensitivity was mainly because of his very warm
face, which made it challenging for the algorithm to distinguish an open mouth from the
surrounding facial tissue. Also, client D completed the mouth open-close test while lying down
in his bed. Hence the infrared thermal camera essentially captured a top view of the client’s
mouth, which was inconsistent with the posture of the other three clients who sat upright in their
wheelchairs while performing the mouth open-close test. Moreover, there were a greater number
of false positives in the case of client D, resulting in lower specificity. Client D’s mouth was
slightly open even when he was prompted to close his mouth. The algorithm detected this
incomplete closure as a mouth open, but since the truth set was temporally in an ‘ignore’ block, a
false positive was registered. Client D’s facial temperature fluctuation was another source of
false positives. In fact, to compensate for those fluctuations, the algorithm’s temperature
threshold had to be recalibrated three times for client D. In contrast, client A sat upright in his
wheelchair, had good lip closure during the rest periods, and exhibited no drastic skin
temperature fluctuation throughout the session. These differences in physical presentation
indicate that the thermal access intervention should not target the disability diagnosis but the
individual’s unique abilities [92].
Among the seven clients, the two (clients A and G) who are regularly fed orally achieved the
highest sensitivity and specificity scores in the mouth open-close test. Oral feeding involves the
ability to suck, chew and swallow [146]. Those clients who can take food and drinks orally are
therefore more likely to be able to perform and maintain lip closure, and control their saliva.
Consequently, it is expected that this group of clients may activate the infrared thermal switch
more proficiently than those who are primarily fed through a gastric tube.
79
Clients B, C, and F failed to open and close their mouths reliably for different reasons. Client B’s
mouth is normally open. He has been trained to close his mouth for a very short time (almost 1
second) in order to swallow food for occasional oral feeding. Client C cannot close her mouth
because of insufficient facial muscle control and attempting to do so is extremely effortful and
fatiguing. Client F seemed to have the physical ability to open her mouth with effort but she
showed no response to the mouth open-close cues. Lack of upper body control was another
factor that impeded the use of the infrared thermal switch for clients B and F. Because of his low
muscle tone, client B’s neck often flopped forward (Figure 5.2a). Similarly due to minimal trunk
control, client F leaned forward frequently. In both cases, the client’s mouth was no longer in the
infrared thermal camera’s field of view. Also, client F’s habitual rubbing of her eyes tended to
occlude parts of her face (Figure 5.2b).
(a) (b)
Figure 5.2: Sample postures in which the client’s mouth is obscured from the thermal camera’s
view. (a) Client with low muscle tone, unable to consistently hold his head up; (b) Client with
habit of frequently rubbing her eyes.
80
5.5.1 Potential Solutions
Identifying factors that may affect the viability of an assistive technology is an important part of
matching users to appropriate technologies [140]. Steps four and five of the Matching Person and
Technology Assessment Process recommend that the assistive technology consumer and
professional discuss challenges to the use of the technology, and work to identify specific
intervention strategies [140].
The purpose of the infrared thermal switch is not to alter the client in any way, but to provide
‘hard and soft’ technologies to enable the client with disability to be functional in activities of
daily living [3] and thereby more fully participate in life situations [92]. With this mindset, we
offer potential solutions, i.e., changes to the technology and physical environment, to mitigate
the negative effect of the impairments in body function and structures identified in Table 5.4. In
keeping with the categorization of impairments in Table 5.4, potential solutions are presented as
physiological, motoric, cognitive or sensory.
Physiological
A dynamic auto calibration module [147], [148] can be added to the infrared thermal switch
algorithm to address the problem of frequent body temperature fluctuations. The system would
continuously update its threshold values based on the intensity of the most recent detections.
Also, a re-calibration of the temperature thresholds would be executed whenever the infrared
thermal switch algorithm detects no activation within a long period of time or detects too many
activations within a short period of time.
Motor
For clients with limited upper body control, harnesses, vests, straps, or belts can be used to
secure the client’s trunk in his/her wheelchair and prevent him/her from falling (e.g., flexible
81
chest harness, shoulder holder [152]). Head support systems can be deployed to help the client
maintain a stable head position [153].
For involuntary body movements, face localization and tracking algorithms [39] can be
implemented to focus on the client’s face as the region of interest and ignore other body parts.
Also, motion analysis algorithms [39] can be used to filter involuntary body movements. These
features are incorporated in the current infrared thermal switch prototype.
In order to mitigate the problem of the client’s face leaving the single camera’s field of view due
to spasticity, multiple cameras can be used to capture client’s facial region from different angles
[16]. Based on [16], we suggest that two non-radiometric infrared thermal cameras each
positioned to one side, anterior and lateral to the client at a 45° angle should provide adequate
facial coverage for a client in an upright or semi-supine position. The cameras can be mounted
on the wheelchair with poles such as the Slim Armstrong mounting system. This placement is
intended to capture the clients face in spite of head rotation, while leaving the client’s field of
view unobstructed. The cost of this solution would not be outlandish as the thermal switch can be
implemented with non-radiometric cameras. We acknowledge that this suggested solution is
speculative and remains to be tested clinically.
Atypical mouth posture can be addressed by implementing adjustable morphological analysis
filters to accommodate the unique mouth open shape of each client [39]. To activate the infrared
thermal switch, the client must be able to voluntarily generate two distinct orofacial thermal
signatures. In this study, we sought mouth opening for 1 second and closing to the point that the
lips are touching. However, other variations, such as partial and full opening may also be
possible motion dyads. If the client’s mouth is usually open, the infrared thermal switch trigger
can be reversed, such that detection of a mouth close would activate the binary switch.
82
The current prototype of the infrared thermal switch requires mouth open activity as the switch
trigger. If a client does not retain the ability to control his/her mouth movement reliably or if a
client’s mouth is concealed with an external object like an oxygen mask, other voluntary facial
activities that involve motion and temperature change can be harnessed to trigger the switch.
Examples of such activities are voluntary eye blinking and strong exhalation. The periorbital
regions, i.e., soft tissue situated around the orbit of the eyes, have been reported to be among the
warmest regions of the human face [154], [56]. Voluntary eye blinking provokes motion in the
periorbital regions which can serve to trigger the infrared thermal switch. Furthermore, it has
been reported that exhalation can be tracked by using infrared thermal imaging [74], [56]. The
air that is exhaled has a higher temperature than the typical background of indoor environments
(e.g., walls). Therefore, the particles of the expired air possess a distinct thermal signature in the
infrared image [74]. Should the client possess the ability to forcibly expire, the infrared thermal
switch can be geared to activate with this activity instead of mouth opening. Again, this
suggestion while motivated by published evidence on thermal image processing, remains to be
tested empirically.
To minimize user fatigue due to prolonged durations of mouth open-close activity, the algorithm
can be adjusted such that smaller and slower mouth openings can trigger the switch [39].
Moreover, based on the nature of the target task, optimized scanning patterns (e.g., group-row-
item scanning, circular scanning) can be employed to minimize the number of switch activations
required to make a selection.
Cognitive and Sensory
Clients, who have difficulty following cues, can be trained with timed exercises using interactive
programs such as the Compass Assessment software [149]. Using engaging audiovisual cues
may also facilitate cue-tracking for clients. Contingency awareness paradigms can be exercised
to help the client better grasp cause and effect [150], [151].
83
It is worth mentioning that contingency awareness and the ability to follow cues and are required
for all user-driven access technologies and are not specific to the infrared thermal switch.
5.5.2 Other Factors Pertinent to Infrared Thermal Access
This paper has focused on the effect of impairments in body functions and structures that limit
infrared thermal switch access. However, it is also important to consider personal contextual
factors [90], such as race, gender, lifestyle, and habits that relate to the background of an
individual’s life and living, independent of his or her health condition or health state [155]. For
thermal switch access, important personal contextual factors may include the cultural
acceptability of publically opening and closing one’s mouth and the financial viability of
investing in a thermal camera.
Environmental contextual factors are also paramount to one’s overall health and function.
Environmental factors refer to the physical, social and attitudinal environments in which people
live and conduct their lives [90], [92]. For thermal switch access, the regional climate (e.g.,
tropical versus temperate) and habitual social consumption of hot or cold foods may present as
important environmental factors. Separate studies should be conducted to explore the impact of
personal and environmental factors that may affect infrared thermal switch usage by the target
population. In the present study, the infrared thermal switch was tested in an indoor environment,
i.e. in a hospital setting. Since infrared thermography is sensitive to ambient changes in
temperature, heating from the sun may interfere with algorithmic performance. A future outdoor
study is required to quantify this solar effect.
5.6 Conclusion
We conducted a case series to identify body functions and structures that are important to the use
of an infrared thermal switch, a new access technology for clients with severe motor disabilities.
84
This access technology translates local temperature changes associated with voluntary mouth
opening to activations of a binary switch. Based on structured switch testing and interviews with
seven clients and their primary caregivers, we derived a list of physiological, motoric, sensory
and cognitive impairments that may negatively affect infrared thermal switch use. Where
applicable, enhancements to the access technology and environmental supports were suggested
as ways to mitigate the effects of these impairments of body functions and structures.
85
Chapter 6
6 Customizing the Infrared Thermal Switch for an
Individual with Severe Motor Impairments
In light of the recommendations discussed in chapter 5, this chapter reports the process of
tailoring the infrared thermal switch to a client with severe spastic quadriplegic CP, who has
never had a reliable means of access. I adopt a client-centred approach, where the tasks the client
wishes to perform are taken into consideration throughout the access switch design,
development, and training.
The entirety of this chapter is reproduced from the following journal article: Memarian N,
Venetsanopoulos AN, Chau T. Client-centred development of an infrared thermal access switch
for a young adult with severe spastic quadriplegic cerebral palsy. Disability and Rehabilitation:
Assistive Technology- In press.
Reprinted by permission of the publisher Informa Healthcare, http://informahealthcare.com/
6.1 Abstract
Purpose: This study reports the client-centred development of a non-contact access switch based
on infrared thermal imaging of mouth opening-closing activity of an individual with severe
spastic quadriplegic cerebral palsy.
86
Method: Over a six month period, the client participated in five test sessions to inform the
development of an infrared thermal switch. The client completed eight stimulus-response trials
(switch test) and eight word-matching trials (scan test) using the infrared thermal switch and
provided subjective feedback throughout.
Results: For the switch test, the client achieved an average correct activation rate of 90% and
average response time of 2.4 seconds. His mean correct activation rate on the scan test improved
from 65% to 80% over the course of system development, with an average response time of 11.7
seconds.
Conclusions: An infrared thermography switch tuned to a client’s extant orofacial gestures is a
practical non-invasive access solution and warrants further research with clients with severe
physical disability.
6.2 Introduction
According to the United Nations, in 2006, around 10 percent of the world’s population, or 650
million people, lived with a disability [156]. They are the world’s largest minority. A significant
fraction of this population live with severe and multiple disabilities. For example, of the 51.2
million people living with disability in the US, approximately 32.5 million have severe disability
[157]. Certain conditions such as cerebral palsy, spinal cord injury, muscular atrophy, multiple
sclerosis, and amyotrophic lateral sclerosis could result in severe functional limitations [1].
While these conditions may not be associated with cognitive limitations, they do result in severe
mobility limitations. Thus, people with severe motor disabilities rely heavily on caregiver
assistance to facilitate communication. With the aim of providing the client with some degree of
independence, much research in rehabilitation engineering has focused on the development of
access technologies that translate the client’s intent into a functional activity. Among the most
widely used access technologies are various forms of push buttons (e.g. Big Macs, head
switches, chin switches) [2], [158], [159], micro switches [160], [161], [162], sip and puff
switches [36], and electromyography-based switches [7]. All of these access technologies
function as a binary switch; the switch is normally off, and is turned on upon detection of a
87
voluntary change in the client’s physical movement or physiological signals. Examples of
emerging access switches are the multiple-camera tongue switch [16], and the vocal cord
vibration switch [143]. A simple binary switch can open many doors to a person with severe
motor disabilities [31]. For example, he/she can use it to select characters from a scanning
keyboard (typing), or move his/her wheelchair (mobility).
Because of the unique needs of clients with severe disabilities, when designing new access
solutions it is important to consider the client not only as the end user of the technology but also
as a partner in the design process [3]. While access solution designers analyse and forecast how
clients are likely to use the technology, it is often very difficult to predict the behaviour of a
novice user and individual learning curves [163]. If the consumer’s preference, opinions and
needs are not considered, the ensuing access solution may not meet the client’s expectations,
cause client frustration and disappointment [164] and eventually lead to device abandonment
[165], [144]. This risk may be mitigated by adopting a client-centred approach where the
opinions of the client are solicited and the client is empowered as an active decision-maker about
the tasks he/she wishes to perform and the context within which the access technology will be
used [166]. As outlined in the International Classification of Functioning, Disability and Health
context involves both personal factors (e.g. client’s physical and cognitive capacities) and
environmental factors (e.g. the physical environment, the communication partners) [90]. The
Human Activity Assistive Technology (HAAT) and the Matching Person & Technology (MPT)
models promote such a perspective for design and delivery of assistive technologies to clients.
HAAT concentrates on the performance of a human operator in a given task (activity) within a
given situation (context) [3]. MPT takes into account the client’s needs and preferences, the
functions and features of the most desirable and appropriate technology, and the milieu-related/
environmental factors influencing use [139].
88
The objective of the present study was to develop a single-switch access solution for a young
man with severe spastic quadriplegic cerebral palsy9. We adopted an iterative design process to
obtain optimal switch performance based on the client’s preferred means of interaction, rather
than to require the client to alter his behaviour to accommodate technical constraints of the
switch. In the following sections, we present a detailed description of the client and the process
of developing a customized infrared thermography-based access solution for him.
6.3 Client Profile
Demographic. Pithom (pseudonym meaning ‘dilatation of the mouth’ [167]) was a 26 year old
male with spastic quadriplegic cerebral palsy. He lived with his family and his primary caregiver
was his mother. He had finished high school and attended college for post-secondary education
for a year, but voluntarily withdrew due to his lack of computer access, which made it extremely
difficult to complete examinations even with a personal facilitator.
Physical function. Pithom’s family had first noticed signs of physical disability subsequent to a
febrile seizure, when he was an infant. Because of severe spasticity, the muscles in Pithom’s
arms, legs and back are chronically hypertonic. As a consequence, Pithom was not able to move
his limbs or trunk voluntarily and often experienced pain in his elbows, legs, and back. His most
comfortable posture was a semi-supine position in his wheelchair with his neck laterally rotated
and extended such that his head faced either left or right. He rarely maintained a forward-looking
head posture. He had some control of his neck and could change the direction of his gaze with
significant effort. He had control of his jaw and could open and close his mouth quite reliably.
He could also smile and blink on cue. Pithom could be fed orally and could drink from a straw.
9 I would like to clarify the objective: The objective of the present study was to test an infrared thermography-based access switch in depth with a young man with severe spastic quadriplegic cerebral palsy (this footnote has been added for clarification and does not appear in the related journal article).
89
Over the years he learned to chew his food long enough before swallowing to minimize the risk
of aspiration.
Cognitive function, language, and communication. Pithom was bright, understood cause and
effect, and could follow cues diligently. His only means of communication was through
vocalizations that could be interpreted exclusively by his mother. He attempted to answer ‘yes’/
‘no’ questions verbally, but his speech was not generally intelligible to the unacquainted
communication partner10. Pithom had no vision or hearing impairments. To the familiar
caregiver, his emotions could be inferred from his facial gestures, vocalizations, and body
movements. For example, when he was happy, he laughed, tilted his head backwards and gazed
upwards. His limbs tensed up and moved involuntarily when he became excited. He
communicated pain and discomfort by frowning and crying. He attempted to say a word or
phrase (e.g. ‘no more’) repeatedly to express boredom or to refrain from continuing an activity.
History of switch use. At the time of study enrolment, caregivers and healthcare providers had
never found a reliable access switch for Pithom. Conventional mechanical switches like push
buttons, or cap switches had not worked due to his spasticity, involuntary movements, and
muscle fatigue. Voice switches had not been feasible because of his frequent moaning from pain
and discomfort.
Needs and expectations. Pithom’s foremost desire was to perform tasks independently. Given the
absence of cognitive limitations, he could grasp new concepts quickly. However, he was not able
to express himself due to his severe physical disability, requiring his mother to be with him as a
facilitator at all times. With his mother’s help, Pithom indicated that he wished to have a means
10 Pithom understood English and Vietnamese but no formal cognitive test score (e.g., IQ test score) or language proficiency test score was available for him (this footnote has been added for clarification and does not appear in the related journal article).
90
to (1) answer multiple choice questions (i.e. be able to select his desired choice from a list of
available choices), (2) browse his music tracks or the pages of his electronics books, and (3) dial
his mother’s pager or cell phone number whenever he required immediate assistance. We learned
that Pithom was very keen about returning to college to continue his education. His mother
explained that finding a way to independently browse electronic course material and complete
multiple choice exams would be key prerequisites to the resumption of his studies. Pithom
assented to the study and his mother provided written consent on his behalf.
6.4 Rationale for an Infrared Thermographic Solution
The infrared thermal switch was designed and implemented based on a client-centred approach
[138], [145], [166]. Given Pithom’s reliable control of mouth opening and closing, Pithom and
his mother wanted to exploit this voluntary motion for access. While mechanical chin switches
have been proposed in the past for capturing this type of movement (e.g. [29]), concerns about
hygiene and mounting led the team to consider a non-contact solution. Specifically, we proposed
to capture the local temperature changes associated with voluntary mouth open-close activity via
a strategically placed infrared thermal camera and to translate this motion into switch activation
(e.g. mouse click) through a computer algorithm.
We chose infrared thermography because it is a non-invasive and non-contact imaging modality.
Unlike other video-based access technologies, infrared thermography is lighting and skin colour
invariant. Infrared thermal cameras measure the radiation emitted from the surface of an object
in the infrared range of the electromagnetic spectrum, i.e. between wavelengths of 0.8 µm and
1.0 mm [46]. These cameras produce an image that is a spatial two-dimensional (2-D) map of the
3-D temperature distribution of the object [45]. Recently infrared thermal imaging of the human
face has been investigated as a non-contact method of monitoring emotional state. Increased
blood perfusion in the periorbital muscles, i.e. the small muscles around each eye and the bridge
of the nose, has been associated with stress [57], anxiety [58], and emotional arousal [55] in
91
humans. The increasing availability and affordability of consumer thermal infrared cameras
further motivated the exploration of a thermographic solution.
6.5 Methods
6.5.1 The Infrared Thermal Switch
Infrared thermal video of Pithom’s face was captured with a ThermaCAM SC640 by FLIR
[131]. The camera had a thermal sensitivity of 60≤ mK and a spatial resolution of 640×480
pixels. It was placed anterior and lateral to the participant at a 45° angle. The acquired video was
non-radiometric and grey scale, with bright intensities corresponding to warm regions and dark
intensities corresponding to cold regions (Figure 6.1). The video was sent to a DELL Inspiron
1560 laptop (Intel® Core™2 CPU T7200 @ 2.00 GHz and 2.00 GB of RAM) via a fire wire
cable for real-time processing.
The algorithm for detecting mouth opening activity from the infrared thermal video was
implemented in Simulink R2008a and consisted of three main modules, namely face
segmentation, motion and intensity analyses, and filtering non-mouth objects. The technical
intricacies are based on the algorithm proposed in [39]. Here, we only present an overview of the
key concepts and refer the reader to the above references for further details.
92
Figure 6.1: Image acquired by an infrared thermal camera. Darker intensities represent colder
regions and brighter intensities represent warmer regions. Inside the mouth is clearly warmer
than the surrounding facial tissue. Reproduced from [Memarian et al. (2010) [43]].
Since mouth opening and closing are facial gestures, the first step was to accurately isolate the
client’s face from all other objects appearing in the video frame (e.g. client’s wheelchair, people
or objects in the background). Recognizing that the face is generally warmer than the
surrounding environment, we used adaptive intensity thresholding and identified the face as the
largest semi-round object in the resultant image. Once we had the face, we localized the mouth,
exploiting the fact that the mouth is typically warmer than the rest of the face and that opening
and closing necessitates motion. We thus deployed motion detection and adaptive intensity
thresholding to uncover facial regions that were both warm and in motion as candidate mouth
areas. This procedure occasionally identified some non-mouth candidates (e.g. forehead or chin).
To remove these non-mouth objects, we applied various morphological (e.g. shape and
hollowness) and anthropometric (e.g. size and typical location of the mouth in a human face)
filters. Upon detection of a mouth open-close sequence by the algorithm, a mouse click was
emulated using a module latching relay by DLP Design Inc. and a Swifty USB switch interface
by Origin Instruments™.
93
6.5.2 Switch Testing
Over a period of six months, Pithom attended five sessions to test the switch and to inform its
development. As detailed in the following sections, the first session focused on access site
reliability testing, and the subsequent four sessions concentrated on switch performance testing.
Aspects of performance were measured using the Compass Assessment Software11 [149], which
provides consistent testing and quantitative analysis of a person’s computer access skills.
Compass was developed through surveys and interviews with rehabilitation professionals [168]
and has documented test-retest reliability and validity in the computer skills domain [169]. The
switch algorithm was iteratively refined based on Pithom’s performance and feedback during
those sessions. For example Pithom indicated that he wished to receive auditory feedback upon
activation of the switch. Pithom’s comment motivated us to deploy a latching relay that made a
distinct clicking sound when it received a signal indicating the detection of mouth opening. This
simple addition not only satisfied Pithom but also facilitated troubleshooting. When the switch
was not functioning as expected, the presence of an audible click upon mouth opening suggested
that the problem rested with the downstream USB switch converter. On the other hand, the
absence of an audible click implicated the algorithm as failing to detect Pithom’s mouth open
activity. As another example of where Pithom’s performance informed a subsequent
modification, consider the first session, i.e., access site reliability testing. We noticed that Pithom
tended to open his mouth more horizontally than vertically. This motivated the addition of a
customizable mouth shape analysis filter (based on eccentricity of the open mouth candidate and
its orientation with respect to the x-axis) in the final filtering module of the algorithm. In
addition, the switch performance tests were selected such that they were appealing to the client.
For example we learned from Pithom’s mother that he prefers engaging audiovisual stimuli over
unimodal stimuli for the test activities. Therefore, we customized the Compass tests such that
both the stimuli and rewards were presented in engaging audiovisual modalities. Inspired by the
11 The activation period and scan rate (where applicable) of tests were the default values from the Compass Assessment Software. The client was comfortable with these default values (this footnote has been added for clarification and does not appear in the related journal article).
94
sequential steps in motor training for switch use suggested in [3] and to ensure that the switch
could facilitate the high priority activities identified by Pithom, testing was organized into three
categories as detailed below.
Access Site Reliability Test. The first test was to verify that Pithom could reliably perform the
physical activity required to trigger the infrared thermal switch, i.e. opening and closing his
mouth. This test occurred in session one, primarily to assess the level of fatigue associated with
consecutive mouth opening-closing and to ensure reliable algorithmic detection of Pithom’s
mouth open-close activity. In this test, infrared thermal video of Pithom’s face was recorded,
while he was cued to: open his mouth, hold it ajar for 1 second, and finally close his mouth. This
was repeated 30 times. He was given an auditory prompt upon every open and close action. The
infrared thermal videos were processed offline to evaluate algorithm performance and Pithom
was asked to rate the activity as ‘little or not fatiguing’, ‘moderately fatiguing’, or ‘very
fatiguing’.
Stimulus-Response Test. In the second type of tests, Pithom used the infrared thermal switch to
perform a stimulus-response test using the ‘switch test’ from Compass Assessment Software
[149] to simulate the switch activity required to turn pages of an electronic or audio book or to
initiate a speed-dial. The switch test was designed to evaluate a client’s ability to activate a
switch in response to a prompt. For each iteration, Pithom was prompted aurally and visually to
activate the switch (i.e. to open and then close his mouth). The visual prompt was the phrase
‘press the switch’ written inside a yellow square on the computer screen, accompanied by an
engaging sound. The prompt persisted for a maximum of 10 seconds and was followed by a
pause of a random duration between 1 and 4 seconds, during which there were no stimuli
presented. The next prompt immediately followed this pause. All prompts were actionable, in
that, every prompt cued the user to activate the switch. For a mouth open and close to be
considered valid, the switch had to be activated for a minimum time of 0.3 seconds. Upon switch
activation, the prompt disappeared and the screen was cleared, in preparation for the next
prompt. If the switch remained inactivated, the screen was cleared after the maximum allowable
95
time (10 seconds) elapsed. The software provided audiovisual feedback to Pithom by rewarding
him with positive phrases such as ‘good job!’ or ‘well done!’ for correct activations and
informed him of the instances he missed with phrases such as ‘sorry’ or ‘no’. The number of
correct activations and misses along with Pithom’s response time to all prompts were
automatically logged by the software. Each trial of the switch test included 20 prompts and
Pithom completed two trials per session.
Scanning Test. The scanning test, also from Compass Assessment Software [149], was used to
simulate the switch activity required to make a selection from a set of multiple choices. A target
word was displayed in a window at the top of the screen, below which appeared an array of four
word choices. The goal was for Pithom to select the target word from the scanning array.
Scanning (i.e. repetitive, sequential highlighting of one word of the array at a time) started when
Pithom performed a mouth open-close action, which emulated a left mouse click. When the
desired word was highlighted, Pithom needed to perform another mouth open-close to select it.
The scan rate (i.e. the pace at which individual words were highlighted) was set at 4 seconds per
word and the maximum time per target word (i.e. the maximum time allowed to complete each
word matching iteration) was 100 seconds. If no selection was made within this time, the target
word timed out and advanced to the next target word. The pause between iterations (i.e. the
length of time between switch activation or time out and the start of the next iteration) was 2
seconds. Each trial of the scanning test consisted of 10 target words. Pithom completed two trials
per session. The data collection setup for the switch and scanning tests is depicted in
Figure 6.2.
The protocol was approved by the Research Ethics Board of the hospital and university where
the authors are affiliated with. The study was carried out in the hospital.
96
Figure 6.2: Infrared thermal switch testing setup. Reproduced from [Memarian et al. (2010) [43]].
6.6 Results
Offline analysis of the infrared thermal videos from step one (access site reliability test) yielded
26 correct activations, four misses, and two false alarms, or 86.7% sensitivity and 93.5%
specificity. Pithom rated the activity as little or not fatiguing.
Table 6.1 summarizes the results of the switch test. The third column indicates the number of
times the switch was activated properly. This is shown both as a percentage of total number of
prompts in the trial, as well as the exact number of correct activations. Average response time
was measured as the average time from when the prompt was first presented until the switch was
activated. The last column represents the number of times where there was no switch activity
during the allowable time (10 s) for responding to the prompt. For the switch test, the client
achieved an average correct activation rate of 90% and average response time of 2.4 seconds.
97
Results of the scanning test are shown in Table 6.2. For every trial the percentage of correct
activations, the average response time, and the percentage of timing errors are reported. As
shown in Figure 6.3, Pithom’s average percentage of correct activations in the scanning test
improved from 65% in session one to 80% in session four. His average response time12 for the
scanning test over all the sessions was 11.7 seconds.
Table 6.1: Result of the stimulus-response switch test. Reproduced from [Memarian et al.
(2010) [43]].
Session Trial % correct activations (# correct/total)
Average response time (s)
% missed activations due to switch not being activated within allowable
time (# missed/total)
1 1 90% (18/20) 2.1 10% (2/20)
2 95% (19/20) 3.45 5% (1/20)
2 1 90% (18/20) 2.42 10% (2/20)
2 100% (20/20) 2.15 0% (0/20)
3 1 80% (16/20) 2.39 20% (4/20)
2 95% (19/20) 1.32 5% (1/20)
4 1 75% (15/20) 2.73 25% (5/20)
2 95% (19/20) 2.37 5% (1/20)
12 Average response time was measured as the average time from when the target word was first presented until the switch was activated. This included the scan times between words in the list (this footnote has been added for clarification and does not appear in the related journal article).
98
A two-sample Kolmogorov-Smirnov (KS) test between the columns in Table 6.2 labelled
‘Correct’ and ‘Incorrect’ activations showed that Pithom’s average response time for correct
activations was significantly different from his average response time for incorrect activations
( 0497.0=p ; 5% significance level). In other words, correct responses were generally generated
faster than incorrect responses. However no statistically significant difference was found
between the number of timing errors during correct and incorrect activations ( 5189.0=p ; 5%
significance level; two-sample KS test). The two-sample Kolmogorov-Smirnov test is a
nonparametric test for comparing probability distributions and considered appropriate for this
study because our data violated the normality criterion required for parametric equivalents (e.g.
two-sample t-test).
A timing error occurred when a word was not selected when it was highlighted during the first
scan of the array. For example, if the client scanned through the array twice before making a
switch activation, that iteration counted as a timing error. A maximum of one timing error was
counted per word matching iteration.
6.7 Discussion
Results of the access reliability test confirmed that Pithom had good control of the mouth open
and close motion and that the infrared thermal switch algorithm could detect this activity with
adequate sensitivity and specificity. On two occasions, Pithom held his mouth open longer than
the cue dictated, resulting in false alarms. From the radiometric images, we also noticed that
Pithom had a relatively warm face, and hence distinguishing between an open mouth and the
nearby facial tissue could be difficult. Pithom expressed little to no fatigue and was interested in
coming back for more sessions to try the switch and scan tests.
99
Table 6.2: Results of the scanning test. The numbers in parentheses are the actual counts.
Reproduced from [Memarian et al. (2010) [43]].
Sess
ion
Tri
al
% correct activations
(#correct/total) Average response time (s) Timing errors
% missed activations
(#missed/total)
Ove
rall
Cor
rect
ac
tivat
ions
Inco
rrec
t ac
tivat
ions
Ove
rall
Cor
rect
ac
tivat
ions
Inco
rrec
t ac
tivat
ions
1 1 70% (7/10) 12.11 7.18 23.62 0% (0/10)
0% (0/7)
0% (0/3)
0% (0/10)
2 60% (6/10) 14.29 11.31 18.76 10% (1/10)
16.67% (1/6)
0% (0/4)
0% (0/10)
2 1 70% (7/10) 12.04 10.11 16.55 10% (1/10)
14.29% (1/7)
0% (0/3)
0% (0/10)
2 80% (8/10) 12.99 13.74 10.01 10% (1/10)
12.5% (1/8)
0% (0/2)
0% (0/10)
3 1 60% (6/10) 7.76 9.52 5.12 0% (0/10)
0% (0/6)
0% (0/4)
0% (0/10)
2 80% (8/10) 11.99 11.46 14.11 0% (0/10)
0% (0/8)
0% (0/2)
0% (0/10)
4 1 70% (7/10) 13.03 9.17 22.02 10% (1/10)
0% (0/7)
33.3% (1/3)
0% (0/10)
2 90% (9/10) 9.73 10.53 2.49 10% (1/10)
11.11% (1/9)
0% (0/1)
0% (0/10)
From Table 6.1, we note that in every session of the switch test, the number of correct activations
increased from the first to the second trial. Also in all sessions except the first one, Pithom
responded faster to the prompts in the second trial. This pattern suggests a constructive effect of
practice on Pithom’s performance with the infrared thermal switch. It should be noted that in the
switch test, misses may have occurred for two reasons: (1) the switch was not activated for a
sufficient length of time, or (2) the switch was not released within the maximum allowable time.
By comparing the 3rd and 5th columns in Table 6.1we see that the number of missed activations
100
in each trial matched the number of missed activations due to the switch not being activated
within allowable time. This reveals that none of the misses were due to a missed mouth closing
and implies that Pithom was able to close his mouth in a timely a manner after each time he had
opened it. From direct observation, we note that most of the missed mouth openings occurred
because Pithom had involuntarily turned his face away from the camera due to spasticity. By the
time Pithom managed to face the camera again, the maximum available time for the prompt had
already passed. This issue of involuntary spastic neck movement highlights a limitation of single
camera computer vision-based access technologies.
From the last column of Table 6.2, it can be seen that during the scanning test, Pithom always
activated the switch within the maximum time per iteration, implying that the 100 second
iteration time was sufficient for the client to make a selection. Figure 6.3 shows the average
percentage of correct activations per session. Pithom’s average percentage of correct activations
improved from 65% in session one to 80% in session four. This rise in performance reflects the
refinement of the infrared thermal switch algorithm over the course of the testing sessions as
well as increased proficiency on part of the client. Average response time of the client improved
through the scanning test sessions except for the very last session where it increased slightly
compared to the prior session. We postulate that this might be due to Pithom’s frequent neck
spasms during the last session, which locked his gaze in a direction opposite to that of the
stimulus presentation computer. Consequently, longer times elapsed before Pithom could turn his
head back to face the screen and generate a response.
It should be noted that the main purpose of this case study was to develop a useful access
solution (i.e., an infrared thermal switch for computer access) for an actual client with severe
disability. A study with more frequent test sessions (e.g. three test sessions per week over a
period of two months) is required to investigate the effect of training on a client’s switch
performance [170], [171].
101
Figure 6.3: Percentage of correct activations per session. Reproduced from [Memarian et al.
(2010) [43]].
6.7.1 User Feedback and Maintenance Issues
We received very positive feedback from Pithom and his mother throughout this study. When we
asked Pithom about the reasons he liked the infrared thermal switch, with his mother’s help as
the facilitator, he pointed to convenience of switch activation and the potential for more
independence in performing his desired activities. Pithom’s mother was extremely happy to
observe his improvement with the scanning tests and she was very hopeful that this new access
solution may help her son resume his post-secondary education.
The infrared thermal switch algorithm processes the greyscale intensity variations corresponding
to changes in the client’s facial temperature, and hence it does not require a fully radiometric
infrared thermal camera (a fully radiometric imaging system is one that measures the absolute
102
temperature of every pixel). This is a plus for the infrared thermal switch, as the price of non-
radiometric infrared thermal cameras is much lower than their radiometric counterparts. In
addition to the camera, very simple and inexpensive hardware (e.g. module latching relay and
USB switch converter) have been used in the design of the infrared thermal switch. Infrared
thermal cameras are usually built to work even in harsh climates and under tough conditions
[131]. Nonetheless, repairing an infrared thermal camera that is beyond its warranty period can
be a costly undertaking.
6.7.2 Future Prospects
A portable version of the infrared thermal switch using a RAZ-IR SX handheld thermal camera
[172] has been developed for Pithom to use at home. Pithom will continue practising with the
Compass Assessment Software using this portable system. We hope that with sufficient training,
he will move on to more difficult Compass tests such as word typing and sentence typing by
row-column scanning of an onscreen keyboard. Also, Pithom will hopefully use the infrared
thermal switch to perform his desired activities like browsing his music tracks or the pages of his
electronics books. It is important to monitor the effectiveness of the infrared thermal switch for
the client over time and revise the system according to potential changes in his available skills,
life roles and performance areas, or the physical and social context in which the system needs to
function [3].
Future research may further enhance the infrared thermal switch. In order to eliminate the need
for manual calibration, a dynamic auto-calibration module can be added. The module can
initialize the system parameters (i.e. intensity and morphology thresholds) by processing a
thermogram (infrared thermal snapshot) of the client while he is holding his mouth open.
Automatic re-calibration may then be invoked whenever the infrared thermal switch algorithm
detects no activation within a long period of time or detects excessive activations within a short
period of time. Several improvements can also be made to address Pithom’s awkward posture or
spastic head movements, which may degrade performance: (1) the algorithm can be updated to
103
track and focus on the region of interest (participant’s face) more accurately; (2) multiple
cameras can be used to capture Pithom’s face from different angles, so that involuntary spastic
head movements would less likely cause a complete departure from the cameras’ fields of view;
and (3) the user can undergo long-term training. With regular practice, it is conceivable that the
quality and speed of the client’s movement may improve [3].
6.8 Conclusion
This case study discussed the development of a novel non-invasive and non-contact access
solution for a client with spastic quadriplegic cerebral palsy. Although the client’s cognitive and
sensory functions were intact, his communication and participation were very limited because of
his severe physical disability. Conventional access switches had not been feasible for this client
because of insufficient fine motor control, involuntary movements, and spasms. Inspired by the
client’s ability to control his jaw motion reliably, the proposed access solution utilized infrared
thermography to capture the local temperature change associated with client’s voluntary mouth
opening/closing in order to trigger a binary switch. Through five experimental sessions, the
client completed multiple trials of stimulus-response and word-matching tests with the infrared
thermal switch. The client’s performance improved as the sessions progressed. The results
support further development of the infrared thermal switch for clients with severe and multiple
disabilities who retain voluntary control of mouth open-close activity.
104
Chapter 7
7 Summary of Contributions
This thesis makes several original contributions to assistive technology research and
rehabilitation engineering, as summarized below. Contributions 1-4 are engineering
contributions and contributions 5-6 are clinical contributions.
1. Inaugural application of infrared thermal imaging in rehabilitation engineering. For the
first time, infrared thermal imaging was employed to create a non-invasive and non-
contact single switch access technology for clients with severe motor disability. The
technology exploits the temperature differences between the inside and surrounding areas
of the mouth as a switch trigger, thereby allowing voluntary switch activation upon
mouth opening. The two main technological contributions are:
a. Design and development of a video processing algorithm for automatic detection
of mouth open activity in infrared thermal video [39]. An algorithm consisting of
face localization, intensity analysis, motion analysis and false positive filtering
was designed to detect a client’s voluntary mouth opening in infrared thermal
video. Pilot testing of the algorithm resulted in 88.5% ±11.3 sensitivity and 99.4%
±0.7 specificity. The algorithm performance was robust to participant motion and
changes in the background scene.
b. A real-time implementation of the infrared thermal switch [40]. The algorithm in
step (a) was enhanced, by using a different motion estimation method and
introducing a new anthropometric filter of non-mouth objects to reduce false
positive activations. Additional software and hardware were invoked to realize a
real-time infrared thermographic single switch. The switch output can be set to a
mouse click or any key on the computer keyboard.
105
2. Introduction of a novel measure of binary switch user performance [134]. A mutual
information measure was proposed that encapsulates the switch user’s performance (hits,
misses, false alarms, correct rejections) into a single number. This measure takes into
consideration the entropy of the stimulus, the entropy of the switch, and their mutual
entropies. It is beneficial for measuring the impact of contextual factors on a user’s
switch performance. One important contextual factor is time; thus the proposed mutual
information measure can be conveniently used to monitor the effect of time (training) on
switch use.
3. Validation of the infrared thermal switch [41]. I gauged the infrared thermal switch
validity with respect to a conventional chin switch. Using the mutual information
measure discussed in contribution number 2, user performance with the chin and infrared
thermal switches was quantified. No statistical differences in user performance or
response time were found between the two switches (p = 0.1611; p = 0.249). The infrared
thermal switch is a valid single switch alternative for individuals with disabilities who
retain voluntary mouth opening/closing.
4. Quantification of the effect of information presentation modality on infrared thermal
switch use [41]. Our study with ten able-bodied participants showed that there was no
significant effect of presentation modality (i.e., visual, audio, or audiovisual) on the
performance (as measured by mutual information efficiency) of the infrared thermal
switch users (p = 0.777). Also there was no significant effect of presentation modality on
the switch users’ response time (p = 0.824). The infrared thermal switch can be useful
regardless of the sensory modality in which the information is presented to the switch
user.
5. Identification of key impairments in body functions and structures that limit infrared
thermal switch access [42]. Based on structured switch testing and interviews with seven
clients and their primary caregivers, a list of physiological, motoric, sensory and
cognitive impairments that may negatively affect infrared thermal switch use was
derived. Keeping in mind that the assistive technology should accommodate the user (not
vice versa), potential solutions, i.e., changes to the infrared thermal switch and physical
106
environment, were suggested to mitigate the negative effect of the impairments in body
function and structures.
6. Provision of a novel means of access (i.e., the infrared thermal switch) to an individual
with severe spastic quadriplegic cerebral palsy [43]. The infrared thermal switch was
customized for a client with spastic quadriplegic cerebral palsy. Prior to this technology,
the client never had a reliable means of access. Although his cognitive and sensory
functions were intact, his communication and participation were very limited because of
his severe physical disability. Based on a client-centred approach, the high priority
activities of the client were identified. Stimulus-response tests (average correct activation
rate: 90%; average response time: 2.4 s) and scanning tests (average correct activation
rate: 72.5%; average response time: 11.7 s) were carried out to simulate the client’s high
priority activities and to train the client on using the infrared thermal switch to perform
those activities. At the time of writing, a portable version of the infrared thermal switch is
being implemented for the client for his day to day activities.
The thesis does not include studies to explore the impact of personal and environmental factors
that may affect infrared thermal switch use by the target population. For the infrared thermal
switch to serve as a robust access technology, consideration of those contextual factors is crucial.
Another limitation is the relatively small sample size of the studies reported in this thesis. More
studies with larger numbers of clients are required to further elucidate potential advantages or
disadvantages of this new access technology. Another evaluation that could not be completed
within the time frame of this thesis is the long term effectiveness of the infrared thermal switch
for the client with severe spastic CP and the potential role of this access technology in improving
his social participation and quality of life.
More versatile access technologies can stem from the infrared thermal switch. The algorithm can
be enhanced such that various patterns or sequences of mouth opening would translate to a
specific message or activity. For example one long mouth opening followed by a short one may
107
translate to the announcement of “I need help” on client’s computer/AAC device or three
consecutive mouth openings might initiate the movement of the client’s power wheelchair. Such
a coding technology will expand the usefulness of the infrared thermal switch beyond its current
binary capabilities and may offer a faster solution to clients who prefer direct communication
over indirect scanning-based communication.
In addition to mouth opening, there are other voluntary facial activities (e.g., strong exhalation,
blowing air in or out, opening and closing the eyes) that elicit temperature change. Expanding
the infrared thermal switch algorithm to distinguish among those activities will result in a multi-
output access switch. Different activities by the client will trigger different modes of such a
switch. For example mouth opening will make the wheelchair move forward, while a strong puff
will reverse the direction of the movement, i.e., make the wheelchair go backward.
108
References
[1] Craig A, Tran Y, McIsaac P, Boord P. The efficacy and benefits of environmental control
systems for the severely disabled. Med Sci Monit. 2005; 11(1):32-9.
[2] Tai K, Blain S, Chau T. A review of emerging access technologies for individuals with
severe motor impairments. Assist Technol. 2008; 20:204-219.
[3] Cook AM, Hussey S. Assistive technologies: principles and practice. 2nd ed. St. Louis
(MO): Mosbey, Inc.; 2001.
[4] NUANCE Dragon NaturallySpeaking Solutions [Internet]. Burlington (MA): Nuance
Communications Inc.; c2002-2010 [cited 2010 Apr 19]. Available from:
http://www.nuance.com/naturallyspeaking/
[5] TrackerPro [Internet]. Edmonton (AB): Madentec Inc.; c2009 [cited 2010 Apr 19].
Available from: http://www.madentec.com/products/tracker-pro.php
[6] Oklahoma’s Assistive Technology Program (Oklahoma Able Tech) [Internet]. Oklahoma
(OK): State of Oklahoma; c2010 [cited 2010 Apr 19]. Available from:
http://www.ok.gov/abletech/Assistive_Technology/Switch_Handout.html
[7] Huang C, Chen C, Chung H. Application of facial electromyography in computer mouse
access for people with disabilities. Disabil Rehabil. 2006; 28(4):231-237.
[8] Chen Y, Lai JS, Luh JJ, Kuo TS. SEMG-controlled telephone interface for people with
disabilities. J Med Eng Technol. 2002; 26(4):173-176.
[9] Falk T, Chan J, Duez P, Teachman G, Chau T. Augmentative communication based on
realtime vocal cord vibration detection. IEEE Trans Neural Syst Rehabil Eng. 2010;
18(2):159-163.
109
[10] Harada S, Landay JA, Malkin J, Li X, Bilmes JA. The vocal joystick: Evaluation of
voice-based cursor control techniques for assistive technology. Disabil Rehabil Assist
Technol. 2008; 3(1):22-34.
[11] Hainisch R, Platz M. Phonetic control: A new approach for continuous, non-invasive
device control using the vocal tract. In: ICORR 2007: Proceedings of the 10th IEEE
International Conference on Rehabilitation Robotics; 2007 Jun 13-15; Noordwijk,
Netherlands. Piscataway (NJ): IEEE Publishing; 2007. p. 688-692.
[12] Sporka AJ, Kurniawan SH, Slavík P. Whistling user interface (U3I). In: Stary C,
Stephanidis C editors. User-centred interaction paradigms for universal access in the
information society. Berlin/Heidelberg: Springer; 2004. p. 472-478.
[13] Jacob RJK, Eye-gaze computer interfaces: what you look at is what you get. IEEE
Computer Magazine. Jul 1993; (26)7:65-66.
[14] Goosens CA, Crain SS. Overview of non-electronic eye-gaze communication.
Augment Altern Commun 1987; 3:77-89.
[15] Nolan C. Under the eye of the clock. New York (NY): St. Martin’s Press; 1987.
[16] Leung B, Chau T. A multiple camera tongue switch for a child with severe spastic
quadriplegic cerebral palsy. Disability and Rehabilitation: Assist Technol. 2010;
5(1):58-68.
[17] Harwin WS, Jackson RD. Analysis of intentional head gestures to assist computer
access by physically disabled people, J Biomed Eng. 1990; 12(3):193-8.
[18] Betke M, Gips J, Fleming P. The camera mouse: visual tracking of body features to
provide computer access for people with severe disabilities. IEEE Trans Neural Syst
Rehabil Eng. 2002; 10(1):1–10.
[19] Morrison K, McKenna SJ. Automatic visual recognition of gestures made by motor-
impaired computer users. Technol Disabil. 2002; 14(4):197-203.
110
[20] Simpson T, Broughton C, Gauthier MJA, Prochazka A. Tooth-click control of a
hands-free computer interface. IEEE Trans Biomed Eng 2008; 55(8):2050-2056.
[21] Reed KL. Quick reference to occupational therapy. 2nd ed. Gaithersburg (MD):
Aspen; 2001.
[22] Funt B, Barnard K, Martin L. Is machine colour constancy good enough? In:
Burkhardt H, Neumann B editors. Proceedings of the 5th European Conference on
Computer Vision; 1998 Jun 2-6; Freiburg, Germany. London: Springer-Verlag; 1998.
p. 445-459.
[23] Puckett AD, Sauer BW, Zardiackas LD, Entrekin DS. Development of a custom-fit
mouthstick appliance. J Rehabil Res Dev. 1989; 26(4):17-22.
[24] Sammons Preston Aids to Daily Living [Internet]. Bolingbrook (IL): Patterson
Medical Products, Inc.; c2010 [cited 2010 Feb 12]. Available from:
http://www.sammonspreston.com/app.aspx?cmd=get_product&id=50177
[25] Extensions for Independence. Multi-Tip Telescoping H-A MODULAR
MOUTHSTICK (System) – Part I [Internet]. San Diego, CA: Extensions for
Independence; c2005 [updated 2005 Nov 21; cited 2010 Mar 03]. Available from:
http://www.mouthstick.net/mouthstc/mbromain.htm#NEW:%20Titanium%20mouth
piecs:
[26] DeVries RC, Deitz J, Anson DK. A comparison of two computer access systems for
functional text entry. Am J Occup Ther. 1998; 52:656-665.
[27] Anson DK, Glodek M, Peiffer RM, Rubino CG, Schwartz PT. Long-term speed and
accuracy of morse code vs. head-pointer interface for text generation [Internet]. Dallas
(PA): Assistive Technology Research Institute as Misericordia University; 2004 [cited
2010 Apr 19]. Available from:
http://atri.misericordia.edu/Papers/MorseVrsOnScreen.php
[28] AbleData. Ultimate switch with quick disconnect (model 1072) [Internet]. Silver
Spring, MD: AbleData; c2005 [2005 Jan 13; cited 2010 Apr 19]. Available from:
111
http://www.abledata.com/abledata.cfm?pageid=19327&top=11343&ksectionid=19327
&productid=78128&trail=0&discontinued=0
[29] Lancioni GE, O’Reilly MF, Singh NN, Sigafoos J, Tota A, Antonucci M, Oliva D.
Children with multiple disabilities and minimal motor behaviour using chin
movements to operate microswitches to obtain environmental stimulation. Res Dev
Disabil. 2006; 27:290-298.
[30] Lancioni GR, O’Reilly MF, Sigafoos J, Singh NN, Oliva D, Basili G. Enabling a
person with multiple disabilities and minimal motor behaviour to control
environmental stimulation with chin movements. Disabil Rehabil. 2004; 23:1291-
1294.
[31] Blain S, McKeever P, Chau T. Bedside computer access for an individual with severe
and multiple disabilities: A case study. Disabil Rehabil Assist Technol. 2010 Feb 04;
PubMed PMID: 20131978.
[32] New Abilities. UCS 1000 with TTKTM [Internet]. Santa Clara, CA: newAbilities
Systems, Inc.; c? [cited 2010 Mar 02]. Available from: www.newabilities.com
[33] Girardi M. Tongue-touch controls give Ben a more satisfying, self-sufficient lifestyle.
Teamrehab Report [Internet]. 1997 Feb [cited 2010 Apr 05]; 15-17, Available from:
http://www.wheelchairnet.org/WCN_ProdServ/Docs/TeamRehab/RR_97/9702art1.PD
F
[34] Struijk LNSA. An inductive tongue computer interface for control of computers and
assistive devices. IEEE Trans Biomed Eng. 2006; 53(12):2594-2597.
[35] Origin Instruments. Sip/Puff Switch with Headset [Internet]. Grand Prairie (TX):
Origin Instruments Corporation; c2008 [cited 2010 Apr 05]. Available from:
http://www.orin.com/access/sip_puff/index.htm#SipPuffSingleUserPackages
[36] Jones M, Grogg K, Anschutz J, Fierman R. A sip-and-puff wireless remote control for
the Apple iPod. Assist Technol. 2008; 20(2):107-110.
112
[37] Simpson T, Gauthier MJA, and Prochazka A. Evaluation of tooth-Click triggering and
speech recognition in assistive technology for computer access. Neurorehabil Neural
Repair. 2010; 24(2):188-194.
[38] Ju JS, Shin Y, Kim EY. Vision based interface system for hand free control of an
intelligent wheelchair. J Neuroeng Rehabil. 2009; 6:33.
[39] Memarian N, Venetsanopoulos AN, Chau T. Infrared thermography as an access
pathway for individuals with severe motor impairments. J Neuroeng Rehabil. 2009
Apr 16; 6:11 (8 pp).
[40] Memarian N, Leung B, Falk TH, Alves-Kotzev N, Blain S, Chau T. Innovations in
switch access for children and youth with profound disabilities. In: Chau T, Fairley J
editors. Paediatric rehabilitation engineering: from disability to possibility. CRC Press;
Oct 15 2010.
[41] Memarian N, Venetsanopoulos AN, Chau T. Validating an infrared thermal switch as
a novel access technology. BioMed Eng Online. 2010 Aug 05; 9:38 (11 pp).
[42] Memarian N, Venetsanopoulos AN, Chau T. Body functions and structures pertinent
to infrared thermography-based access for clients with severe motor disabilities. Assist
Technol. Accepted for publication.
[43] Memarian N, Venetsanopoulos AN, Chau T. Client-centred development of an
infrared thermal access switch for a young adult with severe spastic quadriplegic
cerebral palsy. Disabil Rehabil Assist Technol. –In press.
[44] Uematsu S. Symmetry of skin temperature comparing one side of the body to the
other. Thermology. 1986; 1:4-7.
[45] Jones BF, Plassmann P. Digital infrared thermal imaging of human skin. IEEE Eng
Med Biol Mag. 2002; 21:41-48.
[46] Jones BF. A reappraisal of the use of infrared thermal image analysis in medicine.
IEEE Trans Med Imaging. 1998; 17(6):1019-1027.
113
[47] Grealish K, Kacir T, Backer B, Norton P. An advanced infrared thermal imaging
module for military and commercial applications. In: EM Carapezza editor.
Unattended Ground Sensor Technologies and Applications VII: Proceedings of the
SPIE - The International Society for Optical Engineering (SPIE Volume 5796); 2005
Mar 28-Apr 01; Orlando, USA. Bellingham (WA): SPIE; 2005. p. 186-192.
[48] Burnay SG, Williams TL, Jones CH. Applications of thermal imaging. Bristol: IOP
Publishing Ltd.; 1988.
[49] Gautherie M, Gros CM. Breast thermography and cancer risk prediction. Cancer. 2006
Jun 28; 45(1):51-56.
[50] Head JF, Wang F, Elliott RL. Breast thermography is a noninvasive prognostic
procedure that predicts tumor growth rate in breast cancer patients. Ann N Y Acad Sci.
1993 Nov 30; 698:153-8.
[51] Gautherie M. Atlas of breast thermography with specific guidelines for examination
and interpretation. Milan: Papusa; 1989.
[52] Macdonald AG, Land DV, Sturrock RD. Microwave thermography as a noninvasive
assessment of disease of activity in inflammatory arthritis. Clin Rheumatol. 1994 Dec;
13(4):589-592.
[53] Viitanen SM, Laaksonen AL. Thermography in juvenile rheumatoid arthritis. Scand J
Rheumatol. 1970; 16(1):91-8.
[54] Spalding SJ, Kwoh CK, Boudreau R, Enama J, Lunich J, Huber D, et al. Three-
dimensional and thermal surface imaging produces reliable measures of joint shape
and temperature: a potential tool for quantifying arthritis. Arthritis Res Ther. 2008 Jan
28; 10:R10.
[55] Nhan B, Chau T. Classifying affective states using thermal infrared imaging of the
human face. IEEE Trans Biomed Eng. 2009 Nov 17; PubMed PMID: 19923040.
114
[56] Zhu Z, Fei J, Pavlidis I. Tracking human breath in infrared imaging. In: BIBE'05:
Proceedings of the 5th IEEE Symposium on Bioinformatics and Bioengineering; 2005
Oct 19-21; Minneapolis, USA. Piscataway (NJ): IEEE Publishing; 2005. p. 227-231.
[57] Pavlidis I, Eberhardt NL, Levine JA. Seeing through the face of deception. Nature
2002 January 03; 415:35.
[58] Pavlidis I, Levine J, Baukol P. Thermal imaging for anxiety detection. In: Proceedings
of IEEE Workshop on Computer Vision Beyond the Visible Spectrum: Methods and
Applications; 2000 Jun 21-22; Hilton Head Island, USA. Piscataway (NJ): IEEE
Publishing; 2000. p. 104-9.
[59] Sellers EW, Kubler A, Donchin E. Brain–computer interface research at the university
of South Florida cognitive psychophysiology laboratory: the P300 speller. IEEE Trans
Neural Syst Rehabil Eng. (Special Issue on Brain-Computer Interfaces). 2006;
14(2):221-224.
[60] Piccione F, Giorgi F, Tonin P, Priftis K, Giove S, Silvoni S, Palmas G, Beverina F.
P300-based brain computer interface: Reliability and performance in healthy and
paralyzed participants. Clin Neurophysiol. 2006; 117(3):531-537
[61] Coyle SM, Ward TE, Markham CM. Brain-computer interface using a simplified
functional near-infrared spectroscopy system. J. Neural Eng. 2007; 4:219-226.
[62] Sitaram R, Zhang H, Guan C, Thulasidas M, Hoshi Y, Ishikawa A, Shimizu K,
Birbaumer N. Temporal classification of multichannel near-infrared spectroscopy
signals of motor imagery for developing a brain–computer interface. J Neuroimaging.
2007; 34:1416-1427.
[63] Naito M, Michioka Y, Ozawa K, Ito Y, Kiguchi M, Kanazawa T. A communication
means for totally locked-in ALS patients based on changes in cerebral blood volume
measured with near-infrared light. IEICE Transactions on Information and Systems.
2007; E90D(7):1028-1037.
115
[64] Blain S, Mihailidis A, Chau T: Assessing the potential of electrodermal activity as an
alternative access pathway. Med Eng Phys. 2008; 30(4):498-505.
[65] Tsukahara R, Aoki H. Skin potential response in letter recognition task as an
alternative communication channel for individuals with severe motor disability. Clin
Neurophysiol. 2002; 113:1723-1733.
[66] Lupo J, Balcerak R. The physical basis of thermal imaging. In: Proceedings of the
World Congress on Medical Physics and Biomedical Engineering; 2000 Jul 23-28;
Chicago, USA. Baltimore (MD): Chicago 2000 World Congress HQ; 2000.
[67] Qi H, Diakides NA. Thermal infrared imaging in early breast cancer detection - a
survey of recent research. In: Proceedings of the 25th Annual International Conference
of the IEEE Engineering in Medicine and Biology Society; 2003 Sep 17-21; Cancun,
Mexico. Piscataway (NJ): IEEE Publishing; 2003. p. 1109-12.
[68] Okada Y, Kawamata T, Kawashima A, Hori T. Intraoperative application of
thermography in extracranial-intracranial bypass surgery. Neurosurgery. 2007;
60(Suppl 2):362-365.
[69] Watson JC, Gorbach AM, Pluta RM, Rak R, Heiss JD, Oldfield EH. Real-time
detection of vascular occlusion and reperfusion of the brain during surgery by using
infrared imaging. Neurosurgery. 2002; 96:918-923.
[70] Madjid M, Willerson JT, Casscells SW. Intracoronary thermography for detection of
high-risk vulnerable plaques. J Am Coll Cardiol. 2006; 47(8 Suppl):C80-C85.
[71] Ring F, Harding R. Infrared thermal imaging in peripheral vascular diseases. In:
Proceedings of the World Congress on Medical Physics and Biomedical Engineering;
2000 Jul 23-28; Chicago, USA. Baltimore (MD): Chicago 2000 World Congress HQ;
2000.
[72] Herry CL, Frize M. Quantitative assessment of pain-related thermal dysfunction
through clinical digital infrared thermal imaging. Biomed Eng Online. 2004 Jun 28;
3(1):19.
116
[73] Rose AD, Kanade V. Thermal imaging study comparing phacoemulsification with the
sovereign with WhiteStar System to the legacy with AdvanTec and NeoSoniX System.
Am J Ophthalmol. 2006; 141(2):322-326.
[74] Murthy R, Pavlidis I. Non-contact measurement of breathing function. IEEE Eng Med
Biol Mag. 2006; 25(3):57-67.
[75] Morimoto T, Takada K, Huiya H, Yasuda Y, Sakuda M. Changes in facial skin
temperature associated with chewing efforts in man: a thermographic evaluation.
Archs oral Biol. 1991; 36(9):665-760.
[76] Tu J, Tao H, Huang T. Face as mouse through visual face tracking. Comput Vis Image
Underst. 2007; 108:35-40.
[77] Steketee J. Spectral emissivity of the skin and pericardium. Phys Med Biol. 1973;
18(5):686–694.
[78] Thermal-Eye 2000B/300D [Internet]. Dallas: L-3 Communications Infrared Products;
c2005 [updated 2005 July; cited 2010 May 11]. Available from:
http://infrared.com/pdf/thermaleye-2000b.pdf.
[79] Watmough DJ, Fowler PW, Oliver R. The thermal scanning of a curved isothermal
surface: implications for clinical thermography. Phys Med Biol. 1970; 15(1):1-8.
[80] Zaproudina N, Varmavuo V, Airaksinen O, Narhi M. Reproducibility of infrared
thermography measurements in healthy individuals. Physiol Meas. 2008; 29(4):515-
24.
[81] Otsu N. A threshold selection method from gray-level histograms. IEEE Trans Syst
Man Cybern. 1979 Jan; 9(1):62-66.
[82] Gonzalez RC, Woods RE, Eddins SL. Digital image processing using MATLAB. New
Jersey: Pearson Prentice Hall; 2004.
117
[83] Barron JL, Fleet DJ, Beauchemin SS, Burkitt TA. Performance of optical flow
techniques. In CVPR 1992: Proceedings of IEEE Computer Society Conference on
Computer Vision and Patter Recognition; 1992 Jun 15-18; Champaign, USA. Los
Alamitos (CA): IEEE Computer Society Press; 1992. p. 236-242.
[84] Bailar JC, Meyer EA, Pool R, editors. Assessment of the NIOSH head-and-face
anthropometric survey of U.S. respirator users. Washington (D.C.): The National
Academies Press; 2007.
[85] DeCarlo D, Metaxas D, Stone M. An anthropometric face model using variational
techniques. In: SIGGRAPH 1998: Proceedings of the 25th International Conference on
Computer Graphics and Interactive Techniques; 1998 Jul 19-24; Orlando, USA. New
York: ACM; 1998. p. 67-74.
[86] Moriyama T, Taguchi H, Mashima Y. Relation between the brain waves, face
temperature and blood pressure using nonintrusive blood pressure monitor and the
environments. In: SICE’96: Proceedings of the 35th SICE Annual Conference; 1996
Jul 24-26; Tottori, Japan. Society of Instrument and Control Engineers (SICE); 1996.
p. 1205-08.
[87] MobIR® M4 [Internet]. Northampton, UK: Armstrong Optical Ltd.; c2006 [cited 2010
May 11]. Available from: http://www.armstrongoptical.co.uk/2/MobIRm4.pdf.
[88] Wang J, Sung E. Facial feature extraction in an infrared image by proxy with a visible
face image. IEEE Trans Instrum Meas. 2007; 56(4):2057-2066.
[89] Blain S, Chau T, Mihailidis A. Peripheral autonomic signals as access pathways for
individuals with severe disabilities: a literature appraisal. The Open Rehabilitation
Journal. 2008; 1:27-37.
[90] World Health Organization. International Classification of Functioning, Disability &
Health: ICF. Geneva: World Health Organization; 2001. 236 p.
118
[91] Sykes C. Health Classifications 1 -An introduction to the ICF [Internet]. London:
World Confederation for Physical Therapy Keynotes; 2006 [cited 2007 September].
Available from: http://wcpt.org/sites/wcpt.org/files/files/KN-ICF-Intro-eng.pdf.
[92] Bornman J. The World Health Organization's terminology and classification:
application to severe disability. Disabil Rehabil. 2004; 26(3):182-8.
[93] Blackhurst AE. A functional approach to the delivery of assistive technology services.
[Internet]. Kentucky: National Assistive Technology Research Institute; 2001 [cited
2007 Sep 22]. Available from:
http://natri.uky.edu/resources/fundamentals/function.html.
[94] Blackhurst AE, Lahm EA. Technology and exceptionality foundations. In: Lindsey
JD, editor. Technology and exceptional individuals. 3rd ed. Austin: ProEd; 2000. p. 3-
45.
[95] Haley SM, Coaster WJ, Binda-Sundberg K. Measuring physical disablement: the
contextual challenge. Phys Ther 1994; 74(5):443-51.
[96] Mihaylov SI, Jarvis SN, Colver AF, Beresford B. Identification and description of
environmental factors that influence participation of children with cerebral palsy. Dev
Med Child Neurol 2004; 46:299-304.
[97] Lawlor K, Mihaylov S, Welsh B, Jarvis SJ, Colver AF. A qualitative study of the
physical, social and attitudinal environments influencing the participation of children
with cerebral palsy in northeast England. Pediatr Rehabil 2006; 9(3):219-28.
[98] Colver A. Study protocol: SPARCLE- a multi-center European study of the
relationship of environment to participation and quality of life in children with
cerebral palsy. BMC Public Health 2006 Apr 25; 6(105):10.
[99] Wang PP, Badley EM, Giganac M. Exploring the role of contextual factors in
disability models. Disabil Rehabil 2006; 28(2):135-40.
119
[100] Edwards E. Information transmission, an introductory guide to the application of the
theory of information to the human sciences. London, UK: Chapman and Hall; 1969.
[101] Shannon CE. A mathematical theory of communication. Bell System Tech J. 1948 Jul
and Oct; 27:379-423 and 623-56.
[102] Hick WE. On the rate of gain of information. Q J Exp Psychol. 1952; 4:11-26.
[103] Hyman R. Stimulus information as a determinant of reaction time. J Exp Psychol.
1953; 45:188-196.
[104] Klemmer ET, Frick FC. Assimilation of information from dot and matrix patterns. J
Exp Psychol. 1953; 45:15-9.
[105] Fitts PM. The information capacity of the human motor system in controlling the
amplitude of movement. J Exp Psychol. 1954; 47:381-91.
[106] Broadbent DE. Perception and communication. London: Pergamon Press; 1958.
[107] Ellison KW, Buckout R. Psychology and criminal justice. New York: Harper & Row;
1981.
[108] Wells GL. What do we know about eyewitness identification? Am Psychol. 1993;
48:553-71.
[109] Wright DB, Davies GM. Eyewitness testimony. In: Durso F, editor. Handbook of
applied cognition. West Sussex: John Wiley & Sons; 1999.
[110] Attneave F. Applications of information theory to psychology (A summary of basic
concepts, methods and results). New York: Henry Holt and Company Inc.; 1959.
[111] Wickens CD, Hollands JG. Engineering psychology and human performance. 3rd ed.
Upper Saddle River (NJ): Prentice-Hall Inc.; 2000.
[112] Card SK, Moran TP, Newell A. The Psychology of human-computer interaction.
Hillsdale (NJ): Lawrence Erlbaum Associates Inc.; 1983.
120
[113] Landauer TK, Nachbar DW. Selection from alphabetic and numeric menu trees using
a touch screen: Breadth, depth, and width. In: Borman L, Curtis B, editors. CHI’85:
Proceedings of the ACM CHI 85 Human Factors in Computing Systems; 1985 Apr 14-
18; San Francisco, USA. New York: ACM Press; 1985. p. 73-8.
[114] Olson JR, Nilsen E. Analysis of the cognition involved in spreadsheet software
interaction. Human-Computer Interaction. 1987; 3(4):309-49.
[115] Seow SC. Information theoretic models of HCI: A comparison of the Hick-Hyman law
and Fitt’s law. Human-Computer Interaction. 2005; 20(3):315-52.
[116] Chan RB, Childress DS. On a unifying noise-velocity relationship and information
transmission in human machine systems. IEEE Trans Syst Man Cybern. 1990a;
20(5):1125-35.
[117] Chan RB, Childress DS. On information transmission in human machine systems:
channel capacity and optimal filtering. IEEE Trans Syst Man Cybern. 1990b;
20(5):1136-45.
[118] Ogawa K. An evaluation method of computer usability based on human-to-computer
information transmission model. Ergonomics. 1992; 35(5 & 6):577-90.
[119] Poock GK, Blackstone SW. Using information theory to measure effectiveness of an
augmentative and alternative communication display. Augment Altern Commun.
1992; 8(4):287-96.
[120] Sanger TD, Henderson J. Optimizing assisted communication devices for children
with motor impairments using a model of information rate and channel capacity. IEEE
Trans Neural Syst Rehabil Eng. 2007; 15(3):458-68.
[121] Cover TM, Thomas JA. Elements of information theory. 2nd ed. Hoboken (NJ): John
Wiley and Sons Inc.; 2006.
[122] Pierce JR. An introduction to information theory: symbols, signals & noise. 2nd ed.
New York: Dover Publications; 1980.
121
[123] Roman S. Coding and information theory. New York: Springer; 1992.
[124] Lathi BP. Modern digital and analog communication systems. 3rd ed. New York:
Oxford University Press; 1998.
[125] Haykin S. Communication systems. 4th ed. New York: John Wiley and Sons; 2001.
[126] Van Trees HL. Detection, estimation, and modulation theory, Part I. New York: John
Wiley and Sons Inc.; 2001.
[127] Laming D. Information theory of choice-reaction times. New York: Academic Press;
1968.
[128] Perini E, Soria S, Prati A, Cucchiara R. FaceMouse: A human-computer interface for
tetraplegic people. Lect Notes Comput Sci. 2006; 3979:99-108.
[129] American Psychological Association. Standards for educational and psychological
testing. 2nd ed. Washington D.C.: American Educational Research Association; 1992.
[130] Black R. Managing the Testing Process: Practical tools and techniques for managing
hardware and software testing. 3rd ed. Hoboken(NJ): Wiley; 2009.
[131] ThermaCAM SC640 [Internet]. Danderyd (Sweden): FLIR Systems Inc.; c2006 -
[cited 2010 Feb 18]. Available from:
http://www.flir.com/uploadedFiles/Thermography_APAC/Products/Product_Literture/
SC640_Datasheet%20APAC.pdf
[132] Koga T, Linuma K, Hirano A, Iijima Y, Ishiguro T. Motion-compensated interframe
coding for video conferencing. In: NTC’81: Proceedings of National
Telecommunication Conference; 1981 Nov 23-Dec 03; New Orleans, USA. 1981. p.
G5.3.1-5.
[133] Wang Y, Ostermann J, Zhang Y. Video processing and communications. Upper
Saddle River (NJ): Prentice Hall; 2002.
122
[134] Memarian N, Venetsanopoulos AN, Chau T. Mutual information as a measure of
contextual effects on single switch use. The Open Rehabilitation Journal. 2009; 2:1-
10.
[135] Hecht D, Reiner M, Halevy G. Multimodal virtual environments: response times,
attention, and presence. Presence. 2006; 15(5):515-523.
[136] Neshige R, Murayama N, Igasaki T, Tanoue K, Kurokawa H, Asayama S.
Communication aid device utilizing event-related potentials for patients with severe
motor impairment. Brain Res. 2007; 1141:218-227.
[137] Krusienski DJ, Wolpaw JR. Brain-computer interface research at the Wadsworth
center: developments in noninvasive communication and control. Int Rev Neurobiol.
2009; 86:147-157.
[138] Bernd T, Van Der Pijl D, De White LP. Existing models and instruments for the
selection of assistive technology in rehabilitation practice. Scand J Occup Ther. 2009;
16:146-158.
[139] Scherer MJ, Craddock G. Matching person & technology (MPT) assessment process.
Technol Disabil. 2002; 14:125-131.
[140] Scherer M, Sax C, Vanbiervliet A, Cushman LA, Scherer JV. Predictors of assistive
technology use: the importance of personal and psychosocial factors. Disabil Rehabil.
2005; 27(21):1321-1331.
[141] Scherer MJ, Cushman LA. Determining the content for an interactive training
programme and interpretive guidelines for the Assistive Technology Device
Predisposition Assessment. Disabil Rehabil. 2002; 24(1/2/3):126-130.
[142] Speropoulos CA, Townson AF. Assistive technology assessments in a tertiary-care
rehabilitation facility. B C Med J. 2001; 43(2):82-85.
[143] Chan J, Falk T, Teachman G, Morin-McKee J, Chau T. Evaluation of a non-invasive
vocal cord vibration switch as an alternative access pathway for an individual with
123
hypotonic cerebral palsy – a case study. Disabil Rehabil Assist Technol. 2010;
5(1):69-78.
[144] Copley J, Ziviani J. Use of a team-based approach to assistive technology assessment
and planning for children with multiple disabilities: A pilot study. Assist Technol.
2007; 19:109-125.
[145] Wielandt T, Scherer M. Reducing AT abandonment: proposed principles for AT
selection and recommendation. e-bility.com [Internet]. 2004 Aug [cited 2010 Mar 15];
[about 3 p.]. Available from: http://www.e-bility.com/articles/at_selection.php
[146] Sleigh G, Sullivan PB, Thomas AG. Gastrostomy feeding versus oral feeding alone for
children with cerebral palsy Cochrane Database Syst Rev. 2004; 2:CD003943.
[147] Ravn O, Andersen NA, Theill Sorensen A. Auto-calibration in automation systems
using vision. In: Yoshikawa T, Miyazaki F editors. Experimental robotics III. Great
Britain: Springer Verlag; 1994. p 206-218.
[148] Lin M, Nanning Z, Qing L, Hong C. Dynamic approach of camera auto-calibration for
vision system on autonomous vehicle. Journal of Xi’an Jiaotong University. 2005;
39(10):1072-1076.
[149] Compass Assessment Software [Internet]. Ann Arbor: Koester Performance Research;
c2007 [cited 2010 Feb 10]. Available from: http://www.kpronline.com/compass-
intro.html
[150] Saunders MD, Timler GR, Cullinan TB, Pilkey S, Questad KA, Saunders RR.
Evidence of contingency awareness in people with profound multiple impairments:
response duration versus response rate indicators. Res Dev Disabil. 2003; 24:231-245.
[151] Saunders RR, Saudners MD, Struve B, Munce AL, Olswang LB, Dowden PA, et al.
Discovering indices of contingency awareness in adults with multiple profound
disabilities. Am J Ment Retard. 2007; 112(4):246-260.
124
[152] Harnesses, Vests, Straps and Belts [Internet]. New Farm (Australia): Cerebral Palsy
League; c2010 [cited 2010 Mar 10]. Available from:
http://www.cplqld.org.au/business/ets/wheelchairs/seating/restraints
[153] Head support [Internet]. Silver Spring (USA): AbleData; c2010 [cited 2010 May 01].
Available from:
http://www.abledata.com/abledata.cfm?pageid=19327&ksectionid=19327
&top=13068
[154] Nhan BR, Chau T. Infrared thermal imaging as a physiological access pathway: a
study of the baseline characteristics of facial skin temperatures. Physiol Meas. 2009;
30(4):N23-N35.
[155] Threats T. Access for persons with neurogenic communication disorders: influences of
personal and environmental factors of the ICF. Aphasiology. 2007; 21(1):67-80.
[156] International Convention on the Rights of People with Disabilities (2006 August 14-
25) [Internet]. New York: United Nations; c2006 [cited 2010 Feb 20]; Available from:
http://www.un.org/disabilities/convention/pdfs/factsheet.pdf
[157] US Census Bureau News [Internet]. Washington, D.C.: US Department of Commerce;
c2007 [updated 2007 May 29; cited 2010 Feb 20]. Available from:
www.census.gov/newsroom/releases/pdf/cb07-ff10.pdf
[158] Anderton N, Standen PJ, Avory K. Using switch controlled software with people with
profound disabilities. In: ICDVRAT 2004: Proceedings of the 5th International
Conference on Disability, Virtual Reality and Associated Technologies; 2004 Sep 20-
22; Oxford, United Kingdom. Reading (UK): University of Reading; 2004. p. 269-
274.
[159] Glickman L, Deitz J, Anson D, Stewart K. The effect of switch control site on
computer skills of infants and toddlers. Am J Occup Ther. 1996; 50(7):545-53.
125
[160] Antonucci M, Lancioni GE, O’Reilly MF, Singh NN, Sigafoos J, Oliva D. Enabling a
man with multiple disabilities and limited motor behaviour to perform a functional
task with help of microswitch technology. Percept Mot Skills. 2006; 103(1):83-88.
[161] Lancioni GE, O’Reilly MF, Singh NN, Oliva D, Piazzolla G, Pirani P, et al.
Evaluating the use of multiple microswitches and responses for children with multiple
disabilities. J Intellect Disabil Res. 2002; 46(4):346-351.
[162] Lancioni GE, Lems S. Using a microswitch for vocalization responses with persons of
multiple disabilities. Disabil Rehabil. 2001; 23(16):745-748.
[163] Stephanidis C. User interfaces for all: new perspectives into human-computer
interaction. In: Stephanidis C, editor. User interfaces for all: concepts, methods and
tools. Mahwah (NJ): Lawrence Associates; 2001. p 3-21.
[164] Vella F, Vigoroux N, Truillet P. A user-centered approach for designing CLAPOTI: an
assistive technology designed for speech and motor disorders. In: Craddock G et al
editors. Assistive Technology - Shaping the Future. AAATE 2003: Proceedings of the
7th Association for Advancement of Assistive Technology in Europe Conference;
2003 Aug 31-Sep 3; Dublin, Ireland. p. 289-293.
[165] Galvin JC, Scherer MJ. Evaluating, selecting and using appropriate assistive
technology. Gaithersburg (MD): Aspen Publications; 1996.
[166] Rigby P, Ryen S, From W, Walczak E, Jutai J. A client-centred approach to
developing assistive technology with children. Occup Ther Int. 1996; 3(1):67-79.
[167] Bolton L. The complete book of baby names. Naperville (IL): Sourcebooks Inc.; 2009.
[168] Koester HH, LoPresti E, Ashlock G, McMillan W, Moore P, Simpson R. Compass:
software for computer skills assessment. In: CSUN 2003: Proceedings of the
International Conference on Technology and Persons with Disabilities; 2003 Mar 17-
22; Los Angeles, USA.
126
[169] Koester HH, Simpson RC, Spaeth D, LoPresti E. Reliability and validity of Compass
software for access assessment. In: Proceedings of the RESNA Annual Conference;
2007Jun 15-19; Phoenix, USA. Arlington: RESNA Press; 2007.
[170] Dromerick AW, Schabowski CN, Holley R, Monroe B, Markotic A, Lum P. Effect of
training on upper-extremity prosthetic performance and motor learning: a single-case
study. Arch Phys Med Rehabil. 2008; 89(6):1199-1204.
[171] Lin K, Chen Y, Chen C, Wu C, Chang Y. The effects of bilateral arm training on
motor control and functional performance in chronic stroke: a randomized controlled
study. Neurorehabil Neural Repair. 2010; 24(1):42-51.
[172] RAZ-IR SX Thermal Camera [Internet]. Las Vegas (NV): SPI Corp.; c2010 [cited
2010 Feb 10]. Available from: www.raz-ir.com/index.php/raz-ir-sx-thermal-infrared-
camera.html
127
Appendix A: Research Ethics Approval
This appendix contains the original research ethics approval letters from the research ethics
boards of Bloorview Kids Rehab and University of Toronto. Subsequent renewal approval letters
and amendment approval letters from Bloorview Kids Rehab Research Ethics Board are also
attached.
128
129
130
131
132
133
134
135
136
137
138
Appendix B: Questionnaire
This appendix contains the list of questions that were asked from the client (where feasible) and
the client’s primary caregiver in the study of body functions and structures pertinent to infrared
thermal switch use, discussed in Chapter 5 of this thesis.
DEMOGRAPHICS
1. How old is your child?
2. Does he/she live at home?
3. Who is the primary caregiver (e.g. mother, father; not the name of caregiver)?
4. Who are the secondary caregivers? (e.g. mother, father; not the name of caregiver)?
5. Does he/she receive special education services?
6. What is his/her school grade level?
7. Diagnosis (type of disability – Please answer thoroughly and include medical names)
8. Date when disability first started (or when you first noticed it)
PHYSIOLOGICAL
9. Does he often have fever or chill?
10. Does he/she take any medication on a regular basis?
If you remember, please name those medications and the reason he/she needs to take them?
Do these medications have any side effect (e.g., shaking, drowsiness)?
11. Does he/she experience seizures? o Are there factors that trigger a seizure for him/her?
MOTOR
12. What is his/her comfortable posture?
13. Does he/she have upper body control? o Trunk o Neck o Arms
139
o Fingers o Eyes
14. Does he/she have involuntary body movement? o Is it frequently or occasionally?
15. Does he/she have spasms? o Is it frequently or occasionally?
16. Can he/she control the movement of any of his body parts reliably?) o What are they?
17. Over the years has his/her functionality progressed or deteriorated?
18. Has he/she ever used any of the following switches? o Push button o Head switch o Chin switch o Sip and puff o Has he/she used any other switches? Please name.
19. For the particular switch or switches that he/she has used, how was his/her: o Accuracy o Operation speed o Fatigue
20. What problems did he/she have with those switches?
21. Does he/she experience increased fatigue as the day progresses? o What kind of activities does he/she find difficult to perform later in the day? o What time of day is his/her functionality at his/her best to try new switches?
22. Is his/her mouth usually closed or open? o How fatiguing is it for him/her to open and close the mouth?
23. Can he/she chew and swallow food? o Is he/she fed orally?
SENSORY AND COGNITIVE
24. Vision o Has he/she been tested for vision acuity? When? o What do you (as the primary caregiver) think about his/her vision acuity?
25. Hearing o Has he/she been tested for hearing acuity? When? o What do you (as the primary caregiver) think about his/her hearing acuity?
26. Does he/she understand cause and effect? (e.g., if I press a push button and he will hear a beep sound and I repeat this several times, will he/she understand what is causing the beep?)
27. Can he/she follow cues? o What are those cues?
140
28. Is he/she verbal at all?
29. Current means of communication?
30. Can other people understand him/her or only you can?)
31. Is he/she able to make facial expressions? Which one of the following can he/she do? o Eye blink o Eyebrows o Smiling o Tongue protrusion
32. Was he/she able to communicate better at a younger age? o When? o How?
