-
FP7 - MOBOT – 600796 1
EU FP7-ICT-2011.2.1
ICT for Cognitive Systems and Robotics - 600796
Work Package 5: Specification of Use Cases, Performance Metrics,
and User Evaluation
Studies
Deliverable D5.2: Report on use cases, performance metrics and
user
study preparations
Release date: 20-02-2014
Status: public
Document name: MOBOT_WP5_D5.2
-
FP7 - MOBOT – 600796 2
Executive Summary
This report includes an update of the first report (D5.1) as
submitted before. In D5.1 a
comprehensive documentation of background, definitions and
assessment strategies has been
given. This update (D5.2) is restricted to amendments and
additional information on future
validation of MOBOT prototypes to be developed.
It includes some minor amendments with respect to user groups
and use cases as presented in
D5.1, which were necessary for the specification of the
technical development. The target
group of MOBOT have been defined by impairment - rather than
disease-oriented definitions.
The selected motor and cognitive criteria in addition to the
clinical setting (geriatric rehab)
was not modified for this update and seem to perfectly fit for
the MOBOT context.
A detailed description of validation studies on existing devices
by a systematic review has
already been given in D5.1 and has been updated for this report
(D5.2). Results of the
systematic review will be submitted for publication timely and
will allow an evidence-based
development of a validation strategy for MOBOT prototypes.
In addition, this report includes performance metrics for
evaluation studies, depending on the
assessment strategy to be selected. As some features and
functionalities of the MOBOT
device remain to be specified by technical partners, assessments
and performance metrics may
be modified or added for the first evaluation study. The report
completes with the description
of present preparations of the upcoming user evaluation studies
including tasks of clinical
partners such as ethics votes and subject recruitment. An update
of performance metrics and
current status of user evaluation studies will be given in
D5.3.
-
FP7 - MOBOT – 600796 3
Deliverable Identification Sheet
IST Project No. FP7 – ICT for Cognitive Systems and Robotics -
600796
Acronym MOBOT
Full title Intelligent Active MObility Assistance RoBOT
integrating
Multimodal Sensory Processing, Proactive Autonomy and
Adaptive
Interaction
Project URL http://www.mobot-project.eu
EU Project
Officer
Michel Brochard
Deliverable D5.2 Report on use cases, performance metrics and
user study
preparations
Work package WP5 Specification of Use Cases, Performance
Metrics, and User
Evaluation Studies
Date of delivery Contractual M 12 Actual 20-02-2014
Status Version 1.5 Final
Nature Report
Dissemination
Level
Public
Authors
(Partner)
Klaus Hauer (BETHANIEN), Phoebe Köpp (BETHANIEN), Angelika
Peer (TUM), Yiannis Koumpouros (DIAPLASIS), Pavlos Alevras,
Panagiotis Siavelis, Despoina Alexopoulou, Foteini Koureta
(DIAPLASIS)
Responsible
Author
Klaus Hauer Email [email protected]
Partner BETHANIEN Phone 0049-6221-3191532
Keywords Use Cases, Performance Metrics, User Evaluation
Studies
Version Log
Issue Date Rev
No.
Author Change
12-11-2013 1.0 Phoebe Köpp First draft
06-12-2013 1.1 Klaus Hauer,
Phoebe Köpp
Input of revised chapter 2 and 5
06-12-2013 1.2 Yiannis
Koumpouros
Input of chapter 3 + contribution to chapter 4
06-12-2013 1.3 Phoebe Köpp Formal integration of chapter 3
http://www.mobot-project.eu/
-
FP7 - MOBOT – 600796 4
19-12-2013 1.4 Phoebe Köpp Input of revised chapter 4
27-01-2014 1.5 Klaus Hauer,
Phoebe Köpp
Integration of corrections after internal review
-
FP7 - MOBOT – 600796 5
TABLE OF CONTENTS
Executive Summary
..........................................................................................................
2
1. Introduction
...................................................................................................................
8
2. Update on definition of user groups and use cases (BETHANIEN)
............................. 9
2.1. Definition of user groups
........................................................................................
9
2.1.1. Previous definition of user groups
..................................................................
9
2.1.2. Addenda for user groups
.................................................................................
9
2.1.3. Final inclusion criteria and adjusted assessment
............................................. 9
2.2. Definition of use cases
.........................................................................................
12
2.2.1. Previous definition of use cases
....................................................................
12
2.2.2. Addenda for use cases
...................................................................................
13
2.2.3. Final Use cases
..............................................................................................
13
3. Subjective and objective performance metrics for mobility
device assessment
(BETHANIEN)
...............................................................................................................
17
3.1. Technical evaluation
............................................................................................
18
3.2. Clinical Evaluation
...............................................................................................
18
3.2.1. Generic (established) clinical assessment,
performance-based measures ..... 18
3.2.2. Subjective measures for Human-Robot Interaction (HRI)
(DIAPLASIS) .... 19
3.2.3. Specifically tailored assessment tests focusing on
specific functionalities to
document the added value of the device (BETHANIEN)
....................................... 24
4. User study preparations (BETHANIEN)
....................................................................
26
4.1. Results of a review on clinical validation studies of
robotic devices ................... 26
4.2 Abbreviated presentation of results of the systematic review
............................... 31
4.3 Objectives to be achieved for MOBOT- validation studies
.................................. 31
4.4 Ethical approval
.....................................................................................................
34
4.5 Time table and development of validation strategy
.............................................. 34
4.6 Subject recruitment
...............................................................................................
35
5. Summary of report
......................................................................................................
36
6. References
...................................................................................................................
37
Appendix 1
......................................................................................................................
43
Appendix 2
......................................................................................................................
75
-
FP7 - MOBOT – 600796 6
LIST OF TABLES
Table 2-1 Inclusion criteria for gait
.................................................................................
10
Table 2-2 Inclusion criteria for transfer
..........................................................................
11
Table 2-3 Inclusion criteria for cognition
.......................................................................
11
Table 2-4 Summary of categories for user groups according to
motor and cognitive
impairment
......................................................................................................................
12
Table 2-5 Anthropometric characteristics
.......................................................................
12
Table 2-6 List of use cases
..............................................................................................
13
Table 3-1 Different types, purposes and responsibilities of
evaluations ......................... 17
Table 4-1 Evaluation studies of a mobility assistance robot
........................................... 27
Table 4-2 Objectives and solutions for evaluation
.......................................................... 31
Table 4-3 Specific functions of the device to be evaluated in
first user evaluation study
as described in the study proposal
...................................................................................
32
-
FP7 - MOBOT – 600796 7
LIST OF ABBREVIATIONS
Abbreviation Description
PR Public Report
WP Work Package
Partner Abb. Description
TUM TECHNISCHE UNIVERSITAET MUENCHEN
ICCS INSTITUTE OF COMMUNICATION AND COMPUTER
SYSTEMS
INRIA INSTITUT NATIONAL DE RECHERCHE EN
INFORMATIQUE ET EN AUTOMATIQUE
ECP ECOLE CENTRALE DES ARTS ET MANUFACTURES
UHEI RUPRECHT-KARLS-UNIVERSITAET HEIDELBERG
ATHENA RC ATHENA RESEARCH AND INNOVATION CENTER IN
INFORMATION COMMUNICATION & KNOWLEDGE
TECHNOLOGIES
ACCREA ACCREA BARTLOMIEJ MARCIN STANCZYK
BETHANIEN BETHANIEN KRANKENHAUS - GERIATRISCHES
ZENTRUM - GEMEINNUTZIGE GMBH
DIAPLASIS DIAPLASIS REHABILITATION CENTER SA
-
FP7 - MOBOT – 600796 8
D5.2: Report on use cases, performance metrics and user study
preparations
1. INTRODUCTION
Changes due to the aging process result in manifold limitations
crucial for quality of
life. An increasing number of frail older persons are not able
to perform basic activities
of daily living such as walking, transfer, personal hygiene or
shopping independently.
Acute diseases or hospitalization are associated with worsening
of function and
abilities. The loss or decrease of motor and cognitive functions
represents the main risk
factors for loss of autonomy, the most feared condition/event in
older people`s life.
The use of rollators is established to support mobility,
autonomy and independent living
of older frail persons. A lot of studies prove the benefit of
rollator use such as support
by weight unloading and postural control [Bateni and Maki,
2005], reduction of risk for
falls [Graafmans et al., 2003] and support of mobility, activity
and participation
[Salminen et al., 2009]. But there are also studies that
indicate a higher risk for falls
while using a walker [Stevens et al., 2009]. So far,
rollator-like devices have not, or
only in a limited way, targeted cognitive support.
The aim of the MOBOT project is to develop an intelligent active
mobility assistance
robot for indoor environments (geriatric rehab) that provides
user-centred, context-
adaptive and natural support. Such a robotic rollator can
support older persons in a
number of every-day challenges. Features like automatic
collision avoidance can
prevent falls, enhanced physical support can prevent exhaustion,
navigation support can
prevent disorientation in unaccustomed settings such as
geriatric ward-based
rehabilitation.
As stated in the DOW, technical features such as application of
computer vision
techniques with modalities such as range sensor images, haptic
information, as well as
command-level speech and gesture recognition and other
functionalities will allow such
support.
The aim of WP 5 (clinical partners) is amongst other tasks
(analysis of user needs,
definition of use cases, review of existing devices) the
preparation and execution of
formal user evaluation studies and the definition of objective
and subjective
performance metrics and adequate assessment strategies. With the
user evaluation trials,
the specific improvements of the newly developed devices will be
pointed out clearly
and the additional value for the user will be documented. The
trials will be conducted
with potential users of the target sample of patients in
geriatric rehab, who interact with
prototypes of the developed devices. The selection of adequate
clinical assessment
strategies and associated metrics depend on the technical
development of specific
functionalities of the MOBOT device. Based on the information as
given by technical
partners, MOBOT working groups will assess technical functions
(responsibility of
technical partners), clinical performance-based functions
(responsibility of clinical
partner Bethanien), user specific perception (responsibility of
clinical partner Diaplasis)
and mixed clinical–technical functionalities as developed for
the MOBOT device
(responsibility of technical and clinical partners).
-
FP7 - MOBOT – 600796 9
2. UPDATE ON DEFINITION OF USER GROUPS AND USE CASES
(BETHANIEN)
In this chapter the final definition of user groups and use
cases is provided. In the
following we relate to the previous report D5.1 (Preliminary
report on use cases and
user needs).
2.1. Definition of user groups
2.1.1. Previous definition of user groups
The MOBOT project consortium decided to focus on moderate to
mild impaired
persons in institutionalized settings such as geriatric
rehabilitation and seniors‟ homes.
This target user group is defined as frail, multi-morbid
persons, partly with acute
impairments as in geriatric rehabilitation, with decreased motor
status based on multiple
impairments and advanced age/frailty, and high incidence of
cognitive impairment. This
highly impaired group ascertains that the devices to be
developed and validated by
MOBOT will meet with a high demand for comprehensive support as
induced by
multiple impairments.
With cognitive and motor status we chose two main criteria to
define the sample as both
criteria will dominate the options and limitations in using the
projected devices to be
developed by the MOBOT project consortium. Cognitive as well as
motor impairments
leading to falls represent main risk factors for loss of
autonomy. As established in
rehabilitation research we focus on impairments rather than
single diseases as devices
will not be specified for disease symptoms but specific
impairments.
The definition of user groups is required in this context to
further describe the target
population for use of the projected device and as inclusion
criteria for validation studies
as projected in MOBOT.
2.1.2. Addenda for user groups
In addition to the first version of the user definition as
documented in our previous
report (cognitive and motor status), we decided to add
anthropometric data (body
weight and body height) as safety, usability and functionality
of the new device are
strongly associated with individualized customization.
2.1.3. Final inclusion criteria and adjusted assessment
We decided finally to focus on 3 criteria for the use of the
device including motor and
cognitive impairment and specific anthropometric
characteristics.
Motor status (See D5.1, page 110-115)
The development of MOBOT devices will focus on two motor key
features: walking
ability and transfer ability in sit-to-stand situations. We
therefore decided to use motor
assessment criteria which distinguish between gait assistance
(rollator) and transfer
assistance (e.g. specific rollator type for sit-to-stand
transfer) to achieve a specific motor
definition of the user group. We suggest combined assessment
criteria which are based
on standardised clinical observation (rollator use, sit-to-stand
transfer/ seat elevation) or
-
FP7 - MOBOT – 600796 10
are based on internationally established tests such as Gait
tests or the Short Physical
Performance test (SPPB).
To ascertain that the case definition addresses the target
population of persons using a
gait assistive device we will use a clinical marker/observation
(current use of rollator:
yes vs. no) as a dichotomous clinical criteria.
We further suggest an established performance-based measure
which is specific for gait
impairments (gait speed >0.6 m/sec without walking device)
[Cesari, 2011; Fritz and
Lusardi, 2009; Studenski et al., 2003; Studenski, 2009]. Using
established cut-offs as in
MOBOT, a classification of patients is feasible.
This selected assessment-based cut off represents a clinical
relevant parameter for
indoor performance representing the main activity (indoor
walking) in the target sample
of rehab patients. A number of other references use higher max
speed especially for
outdoor performance/ or higher functioning persons, or lower
values for higher
impairment levels. Gait speed as a continuous variable is highly
associated with other
clinical outcomes [Guralnik et al., 1995] and allows evaluation
of functional status in a
key motor performance for autonomy without ceiling or floor
effects.
In table 2-1 the gait-related classification for in/exclusion of
MOBOT user groups are
given. Patients without clinically relevant impairments are
excluded (no demand for
rollator use), as are patients with severe impairment (no longer
able to use a rollator
because of high risk exposure)
Table 2-1 Inclusion criteria for gait
Category Impairment level
Category
1
Persons with no or minor motor
impairment (no use of rollator, max
gait speed unassisted >0.6 m/sec)
excluded
Category
2
Persons with moderate motor
impairment (habitual use of rollator,
gait speed
-
FP7 - MOBOT – 600796 11
Table 2-2 Inclusion criteria for transfer
Categories Impairment level
Category 1 Patients with no or minor impairment
(Person is able to stand up and sit
down unassisted on a normal chair
without problem, 5-chair stand 16,7 sec or
able to stand up from a chair from
elevated chair (120% lower leg length)
included
Category 3 Patients with very severe transfer
impairment (unable to use transfer aid
without supervision, unable to stand up
from an elevated position (120% lower
leg length)
excluded
The majority of patients admitted to geriatric rehabilitation
will meet with our inclusion
criteria. However, we expect a noticeable number of patients
rated in category 3 to fail
in sit-to-stand tasks (floor effects for STS testing).
Cognitive status (See D5.1, page 110-115)
We will determine the cognitive status with the internationally
well established
screening test for cognitive impairment (MMSE, Mini Mental State
Examination). This
allows comparability to other studies also including cognitive
criteria. The test has
proven high validity, reliability, feasibility to be used as a
screening instrument
[Folstein et al., 1975; Tombaugh and McIntyre, 1992] and allows
a rather
comprehensive documentation of cognitive sub-performances. We
used established cut-
offs for the definition of the user groups with 27-30 scores to
define persons without
cognitive impairment and scores ranging from 17-26 to define
persons with mild to
moderate impairment. We used the cut-off value of 26 as a more
sensitive (but less
specific) cut-off compared to another established value in MMSE
assessment (n=24) to
include less impaired persons.
As we start with the development of a new clinical device we
suggest not to include
persons with more severe impairment levels (MMSE
-
FP7 - MOBOT – 600796 12
Category 2 Mild to moderate impairment (MMSE
17-25)
included
Category 3 Advanced to severe impairment
(MMSE
-
FP7 - MOBOT – 600796 13
The use cases as listed in the table below identify tasks or
situations which differ with
respect to duration, frequency of activity and clinical
relevance. Some use cases will
only take seconds while other may take minutes, some situations
will occur frequently
during a day while others will only occur once a year. Some of
the situations occur
rarely but are crucial for the task of the device (to support
motor stability) to the user (to
prevent injurious falls).
The suggested use cases are not strictly specified for the MOBOT
device. Some of them
as described below will be useful for validation of
functionalities of the device while
others may be less specific and more standardised use cases will
have to be established.
As at the current state of the project not all projected
functionalities have been fully
developed, a number of additional or modified use cases may be
included depending on
availability of functionalities and the validation strategies
chosen.
2.2.2. Addenda for use cases
We decided to delete some of the less specific use cases of our
previous report and add
some others.
2.2.3. Final Use cases
The following table presents a final list of use cases we will
focus on.
Table 2-6 List of use cases
No. Use case Description/Background
1 The user moves to
get up from a lying
position
The user is moving from a lying position to a seated
position, indicating an intention to get up
2 The user moves to
stand up from a
seated position
The user is initiating movements from a seated position
indicating an intention to stand up
3 The user calls for
the device
gesturally and/or
vocally from a
seated position
The user is sitting and is in need of the device, which is
out of reach; the user performs specific calling (come
here type) gestures and/or calls for the device issuing
specific vocal commands
4 The user calls for
the device
gesturally and/or
vocally from a
standing position
The user is in a standing position and is in need of the
device, which is out of reach (in a near stand-by or
parked mode); the user performs specific calling (come
here / come closer) gestures and/or calls for the device
issuing specific vocal commands
5 Mobility assistant
approaches user
while being in a
seated position
Mobility assistant approaches user after being called,
user is in a seated position
6 Mobility assistant
approaches user
while being in a
standing position
Mobility assistant approaches user after being called,
user is in a standing position
-
FP7 - MOBOT – 600796 14
7 The user starts
accelerating the
mobility assistant
Smooth acceleration is required from 0 to an adequate
walking velocity
8 The user walks
straight with the
device in obstacle-
free area
Stability and safety while walking are very important as
those older frail persons are vulnerable and without
adequate postural stability, especially at the beginning of
walking users may feel dizzy (common on older persons
due to balance problems like orthostatic problems). The
device has to observe the walking pattern and stabilize
the gait.
9 User changes the
velocity (e.g. in
dual-task situation)
or changes
orientation
Changing the velocity is a function that has to be
provided as user could be tired or in a dual task situation
(like talking, making a decision, avoiding obstacles while
walking). Changes of orientation or direction could occur
while avoiding obstacles or walls or if the user changes
his mind about the goal
10 User does not
recognize static or
dynamic obstacles
Avoiding obstacles if user does not recognize obstacles
as mentioned before is part of walking. Especially older
persons with sensory impairment (visual impairment)
detect these situations later and more inexact. Compared
to objects, other persons move, possibly very fast, and
change their position.
11 User walks on
slopes
Walking on slopes supposed to be a rare event especially
in indoor situation, but is a very difficult situation in
normal rollators or devices. To support this situation may
be useful for the frail user as it demands for braking
while walking and controlling if going downwards,
increasing pushing force to move and control the rollator
if going upwards and tracking path and controlling the
rollator on a sideway slope.
12 User walks/has to
manoeuvre in
narrow area
User needs support in collision avoidance, obstacle
avoidance and steering as narrow spaces are very
difficult to pass especially if there is visual or balance
impairment
13 User has to walk
through narrow
passages like doors
User needs support in collision avoidance, obstacle
avoidance and steering as narrow spaces are very
difficult to pass especially if there is visual or balance
impairment
14 User stands up or
sits down
Support in sit-to-stand or stand to sit transfer
15 User has to
open/close doors
Opening and closing doors is a typical everyday task
16 User takes
something and tries
to walk using one
Stability is required while holding the device with one
hand (partial pressure), no goal-directed movement with
one hand should occur
-
FP7 - MOBOT – 600796 15
hand only
17 User wants to lean
or sit on device
Device is blocked and provides leaning/sitting support if
needed
18 User wants to walk
with indirect
support by the
device
A following mode to support user if direct contact is not
needed or impossible. The device only approaches the
user in case of a potential fall or insecurity and otherwise
follows him/her at a safe distance.
19 User wants to stop
or brake or has to
stop or brake
accidently
Braking and stopping the rollator has to be smooth and
safe to reduce the risk for the user.
20 User does not want
to use or need the
device
Mobility assistant is in stand-by mode and waits for user
to call it in a safe parking position.
21 User is not oriented Orientation support in unfamiliar
environments to enable
user to participate and to acclimatize and orient in the
new environment. Support of orientation is also
important to persons with cognitive impairment as
orientation is particularly difficult for example if a
demented patient is hospitalized.
22 User does not know
the way, and needs
to be guided to
desired location
User is guided to desired location after having
communicated it to mobility assistant
23 User acts hasty User needs to calm down and has a special
need for
stability
24 User needs advice/
wants to advise
something
Communication advises if reorientation is required or if
user wants to command the device
25 User is insecure If the user is insecure or does not accept
the suggested
movement by the device comment or explanation could
be convenient.
26 User does not
notice instructions
due to his/her
impairment or
distraction
The user is cognitively impaired or does not notice
instructions due to distraction. Repetitions are needed.
27 User does not hear
the instruction
The user has possibly hearing impairment or in case of
loud environment the user cannot hear instructions by the
device.
28 User falls down User falls down and is not able to stand up
alone
29 Misuse of the
device
Protection of impairment of the persons or damage of
device
-
FP7 - MOBOT – 600796 16
30 External problems,
like uneven or slick
floor
Monitoring of environment
31 User or caregiver
wants to change
base setting
Operator mode to modify base setting or support specific
modes. The mode is important for basic instrument
adjustment and machine care.
32 User tests all
support modes of
the device in a
demo mode
User is guided through the different modes of the device
and instructed on the usage of it
33 User has to
recharge the device
Charging mode and simplicity of handling and operating
of charging. The mode is important to guarantee optimal
support while using the device.
*the use cases marked with light colour may not be evaluated
during validation studies,
as they are limited in feasibility or would impair ethical
requests.
*the use cases crossed have been deleted out of the original set
because these represent
very special cases that require very tailored and individualized
solutions.
-
FP7 - MOBOT – 600796 17
3. SUBJECTIVE AND OBJECTIVE PERFORMANCE METRICS FOR MOBILITY
DEVICE ASSESSMENT (BETHANIEN)
For technical devices as projected by MOBOT to support mobility
and independence in
a high risk population, a clinical evaluation is mandatory to
ensure the safety, usability,
efficacy, and acceptance. Identified user needs should guide the
development and
evaluation process to provide structured data of realistic
experience with the device
(Guidelines on medical devices, 2009). The development of
robotic mobility devices
enfolds different perspectives and has to bring together the
technical developments as
well as human requirements. Some of the requirements may be
mandatory for a
mobility device such as safety aspects, while others may
represent service
functionalities. Hierarchical aspects of the development derive
from the hierarchy of
user needs. For instance, mobility robots are developed mainly
for older, disabled or
impaired persons with crucial mobility impairments. For these
persons safety is
fundamentally important due to their vulnerability and has to be
provided in every
possible situation (Feil-Seifer et al., 2007).
To ensure usability of a supportive device not only technical
but also the subjective user
perceptions and perspective may be relevant including handling
and usability of the
device, the satisfaction and acceptance, or the potential impact
on activities of daily
living. In technically driven developments such user-specific
demands have frequently
not or insufficiently been considered.
In the following we focus on technical and performance-based
clinical evaluation. As a
prerequisite for a clinical evaluation, adequate assessment
strategies will have to be
identified including associated metrics related to such
assessments. For a clinical
validation the user perspective will be most relevant and will
guide the selection of
measurements and associated metrics. Different aspects influence
the use and
acceptance of a robotic device.
Beside the technical functionality and (task) performance in
motor key performances as
supported by the MOBOT device (walking and sit-to-stand
transfer), subjectively
perceived usefulness, user experience and usability are
important. We therefore propose
qualitative and quantitative metrics that measure outcomes of
the human-MOBOT
interaction. The outcomes will be consistent with the
predetermined objectives of the
MOBOT project. Particularly those functions that are specific
for the MOBOT device
may have to be targeted to document the added value. To address
different technical and
clinical perspectives, the evaluation of the MOBOT device will
be separated into
different types of evaluation. It will be further specified how
clinical evaluation will be
attributed to clinical partners.
Table 3-1 Different types, purposes and responsibilities of
evaluations
Type of
evaluation
Purpose/matter of evaluation Responsibility
Technical Test of technical systems/subsystems
and technical functionalities
Technical partners
Clinical Generic (established) clinical
assessment, performance-based
measures
Bethanien (clinical partner)
Subjective perception, feasibility Diaplasis (clinical
partner)
-
FP7 - MOBOT – 600796 18
Specifically tailored assessment tests
focusing on specific functionalities
to document the added value of the
device
Validation I/rollator:
clinical partners/ technical
partners
Validation II/ rollator:
clinical partners/ technical
partners
Validation 3: nurse-type:
clinical/ technical partners
3.1. Technical evaluation
The specific functions of the MOBOT device are documented in the
DoW. Technical
functionalities will have to be assessed with respect to
accuracy, validity and reliability
of technical function. Both the single technical functionality
as well as the function
within the device has to be ascertained. Responsibility for such
assessment lies with the
technical partners responsible for the specific functionality.
Specific technical
assessment strategies may translate into the specific clinical
validation to document the
specific added value of the functionality. Optional integration
of technical assessment
strategies into the validation studies will have to be discussed
between clinical and
technical partners when functionalities have been fully
developed. The technical
evaluation represents a first step to ensure the suitability for
the intended use and will be
a prerequisite for clinical validation of a mobility aid
classified as a medical product.
3.2. Clinical Evaluation
Clinical evaluation targets at the interaction between a human
and the device. For
development of a comprehensive validation strategy we propose 3
different approaches.
3.2.1. Generic (established) clinical assessment,
performance-based measures
As already detailed in Deliverable 5.1., we suggest established
clinical measures such as
the Timed-up-and-go test (TUG) and the Short Physical
performance battery (SPPB) to
document performance in motor key functions as supported by the
MOBOT
development. Such motor key features include walking
performance, static and dynamic
postural control, and sit-to-stand performance. Those clinical
measures include detailed
metrics based on a validated and standardised assessment (see
previous report D5.1.).
When the first validation study will take place at Bethanien
hospital we will be able to
support those clinical measures with additional technical
assessments (electronic
gaitway “Gait Rite”) and accelerometer-based assessment of
transfer by “Dynaport”
measures) which give a large number of detailed temporo-spatial
metrics for evaluation.
Clinical as well as additional technical measures hold a variety
of metrics specific for
the assessment (for detailed information see also D5.1.) which
we will not summarize in
this context.
As one of the main objectives of the MOBOT device is support in
walking and a
number of use cases developed in the MOBOT project (see also
D5.1 and chapter 2 of
this report) include walking, different walking tasks may be
used as test scenarios in the
validation. In the following we give a list of scenarios which
are typically required in
daily routine: (see also list of use cases as reported in
D5.1)
-
FP7 - MOBOT – 600796 19
- Starting the rollator: Acceleration from 0 to
adequate/individual velocity
- Changing the velocity/adjust motor driven speed of rollator to
individual speed
of user
- Changing orientation or direction while walking
- Walking up/down slopes
- Steering the device/ avoiding obstacles
- Decelerating/ stopping the rollator
Those specific tasks are not part of the standardised clinical
assessment as reported
above (TUG/SPPB), but may be included in the design to evaluate
specific
functionalities of the MOBOT development (see chapter
below).
3.2.2. Subjective measures for Human-Robot Interaction (HRI)
(DIAPLASIS)
3.2.2.1. Human-Robot interaction metrics
In the early years of many technical fields, the research
community often utilizes a wide
range of metrics that are not comparable due to a bias towards
application specific
measures. The primary difficulty in defining common metrics is
the incredibly diverse
range of human-robot applications. Thus, although metrics from
other fields (HCI,
human factors, etc.) can be applied to satisfy specific needs,
identifying metrics that can
accommodate the entire application space may not be feasible
[Steinfeld et al., 2006].
Attempts to categorize both objective and subjective metrics
have been made.
According to the USUS Evaluation Framework for Human-Robot
Interaction [Weiss et
al., 2009] the factors usability, social acceptance, user
experience, and societal impact
are considered the main categories of evaluation factors. Each
category is divided to
specific metrics, either objectively or subjectively
measured:
Usability: Effectiveness, Efficiency, Learnability, Flexibility,
Robustness, Utility.
Social acceptance: Performance Expectancy, Effort Expectancy,
Attitude toward Using
Technology, Self Efficacy, Forms of Grouping, Attachment,
Reciprocity.
User experience: Embodiment, Emotion, Human-Oriented Perception,
Feeling of
Security, Co-Experience with Robots.
Societal impact: Societal impact describes all effects the
introduction of robotic agents
consequences for the social life of a specific community (taking
into account cultural
differences) in terms of quality of life, working conditions and
employment, and
education.
Among these, the authors propose that the following could be
tested using end-user
questionnaires:
Utility (refers to how an interface can be used to reach a
certain goal or to perform a certain task. The more tasks the
interface is designed to perform, the
more utility it has)
Performance Expectancy (the degree to which an individual
believes that using the system will help him or her to attain gains
in performance)
-
FP7 - MOBOT – 600796 20
Effort Expectancy (indicates to which extent the user perceives
a system will be easy to use)
Attitude toward Using Technology (the sum of all positive or
negative feelings and attitudes about solving working tasks
supported by a humanoid robot)
Self Efficacy (relates to a person‟s perception of their ability
to reach a goal)
Attachment (an affection-tie that one person forms between him/
herself and another person or object - a tie that binds them
together in space and endures
over time)
Reciprocity (the positive or negative response of individuals
towards the actions of others)
Embodiment (describes the relationship between a system and its
environment and can be measured by investigating the different
perturbatory channels like
morphology, which has impact on social expectations)
Emotion (As emotion is an essential part in social interaction
it has to be incorporated in the assessment and design of
robots)
Feeling of Security (it is important to investigate how to
design human-robot interaction in a way that humans experience them
as safe)
Co-Experience (Co-experience describes experiences with objects
regarding how individuals develop their personal experience based
on social interaction
with others)
Societal Impact (Societal impact describes all effects the
introduction of robotic agents consequences for the social life of
a specific community -taking into
account cultural differences- in terms of quality of life,
working conditions and
employment, and education)
The above categorization is the most full and detailed,
including aspects that are rarely
taken into account when it comes to evaluating a robotic
assistant. Most researchers,
however, when evaluating an assistive device/technology tend to
use questionnaires that
give information on the aforementioned fields. However, as
Bartneck et al. mention, due
to their naivety and the amount of work necessary to create a
validated questionnaire,
developers of robots have a tendency to quickly cook up their
own questionnaires. This
conduct results in two main problems. Firstly, the validity and
reliability of these
questionnaires has often not been evaluated. Secondly, the
absence of standard
questionnaires makes it difficult to compare the results from
different researchers
[Bartneck et al., 2009].
For example, on a feasibility study of a robotic medication
assistant for the elderly,
Tiwari et al. created a ten-itemed questionnaire (items answered
on a 5-point Likert
scale, 1 indicating strong agreement and 5 indicating strong
disagreement). Some of the
questions were: “I felt very confident using the system”, “It
would be easy to practically
use this system in our living quarters” or “I would imagine that
most people would learn
to use it very quickly” [Tiwari et al., 2011].
In the same way, Heerink et al. examined the acceptance of a
robotic agent by elderly
users. The subjects were asked questions on perceived social
abilities and technology
acceptance. At this case questions were only partly chosen from
an existing model
(UTUAT) and partly created from the researchers. Again, they
were formed in a 5-point
Likert scale [Heerink et al., 2000].
-
FP7 - MOBOT – 600796 21
This type of choosing tailored questionnaires is the rule in
robotics assessment.
However, few valuated tools from the assessment of assistive
technology exist and are
sometimes used for assessing robotic assistants.
3.2.2.2. Existing questionnaires for subjective HRI measures
The existing variety of questionnaires that could be useful for
the evaluation procedure
of the MOBOT prototypes is narrow, for the reasons described
above. The first one that
we should mention is the “Quebec User Evaluation of Satisfaction
with assistive
Technology” (QUEST2.0) [Demers et al., 2006]. According to Holz
et al. the QUEST
2.0 is the only standardized satisfaction assessment tool
designed for assistive
technologies [Holz et al., 2013]. Moreover, it is considered as
the most relevant tool in
functional assessment for assistive Technology [Jardón et al.,
2011]. It is the most
commonly used assessment tool for HRI. An assistive technology
evaluation should, for
the above reasons, include the QUEST2.0 instrument.
A strategy that would target maximum coverage of the subjective
measures spectrum
would require the combined use of two (or more) questionnaires,
since the QUEST2.0
only covers some subjective aspects (mainly: feeling of
security, perceived
effectiveness, ease of use). The only two validated and
relevantly common used
questionnaires we found in the bibliography were:
- The Assistive Technology Device Predisposition
Assessment-Device Form (ATDPA)
[Scherer et al., 2005].
- The Psychosocial Impact of Assistive Devices Scale (PIADS)
[Palmer et al., 2005].
The ATDPA-Device Form was more relevant in context than the
PIADS, targeting on
evaluating overall user experience with assistive technology,
while PIADS only
emphasizes to the psychosocial impact of assistive devices,
without targeting on
evaluating the actual experience of interacting with a robot,
but rather on the impact that
this interaction has in quality of life (QoL).
Also, other questionnaires such as the USE-IT [Michel-Verkerke
and Hoogeboom,
2013] questionnaire were ruled out from the very beginning,
since they were not well-
valuated or not widely used from researchers in the
bibliography.
So, for the first evaluation measurements, we propose the
combined use of the
QUEST2.0 questionnaire and the Assistive Technology Device
Predisposition
Assessment-Device Form (ATDPA), as the combination that covers
most of the
desirable user-experience aspects, with assured validity and
reliability.
3.2.2.2.1. The “Quebec User Evaluation of Satisfaction with
Assistive
Technology” (QUEST2.0) questionnaire
The Quebec User Evaluation of Satisfaction with assistive
Technology (QUEST) is an
instrument specifically designed to measure satisfaction with a
broad range of assistive
technology devices in a structured and standardized way.
Although its experimental
version consisted of 24 items, an item analysis subsequently
resulted in a reduced 12-
item scale: the QUEST 2.0. These 12 items relate to device
characteristics (n=8) and
-
FP7 - MOBOT – 600796 22
assistive technology services (n=4) [Demers et al., 2002]. The 8
device characteristics
items assess the user‟s degree of satisfaction with device
properties and the remaining 4
items are related to assistive technology services. Items are
rated on a 5-point Likert
scale indicating level of satisfaction with the aspect of the
device or service. The user is
asked to identify the satisfaction variables most important to
them. The QUEST was
developed for use with a wide range of assistive technology
devices and not all items
are relevant to every device (appendix 2).The device
characteristics items are the
following:
How satisfied are you with:
1. the dimensions (size, height, length, width) of your
assistive device? 2. the weight of your assistive device? 3. the
ease in adjusting (fixing, fastening) the parts of your assistive
device? 4. how safe and secure your assistive device is? 5. the
durability (endurance, resistance to wear) of your assistive
device? 6. how easy it is to use your assistive device? 7. how
comfortable your assistive device is? 8. how effective your
assistive device is (the degree to which your device meets
your needs)?
The measurement properties of the QUEST 2.0 have been
investigated with respect to
reliability, test-retest stability, alternate form reliability,
construct validity and
applicability [Demers et al., 2002] by its developers. Also, the
QUEST 2.0 has been
used at many studies in order to evaluate perceived satisfaction
among users of assistive
devices.
Zickler et al. used the QUEST2.0 in order to measure user
satisfaction with the revised
prototype of the brain–computer interface (BCI). Brain Painting
application was
evaluated in its target function – free painting – and compared
to the P300 spelling
application by four end users with severe disabilities [Zickler
et al., 2013]. The
Assistive Technology Device Predisposition Assessment (ATD PA)
Device Form was
also used in combination, in order to complement the subjective
measurements. Four
items of the QUEST2.0 (the assistive technology services
category: durability, service
delivery, repairs/servicing, follow-up services) were not
adequate for the evaluation of a
BCI prototype and were, thus, removed from the questionnaire. To
render the
QUEST2.0 more suitable for evaluation of BCI-controlled AT the
four items reliability,
speed, learnability, and esthetic design were added. This BCI
adapted QUEST 2.0 was
referred to as “extended QUEST2.0.” [Scherer and Cushman,
2002].
In the same field, Holz et al. (2013) used the QUEST2.0 in order
to measure user
satisfaction with “Connect-Four”, a new sensorimotor rhythm
(SMR) based brain–
computer interface (BCI) gaming application. The interface was
evaluated by four
severely motor restricted end-users. The authors used the same
extended form of the
QUEST2.0 as Zickler et al., and also complemented the QUEST2.0
with The Assistive
Technology Device Predisposition Assessment (ATD PA) Device
Form.
The QUEST2.0 has already been used for assessing assistive
robots. Jardón et al. (2011)
used the QUEST2.0 on a population of six spinal cord injury
patients, in order to assess
the usability of a sophisticated technical aid, the “ASIBOT”, a
personal assistance robot
totally developed by RoboticsLab at the University Carlos III of
Madrid. Their
-
FP7 - MOBOT – 600796 23
methodology was based on the QUEST test with some specific and
local changes to
adapt the questionnaire to their population and product. The
designed questionnaire was
similar to QUEST regarding device aspects such as usefulness,
training, robustness,
safety, dimensions, simplicity of use, appearance, and effort of
installation. However,
they –as most prototype assessors- chose not to use the service
components for the
“ASIBOT” prototype.
Concerning mobility assistive devices, QUEST2.0 has been used in
several studies, for
assessing user acceptance of mobility aids such as rollators or
electric and manual
wheelchairs. Kirby et al. used the QUEST2.0 combined with a
device-specific
questionnaire (the Wheelchair Skills Test -WST, version 3.2.) in
order to test the
hypothesis that, in comparison with a commercially available
tilt-in-space wheelchair, a
light-weight manual wheelchair equipped with a new, rear
anti-tip device (Arc-RAD)
provides caregivers with improved wheelchair-handling
performance, less exertion, and
greater satisfaction. Again, the authors used only the 8
components for assistive devices
[Kirby et al., 2008].
Hill et al. used the QUEST2.0 to assess satisfaction with the
use of a typical rollator among persons with Chronic Obstructive
Pulmonary Disease. Supplementary, a
structured questionnaire was used to obtain information
regarding daily utility of the
device and barriers to its use [Hill et al., 2008].
Samuelsson and Wressle, in order to follow-up user satisfaction
with and the use and
usefulness of rollators and manual wheelchairs, interviewed a
sample of 262 users, 175
rollator users and 87 wheelchair users. In this study, also, the
QUEST 2.0 was not used
alone: an additional ten-itemed questionnaire was used for
collection of subjective data
[Samuelsson and Wressle, 2008].
Finally, Laffont et al., used the QUEST2.0 in combination with
some basic objective
metrics (trial duration, number of failures, etc) in order to
evaluate a stair-climbing
power wheelchair (“TopChair”), designed for indoor and outdoor
use, including stair-
climbing. The sample was consisted of 25 people with tetraplegia
[Laffont et al., 2008].
3.2.2.2.2. The “Assistive Technology Device Predisposition
Assessment-
Device Form” (ATDPA)
The ATD PA-Device Form is a 12-item questionnaire that examines
consumer‟s
subjective satisfaction with achievements in a variety of
functional areas, when using
assistive technology (AT). It is the last part of the 66-itemed
“ATD PA”, a
questionnaire based at the Matching Person and Technology (MPT)
Model [Fuhrer,
2001]. The MPT process is validated for use by persons with
disabilities (ages 15 and
up) and is applicable across a variety of users and settings.
The measures have been
determined to have good reliability and validity and they have
been used in research
studies within the US, Canada, and Europe [Scherer et al.,
2005]. A complete list of
validation studies of the “ATD PA” instrument and for the MPT
model can be found at
the Institute of Matching Person and Technology
[http://www.matchingpersonandtechnology.com/validation.html].
The questionnaire rates the AT-person match and the expected
support in using the
device, in other words the expected technology benefit. The 12
items (such as “will help
http://www.matchingpersonandtechnology.com/validation.html
-
FP7 - MOBOT – 600796 24
me to achieve my goals”, “will fit in accustomed routine”), are
rated on a 5-point
Likert-scale from 1 (not at all/0% of the time) to 5 (all the
time/100% of the time).
Users have the option to indicate a “0” if the item is not
applicable. By averaging the
total of the items a mean score can be calculated. The highest
possible score would then
be 5.0. The items are:
1. This device will help me to achieve my goals. 2. This device
will benefit me and improve my quality of life 3. I am confident I
know how to use this device and its various features 4. I will feel
more secure (safe, sure of myself) when using this device 5. This
device will fit well with my accustomed routine 6. I have the
capabilities and stamina to use this device without discomfort,
stress and fatigue
7. The supports, assistance and accommodations exist for
successful use of this device
8. This device will physically fit in all desired environments
(car, living room, etc.)
9. I will feel comfortable using this device around family 10. I
will feel comfortable using this device around friends 11. I will
feel comfortable using this device at work 12. I will feel
comfortable using this device around the community.
(Possible answers: 5 = all the time/100% of the time, 4 =
often/around 75% of
the time, 3 = half the time, neutral/about 50% of the time, 2 =
sometimes/around
25% of the time, 1 = not at all/0% of the time, 0 = not
applicable)
According to the MPT Institute
[http://www.matchingpersonandtechnology.com/], the
ATD PA Device Form is compatible with the World Health
Organizations‟ ICF and,
thus, may be considered measures relevant for use in assessing
ICF domains as
impacted by technology use.
It has already been mentioned that the ATD PA-Device Form has
been used in the
evaluation of very recently developed brain–computer interfaces
(BCI) [Zickler et al.,
2013; Holz et al. 2013]. However, although only in some more
researches the ATD PA
has been used, the ATD PA-Device Form, as a reliable and valid
instrument that covers
several aspects of subjective human-robot interaction
measurements, could be used
supplementary to the use of the QUEST2.0, offering items that
give information on
learnability, performance expectancy, effort expectancy,
attitude toward using
technology, perceived effectiveness and self-efficacy [Scherer
and Fischer, 1998].
Concluding, we propose, for the evaluation of the first stage of
MOBOT, the use of
QUEST 2.0 combined with the ATD PA-Device Form. Thus, we could
acquire the most
significant subjective measures utilizing already valid and
evaluated questionnaires.
3.2.3. Specifically tailored assessment tests focusing on
specific functionalities to
document the added value of the device (BETHANIEN)
The Rollator-type device as projected by MOBOT will include
specific functionalities
which allow a number of additional support aspects compared to
low-tech established
rollators. (For a comprehensive list of functionalities and
state of development see study
protocol).
http://www.matchingpersonandtechnology.com/
-
FP7 - MOBOT – 600796 25
Functionalities to be developed will be finalised at different
points in time during the
lifecycle of the project depending on the complexity of the
task, the current
development and preliminary work. For the first validation
studies we may include
those functionalities which will achieve stable functions at
that time (original projected
start of validation I: Month 16, due to delay in development of
device postponed to
M18). Responsibility for reliable functionalities lies with the
technical partners and
represents a precondition for participation in the clinical
validation.
Within the MOBOT project organisation ICCS for technical
partners and Bethanien for
clinical partners have been appointed at the last project
meeting (Munich 11.-12.12;
2013) to organise and develop the validation strategy of such
functionalities. The task
force will address other technical and clinical partners.
However, technical partners will
hold full responsibility for technical reliability and
functionality and tasks associated to
the validation process such as transport and maintenance of the
technical devices for
validation purpose. The task force will start the development of
specific assessment
strategies associated to functionalities in January 2014 to
allow a timely finalisation for
the first round of validation studies.
Specific roles of clinical partners will have to be specified
for the upcoming validation
studies. As functionalities represent innovative approaches, a
tailored validation strategy
may have to be developed. For such a validation, scenarios as
already listed in D5.1 will
be used and extended by additional scenarios specific for the
functionality. A number of
established clinical tests may be modified to be used for the
tailored validation. Results
of the systematic review (see chapter 4 of this report,
validation) as performed by
Bethanien will be used to implement validated, standardised test
procedures as used in
comparable, previous validation studies.
In case specific technical assessment strategies for technical
validation can be used for
clinical validation as well, their usability for validation
purpose will be checked.
Technical partners will have to participate and support the
decision to use and
implement their functionalities in the validation process. A
report on a detailed
assessment strategy and results of the first validation will be
given in Deliverable 5.3.
-
FP7 - MOBOT – 600796 26
4. USER STUDY PREPARATIONS (BETHANIEN)
The technical devices as developed by MOBOT partners will be
evaluated in a clinical as well
as technical validation process. Depending on the technical
solutions and their feasibility to be
included in the prototypes, associated functionalities may also
be part of the clinical
validation. The primary focus of the clinical validation will be
the subjective perception and
usability of devices and performance-based measures as
established in clinical settings and
validations. In our previous report we described different
assessment strategies (e.g. screening
vs. performance-based measures, clinical vs. technical measures)
with focus on the key motor
features as supported by the MOBOT devices (transfer support,
support in walking). We
provided information and requirements on natural human movement
patterns, support and
compensation and established assessment strategies in frail
older persons (see D5.1, pages 49-
82).
4.1. Results of a review on clinical validation studies of
robotic devices
As a first step in developing an evidence-based validation
design we performed a systematic
review on previous clinical validation studies in comparable
mobility support devices. Results
of this review are currently summarized in a manuscript in
detail and will be submitted for
publication timely. Results of the review have already been
documented in the previous report
(D5.1, pages 136-149) and have been updated for the present
report.
-
FP7 - MOBOT – 600796 27
Table 4-1 Evaluation studies of a mobility assistance robot
Name of device:
author
Tec
hn
ical
ev
alu
atio
n/
Use
r-in
tera
ctio
n t
est*
Type of assessment (Standardized/Non-
Standardized; Validated/Non-Validated;
Technical Evaluation/Performance test), description
of assessments
Number of
participants,
description of
sample
Po
ten
tial
u
sers
/ O
ther
un
con
cern
ed p
erso
ns
*² Results of tests (Technical evaluation/ User-interaction
test: objective
quantification or subjective comment or rating;
comment on statistical analysis ( no stats: no statistical
analysis
mentioned, with stats: with statistical analysis mentioned)
1. CAIROW: Mou
et al., 2012
U S/NS; NV:
P: 80 meters walking path → standard deviation of
velocity and step length;
Subjective user opinion
n=6 (persons with
Parkinson Disease)
P U: Presentation of performance results without detailed
subsumption;
Subj (user): positive comments (stimulating force by
device);
no stats
NS; NV:
P: walking with usual device vs. Cairow → abnormal
gait patterns and gait velocity
Subjective quantification of gait by experts
n=7 (persons with
Parkinson disease,
ø 86 years)
U: Obj: smaller deviation of velocity (more stable);
Subj (experts): less episodes of abnormal steps while walking
with
Cairow;
no stats
2. Care-o-bot II:
Graf, 2009
TU S, NV:
T: collision avoidance in natural environment
P: Robotic walker vs. conventional walker, 2
different walking courses/conditions → duration,
distance and collisions
Subjective user opinion (questionnaire)
n=6 (older persons
with need for
mobility aid)
P T: collision avoidance increased (no collision vs. 3);
U: Obj: duration with robot longer and force to control robot
high;
Subj (experts): difficulties to control conventional walker in
slopes;
Subj (user): 80% felt safe and in control of Care-o-bot;
no stats
3. CMU Robotic,
XR4000
platform: Morris
et al., 2003
TU NS; NV:
T: Usability and capacity of navigation system with
different control modes
Subjective user opinion
n=4 (older persons,
not specified; target
group: persons with
cognitive
impairment)
P? T: demonstration of control concept and technical
feasibility;
U: Subj (user): acceptance and interest in robotic walker
high,
difficulties mentioned manipulating the haptic interface;
no stats
4. Cool Aide:
Alwan et al., 2005
T
S; NV:
T: Walking straight and with turns with vs. without
controller; human intent vs. controller conflict
situation
n=22 (15 older + 7
younger; all
persons without
motor impairment)
O T: assisted steering enhanced stability of user when intents
of user and
robot aligned, possibly endangering problems if this is not the
case;
no stats
5. Electric Motor
Based Gait
Rehabilitation
System: Lee et al.,
2004
U S; NV;
P: body weight support of 0%, 10%, 20%, 30% and
40% in own comfortable walking speed: double
limb support time and
single limb support time
n=10 (ø 37 years old) O U: double limb support time decreased
and single limb support time
increased with increased body weight support; heart rate:
tendency of decrease with increasing body weight support;
no stats
-
FP7 - MOBOT – 600796 28
Impact on user: heart rate measurement
6. GRSR: Jang et
al., 2008
TU NS; NV:
T: Test of Walking guide system
P: Impact on user: muscle activities tested while
walking with body weight support mode
n=2 (young, healthy
persons)
O T: small path tracking error due to friction and slip of
wheels;
U: Impact on user: EMG activity decreased in the quadriceps
muscles
while walking without full body weight (-40%);
no stats
7. Guido:
Rentschler et al.,
2008
U S; NV:
P: Guido vs. low-tech mobility aid on 36.6 m
obstacle course → travel time, obstacle/ wall
contacts and reorientations
Subjective user opinion (questionnaire)
n=45 (persons with
visual impairment
and limited
mobility)
P U: no significant performance differences between the
devices;
Subj (user): more positive after experience of Guido;
descriptive stats given, no statistical analysis
8. Hitachi: Tamura
et al., 2001
U S; NV:
P: Usual walker vs. caster walker vs. power-assisted
walker, 2x6-meter straight path → walking speed,
body acceleration
Impact on user: gastrocnemius electromyogram
measured
n=6 (older persons, ø
82 years) state of
frailty not clear
P? U: Best performance results with conventional walker;
Impact on user: Results of electromyogram show gradual decreases
as
the speed decreases;
no stats
9. i-walker: Annicciarico,
2012
U S; V:
Impact on user: 4-weeks training with i-Walker vs.
traditional ambulatory treatment
n=20 (stroke patients) P U: Impact on user: significant
improvements in i-walker training
regarding functional outcome (Tinetti, 10MWT, Barthel);
with stats
10. i-Walker:
Kikuchi et al.,
2010
TU S;NV:
P: walking course → with/without controller
N=7 (frail older
persons)
P U: improvement of trajectory, collision avoidance with
controller;
deficit: not helpful for a blind person;
no stats
11. iWalker:
Kulyukin et al.,
2007
TU S; NV:
T: navigation mistakes
P: different walking routes → time of trials
Subjective user opinion
n=4 (frail older
adults; clients of
senior services
agency)
P T: system is able to provide way finding task with
location
information;
U: Subjective user comments and feedback regarding the
interface;
no stats
12. MOBIL
Walker: Bühler et
al., 2001
TU NS; NV:
T: feasibility
P: walking course and practical trials
Subjective user opinion (questionnaire)
n=?, number of
subjects not clear
(persons in foster
home)
P T: Robot is feasible;
U: Stability and speed satisfying, comments regarding the
braking
system, 2/3 of users found different control modes easy to
handle;
no stats
13. Pam-AID:
Lacey and
MacNamara, 2000
TU S; NV:
P with active demonstrator: Course with turns and
obstacles → two control modes
Subjective user opinion (questionnaire, comments
and rating on usability and support)
n=5 (persons with
visual and partially
mobility
impairment, ø 82
years)
P U: Subj (user): adaptive better than fixed control mode;
safety,
usability and utility of device quite to very useful depending
on
impairment level, with stats: mean rating and standard
deviation;
no stats
-
FP7 - MOBOT – 600796 29
NS; NV:
Subjective user opinion: passive demonstrator 5 days
test (no detailed information)
n=19+? (older
persons, part of
patients not
defined)
U: Subj (user): little heavy and difficult to push, sonar sensor
errors
occurred, device was in great demand;
no stats
14. PAMM Smart
walker: Yu et al.,
2003
TU S; NV:
T: wall distance, path deviation
P: walking path of 35 meters → 3 control modes
Subjective user opinion
n=6 (older persons in
assisted living
facility)
P T: adaptive shared control mode is “effective”;
U: improves user performance by shared control mode and not
noticed
by user,
Subj (user): averseness of full computer control mode;
no stats
15. Robotically
augmented
walker: Glover et
al., 2003
TU NS; NV:
T: navigation mode
P test
Subjective user opinion
n=6 (older persons,
no potential user
group)
O T: Navigation “successful”;
U: Subj (user): redesign of interface due to difficulties,
self-parking
and retrieval mode useful;
no stats
16. Robuwalker: Rumeau et al.,
2012
U S; V and NV:
P: Usual walking vs. regular walking frame vs.
Robuwalker → 4 meter walking test and modified
Timed get up and go test
n=9 (older persons
with motor and
cognitive
impairment)
P U: tested walker is less efficient than a regular walking
frame; results
presented;
no stats
17. RoTa: Mori et
al., 2002
U NS; NV:
P: small test course
Subjective user opinion
n=60 (older persons
with visual
impairment)
P U: Subj (user): helpful for the blind due to protection
against obstacles,
deficit: cannot go up and down stairs;
no stats
18. RT Walker:
Hirata et al., 2007
T NS; NV:
T: Obstacle, step or wall detection/ path follow
mode/environment with stairs, obstacles and down
slope
n=4/5/1 (young
persons,
blindfolded)
O T: Walker provides obstacle/step avoidance, gravity
compensation,
variable motion characteristics, fall-prevention; no detailed
results,
annotation of authors: further validation needed;
no stats
19. Simbiosis:
Frizera-Neto et al.,
2011
U S; NV:
P: Walking path (2x10meter + turn)
Subjective user opinion (manoeuvrability, security,
posture/comfort)
n=8 (persons with
spinal cord injury)
P U: subj (user): results given, trend: positive rating
without
subsumption;
no stats
20. Monimad:
Saint-Bauzel et al.,
2009
TU NS; NV:
T: biomechanical analysis of temporo-spatial forces
P: Sit-to-stand movement test
n=10 (Multiple
sclerosis patients)
P T: natural style of interaction: temporo-spatial course of
forces (shape
of forces similar to human-human sit-to-stand experiments);
U: quick learning rate of adequate use of the device;
no stats
21. Robotic walker:
Chugo et al., 2009
U S; NV
P: Standing up motion with/without body stability
control scheme
Subjective user opinion (effort and fear of falling)
N=7 (older persons,
67 years+ with
need of care)
P? U: subj (user): easy standing up movement, no fear of
falling
-
FP7 - MOBOT – 600796 30
*Aim of test: T = Technical evaluation; U = User interaction,
handling or satisfaction evaluation; Type of assessment: S =
Standardized; N = Non-Standardized; V = Validated;
NV = Non-Validated; T = Technical evaluation; P = Performance
test; *²User group: P = Potential-user of the developed device; O =
Other unconcerned persons: Subjects
employed for the evaluation study; Results of tests T =
Technical evaluation results; U = User-interaction test: objective
quantification or subjective comment or rating
comment on statistical analysis: no stats = no statistical
analysis mentioned; with stats = with statistical analysis
mentioned
-
FP7 - MOBOT – 600796 31
4.2 Abbreviated presentation of results of the systematic
review
We included 21 studies based on predefined inclusion criteria
focusing on evaluation of
a robotic walking transfer device. Three different types of
validation approaches were
identified: 1. technical evaluations (n=2); 2. user-oriented
validation (n=9) and mixed
strategies (n=10).
Assessment methods: Assessment methods varied substantially
according to aim of
specific testing and quality of assessment methods. Most of
assessment tools seemed
not to be validated in general or seem not to be validated for
the specific target group
(19 out of 21), one study used validated and non-validated
measures [Rumeau et al.,
2012] and one study used validated assessment methods
[Annicchiarico, 2012]. A
number of assessments was not clearly described or were based on
subjective ratings
with uncertain or insufficient test quality. We could not
identify studies focusing on
long-term effects of devices such as the impact on habitual
physical activity or quality
of life. None of the studies showed an overall convincing and
comprehensive
assessment strategy and design. No standardised assessment
protocol specifying
primary outcome variables seems to be established for such
clinical validation studies,
severely limiting comparability of results. The added values by
new functionalities of
devices were only partly or not addressed at all in assessment
strategies.
Target samples: In most of the studies it remains unclear
whether tests and the devices
itself were targeted for a specific user group. A clear use case
definition was missing.
Statistical analysis: With the exception of 1 study, results
were all presented without
statistical analysis thereby substantially limiting the value of
validation results by mere
descriptive case presentation without statistical back up. No
sample size calculation was
mentioned or presented.
4.3 Objectives to be achieved for MOBOT- validation studies
The table presents objectives, solutions and the actual state of
preparation.
Table 4-2 Objectives and solutions for evaluation
Objectives Solutions for MOBOT State of
preparation
Clear definition of target sample
including sub-patient categories
according to motor and cognitive
status
See chapter 2.1 √
Clear definition of goals for the
development of the device and use-
case definition
See chapter 4 (status of
development for evaluation
studies)
√-(√)
Clear definition of use cases See chapter 2.2 √
Selection of standardized, validated (at
best for the specific sample e.g.
cognitive impairment) assessment
tools
Concerning inclusion criteria
to reach target group: see
chapter 2.1.3
Concerning assessment tests
√-(√)
-
FP7 - MOBOT – 600796 32
for new device functionalities:
to be determined
Definition of procedures and metrics
for user evaluation studies
See chapter 4 √-(√)
Adjusted assessment strategy to cover
specific targets of development. The
assessment should clearly separate
between technical assessment of
technical features, requesting mainly
engineering standards and assessment
of user oriented perspective
See chapter 4 √-(√)
√ - fulfilled
(√) – partially fulfilled, to be finalised until validation
According to the study protocol a number of functionalities have
been described for the
technical development of the MOBOT device. Depending on the
progress of the
construction of the device and the development of additional
functionalities, appropriate
targets for validation of the prototypes have to be identified
in tight cooperation with
technical partners. Based on study goals as detailed in the
study proposal we list the
following objectives as potential targets for the first
evaluation study. This list may have
to be modified by technical partners.
Table 4-3 Specific functions of the device to be evaluated in
first user evaluation
study as described in the study proposal
Objective,
functiona-
lities
Background Proposed assessment
User
localisation
and activity
detection
3D coordinates of
user
Evaluation will be performed using the
intersection-over-union (IOU) performance metric
based on the November data set, not the data
recorded during evaluation: saying that we provide
a 3D cube A that presumably contains the person,
B (understood as a volume), the IOU metric= |A
and B|/|A or B| measures the extent to which
volumes A and B overlap. If IOU > .7, A and B
overlap substantially, if IOU >.5, they somehow
overlap, but not too much. Having IOU around .8
or .9 is understood as perfect localization.
Basic pose
estimation (state in
which the user is
with respect to the
rollator: distant,
close, in contact)
Detection accuracy, quantified in terms of
precision-recall curves will indicate the trade off
between false alarms and recall; the goal is in
general to have the minimum number of false
alarms at a given level of recall. A scalar metric
that can summarize the performance of a detector
is the Average Precision (AP) which averages the
detector's precision as recall varies.
Recognition of user
gestures and
Recognition performance (success rate, false
-
FP7 - MOBOT – 600796 33
recognizing and
interpreting user
voice commands
detection) will be evaluated off-line.
Detection
of environ-
ment
Obstacle detection
possible if obstacle
is known/static
(static map)
Evaluation metrics include: (a) measurement of a-
posteriori distance from detected obstacles, and (b)
contour matching against known obstacles.
SLAM: off-line evaluation of the similarity of the
created map against the known map.
Localizatio
n of
rollator
itself
Basic localisation
on a static known
map
Evaluation can be performed using known
markers in the map (e.g. a door, a corridor,
known markers on the floor etc.), and measuring
localisation precision in space, especially in loop
closure situations.
Approach
the user
from a
distance
Autonomous
navigation will be
prototype. User
position will be
known. Navigation
will provide
planning, tracking
and approach using
static map.
Approach metrics will consist of evaluating the
difference between the final robot pose with
respect to the user position, as well as the velocity
profile during the near approach. A third metric
can be the time-to-docking, i.e. from first call to
actual user docking.
Assistance
to the user
Transfer/Sit-to-
stand: brakes to
block rollator, fixed
trajectory
Evaluation should assess whether this approach
really improves the functionality (assist sit to
stand) compared to no-actuated rollator handles
and which of the realized versions is best. Precise
estimations of joint torques will not be possible for
evaluation unless we decide to use the Qualisys
tracking system again, so that we can compute
inverse dynamics. Thus, other measures for
evaluation would need to be considered like
subjective measures derived from questionnaires
or objective ones like for example heart rate.
Following the
user`s demand:
Support is passive
but facilitative
Manoeuvrability measured based on interaction
forces, jerk and smoothness of trajectories
Avoiding obstacles Number of collisions with positive
obstacles,
distance to positive/negative obstacles,
manoeuvrability assessed by measure based on
jerk
Assist the user
localization
“Cognitive localization” is trivially performed
using a point-in-polygon test. User feedback
should be used to investigate the performance of
this feature.
Guide/navigation Number of arrivals at desired location, time
to
-
FP7 - MOBOT – 600796 34
desired location compared to no assisted version,
interaction forces as measure of agreement of user
input and autonomous guidance
4.4 Ethical approval
For series of experimental testings which successfully took
place in November 2013 at
Bethanien, a positive ethical vote was given by the ethical
board of the University of
Heidelberg (Ethikkommission I der medizinischen Fakultät der
Universität Heidelberg).
For the validation studies we plan to submit an amendment which
may cover the
projected future validation studies. Based on the comprehensive
discussion with the
ethical board, an amendment may not suffice, as the validation
of the MOBOT devices
may be qualified as MPG-Studies (medical product studies), which
request a much
stricter formal proceeding, comparable to pharmacological
studies. The application for
the ethical vote is planned to be submitted in January 2014 to
allow a timely start of
validation and will cover the projected 2 series of validation
studies (see comment
below). So far no ethical approval for the Greek clinical
partners (Diaplasis) is
available.
4.5 Time table and development of validation strategy
According to the milestones as documented in the project‟s DOW,
3 separate validation
studies (validation I/IIrollator and III/nurse-type) are
projected during the MOBOT
project. Validation I was supposed to start at month 16 (May
2014). In cooperation with
technical partners (lead ICCS) clinical partners (lead
Bethanien) will develop an
assessment strategy for this first validation. Responsibilities
were decided at the Munich
project meeting (11.-12.12.2013)
Since the state of construction of the first version of the
device and associated
functionalities is slightly delayed, the Consortium decided to
postpone the start to
month 18 (July 2014). Depending on the vote of the ethical
board, and the development
of the study design and technical functionalities, the start of
validation may lead to
further delay. The projected duration of validation also depends
on the study design and
the projected sample calculation with consequences on the test
burden for participants,
the number of test sessions and total number of participants to
be recruited.
A decision by technical partners at the last project meeting
(Munich 11.-12th of
December 2013) to include only the 20-80 percentile (based on
anthropometric
parameters) of the potential target group of rehab patients
substantially reduces the
recruitment base of the validation studies, which were already
limited by predefined
inclusion criteria (see case definition) and medical reasons in
this multi-morbid, frail,
acutely impaired sample. As the number of patients to be
included in the validation
study represents the bottle neck of the duration of validation
studies, the decision will
have substantial consequences for the time table of validation.
Based on the long-term
experience of the clinical WP-leader (Prof. Hauer) the projected
duration as documented
in the DOW of 2 months may not suffice to recruit an adequate
number of study
participants in the target population of the MOBOT project
(Geriatric Rehabilitation).
Within upcoming project meetings a solution of such problems
will be found according
to previous agreements of technical and clinical partners.
-
FP7 - MOBOT – 600796 35
The definition of prototypes to be validated still has to be
finalized. According to the
current state of information, the first validation will cover a
first version of the MOBOT
device with focus on support of walking performances and
sit-to-stand transfer support.
At a later time within the project further validations will
cover an advanced prototype,
which may also include additional technical functionalities as
developed during 2014
and 2015 by technical partners. A positive ethical vote for the
Greek clinical setting will
be mandatory for future testing at their premises.
4.6 Subject recruitment
Clinical centres, at which the validation will take place, will
be responsible for
recruitment. The recruitment will include geriatric patients
during rehabilitation as the
target sample for the development according to predefined
inclusion criteria (see
comments above for general classification of study
participants/user definition),
additional ethical (written informed consent) and medical
inclusion criteria (e.g. severe
medical condition excluding participation). Recruitment of
subjects will start when
rollator prototypes and study protocols for validation will be
finalised and positive
ethical votes will be given.
-
FP7 - MOBOT – 600796 36
5. SUMMARY OF REPORT
This deliverable D5.2 presents an update of the comprehensive
D5.1 report and gives an
additional outlook on preparation of user evaluation studies as
described in the DoW for
WP5.
We present an update of user groups with only minor amendments
for the final
inclusion criteria in chapter 2 including motor, cognitive and
anthropometric
characteristics and the final use cases as basis for the
development of specifically
adjusted performance measurements. Assessment strategies to
determine user needs and
limitations of existing mobility devices are given in chapter 3.
Chapter 4 covers a
preliminary conception of performance metrics and types of
evaluation for the
upcoming validation of the first MOBOT prototype. The specific
functionalities of this
prototype will determine a specifically tailored evaluation
strategy apart from a
standardised clinical evaluation. The results of a systematic
review on previous clinical
validation studies or comparable devices as presented
