1

1. HYPOTHESIS and OVERVIEW

We hypothesize that an environmentally-aware, network connected, robotic assistive device will be able

to prevent a number of problem events (particularly falls) and reduce mean response times to several

classes of key health problems by way of risk assessment and unobtrusive monitoring. This will

significantly reduce costs and

lighten the burden on existing

health care resources. Figure 1

depicts the overall proposed

system which will incorporate a

robotic smart walker, an in-

house base station, and a remote

analytics server. The walker will

provide environment-aware

mobility assistance as well as

recording sensor data about the

user and home activities. This

data will be passed wirelessly to

the base station, which will

compress and transmit data over

a wired Internet connection for

analysis, visualization, and secure

storage at the remote server, as well as storing navigational knowledge for later use, by the walker.

In order to ensure maximum accuracy in fall detection, health condition prognosis, and

environment awareness, the sensors involved will include at least visible and infrared light imaging,

omnidirectional and pan/tilt directional audio, accelerometers, and pulse/heartbeat sensors. Research

performed will include cutting-edge feature preprocessing and adaptation, navigation in dynamic

cluttered environments, multimodal signal compression, and statistical models of day-to-day health.

These will support generation of symptomatic alerts of both long-term problems (tremor progression,

changes in respiration or sweating, altered voiding regimen) and immediate problems (falls, distress).

Patient health histories will be stored in compliance with emerging medical standards.

2. OBJECTIVES FOR PERIOD OF PROPOSED WORK

Listed here are the objectives for the proposed project, with additional description in Sections 3 and 4:

Research, design, and test a novel unobtrusive smart walker and a base station

Research, implement, and test navigation algorithms for the smart walker based upon

unobtrusiveness criteria

Research, implement, and test novel algorithms for smart walker environmental awareness

Research, implement, and test biosignal modeling and person-to-person adaptation

Research, implement, and test novel algorithms for biosignal processing

Research, implement, and test novel post-fall detection algorithms based upon audiovisual event

detection and visual pose estimation

Research, implement, and test novel medicine dosing event detection algorithms based upon

audiovisual object recognition and visual pose estimation

Research, implement, and test novel boosted features based upon SLAM and visual context

Research, implement, and test novel long- /short-term biosignal alerts based upon sensor history

Develop an implementable architecture that lays out the major alternatives for partitioning of

work among processing nodes and associated information exchange among nodes

Figure1. Proposed system (smart walker, base station, server)

2

Develop implementable criteria for processing of sensed data for local handling versus

forwarding downstream to meet effectiveness and efficiency objectives of the overall architecture

Develop implementable approach to metadata characterization of patient data compatible with

existing health information technology standards

3. INTELLECTUAL MERIT

Research performed will include cutting-edge feature preprocessing and adaptation, navigation in

dynamic cluttered environments, activity recognition, fall risk assessment based upon pose, audiovisual

fall detection, statistical models of day-to-day health, symptomatic alerts of long-term (medicine

noncompliance) and immediate problems (falls), and health data management. Results in all areas are

expected to build upon the state of the art and improve the fields of robotics and health monitoring in the

context of service-oriented architectures for ambient assisted living.

4. LITERATURE REVIEW AND RESEARCH CHALLENGES

As this proposal is a collaborative proposal, there are four main components (teams) of the proposal:

Geriatric Need, Unobtrusive Smart Walker Design, Event Understanding and Alert Generation, and

Service Oriented Architecture for Health IT.

4.1 Geriatric Need (Rochester General Hospital System – RGHS)

The population of the United States is an aging population: the US Census Bureau estimates that the

population of people age 65 years or older will increase from 39.6 million (12.9%) in 2009 to 72.1 million

(19.3%) by 2030 [1]. With this shift in demographics comes an increased demand for health care resources,

with the availability of these resources recognized as a key indicator of well-being [2]. Clinical studies

shows that 33% of U.S. adults aged 65 of greater suffer falls in a given year [3, 4], with almost one-third

requiring medical attention and further falls occurring for close to 60% within a year. Direct and indirect

costs of falls are on the order of billions annually (e.g., $19 billion in 2000 [5]), some or all of which can be

prevented through risk factor minimization and immediate responder notification [6, 7, 8]. Likewise,

nonadherence to prescribed medical regimens is estimated to be as high as 38% [9], in spite of existing

methods of determining capacity to manage medications [10], resulting in reduced effectiveness and

potential for relapse. Our proposal is aimed at minimizing high-cost independent living health risks (such

as falls and medication nonadherence) for the elderly through robotics.

As it stands, a large proportion of the elderly population makes use of assistive technology (26% of

the chronically disabled without personal care, 58% with some personal care [11]). Mobility devices are

by far the most common, surpassing all anatomical, hearing, and vision devices. Therefore, we propose

the research and development of a robotic system that duplicates the benefits of a mobility device

(exercise, stability) while unobtrusively identifying risks and maximizing information for medical staff.

This research will advance the state of the art across several fields while meeting a key societal need.

4.2 Unobtrusive Smart Walker Design (Electrical and Microelectronic Engineering, RIT)

4.2.1 Assistive Robotics

Assistive devices have remained crucial throughout the advancement of human kind. Canes, eye glasses,

hearing aid, and work tools have been instrumental in technological and intellectual development.

Recently, intelligent robotic systems are being used for helping the elderly, disabled, blind, and injured.

Research has been done on aiding the blind through other senses than vision via a mobility aid called

a Smart Blind Cane that uses a smart phone based collision warning system (built around sensors like

ultrasonic range and temperature) and text -to- speech to notify the user of their surroundings [37]. Other

related devices include a thermal digital device that lets visually impaired individuals visualize works of

3

art by touch, glasses that measure time between blinks, and a device that uses sensors to smell [39]. For

elderly visually impaired people that have difficulty using guide dogs or long canes, walkers or walking

frame with wheels are designed to help them navigate; one such device is called PAM-AID [38].

In general, there have been many devices and robots researched for walking assistance and helping

the elderly in nursing homes and hospitals, with wheelchairs and walkers the most commonly used.

Researchers have tried to augment these walking aids with robots and sensors to make them smart and

autonomous. Exoskeleton devices [42] and robotic suits [55] [56] [57] have also been used as mobility and

rehabilitation aids. In one of these experiments, an anthropomorphic approach was used by mounting a

robotic Manus® arm [41] on a wheelchair and letting the user control the arm through a user friendly

interface [40]. Along with walking, assistance is also needed in standing and sitting. Devices have been

prototyped that help with all three operations to prevent injuries like fractures and dislocation of bones

while trying to stand or sit [43] [45]. Intelligent wheelchairs with obstacle avoidance and independent

control can be used to carry the elderly and/or disabled or move independently in a smart environment

as a sensorial extension, and have been implemented on at least one occasion [44]. Another such

implementation is a call-to-service walking helper robot that comes and provides assistance by locating

the user in a smart environment [35]. But, in many cases, the user desires assistance that respects their

freedom and control within their ability, yet also within an envelope of safety [49, 50]. For this reason,

semi-autonomous mobility aids have been designed that take into account intent [46, 47] and also

interactive [48]. Figure 2 shows some of these devices.

Figure 2. Examples of assistive walking and posture devices, left to right [35, 26, 49, 50]

Further examples explicitly requiring patient effort include exoskeletons supporting leg alignment

[42], full robotic suits [54, 55], robotic arms and exoskeletons for rehabilitation of upper limbs with weak

muscles [58] and patients post-stroke [52], orthotic ankle devices [53] and pelvic assistance [59] for gait

disorders, and cable-driven machines for rehabilitation [56]. Also developed is an exoskeleton for eating

that simulates upper limb motor patterns in a healthy person and can be controlled by wrist action [51].

Many such-assisted elderly/disabled live in nursing homes or in assisted living facilities with

constant assistance and monitoring. Moreover, caregivers need help in managing assistive devices. For

this reason, there has been work in robotics to push and steer wheelchairs on a fixed path [60] or come

when called [35], and dual-arm robots to help dress patients [64]. Models have been suggested to make

homes and/or hospitals context-aware environments sensitive and responsive to the presence of humans

using embedded systems, sensor networks, and robots [62]; based on feedback from health care

organizations, effort is being made on improving the behavior, appearance, and functionalities of such

systems [59]. Other related efforts include mapping fleets of helper robots, camera-based patient tracking

[61] and audiovisual human-robot interaction [65]. Studies on the social and psychological effects of

robotic-assisted activities on elderly people [66, 67] show improved moods and less depression. Research

on socially-acceptable robots [68] and graphical user interfaces for the elderly are underway [69]. This

extends to wearable life sign sensor vests for prognosis [73], shoes with acceleration sensors to detect fall

4

and gait disabilities [74], and floors with sensors to determine positioning and falls [63]. Children with

disabilities and suffering from autism or cerebral palsy have also benefited from robots and robotic toys

that help them learn and interact [70, 71, 72].

Research Challenge

There have been several efforts in developing smart assistive mobility devices for the elderly. However,

these efforts have been limited to one or two aspects of care and are obtrusive. In aiding the elderly,

special care needs to be taken in designing socially-acceptable devices which are minimally obtrusive.

Furthermore, most prototypes have not been networked, limiting the use and reporting. In an effort to

assist elderly without disrupting their daily lives, the assistive device needs to be designed so as to be

unobtrusive, semi-autonomous, and networked. Thus, in this proposed work we plan to design a walker

which is functionally similar to a manual walker but also networked to collect, monitor, and report life

signs.

4.2.2 Mobile Robot Navigation

In mobile robotics, navigation is one of the most fundamental and challenging tasks in robot autonomy;

hence, a lot of research has been done to address challenges such as positioning and orientation,

analyzing surroundings (mapping), conveying this information to the robot, and choosing a subsequent

appropriate action or path to take. Navigation is twofold: local and global. Local navigation, often called

reactive control, is how the robot learns or plans local paths using the sensory inputs without prior

complete knowledge of the environment. Global navigation, also called deliberate control, is where the

robot learns or plans the global paths based upon complete knowledge about the environment [26].

Additionally, challenges in navigating indoors and outdoors are addressed differently. An indoor

environment can be made smart by using a network of sensors or by placing identifying landmarks.

However, in an outdoor environment, all the sensors generally have to be on the robot and hence the

robot has to be independent of the environment to navigate.

Sensors generally used include RFID (Radio Frequency Identification) tags, sonar or ultra-sonic range

sensors, cameras, magnetic sensors and laser or infrared scanners. Vision-based navigation is the most

common and many research efforts use image processing with range sensing (odometry). For example, in

an unknown indoor environment, local navigation is achieved either through monocular vision-based

simultaneous localization and mapping (SLAM) [16] [17] by mounting a camera on the robot facing

upwards and extracting features like lamps, doors and corners and laser scans for odometry [12] [13]; or

by using stereo vision and extracting 3D features from static and dynamic obstacles [14] [15]. For

navigation in unknown environments without vision, networks of RFID tags or passive RF sensors [18]

[19], magnetic landmarks [22] and ultrasonic sensors [23] have been used. A sensor network can also be

used for SLAM by using triangulation localization methods [20] [21], including coupling with limited

sensing [24]. SLAM variants include complete coverage navigation (CCN) where the robot passes

through all points in a changing environment [25], hierarchical Q- learning for coupled global and local

navigation for very accurate tasks [26]; predictive SLAM (P-SLAM) for exploration [27]; uncertainty-

based algorithms [28]; algorithm for probabilistic localization using indoor GPS and laser scanners [29],

topological model [31], or occupancy maps [30], and many more. Further research includes algorithms for

multiple robots such as reconnaissance and surveillance for a large urban environment [33] and adaptive

road mapping [32].

Research Challenge

The main challenge in navigation of the Smart Walker is to move around with precision with passive

motion where the patients holding/pushing the walker. In addition, environmental awareness is very

critical for the walker to make decision on active versus passive motion. Some of the applications of

5

mobile robot navigation are call-to-service mobile robots for elderly patients (Walbot), that use Zigbee

sensor networks for localization and tracking the caller [35], service robots for transportation in hospitals

[34] and an autonomous vehicle for indoor surveillance [36].

4.2.3 Biosignal Processing and Classification

Analyzing and understanding human biosignals has been an important research area with practical

applications in everyday life. Brain computer interfacing (BCI) is a research area that studies controlling

devices by processing and understanding the brain via Electroencephalogram (EEG) signals [75 – 78].

Similarly, assistive robots [79] are being developed using eye [80] or muscle signals [81 – 87] to provide

control inputs. The efficiency for all of these applications depends on being able to process and classify

biosignals. Generally speaking, the most frequently used biosignals are Electroencephalogram (EEG),

Electroocculogram (EOG), and Electromyogram (EMG). Recorded EEG/EMG/EOG signals can be used to

control inputs for real-time systems such as BCI-based assistive devices or detection systems for brain

abnormalities [76]. Therefore, most of the applications that utilize biosignals require careful consideration

in terms of collection, processing and information extraction [79, 80, 85 – 95]. Biosignal applications

include human computer interfaces [75, 76, 96], emotional state classification [88], mobility device control

[97], prosthetic arm control [81, 82, 86], and detecting driver drowsiness [92].

In preprocessing, filtering is commonplace, coupled with feature extraction methodologies including

Principle Component Analysis (PCA) [77, 78, 93], Analysis of Variance (ANOVA) [77], Linear

Discriminant Analysis (LDA) [86, 93], Independent Component Analysis (IDA) [84, 93], Waveform

Length Transforms [85, 89, 90, 92], and clustering [87, 94, 95]. Other techniques appearing are Mean

Absolute Value, Root Mean Square, Variance, Standard Deviation, Zero Crossing, Slope Sign Change,

and Singular Value Decomposition (SVD) [84, 86]. Common machine learning methods are Support

Vector Machines (SVM) [77, 79, 83, 84, 86, 90, 93], Radial Basis Function Networks (RBFN) [79, 87, 94, 95],

Artificial Neural Networks (ANN) [81, 82, 85, 89, 90, 91, 97], C4.5 [78], Fuzzy Logic [81, 82, 87, 92, 94, 95],

Genetic Algorithms (GA) [81, 82], Particle Swarm Optimization (PSO) [87, 94, 95], K-Nearest Neighbor

Classifier [88], Random Forest Classifier [88], Decision Trees [89], and the Gaussian Classifier [96].

Research Challenge

Based on the literature, success rates are nearing the 80-95% range [87, 94 – 96] for EEG, 90-97% for EMG

[83], and above 95% for hybrids [92, 96]. While not directly applicable to this task, successes by PI Ferat

Sahin and his students inspire the use of biosignals for vital sign tracking [87, 94, 95].

4.3 Event Understanding and Alert Generation (Sustainability Institute at RIT)

4.3.1 Fall Detection Algorithms

Visual fall detection is a well-established field of study, yet maintains poor detection in clutter and

insufficient false alarm rates [98]. Current state-of-the art is based around estimating human pose, using

the positioning of body parts to tell if a fall is occurring or has occurred. While the data used may vary

from traditional monocular, visible-light imaging (including infrared [99] and 360° imaging [100]), the

procedure remains largely the same: identify features of human body parts, fit a parameterized model of

human structure to these feature positions, and perform classification based upon the resulting model

parameters, often making use of context cues to weigh the final decision [101 – 103].

Of note is the division between research that identifies fall events as they occur and research that

identifies humans in a post-fall position. While the former [104, 105] suggest a shorter response time, low

false alarm rate requirements have pushed a significant portion of the research community towards

detecting immobilization following a fall [106 – 108], as it is possible to build effective confidence metrics

based upon long-term observation, rather than relying upon a limited window of activity.

6

Research Challenge

For the purposes of this project, we will focus on post-fall detection, both for reduction of false alarm

rates and to allow for investigation of suspected falls that may be outside of the robot’s field of vision.

This last point emphasizes the utility of the proposed system as well as one of our directions of

research: active, multimodal investigation. Much of the current state-of-the-art relies on static, large

baseline deployment for 3D pose estimation, and only makes use of visible light. While there is a push

for the inclusion of audio in sensor networks [146, 147], it has been used only to segment event data [148]

and requires static sensor placement [149]. Instead, we are interested in mobile investigation of potential

fall events, as we can obtain a variety of viewpoints to confirm/reject suspected fall events that occur

within or outside of the robot’s field of vision. Investigation is via floor vibration through accelerometer

readings, coupled with an omnidirectional microphone and stereo video (multiple wavelengths, visible

and otherwise), identifying suspicious occurrences whereupon all data available will be studied to obtain

a more complete picture of the room context, human pose (prone, standing, etc.), and signs of panic. The

research challenges posed, therefore, are: identification of suspicious events (visible and out-of-view),

visual/auditory/vibration/biosignal context, and multi-sensor fusion in a supervised fall/no fall classifier.

In total, work on fall detection algorithm development for our system is expected to take 1.5 to 2

years, and serve as the research cornerstone for a doctoral candidate in computer vision and machine

learning at the Rochester Institute of Technology, with the practical aspects of said research thoroughly

explored by the final year of human subject experimentation.

4.3.2 Activity Recognition Algorithms

Noncompliance with prescription medication is a serious problem for independent-living seniors, and is

an indicator of declining ability to take care of oneself. In order to determine compliance with medication,

we propose the use of activity recognition through video. By identifying when patients do not take

medication on schedule, we can generate noncompliance alerts for medical staff.

Eating/drinking/dosing events, not often the focus of human activity recognition [109], still benefit

from general video-based recognition improvements [110]. For instance, high-res., real-world training

data [111] has allowed data-driven semantic clustering and labeling of repeated activities [112], replacing

traditional parametric algorithms. We will build upon this, emphasizing context and sensor fusion.

Top activity recognition techniques today make use of spatio-temporal context to distinguish

between activities with similar subcomponents [113, 114] and discount activities that are unlikely in a

location or in conjunction with particular objects [115, 116]. By training models of context from a variety

of recorded video, it is possible to automatically learn behavior patterns across days and subjects [117].

Research Challenge

In the proposed independent living domain, we intend to focus on the isolation of prescription dosing

from normal eating and drinking by way of novel context models, particularly the repetition of normal

eating and the spatial/object context of drinking.

In addition to traditional visual feature-based context discernment of dosing, we intend to make use

of additional modalities of sensor data, particularly audio. Intuitively, bottles of pills make sounds that

are distinct from foodstuffs and beverages, allowing us to identify them [150]. The proposed work will

expand upon current techniques in audio context modeling, emphasizing the variety in whole-house

independent living, the presence of other humans and animals, and robot positioning during recording.

Dosing detection algorithms are expected to take 1 to 1.5 years of research and development, serving

as a key focus for a doctoral candidate in audiovisual signal processing at the Rochester Institute of

Technology which, coupled with research work in biosignal anomaly detection and bolstered by

extensive human subject experimentation in year four, will serve as the foundation for their thesis.

7

4.3.3 Vision Preprocessing

In order to maximize the effectiveness of activity recognition and fall detection algorithms, it is necessary

to perform preprocessing of input video data to identify the location of the robot relative to the rest of the

world and to point out recognizable objects. Each of these tasks is well-studied in the literature, and

software solutions could be quickly bootstrapped from existing open-source components.

Beyond odometry (covered in 4.2.2), we are interested in pure vision-based SLAM [118, 119].

Solutions run the gamut of waypoint identifiers and reasoners [120, 121], but the most popular approach

is FastSLAM [122 – 125], which finds the posterior likelihood of an estimate by factorization into

conditional likelihoods for each landmark and path continuity. In particular, earlier research on SLAM in

homes [126 – 128] has led to FastSLAM with added tracking of moving objects for people detection [129].

We will build upon this, adding meta-reasoning that integrates feature extraction for object detection.

Recent advances in feature detection for learnable tasks (such as the detection of particular, pre-

known objects) have hinged upon the immense success and popularity of Adaptive Boosting [130] in

constructing highly-descriptive, complex image features from many simple features [131 – 137].

Research Challenge

Our work will make use of boosted features learned experimentally, pushed further: since we will be

performing SLAM, we can use spatial maps to associate features from multiple frames/viewpoints with

each object, feeding back to improve feature selection across viewings. Research will focus on this

process, especially learning rate and exact feedback mechanisms in the presence of real-world data.

This combined system will be assembled and trained in 6 months to 1 year for use in fall detection

and activity recognition. Doctoral candidates working on those components would collaborate on this

effort, with the publication of results serving as an introduction to the publication and review process.

4.3.4 Vital Sign Anomaly Detection Algorithms

Standard medical practice holds with four key vital signs: body temperature, blood pressure, pulse rate,

and respiratory rate; these four signals are indicative of health and reveal symptoms of health problems.

As the proposed system is intended to inform clinical staff, vital signs will be recorded (when available)

and logged. Temperature, for example, can be captured via infrared thermography [138 – 141], with high

values indicating fever [142 – 145]. Beyond recording of biosignal, it is possible to detect anomalies that

will alert attending doctors to any sudden or long-term extremes or changes in vital signs.

Research Challenge

The proposed system will include the detection of short-term (sudden) changes in vital signs as well as

long-term (gradual) changes indicative of declining health.

For sudden changes, we will build algorithms around signal anomaly detection [151], particularly

feed-forward neural networks [152]. Popular in network intrusion detection [153 – 157], neural networks

have yet to be extensively studied for use in medical anomalies, despite health models serving to

motivate network research [158, 159]. Generally, neural network approaches are based upon the

deviation of inputs from past observations and training data, such as the residuals of a replicator neural

network [160, 161].

We propose the study of network topologies for vital sign signals (number and size of neuron layers,

transfer function design, connectivity), as well as the use of a single network on all modalities

simultaneously. Of particular note is that success in this domain would also allow for improvements in

automatic patient alerts other elderly care settings, including hospital, nursing home, and hospice care.

For longer-term changes, such as gradual reduction in pulse or breathing rate, we intend to create

algorithms that model history to predict future readings. This research will span avenues of study,

8

particularly sequential Monte Carlo simulation [162 – 167]. Like other alerts, we are interested in a single

model across signal types; in this case, most novel work will be in kernel design and selection, focusing

on the automatic learning of kernel combinations [168, 169] and patient-by-patient adaptation thereof.

Research and development in biosignal anomaly detection is expected to take place for six to nine

months, as a joint effort of the two doctoral candidates earmarked for fall detection and activity

recognition (as mentioned earlier). This work is intended to serve as an expansion of research interests,

with publication increasing awareness of studies in sensor reliability in the presence of noisy data.

4.4 Service Oriented Architecture for Health IT (RTSync Inc. of Arizona)

Our overall proposal can be viewed as providing Ambient Assisted Living (AAL), active telehealth for

elderly based upon point-of-care health management within a well-defined service oriented architecture.

4.4.1 Architectural Framework

AAL encompasses technical systems to support elderly people to allow an independent and safe lifestyle

for as long as possible [170]. These systems make use of inhomogeneous technologies such as wireless

networks, machine learning algorithms and sensory devices. Kurschl et al. [171] advocate Model Driven

Architectures [172] that support both expert-based top-down representation of the envisioned system as

well as support developers in bottom-up synthesis [173]. Neßelrath et al. [174] present a related vision

based on open standards and architectures in a smart home.

Research Challenge

Although the Model-Driven Architecture approach advocated by Kurschl et al. [171] is a good starting

point, it does not provide tools for simulated model representation of architecture alternatives with the

ability to test prior to implementation. We will employ proprietary RTSync DEVS-based modeling and

simulation environment [206] to develop, test, and automatically generate our AAL code.

Network Layer: The Internet of Things expands today's Internet to connect smart objects capable of

identifying, locating, sensing, and communicating in support of AAL [175]. Martin et al [176] propose the

combination of ZigBee technology with a service oriented architecture. Pal et al. [177] advocate using

compressed sensing [178] for data processing in low-power networks in the presence of noise.

Services Layer: Service-Oriented Architecture (SOA) is a concept for integration/implementation of

networked systems [179] which separates functions into independent, reusable services that flexibly

interoperate together [180 – 183]. SOA is starting to infiltrate into health care and clinical decision

support [184-186]. Ongenae et al. show how a combination of expert systems and SOA is able to support

physicians in continuing patient care [187].

Data and Modeling Layer

Personal Models: Modeling of the assisted person’s particular characteristics is essential for monitoring

and adaptive intervention [188]. One challenge of AAL is recognition of the assisted person's routine

[189]. Hossain et al. [190] develop an approach to modeling the assisted person's current activity for

detection of changes of capabilities and preferences, including user’s preferred time, location, and level of

effort. Based on World Health Organization’s framework for an individual person or population [191,

192], patient capabilities focus on impairments of body functions, body structures and limitations on

certain activities. Frey et al. model medication calendar tasks for patients with cognitive disabilities [193].

9

4.4.2 Electronic Health Records

Beyond activity models, individualization has been sparse [194], a situation likely to change with

widespread use of Electronic Health Records (EHR) and the push for “meaningful use” of Health

Information Technology [195]. Currently, the most popular HIT standard for EHRs is HL7 v2 [196].

Unfortunately, HL7 v2 messages have no explicit information model and poor definitions for many

fields. To comply with meaningful use criteria, applications are expected to implement all interactions of

the role they conform to. Standards expertise is needed to keep abreast of the moving target for

meaningful use and to enable AAL to comply with emerging EHR standards.

Research Challenge

Given evolution in federal standards for medical records, our data archiving must meet current standards

while still retaining flexibility and fidelity for future analytics. We will use the data engineering approach

of [203, 204, 205] to provide metadata characterization and representation of data for archiving that meet

emerging HIT standards. This ontology-based approach will also support investigation of data handling

principles and implementation in networked environments mediated by our simulation environment.

4.4.3 Multi-sensor information fusion and data engineering

Multi-sensor data fusion, including Kalman filtering, Bayes estimation, Dempster-Shafer evidential

reasoning, fuzzy logic and neural networks, is often used for modeling in robot control [197]. The most

common medical application is diagnostics, albeit with little fusion at levels higher than raw signal, such

as features or classifiers [198]. Use in AAL is in its infancy and offers a chance to apply mature concepts

from other domains [199]. Blach [200] extended military data fusion [201] to account for consumer needs

for processing. Kokar introduced web ontology methodology to support situation awareness using

fusion [202]. Lee [203] applied the information exchange framework based on the System Entity

Structure (SES) ontology [204, 205, 207] to extract / fuse data for commanders with different requirements.

Research Challenge

Challenges arise in the proposed system due to the nature of the walker-generated data: its composition

(motion/vision/audio/bio signal), dynamic, and intermittent nature. Two principals will be investigated:

Early characterization of data quality/utility. For example, data that will not have value to

downstream analytic processing should be identified at the base station and discarded

Early processing of data to generate alerts. For example, using individual patient models, the base

station may be able to off-load work from the analytic server, saving communication bandwidth.

Detailed investigation of these principles in the proposed AAL system will feed back to the architectural

decomposition of work and assignment of processing/communication roles to system nodes.

4.4.4 Targeted information flow with selection among data transmission/distribution technologies

The proposed AAL architecture will focus on the following components:

Data products: walker-generated (motion/vision/audio/bio) data, walker-generated alerts (shouts),

data retained for analysis, base station-generated alerts (phone call, text, email), server-generated

alerts (analytic content displayed on console screens), archived data/metadata, patient health records

Consumers of data products: assisted person, emergency responders, clinical/medical staff

Platforms or processing nodes that must be considered in the architecture: walker, base station, analytic

server, fixed computer workstations, mobile hand held devices (PDAs, smart phones, etc.)

Data transmission media include wireless, internet, phone lines, social media (e.g., Twitter)

A new design will be developed which delivers data products to the consumers over media to meet the

challenges spelled out earlier. Our design will follow the data engineering SES ontology framework via

10

our modeling and simulation environment. We will exploit the ability of SES to represent complex data

rigorously while affording flexibility to transform the structure according to a set of operations; for

example, pruning structure that is not relevant to interests of a consumer based on a specified “pragmatic

frame”. We will also use the implementation of the SES as an XML schema and its generated data as

XML documents to implement on XML-compliant data exchange/processing platforms.

The heart of the design will be a master SES that characterizes data of interest to all consumers and

operations that can be used to tailor the data forwarded to each consumer thereby reducing bandwidth

and downstream processing. Thus, the walker is the primary producer while the base station and the

server are secondary producers that turn processed data into information. The information requirements

of such producers and consumers will be expressed as pragmatic frames able to appropriately tailor the

master SES to meet such requirements. Based on this model, we will investigate information technologies,

to build upon the SOA Web Server-Client, Data-distribution Service/Publish/Subscribe technologies in

the literature for implementing AAL systems.

4.4.5 Acceptability categorization of sensed data for downstream transmission

As a primary consumer of walker-generated data, the server should receive only data that support its

diagnostics. Thus, the pragmatic frame for the server should distinguish actionable and non-actionable

data, transmitting only the former. Criteria for non-actionable data include absence of people, no signal

change, and signal corruption. We will study the fusion of data to implement these criteria. To detect

absence of people, we will use portions of infrared images containing "people" heat signatures so that

thresholds can be developed for human occupancy. For constancy of signals, we will adapt value

quantum methods [208], generalized to accommodate the multi-source walker data and account for noise

using rapid change detection [209]. To handle signal corruption, we will develop models of good and

bad data, parameterized from design and improved experimentally. Having discarded non-actionable

data, the base station will take the data, package it in terms of the standardized XML representation

derived from the pragmatic frame formulation, compress it, and transmit it through an Internet

connection to the server. We will investigate compression methods such as [177] and their interaction

with quantization.

4.4.6 Data-driven modeling, model-based prioritized generation of alerts.

Having established a baseline implementation, we will investigate alternative assignments of processing.

One possibility is the detection of a subset of short-term biosignal anomalies at the base station and

generation of alerts to emergency personnel; such anomaly detection will be based on individualized

models. We will investigate activity capability and preference models [190] for suitability to support this

change. We will also develop new methods for deriving normal, individual biosignal limits. Detection of

values outside these ranges will generate the mentioned alerts intended for emergency responders.

5. STATEMENT of WORK and TIMELINE

Phase I (year 1) of a Four Year Program

1. Purchase and set up analytics server hardware, write data storage software and test with junk data.

2. Graduate students study state-of-the-art in Simultaneous Localization and Mapping (SLAM) and

visual feature detection and state-of-the-art in assistive robotics.

3. Design “smart walker” robotic system and base station, including programmability, power use and

storage, sensor selection and placement, mobility, and communication.

4. Design data representation for storage and transmission.

5. Purchase hardware components for “smart walker” and base station.

6. Construct initial “smart walker” and base station hardware prototypes.

11

7. Implement existing SLAM and feature detection algorithms for server and test with junk data.

8. Start writing SLAM / feature detection paper for scholarly publication.

9. Develop detailed Phase II plan

Phase II (year 2) of a Four Year Program

1. Gather sample sensor data using sensors detached from smart walker.

2. Design, implement, and test novel SLAM and feature detection algorithms, experiment with sample

sensor data, and correct algorithms accordingly.

3. Finish writing SLAM / feature detection paper.

4. Implement existing data triage and compression algorithms for base station.

5. Graduate students study state-of-the-art in mobile robot navigation and environment awareness, fall

event detection and pose estimation.

6. Implement existing robot navigation and environment awareness algorithms for “smart walker”, fall

event detection and pose estimation algorithms for server, and test with sample sensor data.

7. Design, implement, and test novel robot navigation, environment awareness, fall event detection and

pose estimation algorithms, experiment with sample sensor data, and correct algorithms accordingly.

8. Write robot navigation paper.

9. Start writing environment awareness, fall event detection / pose estimation (fall detection) papers.

10. Develop detailed Phase III plan.

Phase III (year 3) of a Four Year Program

1. Finish writing environment awareness and fall detection papers.

2. Refine “smart walker” and base station hardware prototypes.

3. Test and refine data compression, transmission, and representation based upon algorithm use.

4. Graduate students study state-of-the-art in short-term biosignal modeling, person-to-person biosignal

adaptation, risk analysis based upon pose and obstacle detection, long-term biosignal modeling and

specific object and activity recognition.

5. Implement existing short-term biosignal and risk analysis algorithms for “smart walker”, long-term

biosignal, object and activity recognition algorithms for server, and test with sample sensor data.

6. Design, implement, and test novel short-/long-term biosignal, risk analysis, object and activity

recognition algorithms, experiment with sample sensor data, and correct algorithms accordingly.

7. Write short-term biosignal (emergency detection) paper.

8. Start writing object and activity recognition (medicine noncompliance) paper.

9. Develop human subject experiment plan, including specific activity script.

10. Run experiment plan with graduate students and refine accordingly.

11. Develop detailed Phase IV plan.

Phase IV (year 4) of a Four Year Program

1. Write long-term biosignal (health degradation) paper.

2. Confirm all software complete and operational on graduate student experimental data.

3. Run experiment plan with PI/co-PIs and refine accordingly.

4. Finalize human subject experiment plan.

5. Run experiment plan with first set of elderly human subjects.

6. Update algorithms based upon elderly subject experimental data.

7. Run experiment plan with second set of elderly human subjects.

8. Analyze results from both sets of elderly subject experimental data.

9. Write human subject experiment paper.

10. Write follow-up journal papers for any or all earlier papers.

12

11. Compile all findings in final project documentation.

Figure 3 presents the timeline and the main task division among the four partners.

Figure 3. The timeline of the research efforts for four years and four partners

6. RESULTS FROM PRIOR NSF SUPPORT

The PI Sahin has previous worked with colleagues at Rochester Institute of Technology through NSF

funding. He was a co-PI on an MRI grant titled “MRI: Acquisition of an Optical Profiler for Surface

Characterization and Dynamic Analysis of MEMS Devices” (ECS#0619676, PIs: Hirschman, Raisanen,

Sahin, Boedo, Fuller, Walter (Senior Personnel); $223,717, 9/13/2006 – 11/29/2009). This award resulted in

the acquisition of a Veeco Wyko NT1100 DMEMS system, with final reporting on 11/29/2009. The system

is used extensively for research and education at RIT, with almost unlimited availability for device

characterization and otherwise-impossible publications [210 – 214]. The PI Sahin was also a Co-PI of an

NSF grant titled, “MRI: Acquisition of a High Resolution X-Ray Diffractometer for Engineering Research

and Education” (MRI # 0521341, PIs: Gupta, Raisanen, Sahin, Hirschman, Kurinec; $253,746, 9/1/05-

9/1/06). The equipment is also used by doctoral students in Microsystems Engineering PhD Program

where PI Sahin serves in the core faculty, resulting in numerous publications [215 – 218].

7. PRELIMINARY WORK DONE

The Multi-Agent Bio-Robotics Laboratory (MABL), led by PI Ferat Sahin, has completed two studies

related to the proposed work: mobile robotics for hospitals [219] and biosignal classification [87, 94, 95].

7.1 Autonomous robot design for automatic loading and unloading of medical carts in hospitals

In this corporate-funded project, an autonomous mobile robot was designed for loading and unloading of

medical carts in hospital washing machines. The robot was designed to work with the hospital system

and washing machines through wireless communication. RFID tags and cameras were used as

13

navigational sensors to precisely locate the robot and the medical carts. In addition, a barcode scanner

was used to determine the medical cart location and type. The robot was successful in the operation in

the test area, and is planned for inclusion with the washing machines Getinge Inc. [220] is manufacturing.

7.2 Real-Time Robot Control with the Designed BCI using EEG signals

Recent publications include a transaction [94] and two conference papers [87, 95] in classification

techniques for biosignals and demonstrating a real-time application of a robot control using biosignal

acquisition tools: ClevMed Bioradio 150 [221] and Emotiv Epoc Headsets [222]. Currently, we are

working on controlling a wheelchair using EEG signals. In this work, we propose to employ our novel

biosignal classification techniques [94] as well as recent hybrid BCI techniques [96]. Based upon the

methods in [87, 94, 95], PSO- and RBFN-based classifiers present the best opportunity for real-time

classification through reduced computational complexity post- training. In order to train and test the real-

time system, a graphical user interface was created in Matlab. This GUI instructs subjects to think “left”

and “right” while visual stimuli by EmoCube [222] is presented. Once data regarding left and right

thinking is collected, it is fed into PSO-RBFN classifier for learning the subjects’ left and right thinking

patterns. A block diagram describing the system is shown in Figure 4. The RBFN outputs during real-

time EEG processing are converted into serial port inputs for a walking robot called Hexapod, equipped

with wireless communication; commands are sent from Matlab. The real-time control of the robot with

two commands was successful in about 90% of the subject trials [94].

Figure 4. Real-time robot control with biosignals.

8. MEDICAL RESEARCHERS WORKING ON THE PROJECT

In his role as co-PI, Dr. Steven Rich, the Medical Director of Long Term Care & Senior Services at

Rochester General Health Systems will be providing medical oversight. This work will include an

evaluation of design documents in accordance with known symptoms, confirmation of human subject

protections, guidance in the selection and study of individuals suffering from geriatric syndromes that

generally require intervention, and a general assessment of the practically and economics of robot

deployment based upon prototype design and construction. As an initial example of this oversight, the

statistics-driven design emphases on early fall intervention and observation of prescription

14

noncompliance were confirmed by high statistical observations during the Director’s 25 years of

experience in geriatric medicine in the Rochester community.

This oversight is of particular emphasis for human subject experimentation in years 3 & 4, ensuring

compliance with federal, state, regional, and organizational guidelines in subject selection, experimental

control, and protection from health and other hazards. During these years, the Director will work with

all personnel to ensure correct experimental setup, completion, and evaluation.

9. RELATION OF PROPOSED WORK TO LONGER-TERM GOALS OF THE PIs

The proposed work will support the long-term goals of the PIs to develop state-of-the-art technological

solutions for an aging population to supplement traditional medicine. This project is intended as one the

first step in an extensive collaboration between Rochester General Health System, Rochester Institute of

Technology, and their new industrial partner, RTSync. Proposed work will provide initial opportunities

in elderly human subject experimentation in the Rochester area, for this and future projects.

10. RELATION OF PROPOSED WORK TO WORK UNDER OTHER SUPPORT

Rochester Institute of Technology (RIT) and Rochester General Health System (RGHS) have recently

established the RIT/RGHS Alliance to capitalize on the synergies between medicine and technology in

quality and cost-efficient health care. The PI / co-PIs are in application for seed funding for smart walker

prototyping, and have begun sensor selection and robot design in anticipation of continued collaboration

within and beyond the scope of this proposal. Related work by co-PIs also includes work in progress in

non-biological (complex system and vehicle) health monitoring under Department of Defense and

Department of Transportation funding, portions of which will be leveraged in the proposed system.

11. BROADER IMPACTS

We will engage K-12 and college students through culturally popular and low-threshold applications

(robotics) and workshops, expand existing K-12 outreach activities at RIT and boost mentoring of

National Technical Institute for the Deaf (NTID) and Engineering students in Senior Design projects.

11.1 Culturally Popular and Low Threshold Application to Engage Students in STEM

With recent low-cost, miniaturized components and increasing computational power, Lego-based

robotics kits have great appeal for children to work, play, and learn with robots [223]. The Intelligent

Brick (later: Lego Mindstorm [224 – 226]) by MIT first received widespread attention. The Carnegie

Mellon University developed Robotics Engineering I and II

curricula [227] that have been widely used to teach science and

engineering concepts to K-12 and college students through Lego

Mindstorm kits in a fun and innovative environment [224, 228 –

235]. There are several national competitions [236] that use Lego

robots (e.g. First Lego Leagues [237]) or other robots (e.g. US First

Robotics [238], Robotics Soccer [239], Firefighting Robots [240]) to

promote interest in STEM fields [233 – 235, 241]. ROBOLAB [247]

and Lego Mindstorm Education NXT software (powered by

LabView [243]) are programming environments well-received by K-

12 students. The PI Ferat Sahin has been using Lego Mindstorm kits

to offer a two-day weekend workshop, called RoboWeekend (See Figure 5) [244] and summer camps

called RoboCamps [244], for K-12 students. These programs use the CMU Robotics Engineering I&II

curriculum and in 2009-10 offered training to 66 students (grades 2-10) from urban, suburban, and rural

districts, with more than a quarter being females. Students found the workshops to offer highly (83%) or

Figure 5. K-12 students at

RoboWeekend.

15

somewhat (17%) valuable knowledge about robotics. Students responded with “definite” (56%) and

“maybe” (42%) to a question of whether they would want to return; similar student feedback was

received from the 2010-2011 workshops. The Robocamps, first run in Summer 2011 for total of 75 middle

and high school students, are designed to have three levels: beginner, advance, and expert. The

satisfaction levels for the beginners, advance, and expert RoboCamps were 95%, 93%, and 100%

respectively. Students stated that they had enjoyed, learned, and would like to study in STEM fields.

We intend to integrate the proposed smart walker prototypes into these RoboWeekend and

RoboCamp workshops.

RoboWekeends: In a 2-day Beginner RoboWeekend workshop, students work in teams of two to learn

basic building blocks of the Lego Mindstorm NXT kit through instructional videos, tutorials, and

presentations from the CMU Robotics Engineering I curriculum. The program design meets many of the

NETS standards [245] to promote STEM interest. In the first day, students first build their robots by

attaching motors, sensors, and other components. By construction and programming of the robot,

students prepare for a challenge in the second day when they build as a team without major supervision.

In Advance RoboWeekends, students build and program sensor and actuator units from the Lego

Mindstorm Kits (sound sensor, infrared sensor, speakers) and third-party sensors (gyroscope and

camera). The students also explore complex programming concepts to handle data from multiple sensors.

RoboCamps: Beginner and Advanced RoboCamps are designed based upon Beginner and Advanced

RoboWeekend workshops, tailored to students with limited experience in robotics and Lego kits. In the

Beginner RoboCamp, students learn the basics of Lego kits and robotics concepts such as sensor

interfacing, line following, navigation, and programming. In the Advanced RoboCamp, similar topics are

covered with harder challenges and more complex programming. In Expert RoboCamps, students

design a servo-based mobile robot controlled by an Arduino microcontroller and infrared (IR) proximity

sensors. Sensor interfaces are explored, such as infrared and ultrasonic proximity and touch sensors. In

addition, students will experiment with the Smart Walkers from this project.

In Fall quarters, we will have four Beginner RoboWeekend workshops. In Winter and Spring, we will

run four Beginner and two Advance RoboWeekend workshops. Thus, we will reach out to 220 K-12

students per year. In summers, there will be six weeks of RoboCamp workshops (three Beginners (15

students per session, two Advance and one Expert camp (10 students per session)) for a total of 75

students. In four years, we will reach out to 1005 K-12 school students. We will provide registration fee

support for 200 underrepresented and minority students.

11.2 Integration of Smart Walker prototypes with other existing K-12 outreach activities at RIT

The WE @ RIT [246] organization holds extensive on-campus and weekend activities for middle- and

high-school girls throughout the academic year and summer. Programs encourage and inspire girls and

young women in grades 4-12 to consider engineering as a career. Several hundred students (K-12),

educators, and parents participate in WE@RIT outreach programs each year. In a one-day program for

grades 10-12, girls learn about emerging engineering topics. Other RIT K-12 programs, such as E3 Fair

[247], and Traveling Engineering Activity Kits [248], will also introduce our Smart Walker prototypes.

11.3 Structured mentoring environment for deaf or hard of hearing students interested in robotics

The National Technical Institute for the Deaf is the world’s largest technical college for deaf students and

is one of eight colleges at RIT. Hearing-impaired students in engineering will be recruited for Senior

Design teams and as undergraduate researchers; funds have been specifically allocated in the budget for

this. Formal training will be created for these undergraduate students with a focus on graduate school

preparation. Training plans will be developed in conjunction with academic advisors at NTID.

16

12. COORDINATION PLAN

While the proposed system will function as a single whole, the final responsibility of each component is

divided based upon area of expertise.

Dr. Sahin (and his students) will be responsible for the software and hardware design and

implementation of the robotic “smart walker” subsystem, including: recording signal data, navigating

autonomously, maintaining awareness of all observable people and animals, and identifying imminent

risks of falling.

Drs. Ardis and Nenadic (and their students) will be responsible for the software and hardware design

and implementation of the analytics server subsystem, including: maintaining infrastructure for

continuous alert generation, generating fall detection alerts, generating medicine noncompliance alerts,

generating biosignal anomaly alerts, and generating biosignal degradation alerts.

Drs. Zeigler and Seio will be responsible for the software and hardware design and implementation

of the base station subsystem, including: managing data transmission between subsystems,

standardizing data representation, generating biosignal “extreme value” alerts, and minimizing data

transmission (during inactivity or sensor dropout).

Dr. Rich will be responsible for the medical oversight of subsystem design and final human subject

experimentation.

Some of these responsibilities, particularly medical oversight of design and standardization of data,

will require coordination between personnel to ensure compatibility of subsystems. For these

responsibilities, the following steps will be taken:

1.) A wiki will be constructed for continuous discussion and refinement of research work. This wiki

will be partitioned by subsystem, with moderation of each subsystem the responsibility of the

corresponding PI/co-PI(s). Login will be limited by unique usernames and passwords for all

personnel (including students), and the terms of usage will include continuous logging of edits

and restriction of data discussions to cross-subject/non-subject statistics without revealing any

raw signal data (pre- or post-anonymization). This wiki will be maintained by Drs. Ardis and

Nenadic, in addition to their responsibilities above, running on the server subsystem hardware.

2.) Meetings between all senior personnel will occur every second week, on a schedule yet to be

determined. These meetings will include presentations of recent progress, discussion of existing

problems, brainstorming of solutions, and confirmation of deadlines. In order to accommodate

personnel being located in different parts of the country, these meetings will be conducted via

Skype ™ and FreeConference ™ with audio logged for later reference (accessible through the

wiki). Raw signal data will not be presented at these meetings.

3.) Meetings between senior personnel and their students and staff will occur at least once every

week, on a schedule yet to be determined, for all senior personnel with such support. These

meetings will parallel the senior personnel meetings, but with an emphasis on work interplay

within each subsystem and a finer granularity of progress. These meetings will not have

recorded audio, and raw signal data may be presented and discussed freely.

4.) Meetings between all personnel will occur once every six months, on a schedule yet to be

determined. These meetings will parallel the other meetings, but with an emphasis on significant

research milestones achieved and any issues requiring additional collaboration beyond the scope

of the above meetings. All-personnel meetings will take place at the Rochester Institute of

Technology, with personnel traveling as necessary. Presentation materials for these meetings

will be made available through the wiki, along with audio logged for later reference. Raw signal

data will not be presented, although its transfer will be allowed in person (offline, encrypted).

5.) Irregular communication between personnel will be available through email and phone calls, as

needed. Raw signal data will not be discussed through these means.

17

In addition to the above steps to perform cooperative work on the project, we intend to publish all

findings via a series of presentations in scholarly conferences (with follow-up articles in applicable

journals), at the annual Imagine RIT Innovation & Creativity Festival, and at local K-12 schools within the

Rochester, NY area. Furthermore, we intend to make all design and presentation documents public via

web hosting, including software source and libraries (as described in the Data Management Plan).