Low-cost Visual Tracking with an IntelligentWheelchair for Innovative Assistive Care

Jaime Valls MiroFaculty of Engineering and IT

University of Technology, SydneyAustralia

Email: jaime.vallsmiro@uts.edu.au

James PoonFaculty of Engineering and IT

University of Technology, SydneyAustralia

Email: james.poon@student.uts.edu.au

Shoudong HuangFaculty of Engineering and IT

University of Technology, SydneyAustralia

Email: shoudong.huang@uts.edu.au

Abstract—This paper presents the development of a low-cost vision-based robotic wheelchair system towards autonomousconvoying. The non-holonomic follower vehicle obtains visualreal-time pose data of a known coplanar target installed onthe back of the leading vehicle. This allows the tracking vehicleto mimic the path of the preceding vehicle, while maintaininga safe distance behind it with the aid of a controller basedon the robot’s kinematics constraints. A back-end visual filteris proposed in the planning strategy to overcome the noisyenvironmental information acquired from the camera as it tracksthe vehicle in front. The effectiveness of the approach is evaluatedin an indoor setting using data obtained from an instrumentedwheelchair platform and a low-cost camera, and validated withobservations from a laser range finder and derived (known) mapsof the environment.

I. INTRODUCTION AND MOTIVATION

There is a marked demographic shift at a global levelindicating that the worldwide proportion of people aged over60 is expected to double between 2000 and 2050 [1], [2], atrend pictorially shown by Fig. 1. The challenges associatedwith this healthy longevity paradigm are driving the need forimprovements in the range of services related to aged care.However, while the number of people requiring care is in theincrease, the projected number of active people being able toprovide that care is in decline. This paradigm is accentuatedby the fact that, although older people are often reluctant toidentify themselves as disabled, the truth of the matter is thatthey exhibit similar declining physical and cognitive capacities,and there is a clear and significant overlap in the needs ofthese two populations from care organisations. In the nursinghome for instance, where the prevalence of impaired mobilityis highest, cognitive impairment is equally high, and the needfor individual caregiver assistance is ever present.

A recent report by the Australian Academy of TechnologicalSciences and Engineering canvassed various options based

Fig. 1: Changes in the population structure [2].

on the use of emerging innovative technologies to addressthese challenges [3]. Areas such as telemedicine, elderly-friendly housing, remote monitoring systems and sensors, orwander management systems form part of the technologicalaids becoming more readily available to the frail, disable andchronically ill community to enhance their quality of life.

The work hereby proposed is driven by the needs of acollaborative care organisation supporting people with disabil-ities and complex health needs in the community to lowerthe demands on their carers so that better services can beadministered. In providing support to cognitive and physicallydisabled patients confined to wheelchairs, a carer is requiredto work with individual patients one at a time. This workproposes a tracking system to enhance day-to-day mobility-oriented tasks. A convoy system could provide a less staff-intensive alternative where one carer is capable of leadingmultiple autonomous followers, greatly reducing required man-hours which could be better utilised in other aspects of thepatient’s care. Given the community-oriented nature of theorganisation, where clients have been moved from institutionsto live in group homes, one of the key intended application ofthis system is in enhancing their quality of life by being able toenjoy increasing outdoor excursions of the surrounding area.

A low-cost vision-based solution suitable to be reliablydeployed on a tracking non-holonomic wheelchair vehicle hasbeen developed for that purpose, and the initial experimentalresults are presented in this work. Detection (in the imagespace) and tracking of a reference pattern located in the leadingvehicle results in a trajectory to be followed based upon thepose (position and heading) of the vehicle in front relativeto the follower. Using vehicle kinematics, a simple controlleris then proposed for the tracking vehicle to closely followthe estimated trajectory of the leader unit, keeping also asafety distance. In this initial phase of the work, a motorisedwheelchair has been instrumented, and indoor experimentscarried out and validated by using a laser range finder andgeometric maps of the environments. The main contributionof this paper is the demonstration of how a probabilisticvisual tracking wheelchair system is capable of modellingand tracking the behaviours of the vehicle in front, therebyvalidating the effectiveness of this low-cost embodiment ofmachine intelligence to support some of the activities that

978-1-4673-1872-3/12/$31.00 ©2012 IEEE 1540

Fr34.22012 12th International Conference on Control, Automation, Robotics & VisionGuangzhou, China, 5-7th December 2012 (ICARCV 2012)

a carer performs on a daily basis for the frail and disabledcommunity.

The remainder of the paper is organised as follows: anoverview of related work from the bibliography is first givenin Section II, followed by a description of the robotic systemarchitecture in Section III. Section IV illustrates the core ofthe proposed visual tracking assistive robotic application, whileSection V presents the proposed wheelchair controller to guidethe vehicle along the infered visual trajectory. Section VIdescribes the experimental data collection set-up employed andthe results attained to validate the proposition in this paper.Section VII summarises the proposition of the paper and looksat limitations and future plans.

II. RELATED WORK

Impaired mobility is one of the most significant problemfor the frail and disabled populations, largely the result of theage-related decline in the musculo-skeletal and neurologicalsystems. Given its widespread use and social acceptance, theconcept of mobility aids to overcome motor impairments ispossibly the most prominent illustration of the substantialbenefits that gaining access to such assistive devices can bringto the the elderly and disabled community. While immobilityis mainly associated with physical complications, e.g. bedsores, osteoporosis, deep vein thrombosis etc. it also has asignificant social impact. Given their social acceptance andubiquity, conventional powered wheelchairs in particular havesuccessfully provided the means by which a variety of peoplewith gait disorders can maintain stable mobility, and studieshave shown how a wheelchair system intervention in carefacilities provides a vehicle for sustaining social interactionsand physical and mental activity for longer, thereby facilitatingtheir ability to live more independently and consequentlyimproving their overall quality of life [4].

Despite the functional independence warranted by manualand electric wheelchairs to perform a large array of dailyactivities, there is nevertheless a clear appreciation of theexpanding ways technology can support the independence ofthe older and disabled population [5], and this has motivatedresearchers to develop more advanced technological aids ca-pable of providing enhanced support to people suffering frommobility impairments [6], such as robotic wheelchairs [7],[8], [9], walkers or smart canes. Many mobile wheelchairrobotic systems for instance have been predicated on the useof laser range finders as the primary sensing method [10].Stereo camera solutions have also been developed mainlyaimed at generating maps where to localise the vehicle andplan paths in the context of the simultaneous localisationand mapping (SLAM) problem, as well as to extract limitedcontextual information from signs in the environment [11].Monocular vision-based tracking systems have been used ina variety of outdoor autonomous vehicle scenarios, such asmilitary vehicles [12], harvesters [13], or outdoor multi-vehicleconvoy systems [14] [15] for tracking the vehicle ahead.In convoying, visual tracking provides a sensing techniquewhich is cost-effective, well distributed and easily adaptable.

Fig. 2: The robotic wheelchair platform, with the cameralocated inside the high dome at the back.

The work hereby proposed also exploits monocular camerasas the primary sensing method to progress the concept ofassistive care further, effectively advancing the emerging ideaof ‘intelligent robotic mobility aids’. This is a promising fieldof development looking at narrowing the gap between anindividual’s ability and their environment. Specifically, thiswork is an attempt to validate the potential of smart intelligentmachines to viably support care in the community to benefitthe many who would otherwise struggle in everyday mobilitytasks, increasing patients’ quality of life and independence, aswell as decreasing the workload on their circle of carers inrudimentary tasks such as taking patients for a stroll.

The objective of the proposed tracking solution is thusan affordable system that can safely guide an instrumentedwheelchair vehicle along the route of a leading vehicle, whilemaintaining a safety distance between the two vehicles. This ispredicated on an image processing system capable of extractingpose data from a coplanar target of known dimensions.

III. SYSTEM ARCHITECTURE

Experiments were conducted on a modified Invacare RollerM1 motorised wheelchair, fitted with an on-board computer,wheel encoders and an automatic battery management system.The proposed solution is based on the use of monocularcameras. While other sensors, such as laser scanners or radarscould have also been used, these are often expensive, po-tentially heavy and have a substantial power draw, whereasmodern cameras are power efficient, far less costly, and canbe easily interfaced with regular computers. Hence, the instru-mented robotic wheelchair relies on wheel odometry and posedata gathered from a calibrated monocular camera - a LogitechQuickCam USB webcam providing 1280 × 960 resolutionimages at 10 frames per second. The camera was located ona raised mounting bracket at the back of the vehicle to afforda wider visual field, as illustrated in Fig. 2.

A distinctive green landscape sheet of known dimensions,depicted in Fig. 3, was chosen as the test target mounted onthe leading moving vehicle to be tracked.

All processing, fundamentally running under the roboticsmidleware ROS 1 and the computer vision library OpenCV 2,

1www.ros.org2code.opencv.org, opencvlibrary.sourceforge.net

1541

(a) Poor segmentation (b) Segmented target over origi-nal image

(c) Good segmentation (d) Segmented target over origi-nal image

(e) Noise in corner data (red),and the smoothing effect of theKalman filter (green)

Fig. 3: Examples of tracking pattern segmentation process(a,b,c,d) and noise smoothing (e).

Algorithm 1 Tracking Algorithminput: image of target on back of leader vehicle.Compute hue/saturation histogramFor each incoming frame1. Compute contour target from histogram back-projection.3. Extract corners from resulting contours.4. Reduce raw corner calculations noise via a Kalman filter.5. Find target pose, relative to follower wheelchair (initialdistance between two vehicles known).6. Feed poses to wheelchair controller.output: follower wheelchair robot guided along route of lead-ing vehicle.

was performed in the on-board computer: a low-cost 1.8 GHzIntel Atom processor, 2 GB RAM and several serial/USB portsfor device interfacing and robot control. This relatively low-spec set-up was capable of processing data at a rate above1Hz, which proved sufficient to avoid any compromises withthe wheelchair controller performance, as will be seen inSection VI when the experimental results are analysed.

IV. TRACKING SYSTEM DESIGN

An overview of the proposed tracking procedure is de-scribed by Algorithm 1. It is a sequential process where eachof the incoming frames is processed to extract the pose of thetarget (leading) vehicle to be tracked. The resulting pose ismathematically calculated in 6D. However, for the intendedpopulation motions can be safely regarded as restricted to pla-nar environments, 2D displacement and headings are assumedby the controller without loss of generality.

A two dimensional histogram of the green test target is firstcomputed, and its hue and saturation characteristics derived.This is generated from a mask area manually extracted fromthe RGB video stream. Incoming frames are then convertedinto the HSV colour space, and a back-projection of the targethistogram is calculated [16] on the hue and saturation images.This back-projection is then converted into a series of contours,and the contour with the largest area is taken as the targetoutline.

Hough transform line detection [17] is then used to identifythe possible borders of the isolated contour. These potentialborder lines are then sorted by both gradient and positionrelative to the contour’s centroid, to obtain the candidate fouredges of the target rectangle. Intercepts are then calculatedbetween these characteristic lines to find the image coordinatesof the corners, which are then sorted by distance with respectto the previous set of corners to identify them as top-right,bottom-left etc. An example of good and poor segmentationprocess is depicted in Fig. 3.

As the segmentation step may fail to completely outlinethe target due to sharp lighting changes or momentary obstruc-tions, any calculated points may not properly reflect the target’scorners. This would make the current computed target poseinvalid and potentially cause erratic or dangerous movementof the wheelchair platform. To counter this it is necessaryto have a mechanism that filters out ordinary measurementnoise (as seen in Fig. 3e), and also check for sanity incomingtarget pose data. This allows the wheelchair to be stoppedif necessary until normal driving can resume, such as inthe case of obstruction between wheelchair and target, andhelps reduce small jitters in movement. A recursive Bayesianestimator in the form of a Kalman filter has been used in thisapplication with the latent state being the corners of the targetbeacon being tracked. The coordinates of each corner pointand their velocities along the x and y image axes are taken asmeasurement updates, with set covariances sufficiently large toallow for abrupt but realistic changes in position and velocity.

To increase robustness, the Kalman filter’s posterior densi-ties of the latent state are also used as a comparison to eachset of incoming corner location measurements for consistency:if there is a major difference between any two points, the lastvalid dataset is held up, and the filter is temporarily suspended.When stability has been re-established, the filter is re-initialisedto the new set of points. This approach ensures that the filteris not thrown off by largely skewed measurements. It alsoreduces the time period before valid poses are to be filteredagain following one or several unsuccessful segmentations, as

1542

(a) Performance of leading vehicle posedetection (real pose (green), pose inferredfrom visual algorithm (red))

(b) Tracking performance (leading vehi-cle pose from visual algorithm (red), realpose of tracking wheelchair (blue))

(c) Overall system performance (real poseof leading vehicle (green), real pose oftracking wheelchair (blue))

Fig. 4: Indoor tracking process - shorter experiment through narrower spaces (6× 7m approx).

the filter’s corrections do not have to naturally shift to the newstable position.

The matrix of points that defines the dimensions of thetarget shape, and the corner posteriors from the Kalmanfilter are then used to estimate the target pose in 6D of theleading vehicle in relation to the camera, hence the trackingwheelchair. This can be accomplished given the known co-planar relationship between the set of points that define thetarget, and their corresponding image projections (as wellas the camera matrix and the distortion coefficients obtainedfrom the initial camera calibration). In OpenCV, this is apose optimisation process that minimises the reprojection error,i.e. the sum of squared distances between the observed andprojected points in the image.

V. TRACKING CONTROLLER

A receding horizon controller strategy purely based on localrelative pose information between the target and the wheelchairis proposed to revisit the leader vehicle trajectory. The mainreason for using this strategy is to improve the robustness ofthe controller given the lack of knowledge about the globallocations of the vehicles.

At each time step when an image is processed, a relativepose information between the leader and the follower isobtained from the image processing. Using this relative poseinformation, a target position of the follower is computed. Thenthe controller is designed such that the follower can reach thetarget position within minimum time. However, this controlleris only executed until the next valid image is processed, andthe next relative pose becomes available. By always using theupdated relative pose information in the calculations for thenew target position, the follower is able to robustly followthe leader even when there are noises and uncertainties in theexecution of the controller, the key benefit of using a recedinghorizon strategy.

In order to compute the target position using the relativepose information, the target is simply set in the direction ofthe leader at a minimal distance dmin = 0.7m for safety.

Hence, given the relative pose (dx, dy) obtained from theimage processing engine, the target position (tx, ty) and thetarget heading th can be computed by

tx = ρ dxty = ρ dyth = atan2(dy, dx)

(1)

where ρ is the ratio given by

ρ =

⎧⎪⎨⎪⎩

√d2x+d2

y−dmin√d2x+d2

y

if√d2x + d2y > dmin

0 if√d2x + d2y ≤ dmin

(2)

Note that all parameters are described in the local coordinateframe of the follower vehicle.

If the target position is beyond a maximum thresholddistance dmax, the controller will direct the wheelchair totravel at a predetermined maximum speed towards the targetposition. If the distance between the leader and follower isbelow the minimal distance, the follower will simply keepstationary. A speed proportional to the distance gap betwenthe vehicles will be chosen for anything in between.

VI. EXPERIMENTAL RESULTS

Two trials have been carried out to validate the proposed ap-proach. In both instances the experiments have been conductedindoors to be able to validate the inferred visual poses withrespect to a global map. Accurate maps have been generatedwith a SLAM algorithm from a laser range finder and odometryfrom the wheelchair. In the first experiment, a smaller and morecluttered office-like area is considered, whereas in the secondwider spaces in a much larger close loop corridor area areconsidered.

For the experiments, the leading vehicle was manuallypulled at walking speed, while the wheelchair robot followedautonomously. The resulting paths eventuated by both vehiclesare shown in Fig. 4 and 5. Poses (2D location plus orientations)are colour coded for ease of interpretation, where the true pose

1543

(a) Performance of leading vehicle pose detection (realpose (green), pose inferred from visual algorithm (red))

(b) Tracking performance (leading vehicle pose fromvisual algorithm (red), real pose of tracking wheelchair(blue))

(c) Overall syetm performance (real pose of leadingvehicle (green), real pose of tracking wheelchair (blue))

Fig. 5: Indoor tracking process - larger experiment throughwider spaces (27× 15m approx)..

of the leading vehicle is depicted by green arrows, the pose ofthe leading vehicle inferred by the visual engine is shown byred arrows, and the true location of the following wheelchairvehicle is shown in blue. The real pose of the leading vehiclehas been extracted as a distinctive length segment from thelaser scans, and leading vehicle poses are shown transformedwith respect to the wheelchair true location from SLAM.

The attained results therefore show:

• how good is the proposed mechanism to locate the targetaccurately, Fig. 4a and 5a.

• how good is the proposed mechanism to track the target,Fig. 4b and 5b.

• the overall system accuracy, Fig. 4c and 5c.

It can be seen how in both instances the combined visualtarget location and the controller that ensues are able to keepthe follower vehicle relatively close to the path described bythe preceding vehicle. The errors are quantified in the graphs

shown by Fig. 6a, 6d, 6b and 6e, where the deviations errors in2D for both experiments are shown. These graphs effectivelyquantify how close the trajectory followed by the wheelchairwas to that of the leading vehicle. Likewise, graphs depicted inFig. 6c and 6f measure the controller performance in keepingthe gap between the two vehicles beyond the specified minimalsafety distance of 0.7m. Measurements between both vehicleshave been derived from the laser scans for ground-truth. It canbe seen how this distance is guaranteed throughout the motion,with a small number of instances in the long run where thefollower vehicle stopped as the gap between the two vehicleswas perceived to be getting below the safety distance. Thissituation is mainly attributed to the poorer lighting conditionsexperienced by the wheelchair camera in the corridors ofthe longer experiment, which meant that the visual engineeffectively discarded a number of poses, and the wheelchairnavigated towards the last known pose without visual poseupdates until the safety distance was compromised. As thevehicle in front kept moving away, when the visual enginelocked again into a new target, tracking was re-established.

VII. CONCLUSIONS AND FUTURE WORK

The first step towards an autonomous vision-guided convoysystem aimed at enhancing the quality of life of elderly anddisabled people reliant upon mobility aids has been describedin this work. Beyond the psychological [18] and physical [19]benefits of being exposed to the outdoors, wheelchair outdoorexcursions are widely regarded as a non-exertive means ofentertainment and relaxation for clients with limited mobility.The proposed solution is motivated by the needs of careproviders to conduct more regular outdoor wheelchair toursof surrounding areas for their clients with increasingly limitedsupport care personnel.

Results from the initial phase of the work on indoor exper-iments have demonstrated how a probabilistic tracking systemsolely based on visual feedback was capable of modellingand tracking the trajectories of the vehicle in front with goodaccuracy. The experiments, validated with a laser range finderand geometric maps of the environments, have shown errorsbelow 0.5m of the trajectories followed by the leading vehicle,and set safety distances between vehicles being maintained.

Given the substantial savings that an autonomous navigationsolution mainly reliant on vision can offer, these are encour-aging results to further trials under more realistic semi-urbanoutdoor settings where the proposed solution will be eventuallydeployed. It is however expected that for outdoor use cameraexposure will need to be dynamically adjusted to overcomethe difficulties associate with changing lighting conditions. Tothat end, either a camera with integrated high dynamic rangecapabilities will be used, or a mechanism to set variable regionsof interest within the image may be preferred instead [20].

Loss of line of sight is currently the primary cause of failurefor this system, whether it be from physical obstruction suchas a person walking behind the target, or from unpredictablelighting conditions such as entering a region of full/mottledshade, or direct sunlight. Hence, despite the proposed safety

1544

(a) Error in X(m) (b) Error in X(m) (c) Shorter experiment

(d) Error in Y(m) (e) Error in Y(m) (f) Longer experiments

Fig. 6: Deviation between tracking wheelchair and leading vehicle - (a,d) shorter experiment, (b,e) longer experiment. Lineargap (m) between tracking wheelchair and leading vehicle - (c) shorter and (f) longer experiment.

checks to minimise the risks associated to the proposed low-cost solution, thoughtful driving of the leading vehicle by thecarer remains key to mitigate the risks, and must be conductedwith full awareness of the limitations of its follower(s).

ACKNOWLEDGMENT

The authors would like to thank Greystanes DisabilityServices 3 for their collaboration in this research project.

REFERENCES

[1] U. Nations, “World populations ageing,” 2007.[2] Congressional Budget Office (C.B.O.), “Global population aging in the

21st century and its economic implications,” 2005.[3] W. J. M. Tegart, “Smart technology for healthy ageing,” Australian

Academy of Technological Sciences and Engineering (ATSE), Tech.Rep., 2010.

[4] E. Trefler, S. G. Fitzgerald, D. A. Hobson, T. Bursick, and R. Joseph,“Outcomes of wheelchair systems intervention with residents of long-term care facilities,” Assistive Technology, vol. 16, pp. 18–27, 2004.

[5] J. Connell, C. Grealy, K. Olver, and J. Power, “Comprehensive scopingstudy on the use of assistive technology by frail older people living inthe community,” Urbis for the Department of Health and Ageing, Tech.Rep., 2008.

[6] G. Wasson, P. Sheth, C. Huang, and M. Alwan, “Intelligent mobilityaids for the elderly,” in Eldercare Technology for Clinical Practitioners,M. Alwan and R. A. Felder, Eds. Humana Press, 2008, pp. 53–76.

[7] G. Pires, R. Araujo, U. Nunes, and A. T. de Almeida, “RobChair -a powered wheelchair using a behaviour-based navigation,” in Interna-tional Workshop on Advanced Motion Control, 1998, pp. 536–541.

[8] C. Mandel, K. Huebner, and T. Vierhuff, “Towards an autonomouswheelchair: cognitive aspects in service robotics,” in Towards Au-tonomous Robotic Systems, 2005, pp. 165–172.

[9] E. Demeester, A. Huntemann, D. Vanhooydonck, G. Vanacker,A. Degeest, H. V. Brussel, and M. Nuttin, “Bayesian estimation ofwheelchair driver intents: modelling intents as geometric paths tracked bythe driver,” in IEEE/RSJ International Conference on Intelligent Robotsand Systems, oct 2006, pp. 5775–5780.

3www.greystanes.org.au

[10] Y. Kinpara, E. Takano, Y. Kobayashi, and Y. Kuno, “Situation-drivencontrol of a robotic wheelchair to follow a caregiver,” in Korea-JapanJoint Workshop on Frontiers of Computer Vision, 2011, pp. 1–6.

[11] M. Bailey, A. Chanler, B. Maxwell, M. Micire, K. Tsui, and H. Yanco,“Development of vision-based navigation for a robotic wheelchair,” inIEEE International Conference on Rehabilitation Robotics, 2007, pp.951–957.

[12] J. L. Giesbrecht, H. K. Goi, T. D. Barfoot, and B. A. Francis, “A vision-based robotic follower vehicle,” in Proceedings of the SPIE Defence,Security and Sensing, vol. 7332, 2009, pp. 14–17.

[13] G. Carducci, M. Foglia, A. Gentile, N. I. Giannoccaro, and A. Messina,“Pneumatic robotic arm controlled by on-off valves for automatic har-vesting based on vision localisation,” in IEEE International Conferenceon Industrial Technology, vol. 2, 2004, pp. 1017–1022.

[14] S. Benhimane, E. Malis, P. Rives, and J. R. Azinheira, “Vision-basedcontrol for car platooning using homography decomposition,” in IEEEInternational Conference on Robotics and Automation, 2005, pp. 2161–2166.

[15] N. Kehtarnavaz, J. S. Lee, and N. C. Griswold, “Vision-based convoyfollowing by recursive filtering,” in American Control Conference, 1990,pp. 268–273.

[16] K. S. Chenaoua, A. Bouridane, and F. Kurugollu, “Unsupervised his-togram based color image segmentation,” in IEEE International Confer-ence on Electronics, Circuits and Systems, vol. 1, 2003, pp. 240–243.

[17] R. O. Duda and P. E. Hart, “Use of the hough transformation to detectlines and curves in pictures,” Commun. ACM, vol. 15, no. 1, pp. 11–15,Jan. 1972.

[18] R. M. Ryan, N. Weinstein, J. Bernstein, K. W. Brown, L. Mistretta, andM. Gagne, “Vitalizing effects of being outdoors and in nature,” Journalof Environmental Psychology, vol. 30, no. 2, pp. 159–168, 2010.

[19] I. R. Reid, D. J. A. Gallagher, and J. Bosworth, “Prophylaxis againstvitamin d deficiency in the elderly by regular sunlight exposure,” Ageand Ageing, vol. 15, no. 1, pp. 35–40, 1986.

[20] M. M. Clothier and M. Bailey, “Overcoming augmented reality trackingdifficulties in changing lighting conditions,” in IASTED InternationalConference on Computer, Graphics and Imaging, 2004, p. 9.

1545