Techll<>logy and Disability 14 (2002) 101-10510S Press

i

I

101

The Quebec User Evaluation of Satisfactionwith Assistive Technology (QUEST 2.0): Anoverview and recent progress

Louise Demers*, Rhoda Weiss-Lambrou and Bernadette SkaResearch Center of the Institut universitairede geriatrie de Montreal, 4565 Queen Mary Road. IHontreal(Quebec),H3W IW5, Canada

~

Abstract. The Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST 2.0) is a 12-item outcome measurethat assesses user satisfaction with two components, Device and Services. Psychometric properties have been tested with respectto test-retest stability. alternate-form equivalence. internal consistency, factorial composition and nomological validity. Examplesof results obtained with the first version of the tool in outcome studies in Europe and North America support the importance andrelevance of the satisfaction measure.

1. Purpose

The Quebec User Evaluation of Satisfaction withAssistive Technology (QUEST 2.0) was designed asan outcome measurementinstrument to evaluate a per-son's satisfaction with a wide range of assistive tech-nology (AT). It was intendedas a clinical and researchinstrument. As a clinical tool,..the rating scale pro-vides practitioners with a means of collecting satisfac-tion data to document the real-life benefitsof ATand to

justify the need for these devices. As a research tool,it can be used to compare satisfaction data with otheroutcome measures such as clinical results, quality oflife, functional status, cost factors and comfort. It can

also serve to compare satisfactionresults obtained withdifferent user groups, in differentsettings and in differ-ent countries. Until now,theQUEST 2.0 has been usedmainly with adults but it may also be administered toadolescents and elderly users of AT.

'Corresponding author: Tel.: +15143403540; Fax: +15143403548; E-mail: Louise.Demers@umontreal.ca.

ISSN 1055-4181/021$8.00 @2(J(J2 - [aS Press. All rights reserved

",,- ---..

[8]

2. Conceptual basis

Due to a vacuum in theoretical knowledge, satisfac-tion determinants are vague in the field of assistive tech-nology. Similar conceptual limitations exist in other

health domains [1-3]. Despite lack of standards, therationale behind the QUEST was drawn from the stateof the art in satisfaction assessment. The relations be-

tween the variables involved in the experience of sat-isfaction with AT are represented within a linear gen-eral framework, inspired by Simon and Patrick [4]. InFig. 1, expressed satisfaction is conceived as a reactionto assistive technology provision. It may also triggera subsequent action or behaviour, for instance, the useor the abandonment of the AT. Satisfaction is defined

as a person's critical evaluation of several aspects of adevice. This evaluation is believed to be influenced byone's expectations, perceptions, attitudes and personalvalues. Accordingly, satisfaction is considered as amultidimensional concept with two underlying dimen-

sions respectively related to assistive technology, De-vice and Services. As shown in Fig. 2, the Device di-mension embraces 8 items related to salient character-

istics of the assistive technology whereas the Servicesdimension encompasses 4 intercorrelated items.

Assistive Technology Experience

L. /J"/I"'/".' ,'t ill. / QUEST 2.0: .-\/1 """/T;,'\\' <111<7/"eel'llt I'n'g/"e.u102

Service Satisfaction Evaluation

delivery dimensions involvingcognition

Exprcssnl... satisfactil\ll

Behavioral. outcomes

Fig. 1. Satisfaction with assitive technology model (inspired by Sillll'lI and Patrick H]).

. .v0'P~~

0°§;.$2

f?fi

t!!!

Fig. 2. QUEST 2.0 Bidimensional satisfaction structure.

3. Description

The firstversionof the QUEST consisted of 24 itemsalthoughthe usercouldalsoadd anyother items consid-ered important [5]. They were drawn and adapted fromcurrent literature on assistive technology assessmentand the Matching Person and Technology model [6].Responses were coded on a 5-point importance scale('of no importance' to 'very important') and on a 5-point satisfaction scale ('not satisfied at all' to 'verysatisfied'). Field testing of this original version wereconducted [7,8]. Based on these data, an item analysiswas performed [9] and those items that ranked best interms of demonstrating optimal measurement perfor-mance were included in QUEST 2.0. The importancescale was removed because it did not prove to be re-liable for discriminating between and among users ofAT [8].

Accordingly,thecurrentversion of the scale includes12 items rated on a 5-point satisfaction scale gradedas previously (see examples in Fig. 3). The itemswere considered applicable for a wide variety of de-vices [9]. For the purpose of identifying the sources ofuser satisfactionor dissatisfaction, space for commentsis provided next to each item, The QUEST 2.0 yieldsthree scores: Device, Services, and total QUEST, cal-culated by averagingvalid responses to assigned items(range 1.00 to 5J)O).Finally, the three most important

items for the device being assessed are identified froma checklist of the l2 satisfaction items. With respect teevaluation procedures, two administration formats ma:be used. In its self-administered format, the QUEST 2.(demands minimal skills to circle or mark the response~on the rating scale and to write one's comments, 11however the pencil and paper format is not appropriatefor an individual user, the assessment tool can be ad.

ministered by an interviewer. For this purpose, someinteractive optional material is provided, including,list of 12 satisfaction items printed in large font an<an enlarged rating scale displaying the 5-point degreeof satisfaction, In either case, for each device bein~evaluated, approximately 10-15 minutes are require!to complete the evaluation. The instrument was developed in Canadian English and French. 1 It was sutsequently translated into Dutch, Swedish, NorwegiarDanish, and Japanese.

4. Reliability

Reliability, which is concerned with error in me,surement, was assessed through test-retest stabilitalternate-form equivalence between self and intervie'administration forms, and internal consistency. J-the test level, some studies were carried out on L1

community-based adults with Multiple Sclerosis uing mobility devices (walkers, wheelchairs, and scooers) [10]. These subjects were assigned to four grouland a second QUEST 2.0 was administered one weelater. Groups differed with respect to the format arthe order in which alternate forms were presented, TImeasures of association for test-retest stability, the itraclass correlation coefficient (lCC), were .82, ,82 at.91, for the Device, Services, and total QUEST, respe

tively. For alternate-form equivalence, the ICC we,89, .76, and .91, suggesting somewhat 100verresu:for the Services subscale, At the item level. indices

t Publisher of QUEST 2.0: Marcia J, Scherer Ph.D. Institute

Matching Person and Technology, 4R6 Lake Road, Webster, 1-York, USA 14580. E-mail: IMPT97@aol.com.

'omHo

:lay2.0Ises

Iflatead-'megamd'ees

ingredlel-.ub-lan,

'ea-ity,lewAt48us-lot-ups~ekmdthein-mdec-ereIlts.of

, for:-lew

I.

...'

L. Delllas ,'t aLl QUEST 2.0: An overview and recent progres." 103

1 3 4 52

not satisfied

at all

more or less satisfied quite satisfied very satisfiednot very satislied

Fig. 3. Examples from the QUEST 2.0.

agreement for the 12items were obtained from a studyof the original 24-item instrument [8]. In this study,85 users of wheelchairs and lower limb prosthesis wereadministered the test twice, with a 7 to lO-day inter-val between measures. Weighted kappa ranged from.51 to .74 and averaged around .61. Internal consis-tency estimates were calculated from the same sampleof subjects and Alpha values of .80, .76 and.82 werefound [9].

ers and consumers to review and critique the prelim-inary versions of the instrument [5]. This led to thedevelopmentof an experimental 24-item version, readyfor field trials. A number of researchers and clinicians(n = 12) from the United States, The Netherlands,and Canada volunteered to test the newly developedoutcome measureand completed a questionnaire on itscontent, administration procedures, and usefulness [7].At the test level, the findings revealed that the QUESTwas sampled adequately in terms of embracing all im-portant facets of satisfaction with AT.At the item level,however, it was shown that changes needed to be madeto item wording and to the administration proceduresto ensure optimal content validity. These results, com-bined with those of a reliability study, were used in

5. Validity

During the early development of the QUEST, con-tent validity was tested by asking a panel of stakehold-

-., ,-..... -.., '.~- .'~.

ASSISTIVE DEVICE

How satisfied are vou with,

I. the dimensions (size, height, length. width) of your asslstive device?Comments:

I 2 3 4 5

2. the weight of your assistive device?Comments: I 2 3 4 5

3. the ease in adjusting (fixing, fastening) the parts of your assistive device? ':-Comments:

I 2 3 4 5

4. how safe and secure your assistive device is?Comments: I 2 3 4 5

SERVICESHow satisfied are you with,

9. the service delivery program (procedures, length of time) in which you obtainedyour assistive device?Comments: I 2 3 4 5

10. the repairs and servicing (maintenance) provided for your assistive device?Comments:

I 2 3 4 5

104 L. nell/a'\" "Il/I. / QUEST ':,0: .4" m'en';""' and I'/'cl.'lIl progn',H

an item analysis [9] and led to the development of theshorter single rating scale QUEST2.0 1111.

Factorial analyses of the QUEST 2.0 were per-formed in two studies using the Principal Axis Factor-ing method. The first study [9) involved 150 Canadiansubjects using mobility devices. Results suggested abidimensional structure of satisfaction with AT, De-

vice and Services, accounting for 40% of the commonvariance. A cross-validation study [121 was conductedwith 243 Dutch subjects using a wide range of hometechnologies. The findings broadly supported the divi-sion into Device and Service components, in spite ofcontrasting assistive technology and services deliverysystems.

In a recent study [10], the expected relationship be-tween the QUEST 2.0 and another outcome measure

of assistive technology was empirically tested. Par-ticipants (n = 81) who were administered the instru-ment also completed the Psychosocial Impact of Assi-tive Devices Scale (PIADS) [13]. This measure cap-tures the concepts of Competence, Adaptability, andSelf-Esteem, all subsumed as fundamental dimensions

under quality oflife. Both measures rely on the individ-uals' subjective experiences but have different stand-points for evaluation. Positive correlations were foundbetween the QUEST 2.0 and the three PIADS dimen-sions. They were fair to moderate for Device and to-tal QUEST (Pearson correlation coefficient [rp] .34 to.45) and fair with Services (r p .27 to .30). This ar-gues that satisfaction with a device is associated, to acertain extent, with lifestyle, behavior, and experientialfactors.

6. Results of application in outcome studies

The following are exemplesof studies in Europe thatused the first version of the QUEST as an outcomemeasure of satisfaction with AT. In the Netherlands,Wessels et a1. [14] implementeda 3-month follow-upstudy with 375 users of toiletadaptations, shower seatsand chairs, wheelchairs,adaptedbeds, stairslifts, homeadaptations and adaptedbeds. Althoughthe majority ofthe respondents reported they were very satisfied withtheir devices, a substantialproportion(199'0)expressedspecific concerns andoveralldissatisfaction. The itemsthat clients were least satisfied with were related toservices issues, such as Service Delivery, Follow-upServices, or Professional Services, whereas the itemsthat clients were most satisfiedwith included Durabil-ity, Effectiveness, Comfort and Dimensions. Brandt

and Iwarsson (15] used the Danish version of the origi-nal QUEST as part of their outcome study on poweredwheelchair used by the elderly. Data from III sub-jects showed that the vast majority considered the ATto be important and that they were satisfied with it as awhole. However, a substantial proportion of users werenot satisfied with some of the technical characteristics.

such as power and speed.In North America, Benedict et a!. [16] used the tool

to examine whether use of AT by young children wasrelated to caregiver satisfaction (n = 37). They foundthat high ranked satisfaction scores were more likelyfor children using the device as intended than for thoseunderutilizing the device. Weiss-lambrou et a!. (17]assessed user satisfaction ,lVith modular-type seatingdevice integrated in a powered wheelchair (n = 23).The results revealed that the item comfort was the most

important consumer criterion yet it was evaluated asthe least satisfying. The QUEST was one of four mea-sures chosen by Bursick et a!. [18] to assess wheelchairseating and positioning outcomes in the elderly nursinghome population. This randomized controlled studyincluded an intervention group (n = 12) receiving anew wheelchair, cushions and a custom seat back and

a comparison group (n = 12). Overall, the subjects inthe custom fitted wheelchairs and seating systems weremore satisfied (3.72 compared with 3.14) with their AT.Use of and satisfaction of upper limb myoelectric pros-theses were studied by Routhier et a!. [19] with a sam-ple of 10 subjects. Although general satisfaction washigh (80%), specific concerns were raised with respectto heat, weight, service delivery procedures, durabilityof mechanisms and battery, follow-up services, profes-sional services, dimensions and loss of tactile sensa-tion. Stickel et a1. [20] conducted interviews with 40users and nonusers of electronic aids to daily living.Simplicity of use and safety were rated as the mostsatisfactory items whereas cost, follow-up services anddevice compatibility were commonly reported to below.

Results from the application of the QUEST 2.0 arebeginning to be published. Vincent and Demers (21]administered the test to 43 community-dwelling sub-jects using twin or double electrical beds. The major-ity of users are very satisfied with their bed, with itemscores averaging close to 4.00. Comfort and ease ofuse were considered the most important items. Inter-estingly, users of double beds were more satisfied withthe item dimension than were users of twin beds. Sev-

eral concerns were raised about the noise of the engine,the maximal and minimal heights, the tilting angles and

.. ----- ~-

I

1

L. Del1lt'r,f et Ill, I QUEST ~.(): :\11oven'iell' and ream progress

the overall resistance of the beds. The QUEST 2.0 hasbeen released recently and more outcome studies areexpected to be published in the coming years.

7. Discussion

The QUEST 2.0 is a pioneer scale for satisfactionmeasurement. It was developed in response to the lastdecade's wake-up call for outcome measurements inthe field of rehabilitation assistive technology. As ageneric assessment, it covers both Device and Servicescomponents of AT. Thus, the items may be applied to awide range of devices. On the other hand, some poten-tial items, relevant to a specific pieces of technology ordelivery systems may be absent. Speed, for instance,is subsumed within the item effectiveness although itmay be considered an important aspect of a poweredwheelchair's performance. Users of the tool are invitedto add a few items of their own, using the 5-point sat-isfaction rating scale and a similar format. However,such data cannot be included in the Device, Service andtotal QUEST scores; they should be added at the endof the questionnaire and analysed separately.

Most studies about the psychometric properties ofthe tool have been conducted by the team of developersthemselves. Nevertheless, the scales and subscales ap-pear to be adequate with respect to test-retest stability,alternate-form equivalence, and internal consistency.More reliability tests for the 4-item Services subscalehowever need to be conducted. Construct validity hasbeen studied through factorial composition and nomo-logical relatedness (expected correlation between mea-sures of related concepts). Although not extensive, va-lidity results are convergent and support the adequacyof the satisfaction measure. Severed outcome studieswere conducted using the first version of the tool. TheQUEST 2.0 has better measurement properties and isan important improvement to the previous version. Itallows both item and test levels results and should be-come a valuable tool to enhance most studies concernedwith consumer satisfaction.

References

[I] S. Bond and L.H. Thomas, Me.asuring patients' satisfactionwith nursing care, Journal of Advanced Nursing 17 (1992),52-63.

[2] R.A. Carr-Hill, The measurement of patient satisfaction, Jour-nal of Public Health Medicine 14 (1992), 236-249.

[3] H. Vuori, Patient satisfaction - Does it matter? Qualit)' Assur-ance In Health Care 3 (1991). 183-189.

~-- - --.---..

!O5

[4] S.E. Simon and A. Patrick. Understanding and assessing con-sumer satisfaction in rehabilitation, Jollmal 4 RehabilitationOutcomes Measurement 1(1997). 1-14.

[5] L. Demers et aI., Development of the Quebec User Evaluationof Satisfaction with Assistive Technology (QUEST), AssistiveTechnology 8 (1996), 3-13.

[6] M.J. Scherer, Li,'ing in tl1,' slIIte ofstuck-HolI' teelmolog." im-pactthe !i\'es of people with disabilities, 2nd ed.. Cambridge:Brookline. 1996.

[7] L. Demers et al., An international content validation of theQuebec User Evaluation of Satisfaction with assistive Tech-

nology (QUEST), Occupational Therapy International 6(3)(1999),159-175.

[8] L. Demers et al., Stability and reproducibility of the Que-bec User Evaluation of Satisfaction with assistive Technology(QUEST), Journal of Rehabilitation Olllcomes Measurement3(4) (1999), 42-52.

[9] L. Demers et al" Item analysis of the Quebec User Evaluation

of Satisfaction with assistive Tec~ology (QUEST), AssistiveTechnology 12(2) (2000), 96-105.

[10] L. Demers et al., Reliability, validity, and applicability of theQuebec User Evaluation of Satisfaction with assistive Tech-

nology (QUEST 2.0) for adults with Multiple Sclerosis, Dis-ability and Rehabilitation 24 (2002), 21-30.

[II] L. Demers et al., Quebec User Evaluation of Satisfaction

with assistive Technology QUEST 2.0 - An outcome measurefor assistive technology devices, Webster (NY): Institute forMatching Person and Technology, 2000.

[12] L. Demers et al., Key dimensions of client satisfaction with as-sistive technology: A cross-validation of a Canadian measurein the Netherlands, Scandinavian Journal of RehabilitationMedicine 33 (2001), 1-5.

[13] H. Day and 1. Jutai, Measuring the Psychosocial Impact ofAssistive Devices: The PIADS, Canadian Journal of Reha-bilitation 9 (1996), 159-168.

[14] R.D, Wessels et al" A Dutch version of QUEST (D-QUEST)

applied as a routine follow-up within the service delivery pro-cess, in: Improving the quality of life for the European Citizen,I. Placiencia Porrero and E, Ballabio, eds, Amsterdam: IOSPress, 420-424, 1998.

[15] A. Brandt and S. Iwarsson, Do certain groups of older people

benefit the most from the use of powered wheelchairs? in:Resna Proceedings 2001, J. Winters, ed., Arlington: ResnaPress, 2001, pp. 212-214,

[16] Benedict et al., Assistive devices as an early childhood in-tervention: Evaluating outcomes. Technology and Disability11(1/2) (1999), 79-90.

[17] R. Weiss-Lambrou et al., Wheelchair seating aids: How satis-

fied are consumers? Assistive Technology 11 (1999),43-53.[18] T. Bursick et al., Wheelchair seating and positioning outcomes

in the elderly nursing home population, in: Resna Proceedings2000, J. Winters, ed., Arlington: Resna Press, 2000, pp. 316-318.

[19] F. Routhier et al., Clinical results of an investigation of pae-diatric limb myoelectric prosthesis fitting at the Quebec Re-habilitation Institute, Prosthetics and Orthotics International25(2) (2001), 119-131.

[20] S. Stickel et al., Toward a comprehensive evaluation of the im-pact of electronic aids to daily living: Evaluation of Consumersatisfaction, Disability and Rehabilitation (in press).

[21] C. Vincent and L. Demers, Les lits electriques 11domicile?Pouvons-nous dormir sur nos deux ore illes, OccupationalTherapy Now 4(2) (2002), 11-14.