Qmc

FI

a

A

R

R

1

A

K

E

U

P

Q

M

W

S

C

DS

1d

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 1 ( 2 0 1 2 ) 303–313

j ourna l homepage: www.i jmi journa l .com

ualitative and quantitative evaluation of EHR-integratedobile patient questionnaires regarding usability and

ost-efficiency

leur Fritz ∗, Sebastian Balhorn, Markus Riek, Bernhard Breil, Martin Dugasnstitute of Medical Informatics, University Münster, Albert-Schweitzer-Campus 1/A11, 48149 Münster, Germany

r t i c l e i n f o

rticle history:

eceived 19 August 2011

eceived in revised form

5 December 2011

ccepted 16 December 2011

eywords:

valuation

sability

atient reported outcomes

uality of life

obile devices

eb-based documentation

ingle source

linical information system

a b s t r a c t

Objectives: The objective of this evaluation study is to assess a web-based application, cur-

rently available on iPad, to document questionnaires regarding patient reported outcomes

such as quality of life. Based on the single source approach, the results of these question-

naires are available in the electronic health record to be used for treatment and research

purposes. The assessment focuses on the usability and efficiency of the system.

Methods: The system usability scale questionnaire with seven additional items was used

to rate the usability by the patients. It was formally validated by a Cronbach Alpha test.

In addition, semi-structured interviews were conducted with patients and medical staff.

Time and cost measures, based on official tables of costs, were taken through workflow

observations. This study was conducted in the department of dermatology at the University

Hospital of Münster, Germany from April to June 2011.

Results: Using the web-based application questionnaire, results about patient reported out-

comes like quality of life are immediately available in the electronic health record and can

be used for treatment or research purposes. 118 patients and four staff members partici-

pated in the study. The usability score reached 80 from 100 points and patients as well as

medical staff stated in the interviews that the usability of the web-based system was high,

and they preferred it to the previously used paper-based questionnaires.

In the setting of our pilot department the mobile devices amortized their costs after 6.7

months. In general, depending on the professional group who are going to post process the

paper-based forms, the earliest break-even point to use mobile questionnaires is at 1737

paper sheets per year.

Conclusion: The mobile patient questionnaires, integrated into the electronic health record,

were well accepted in our pilot setting with high usability scores from patients and medical

staff alike. The system has also proved to be cost-efficient compared to the paper-based

workflow, given that a certain number of questionnaires is used per year.

© 2011 Elsevier Ireland Ltd. All rights reserved.

Abbreviations: CCP, Competence Centre for the Diagnosis and Treatment of Chronic Pruritus; CIS, Clinical information system; DLQI,ermatology Life Quality Index; EHR, Electronic health record; EPC, Event-driven process chain; HADS, Hospital Anxiety and Depressioncale; PBI, Patient Benefit Index; PRO, Patient reported outcome; SUS, System usability scale.∗ Corresponding author. Tel.: +49 251 8355265; fax: +49 251 8355277.

E-mail addresses: Fleur.Fritz@uni-muenster.de, Fleur.Fritz@ukmuenster.de, fleurfritz@gmx.de (F. Fritz).386-5056/$ – see front matter © 2011 Elsevier Ireland Ltd. All rights reserved.oi:10.1016/j.ijmedinf.2011.12.008

i c a l

quasi-experimental field study with an observational charac-ter [22], which was conducted by a neutral evaluator to avoid

304 i n t e r n a t i o n a l j o u r n a l o f m e d

1. Introduction

1.1. Scientific background

Patient reported outcomes (PRO), such as quality of life ques-tionnaires, are completed by patients to assess the influenceof their medical condition on their lives. They are an impor-tant instrument to express the patient’s voice [1] and improvethe communication between physicians and patients [2]. Theresulting information is not only valuable for treatment, butalso for research, as Miguel et al. highly recommended fordrug clinical trials recently [3]. According to the single sourceapproach, medical data, needed for more than one purpose,should be collected only once [4,5]. This implies to makethe data electronically available. As patients usually com-plete the questionnaires on paper, the medical staff needs totransfer the information manually into the electronic healthrecord (EHR). As documentation work already makes up alarge amount of the available work time [6,7], PRO question-naires are rarely used outside those clinical studies in whichthey are mandatory. Both Snyder and Chang analysed thisproblem of “logistics” and emphasized the need of efficientmethods to capture PRO data [8,9]. To overcome these obsta-cles and offer an easy-to-use solution, which also makes thedata available in the EHR, we developed a web-based appli-cation to enable questionnaire completion and data transfer[10]. Mobile solutions for healthcare tasks have been discussedsince the early nineties [11], but are currently getting increasedattention. Similar solutions focus on comprehensive question-naire applications [12]. In our solution, however, we focusedon capturing data from patients and make them available inthe EHR according to the single source approach.

1.2. Rationale for the study

A previous implementation project demonstrated the feasi-bility of the single source approach for EHR-integrated mobilequestionnaires. In this study we intended to evaluate theusability and cost-efficiency of the system to determine theconditions on which it can be used in the long run. Thus stake-holders of this study are clinical staff, medical informaticsscientists and chief financial officers.

1.3. Objectives of the study

The objective of this study was to assess the web-based appli-cation for patient questionnaires as a method of collecting PROin a single source approach. In detail we defined to evaluate:

• How do patients and clinical staff rate the usability of theweb-based application?

• How cost-efficient is the mobile documentation workflowcompared to the paper-based approach?

This study has been approved by the local ethics committee(reference number 2011-290-f-S) [13].

i n f o r m a t i c s 8 1 ( 2 0 1 2 ) 303–313

2. Study content

2.1. Organizational setting

The Competence Centre for the Diagnosis and Treatment ofChronic Pruritus (CCP) of the University Hospital Münster[14] functioned as pilot department of the mobile patientquestionnaires and for this evaluation study respectively.Approximately 2000 outpatients and inpatients are treatedyearly by two physicians and four other medical and adminis-trative staff members. The CCP is one of the leading institutesfor the research of chronic pruritus on a national and interna-tional level [15,16].

2.2. System details and system in use

The evaluated system is an in-house developed web-application to document patient questionnaires with aninterface to a commercial clinical information system (CIS),in order to import PRO data into the EHR and make themavailable for reuse [10]. Currently the following three patientquestionnaires with similar data types (answers as radio-buttons) are at hand: the Dermatology Life Quality Index(DLQI) [17], the Hospital Anxiety and Depression Scale (HADS)[18] and the Patient Benefit Index (PBI) [19]. The applicationwas tested on Apple’s iPad [20]. After the patient identifica-tion details have been inserted by a staff member, the mobiledevice is handed to the patient, who completes the cho-sen questionnaires. One question is displayed per screen andthe system checks upon completion whether all necessaryquestions were answered. If not, the patient may choose tocomplete missing items or to abort the session. No other func-tionality is offered to the patient in order to avoid confusionof inexperienced users. After the completion of the question-naires, an XML file is generated automatically and importedinto the corresponding EHR forms within the CIS. From thereit can be used for treatment, research and further purposes.

The web-based application system has been in routine usein the CCP since the beginning of 2011 and is also tested forthe oncology clinics at the University Hospital in Münster.To satisfy the law of data protection, the application systemhas been approved by the local data protection officer. Theused devices were also tested and approved by the Institute ofHygiene of the University Hospital Münster [21].

3. Methods

3.1. Study design

Since the objective of the study has been a comparisonof usability and cost-efficiency of mobile patient ques-tionnaires to the paper-based version, we performed a

a bias in the study. Regarding the staff evaluation there mightbe a bias as the CCP as a pilot department is open minded to

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 1 ( 2 0 1 2 ) 303–313 305

Fig. 1 – Gant diagram of the study flow during the 3-month-evaluation period. After starting the mobile questionnaires, theworkflows were analysed, time measures were taken and the usability questionnaires were completed by the users. In thef eva

htwmwtaarg

3

PpttTet

3

Tholcuwnwiiwt

ollowing users were interviewed and all collected data were

ealth-care IT and encourages patients to use the new sys-em. Because pre-/post studies have various sources of bias,e set up a control group during the evaluation period. Theethods used, were of quantitative and qualitative characterith time and cost measures, workflow analysis, an evalua-

ion questionnaire (system usability scale (SUS), see below)nd semi-structured interviews [23,24]. The chosen methodsre based on an analysis of similar evaluation studies andecommendations like those stated above and the STARE-HIuidelines [25] respectively.

.2. Participants

atients and staff members of our pilot department, the CCP,articipated in this evaluation study. A group of patients forhe evaluation questionnaire on the mobile devices and theime measures were analysed over a period of several weeks.he control group, following the paper-based workflow, wasstablished by randomly switching off the web-based applica-ion for a few days.

.3. Study flow

he evaluation started in April 2011 after the approval of theardware and peripheral devices by the Institute for Hygienef the University of Münster and the ethical approval of the

ocal ethics committee. The first step was the assessment ofompliance to usability standards (e.g. ISO) with respect to theser interface. In the following observation period of one week,orkflows of the mobile and paper-based patient question-aires in the CCP were analysed and visualized. Based on theseell-defined and illustrated workflow processes, we selected

ndicators for time measurements, which were captured dur-ng a timeframe of 2 weeks. After that the SUS questionnaireas added to the existing mobile patient questionnaires and

he answers were recorded for 5 weeks until mid-June. Shortly

luated.

before and within this period of time we also interviewedparticipating staff members as well as some patients. Anoverview of the whole study flow is provided in Fig. 1.

3.4. Outcome measures and evaluation criteria

The following factors were considered to achieve a compre-hensive view through this evaluation:

1. Front-end usability and user satisfaction of the mobilepatient questionnaires including the hardware by patientsand staff.

2. A monetary analysis and comparison of the mobile to thepaper-based questionnaires, including the time needed forpost processing to have the data available in the EHR.

3. Completion time of mobile and paper-based question-naires.

4. Data quality of mobile questionnaires compared to paper-based questionnaires with regard to errors in data transferand missing answers.

5. Random observation of system stability.

3.5. Methods for data acquisition and measurement

To evaluate the software usability, a German translation ofthe System Usability Scale (SUS) [26] was used. The ten-itemquestionnaire is based on a five-point Likert scale [27] andproduces a maximum score of 100 with respect to the users’impression of the application. We chose the SUS because of itsshortness and the understandable phrasing of questions evenfor non-IT skilled persons like many patients, especially olderones. Despite the relatively small number of questions the SUS

has a reasonable reliability with a Cronbach’s alpha of at least0.85 [28]. Seven questions of our own were added to assessgeneral patient impressions on completion time, patient out-come, use of computers and the preference of mobile patient

306 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 1 ( 2 0 1 2 ) 303–313

Table 1 – Personnel expenditures for three different job categories in Euro per hour.

Over 400D Under 400D Average

Student assistant 9.90

Scientific assistant 17.02

Medical documentation assistant –

questionnaires in a hospital. The results of our modified SUSquestionnaire showed a Cronbach Alpha of 0.841 for reliabil-ity. The complete questionnaire can be found in Supplement1.

To validate the criticisms of the users, gained by the SUS,we observed the workflow processes and performed semi-structured face-to-face interviews with patients and staff assuggested by Oschem et al. [23]. These interview questions canbe found in Supplement 2.

Mobile and paper-based workflows in the outpatient clinicof the CCP were observed and analysed by an experienced yetneutral person. Based on these results we chose indicatorsfor time measures especially to compare the costs of the twomethods:

1. Pre and post processing time of the paper-based question-naires needed by the staff.

2. Completion time of the patient questionnaires using thepaper-based version versus a mobile device.

Measurements were done by open observation of thepatients who completed the paper-based forms in theoutpatient clinic. Completion times of the mobile patientquestionnaires were taken from system log files. The costsof material acquisition and maintenance as well as person-nel expenditures were collected through the administrativeoffices of the hospital. We based the calculation on theassumption that basic IT infrastructure, like printers and WiFi,already exist. Printing costs then include the purchase priceof paper and staples as well as the price per page for tonerand average electricity costs on the basis of industrial rates(D0.086/kwh in 2010) for a common printer in the hospital(900 W, 20 pages per minute). The same electricity costs applyfor the charging of the mobile devices (a 10-W iPad, charged7 h per working day).

To determine the different personnel expenditures for thepost processing of the paper-based questionnaires, we aver-aged hourly costs for student and scientific assistants takinginto account the 400-Euro-threshold, defined as mini-jobs inGermany. This threshold means that the employee pays noincome tax and social security contributions but a flat chargeis laid upon the employer, which consequently leads to dif-ferent costs [29,30]. We also calculated the costs per hour of amedical documentation assistant, who may handle the postprocessing, by averaging the German tariff T7–T10 includingspecial payments within the year of the labour agreementof 2010 [31]. The calculated results of the hourly personnelexpenditures are shown in Table 1.

3.6. Methods for data analysis

Microsoft Excel [32] was used for cost and personnel expendi-ture estimations as well as for diagrams. Cronbach Alpha of

11.52 10.7118.24 17.63

– 24.90

the SUS questionnaire was calculated with IBM SPSS Statistics[33]. SPSS was also used to calculate descriptive data analyseslike time averages and their standard error of measurement(SEM). The workflow was illustrated as an event-driven pro-cess chain (EPC) with Microsoft Visio [34].

Interviews were tape-recorded and then manually relatedto the findings of the quantitative data, namely usability scaleand time measures.

4. Results

4.1. Demographic and other study coverage data

The participating staff members consisted of the medical headof the CCP and one further dermatologist, both taking care ofthe treatment of patients. Furthermore one study nurse anda non-medical scientific assistant, being responsible for thepre and post processing of the patient questionnaires, par-ticipated. All four staff members are female and have beeninterviewed face-to-face.

The mobile patient questionnaire user group comprised118 patients, who were given the possibility to participate inthe evaluation. 111 patients completed the SUS questionnaire,which results in a return rate of 94%. Our additional questionswere answered by 84 patients which equals a return rate of71%. The participants were between 15 and 86 years old (aver-age age of 56.5 years with standard deviation (SD) 16.2 years,SEM 1.5 years) and were almost equal in distribution of sex:50.5% were male and 49.5% female. Time measures regardingthe completion of the three different PRO questionnaires weretaken of those 118 users. The control group for time measure-ments regarding the paper-based questionnaires was smallerand included 20 patients. The average age of the control groupwas 55.6 years (SD 16.5 years, SEM 3.6), the distribution ofsex differed from the first group with 31.6% male and 68.4%female. Three randomly selected patients were interviewedface-to-face.

We analysed 100 completed paper-based questionnairesof different patients with respect to data quality (number ofmissing or wrong answers, e.g. double-checked answers).

After the observation period both workflows were illus-trated as an EPC and can be found as background informationin Supplement 3.

4.2. Study findings and outcome data

4.2.1. Usability – SUS and programme designThe software interface was designed following the relevantrequirements of the ISO 9241-11 standard [35–37]. The high age

of the patients and the fact that many may not be experiencedwith the use of computers was also taken into consideration.Therefore, the graphical user interface was kept simple andconsistent throughout the entire questionnaire. Eye-catching

a l i n

cbnnwi

saBs4

4teom

4RqmC

w(pail

Ft

i n t e r n a t i o n a l j o u r n a l o f m e d i c

olours were used for the buttons, user actions are confirmedy visual signals, only one question per screen is displayed ando further functionality is possible so that the patient doesot get confused or has to scroll. In addition a big font sizeas chosen, which was appreciated by most patients in the

nterviews.The evaluation of the completed SUS questionnaires

howed an overall score of 80.34 from a maximum of 100 with SEM of 1.37 and a standard deviation of 14.39. According toangor et al. this equals a score for better products [31]. Theingle evaluation of all questions can be found in Supplement.

The average scores (from 0 = strongly disagree to = strongly agree) from our additional evaluation ques-ions are shown in Fig. 2. It becomes evident that the patients,ven though they were not familiar with computers, werepen-minded and willing to support the new system ofobile patient questionnaires.

.2.2. Cost-efficiency analysisegarding the monetary comparison of the mobile patientuestionnaires and the paper-based version, we determinedaterial and personnel costs, based on the pilot setting in the

CP, and generalized them in a second step.The printing costs are D0.0177 per paper sheet. Multiplied

ith the number of sheets for all three questionnaires in use4) in the CCP and adding stapling costs (D0.004), the total sum

er patient is D0.075. The CCP has consultation hours on 4 days

week with an average of 10 patients per day. Taking holidaysnto account this adds up to 2020 patients per year, whicheads to total costs of D151.50 per year for the paper-based

ig. 2 – Averaged scores on the additional seven questions askedhe mobile questionnaires.

Table 2 – Time measures in minutes of post processing paper-bconsists of both data entry times. For the total 25 s are added, w

HADS questionnaire DL

Arithmetic mean 00:46

Median 00:44

Standard deviation 00:11

f o r m a t i c s 8 1 ( 2 0 1 2 ) 303–313 307

questionnaires. We did not include the costs for ball pens(D15.00 per month) as they are also used for other tasks.

As hardware for the mobile patient questionnaires threeApple iPads (16 GB with UMTS) with a retail price of D599.00are used in the CCP. All iPads are protected by an “iSkin” case(D40.00) and an “invisible Shield” display protection (D20.00).Moreover, capacitive pens (D25.00) for data entry are availableto the patients. Hospitals in Germany are not entitled for inputtax deduction for items used in treatments [38], thus the totalexpenditures add up to D684 for one mobile device includingequipment. For a depreciation time we chose a usage periodof three years [39,40], resulting in D228.00 per year. Includingyearly electricity costs of D2.12 to charge the iPads, the esti-mated total costs per device per year are D230.12. As the CCPuses three iPads there is a total expenditure of D690.36 peryear.

The average time needed for post processing the paper-based questionnaires is provided in Table 2 (based onobservations with n = 20). The time needed to file these ques-tionnaire sheets into the patient record was 25 s. Thus theaverage post processing time adds up to 1:49 min for HADS andDLQI questionnaires. During our evaluation period the data ofthe PBI was not entered into the EHR. Consequently, it is notlisted here.

For 2020 patients per year a total of 61.5 h are spent on thepost processing of the two paper-based questionnaires. Mul-tiplied with the personnel expenditures calculated above, thecosts are D658.00 for a student assistant, D1083.00 for a sci-

entific assistant and D1530.00 for a medical documentationspecialist. Adding material costs of D151.50, the total costsof the paper-based workflow per year for the use-case in theCCP can be seen in Fig. 3. In comparison the costs of D690.36

to the patients, which show an overall satisfaction with

ased questionnaires (HADS and DLQI) in the CCP. The sumhich are needed to file the forms into the patient record.

QI questionnaire Sum Total (+0:25)

00:41 01:24 1:4900:39 01:27 1:5200:08 00:25 0:25

308 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 1 ( 2 0 1 2 ) 303–313

Fig. 3 – Comparison of total expenditures per year in Euro in the CCP (Competence Centre Pruritus). The three bars on theleft show material and personnel costs for the paper-based workflow. The one bar on the right shows the costs of the

e de

mobile patient questionnaires, based on having three mobil

for using the three mobile devices for the questionnaires areshown.

The figure shows that the costs of the mobile question-naires are lower than those of the paper-based version for allthree staff groups. Without adding any further costs, the CCPeven has the PBI questionnaire available in the EHR, using themobile workflow, which was not the case in the paper-basedworkflow. As in the CCP the paper-based questionnaires arepost processed by a scientific assistant, the iPads amortizeafter 6.7 months.

To generalise the costs for other departments the totalexpenditures from above were split into average costs perquestionnaire sheet as shown in Table 3.

With these numbers a break-even analysis, shown in Fig. 4,was performed. The analysis represents the point at which acertain number of paper-based questionnaires equal the costsfor one iPad under the assumption that one questionnaireconsists of one sheet and the devices are used for 3 years.

The diagram shows for example that 3802 sheets postprocessed by a student assistant or 1737 by a medical docu-mentation specialist come up to the same yearly costs as onemobile device.

4.2.3. Completion time of questionnairesThe average completion time of three questionnaires onthe mobile device (n = 118) was 17:18 min (SD = 9:22 min

and SEM = 0:53 minutes); on paper it was 8:22 min (n = 20;SD = 2:55 min and SEM = 0:53 min). However, in the interviewsthe patients stated that they did not need more time using themobile patient questionnaires compared to the paper forms.

Table 3 – Total costs per paper questionnaire sheet in Euro for t

Student assistant Scie

Personnel costs 0.16 0.27

Material costs 0.02 0.02

Total 0.18 0.29

vices which are depreciated over 3 years.

In the usability questionnaire the patients negated (score 0.8out of 4) that the use of the mobile patient questionnaire tookmore time than the paper questionnaires. Even the staff mem-bers reported that they hat not noticed any time increase.

4.2.4. Assessment of data qualityIn the reviewed paper questionnaires (n = 100) we found thatthree questionnaires were completed falsely due to missing ordouble answers. Furthermore, in 14 questionnaires the samespecific question in the DLQI was missing or answered incor-rectly. During the observation of 20 data transfers into the HISno error occurred.

In the mobile patient questionnaires no errors in datatransfer or missing answers were found as this is preventedtechnically.

4.2.5. System stabilityDuring the observation we found that the WiFi signal was notvery strong and led to instabilities, which caused long load-ing periods and in the beginning even system crashes everyfew days, so that the questionnaire had to be restarted. Withan installation of a further WiFi access point during the eval-uation phase, the signal quality could be enhanced and theproblem, mentioned above, disappeared.

A critical point regarding hardware was the home-buttonon the left edge of the iPad. By accidentally pressing it with

the left hand, holding the iPad, the browser closed and somepatients thought that the questionnaire was finished andhence the staff did not know that it was not completed. In thebeginning this happened once a day but was minimized during

hree different professional groups.

ntific assistant Medical documentation assistant

0.380.020.40

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 1 ( 2 0 1 2 ) 303–313 309

Fig. 4 – Break even analysis of mobile patient questionnaires versus cumulated costs of paper sheets. The point ofi being

thctp

uwra

4Toiitadoicdhot

at

4

Taww

ntersection marks the costs for a certain number of sheets

he evaluation phase by handing out the tablet with theome-button facing on the right. In addition, the staff nowontrols that the login screen is displayed after the return ofhe iPad to ensure that the questionnaires have been com-leted.

The battery runtime was no critical point during the eval-ation phase. It lasted up to 8 h, which is sufficient for aorkday. According to Apple their battery “is designed to

etain up to 80% of its original capacity at 1000 full chargend discharge cycles” [20].

.2.6. Patient outcome and further findingshe interviewed physicians stated that from a medical pointf view the immediate availability of PRO scores in the EHR

s an important advantage. Critical scores are visually markedn the EHR forms and a link for a psychosomatic consulta-ion is provided for the treating physician. Although not beingnalysed in this study, we noticed that medication is chosenifferently if it is known that the patient has critical depressionr anxiety scores. With the paper-based questionnaires such

nformation is only available after the treatment. Thus physi-ians expect that the clinical outcome for the patients profitsirectly from the immediate data availability. Not all patients,owever, thought that the questionnaires had a direct impactn their treatment as is reflected by a score of 2.85 out of 4 onhis question and patient statements during the interviews.

The (re)use of PRO data, being available in the EHR, haslready been described in a previous paper [10] and substan-iated during this evaluation.

.3. Unexpected observations

here were two unexpected observations during the evalu-tion. The first occurred during the analysis of the clinicalorkflow, where a bottleneck in the administration officeas discovered. Long waiting times during the admission of

equal to the costs of an iPad-based system.

the patient led to a delay in using the mobile patient ques-tionnaires, because patient case IDs are needed for patientidentification during the data transfer into the EHR forms.

The second one became apparent during the observationof patient reactions, while completing the mobile question-naires and substantiated in the patient interviews later on.Patients reported that on the mobile device questions wereeasier to understand and to answer than on the paper forms.Apparently, this especially applied for those questions havingnegations and for those patients not having German as theirnative language.

5. Discussion

5.1. Answers to study questions

In this study we wanted to assess the usability of a web-basedapplication to capture patient questionnaires and its cost-effectiveness in consideration of the single source approach.

SUS scores and comments from the interviews show a highsatisfaction with the system. Patients stated that “[. . .] it ismuch better to read and clearly arranged” and “[. . .] the big let-ters help to clearly understand the questions, especially thosewith negations”. Medical staff members also emphasized that“[. . .] the questionnaires are self-explanatory and the menunavigation is very good. Everything is well readable and alsoconveniently to handle”.

The system also proved to be cost effective as it paid-offquite early compared to the paper-based workflow in whichdata needed to be transferred manually.

As double answers and mistakes during data input

are prevented technically and non-answered questions arerepeated at the end of the questionnaire, the electronic sys-tem increases data quality. In pseudonymized format thisdata is very valuable for research questions. The immediate

i c a l

310 i n t e r n a t i o n a l j o u r n a l o f m e d

availability for treatment was mentioned positively during thestaff interviews. One physician said that “[. . .] it has a sig-nificant influence on the treatment. For example, when I seethat a patient has a high depression score, I choose a differ-ent medication. This is not the case, when I use the paperbased version”. It became evident during our study that datacaptured once, can be used for different purposes, whichclearly reflects the idea of the single source approach.

The increased documentation time of the patients seemedto be of no importance, as in all cases waiting times werelonger than documentation time. Furthermore, although thecompletion time was objectively longer, the subjective impres-sion of the users was the opposite, and they stated that it hadnot taken longer than on paper forms. We are not sure whetherthe documentation time was prolonged through other activ-ities like coffee or toilet breaks, which might be avoidable. Inany case, if the mobile patient questionnaires should be usedin time critical processes in more fast-moving clinical settings,these times must be considered as a potential obstacle. Fur-thermore, it can affect the number of mobile devices neededto guarantee that waiting times do not increase because ofpotential bottlenecks in devices.

5.2. Strengths and weaknesses of the study

The patient population in the study covered different ageranges on the one side, but was equal in the distribu-tion of sex on the other side. As our return rate for theSUS was very high, the achieved score can be seen as rep-resentative for this kind of patient population. It has tobe kept in mind, though, that the study was conductedonly in one department. As a pilot department the staffwas open-minded to use healthcare IT. Because of the lownumber of staff members, the interviews, taken there, canonly reflect the perception of this specific department. Inaddition, the interview questions were not validated andtherefore staff and patient citations reflect only the moodamong these user groups. Still we think that the resultscan be applied to other patient populations, also taking intoaccount Nielsen’s findings that small study groups can pro-vide enough usability information [41]. This weakness is alsopartly compensated through the different methods we usedin this evaluation study. A mix of quantitative and qualita-tive instruments like observation, standardized questionnaireand semi-structured interviews were performed to get a broadpicture.

The cost calculations for paper sheets and staples in thisstudy were based on the average costs of the used question-naires in our pilot department. However, other questionnairesmay have more pages, a one-sheet-questionnaire does notneed staples and other hospitals may have different whole-sale prices. We also assumed that paper-based questionnairesare transferred manually into the EHR and filed into the paperrecord, which is reflected by the post processing costs. Nev-ertheless, after talking with other potential users, we thinkthat our analysis reflects a good average and can be general-

ized.

No personnel expenditures for iPad usage were takeninto account, because during the workflow observation wefound out that it equals the paper-based approach. In both

i n f o r m a t i c s 8 1 ( 2 0 1 2 ) 303–313

cases either paper or a device needs to be handed tothe patient and the questionnaires have to be explained.The time needed to insert the patient ID into the mobilequestionnaire equals the time needed to stick the patientID label onto the paper form. Therefore, the expendituresfor the mobile workflow only comprised the costs for theiPads.

5.3. Results in relation to other studies

Many usability studies regarding mobile devices in healthcarefocused on the use of personal digital assistants by medicalstaff members and not by patients. For example, Fischer in2003 reviewed the use of mobile devices [11] and Free pro-vided a review protocol to do so in 2010 [42]. The use-case ofcapturing PRO was not part of these studies.

Different and new methodologies of collecting PRO are dis-cussed by Rose et al. [43]. They mainly rated characteristicsof available methods, provided examples and focused on thelevel of support needed. It is pointed out that the processshould not interfere with the usual clinical workflow and thatusers need to know the purpose of the PRO collection. Withour experience we can fully support this important point,which needs to be considered when implementing web-basedpatient questionnaires.

Recent studies with respect to specific modes of PRO col-lection were done by Holzner [44], Boissy [45], Fromme [46]and Lee [47]. Holzner et al. present a dedicated PRO databasewith many useful functionalities. However, it is not interfacedwith the EHR and no usability evaluation was done. Boissyet al. evaluated the usability of a barcode scanner/personaldigital assistants combination to document PRO. Like in ourstudy they experienced longer documentation times but moresatisfied users. However, also in this study there was noexplicit transfer to the EHR in order to (re)use data. Frommeet al. especially focused on the ease of use for older patients,using tablet PCs for PRO documentation. Although ease ofuse was lower than with younger patients, the overall rat-ing of the application was positive. Neither a data transferto the EHR nor the evaluation of the involved costs wereconsidered. Lee et al. provide no full evaluation study, butrather a review paper, in which aspects of data quality,costs, patients’ preferences and validity of questionnaires areassessed with respect to electronic PRO documentation. Theycome to similar results, as they found that the question-naires were easy to use, had less long-term costs, providedgood data quality and had the advantage of an immedi-ate availability of the results and the possibility of reusingthem.

Common in all those studies and our study is the need ofefficient methods to collect PRO data and the positive resultsregarding different usability aspects. The novelty and benefitfor physicians and patients in our system is the single sourceapproach. We did not aim at developing yet another stan-dalone system but a possibility to capture PRO data, which can

then be combined with other EHR data and be reused for dif-ferent purposes. Furthermore, our findings combine not onlythe usability but also the cost-effectiveness of the system andthus provide a practical recommendation for implementation.

a l i n f o r m a t i c s 8 1 ( 2 0 1 2 ) 303–313 311

5

Asoidodt

iacthwu

5

Psoiuiqs

fp

eottr

ift

e

6

Iortesm

A

Fg

Summary pointsWhat was already known on the topic:

• It is important to collect patient reported outcomesand thus involve the patients’ voice in the treatment.

• Mobile devices can be used in various clinicalsettings, but logistical problems for patient self-documentations still exist.

• A single source approach to document patient reportedoutcomes once and then use it for different purposesis feasible.

What this study added to our knowledge:

• Patients are highly satisfied to use mobile documenta-tion instead of paper forms.

• Mobile patient questionnaires are cost-efficient if usedregularly.

• For physicians the main advantage of our methodto capture patient reported outcomes is the EHR-integration, from where the data can immediately beused during the treatment and later on for researchquestions.

r

i n t e r n a t i o n a l j o u r n a l o f m e d i c

.4. Meaning and generalizability of the study

s discussed above, our findings are in line with similartudies. They are useful for physicians looking for meth-ds of documenting PRO for treatment and research, taking

nto account economical questions on whether to use mobileevices or not. The generalizability was also emphasized byne of our physicians who stated that “I can imagine that otherepartments with different questionnaires could benefit fromhe mobile patient questionnaires. . .”.

Our study was solely based on Apple’s iPad, but afternternal analyses of other mobile devices, we did not findny fundamental differences. Important characteristics whenhoosing the device are price, appropriate functionality, bat-ery runtime and size of screen. Additionally the device shouldave the possibility to scan patient barcodes for identification,hich was not possible with the first generation of iPads, wesed.

.5. Unanswered and new questions

atients reported that the questions were easier to under-tand and to answer by using the mobile device insteadf the paper form. In a further study we could therefore

nvestigate whether patients choose different answers, whensing mobile patient questionnaires. However, previous stud-

es assessed the differences in electronic vs. paper-baseduestionnaires [48] and thus it would only be a validation ofome specific questionnaires.

The piloted questionnaires only consisted of radio-buttonsor data input. We could further evaluate whether more com-lex data types or even a free text answer would be feasible.

Our system proved to have a good usability and cost-ffectiveness for clinics documenting PRO data. It would bef even more value if the study increased the documenta-ion of PRO data and led to more clinics doing so. To assesshe intensity of the stimulus would be interesting for furtheresearch.

Another research topic, following this study, is the med-cal usefulness and the effects on the patient’s outcome if,or example, quality of life information is available during thereatment or if physicians use PRO data for their research.

Based on our results a broad dissemination in our andxternal hospitals is planned.

. Conclusion

t is worthwhile and efficient to document patient reportedutcomes by using a web-based application that transfersesulting data into the EHR from where it can be used forreatment and research purposes. Good usability and costffectiveness of the system were proven by our evaluationtudy and the implementation in other clinics may be recom-ended.

uthors’ contributions

leur Fritz designed the system, implemented the CIS forms,uided the evaluation and wrote the manuscript. Sebastian

Balhorn performed data acquisition, observations, ques-tionnaires and interviews. Markus Riek implemented theweb-based application and provided logfile data. BernhardBreil assisted in system implementation and evaluation meth-ods. Martin Dugas supervised the evaluation. All authors readand approved the paper.

Competing interests

The authors declare that they have no competing interests.

Acknowledgments

Thanks to Prof. Dr. med. Sonja Ständer and her team of theCCP for participating in this evaluation study.

Appendix A. Supplementary data

Supplementary data associated with this article can be found,in the online version, at doi:10.1016/j.ijmedinf.2011.12.008.

e f e r e n c e s

[1] E. Basch, The missing voice of patients in drug-safetyreporting, N. Engl. J Med. 362 (10) (2010 March) 865–869.

[2] G. Velikova, A. Keding, C. Harley, K. Cocks, L. Booth, A.B.Smith, P. Wright, P.J. Selby, J.M. Brown, Patients report

improvements in continuity of care when quality of lifeassessments are used routinely in oncology practice:secondary outcomes of a randomised controlled trial, Eur. J.Cancer 46 (13) (2010 Sep) 2381–2388.

i c a l

312 i n t e r n a t i o n a l j o u r n a l o f m e d

[3] R.S. Miguel, A.M. López-González, E. Sanchez-Iriso, J. Mar,J.M. Cabasés, Measuring health-related quality of life in drugclinical trials: is it given due importance? Pharm. World Sci.30 (April (2)) (2008) 154–160.

[4] M. Dugas, B. Breil, V. Thiemann, J. Lechtenbörger, G. Vossen,Single source information systems to connect patient careand clinical research, Stud. Health Technol. Inform. 150(2009) 61–65.

[5] H.U. Prokosch, T. Ganslandt, Perspectives for medicalinformatics. Reusing the electronic medical record forclinical research, Methods Inf. Med. 48 (1) (2009) 38–44.

[6] E. Ammenwerth, H.P. Spötl, The time needed for clinicaldocumentation versus direct patient care. A work-samplinganalysis of physicians’ activities, Methods Inf. Med. 48 (2009)84–91.

[7] M. Tipping, V. Forth, K. O’Leary, D. Malkenson, D. Magill, K.Englert, M. Williams, Where did the day go? A time-motionstudy of hospitalists, J. Hosp. Med. 5 (6) (2010) 323–328.

[8] C.F. Snyder, N.K. Aaronson, Use of patient-reportedoutcomes in clinical practice, Lancet 374 (9687) (2009August) 369–370.

[9] C.H. Chang, Patient-reported outcomes measurement andmanagement with innovative methodologies andtechnologies, Qual. Life Res. 16 (Suppl. 1) (2007)157–166.

[10] F. Fritz, S. Stander, B. Breil, M. Riek, M. Dugas, CIS-basedregistration of quality of life in a single source approach,BMC Med. Inform. Decis. Mak. 11 (1) (2011 April) 26.

[11] S. Fischer, T.E. Stewart, S. Mehta, R. Wax, S.E. Lapinsky,Handheld computing in medicine, J. Am. Med. Inform.Assoc. 10 (2) (2003 March–April) 139–149.

[12] A. Rogausch, J. Sigle, A. Seibert, S. Thüring, M.M. Kochen, W.Himmel, Feasibility and acceptance of electronic quality oflife assessment in general practice: an implementationstudy, Health Qual. Life Outcomes 7 (2009 June) 51.

[13] Ethics committee Münster.http://campus.uni-muenster.de/ethik allgemeines.html (lastaccessed on 04.08.11).

[14] CCP: Competence Center for the Diagnosis and Therapy ofChronic Pruritus, University Hospital Münster, Germany.http://www.klinikum.uni-muenster.de/index.php?id=kompetenzzentrum pruritus#c4441 (last accessed on04.08.11).

[15] S. Ständer, E. Weisshaar, T. Mettang, J.C. Szepietowski, E.Carstens, A. Ikoma, N.V. Bergasa, U. Gieler, L. Misery, J.Wallengren, U. Darsow, M. Streit, D. Metze, T.A. Luger, M.W.Greaves, M. Schmelz, G. Yosipovitch, J.D. Bernhard, Clinicalclassification of itch: a position paper of the InternationalForum for the Study of Itch, Acta Derm. Venereol. 87 (4)(2007) 291–294.

[16] S. Ständer, I. Schäfer, N.Q. Phan, C. Blome, K. Herberger, H.Heigel, M. Augustin, Prevalence of chronic pruritus inGermany: results of a cross-sectional study in a sampleworking population of 11,730, Dermatology 221 (3) (2010)229–235.

[17] A.Y. Finlay, G.K. Khan, Dermatology Life Quality Index(DLQI): a simple practical measure for routine clinical use,Clin. Exp. Dermatol. 19 (1994) 210–216.

[18] C. Herrmann, U. Buss, R.P. Snaith, HADS-D: Hospital Anxietyand Depression Scale—Deutsche Version, Hans Huber, Bern,1995.

[19] C. Blome, M. Augustin, Evaluation des therapeutischenNutzens aus Patientensicht: Der Patient Benefit Index (PBI)als Beispiel für zielorientierte Präferenz- undOutcome-Messung, Gesundh. Ökon. Qual. Manag. 15 (2010)

236–240.

[20] Apple Inc., http://www.apple.com/ipad/ (last accessed on04.08.11).

i n f o r m a t i c s 8 1 ( 2 0 1 2 ) 303–313

[21] Institute for Hygiene, University Hospital Münster, Germany.http://klinikum.uni-muenster.de/index.php?id=hygiene start(last accessed on 04.08.11).

[22] A.D. Harris, J.C. McGregor, E.N. Perencevich, J.P. Furuno, J.Zhu, D.E. Peterson, J. Finkelstein, The use and interpretationof quasi-experimental studies in medical informatics, J. Am.Med. Inform. Assoc. 13 (1) (2006 January–February) 16–23.

[23] M. Oschem, V. Mahler, H.U. Prokosch, Objectifying usercritique. A means of continuous quality assurance forphysician discharge letter composition, Methods Inf. Med.50 (2011) 23–35.

[24] T. Bürkle, E. Ammenwerth, H.U. Prokosch, J. Dudeck,Evaluation of clinical information systems. What can beevaluated and what cannot? J. Eval. Clin. Pract. 7 (4) (2001November) 373–385.

[25] J. Talmon, E. Ammenwerth, J. Brender, N. de Keizer, P.Nykänen, M. Rigby, STARE-HI-statement on reporting ofevaluation studies in health informatics, Yearb. Med.Inform. (2009) 23–31.

[26] J. Brooke, in: P.W. Jordan, B. Thomas, B.A. Weerdmeester, I.L.McClelland (Eds.), SUS—A ‘Quick and Dirty’ Usability Scale,Usability Evaluation in Industry, 1996, pp.189–194.

[27] R. Likert, A technique for the measurement of attitudes,Arch. Psychol. 140 (55.) (1932).

[28] A. Bangor, P. Kortum, J. Miller, An empirical evaluation of theSystem Usability Scale, Int. J. Hum. Comput. Interact. 24 (6)(2008) 574–594.

[29] Europa-Universität Viadrina Frankfurt (Oder) (Ed): KostenSHK ab 2011, http://www.europa-uni.de/de/struktur/verwaltung/dezernat 2/personalkosten/Kosten SHK 01 2011.pdf (last accessed on 04.08.11).

[30] Europa-Universität Viadrina Frankfurt (Oder), (Ed.): KostenWHK MA ab, 2011, http://www.europa-uni.de/de/struktur/verwaltung/dezernat 2/personalkosten/Kosten WHK MA01 2011.pdf (last accessed on 04.08.11).

[31] Fern-Uni-Hagen (Ed.): TV-L Arbeitgeberberechnung2009-2010 web version, http://www.fernuni-hagen.de/imperia/md/content/arbeiten/personal-undarbeitsthemen/bezahlung/pauschalsaetze 2009-2010 web version .xls (lastaccessed on 04.08.11).

[32] Microsoft Excel, http://office.microsoft.com/en-us/excel/(last accessed on 04.08.11).

[33] IBM SPSS Statistics,http://www.spss.com/software/statistics/ (last accessed on04.08.11).

[34] Microsoft Visio, http://office.microsoft.com/en-us/visio/ (lastaccessed on 04.08.11).

[35] M. Riek, Design und Implementierung webbasierterPatientenfragebögen, Bachelorthesis, Institute of MedicalInformatics, University Münster, 2010.

[36] ISO 9241-11:1998, Ergonomic requirements for office workwith visual display terminals (VDTs)—Part 11: Guidance onusability, 1999.

[37] ISO 9241-110:2006, Ergonomics of human-systeminteraction—Part 110: Dialogue principles, 2008.

[38] Umsatzsteuergesetz (UStG) in the copy and proclamation ofFebruary 21, 2005 (BGBl. I p. 386) that was changed by thearticle 6 of the law of June 16, 2011 (BGBl. I p. 1090). § 4 No. 16.

[39] Einkommensteuergesetz (EStG) in the copy andproclamation of October 8, 2009 (BGBl. I pp. 3366, 3862), thatwas changed by article 7 of the law of June 22, 2011 (BGBl. Ip. 1126). § 7 sect. 1.

[40] Bundesministerium der Finanzen: AfA-Tabelle für dieallgemein verwendbaren Anlagegüter, 2002.

[41] J. Nielsen, Why you only need to test 5 users. Alertbox, 2000,http://www.useit.com/alertbox/20000319.html (last accessedon 04.08.11).

a l i n

electronic and paper-and-pencil administration of

i n t e r n a t i o n a l j o u r n a l o f m e d i c

[42] C. Free, G. Phillips, L. Felix, L. Galli, V. Patel, P. Edwards, Theeffectiveness of M-health technologies for improving healthand health services: a systematic review protocol, BMC Res.Notes 3 (2010 October) 250.

[43] M. Rose, A. Bezjak, Logistics of collecting patient-reportedoutcomes (PROs) in clinical practice: an overview andpractical examples, Qual. Life Res. 18 (1) (2009 February)125–136.

[44] B. Holzner, G. Schauer-Maurer, G. Stockhammer, A. Muigg,M. Hutterer, J. Giesinger, Patient reported outcomemonitoring using a tablet PC is suitable for measuringquality of life in patients with gliomas, Wien. Med.

Wochenschr. 161 (1–2) (2011 January) 6–12.

[45] P. Boissy, K. Jacobs, S.H. Roy, Usability of a barcode scanningsystem as a means of data entry on a PDA for self-reporthealth outcome questionnaires: a pilot study in individuals

f o r m a t i c s 8 1 ( 2 0 1 2 ) 303–313 313

over 60 years of age, BMC Med. Inform. Decis. Mak. 6 (2006December) 42.

[46] E.K. Fromme, T. Kenworthy-Heinige, M. Hribar, Developingan easy-to-use tablet computer application for assessingpatient-reported outcomes in patients with cancer, SupportCare Cancer 19 (6) (2011 June) 815–822.

[47] S.J. Lee, A. Kavanaugh, L. Lenert, Electronic andcomputer-generated patient questionnaires in standardcare, Best Pract. Res. Clin. Rheumatol. 21 (4) (2007 August)637–647.

[48] C.J. Gwaltney, A.L. Shields, S. Shiffman, Equivalence of

patient-reported outcome measures: a meta-analytic review,Value Health 11 (2) (2008 March–April)322–333.