Evaluation of

Patient Reported Outcome (PRO)in General Practice

Final brochure of the project PRO in General Practice (2016 2018)

2

The brochure is based on the report ”Evaluation of Patient Reported Outcome (PRO) in General Prac-

tice”.

The project and evaluation are established and financed by the Ministry of Health in collaboration with

the Organisation of General Practitioners in Denmark (PLO), MedCom and Centre for Innovative Medi-

cal Technology (CIMT).

The brochure summarises the implementation of electronic PRO in general practice during 2016-2018

and presents the most important results from the evaluation report.

For more information, please contact:

Project manager, Tina Aa. Bjørnsholm, MedCom

tab@medcom.dk - +45 65 43 20 30 / +45 51 15 72 27

For the full report and more information about the project, please visit:

www.medcom.dk/projekter/pro-i-almen-laegepraksis

3

Table of contents

Introduction Page 4

About the project Page 5

The use of electronic PRO Page 6

Evaluation of electronic PRO in general practice Page 7

Implementation results Page 7

Results from the questionnaires Page 8

Experiences with the electronic PRO system Page 10

Statements from general practitioners and patients Page 11

General Practice: Tasks and time consumption Page 12

Patients: Usability Page 13

Conclusion Page 14

4

Introduction

In recent years, there has been an increased focus on development and use of Patient Reported

Outcome (PRO) in the Danish healthcare system. PRO-data is a subgroup of patient reported data

that involves the patient’s health condition and is defined in the national PRO-work (PRO-

secretariat, 2019, Danish Health Data Authority, 2017, ViBIS, 2016) as follows:

“Patient reported data that concern the patient’s health, including physical and mental health,

symptoms, health related quality of life and level of function”.

Essential for PRO is that the patients report their data from home, which is systematically collected

through PRO questionnaires. In this project, PRO covers the term Patient Reported Outcome and

PRO questionnaires covers validated questionnaires and/or measuring tables with relevant deci-

sion algorithms (PRO-secretariat, 2017, Danish Health Data Authority, 2017).

5

As part of the Danish Government's objective to strengthen collaboration and ensure a close and

coherent health service, the Ministry of Health initiated in 2016 the establishment of the project

"Patient Reported Outcome (PRO) in General Practice" with the aim to expand the use of an elec-

tronic PRO system for managing patient reported outcomes in general practice. As replacement

for paper-based forms, a web-based solution is used. General practice can prescribe home-

monitoring to a patient through the system and, thereby, electronically send, receive and process

PRO questionnaires with information about the patients’ health.

On that basis, an evaluation report has been written, aiming to clarify the implementation of the

system and assess general practitioners, GP staff and patients’ experiences with the system.

The project is initiated by the Ministry of

Health as a 3-year project (2016-2018) as

part of the Government's objective to

strengthen collaboration and ensure a

close and coherent health service. By

agreement with the Ministry of Health,

the project has been extended to 2019.

Collaborating partners

• General Practice

• Patients

• Regional data consultants

• Organisation of General Practitioners in Denmark (PLO)

• Ministry of Health

• MedCom

About the project

6

The use of electronic PRO

Figure 1: Workflow when using electronic PRO

General practitioners (GPs) and their staff are able to order PRO questionnaires for a patient and pre-

scribe home-monitoring through their joint laboratory requisition system. The patient is automatically

notified by email or text message and is able to access the system through electronic identification and

electronically report the measurements, he or she took at home or answer the questions that exist within

the PRO questionnaires. If relevant, an automatic validation of the patient’s data and an estimated result/

score is done in the electronic PRO system. This data is sent to the GP system as a laboratory result, after

which the GP or other staff can decide if action should be taken based on the reported data.

7

Figure 2: National implementation in % of general practice

Figure 3: Regional implementation in % of general practice

The project’s ambition was to achieve an implementation rate of 65% by end of 2018. In December

2018, the national implementation rate was 63% (Figure 2).

The implementation rate in general practice differs between the five regions (Figure 3).

Evaluation of PRO in General Practice

Implementation results

The project was evaluated in Fall 2018 in order to shed light on the implementation process of the

system and to assess the users (GPs, GP staff and patients) experiences with the system. The imple-

mentation rate of the electronic PRO system was captured from monitoring data from the system.

Experiences with the use of the electronic PRO system was investigated first through interviews in

one general practice from each of the five regions in Denmark, followed by a national questionnaire

survey in the same target groups.

8

By December 2018, 800 out of 1,754 general practices were using electronic PRO (Figure 4).

During 2016-2018, general practice ordered more than 160,000 questionnaires, of which 70% were

questionnaires for home blood pressure measurement (Figure 5).

Figure 4: Monthly number of general practices that have used the electronic PRO system

Figure 5: Quarterly number of ordered PRO question-naires in general practice in the electronic PRO system

Results from the questionnaires

Most respondents from general practice state that they do not need further training in use of the sys-

tem. The evaluation shows, however, also that some GPs and their staff ask for already existing func-

tions to be added and do not know the full selection of available PRO questionnaires in the system.

Furthermore, more than half of the GPs and their staff that have not used the electronic PRO system

state that they are not familiar with the system. This indicates that more communication and

knowledge about the system is needed to obtain a higher implementation rate and increased profit

of the electronic PRO system.

9

The majority of GPs and their staff estimate that using the electronic PRO system saves them time.

However, a potential reduction in time has showed difficult to quantify and will require different

analysis methods. In addition, the questionnaire survey is based on a small number of respondents

and the subsequent results should, therefore, be interpreted carefully.

Some of the GPs, their staff and patients express their concern, whether the electronic PRO system

can be used by elderly patients without IT skills. The results of the evaluation show, nevertheless,

that the system is usable, also by elderly patients. However, these patients make up a selected

group of patients with a good level of IT skills, and based on the GPs’ experience from the interview

survey, the use of paper-based questionnaires cannot be fully replaced.

10

General practitioners and GP staff Response rate: 2-12 % (individual and practice), n=203

Patients

Response rate: 22,6 %, n=1.771

• Most use the questionnaires daily or weekly.

• PRO questionnaires for home blood pressure measurements are most frequently used.

• The majority estimate that they do not use more questionnaires now.

• The majority experience the system as easy to use and estimate that most would be able to quickly learn how to use it.

• A great part of GPs have not received training in how to use the system.

• The majority estimates that they do not need the training.

• The system is perceived as reliable and only a few have experienced technical issues.

• A minority estimates that the system has given them new work tasks.

• The majority of the GPs estimate that they save time and their PRO tasks in general have gotten easier.

• The majority estimates that the data quality in has improved by use of the system.

• The majority of the health staff estimate that the risk of adverse events is reduced.

• The majority have used the system 1-2 times, primarily PRO questionnaires for home blood pressure measurement and depression.

• The majority have filled out the PRO questionnaires themselves without help.

• Most have not received training in using the system and have not needed it either.

• The majority experience the system as easy to use and state that more people are able to quickly learn how to use it.

• The majority perceive the system as being reli-able and a few have experienced technical is-sues.

• The majority feel safe in sending health related information to their own GP via the system.

• The majority are satisfied with the system and would recommend it to others.

Experiences with the electronic PRO system

Figure 6: Questionnaire survey: Experiences with electronic PRO

11

Statements from general practitioners and patients

”When I have thought about things and filled out the questionnaire at

home, I feel like it gives a better picture of my condition than when I

say—today I’m not feeling well” (Patient)

”This is the first time I’m using the system. It seems easy to use, even

for me, who am only using my computer for emails and paying

bills” (Patient)

”The questionnaires work well for educated patients –especially well-educated patients. I have experienced that, especially, the

depression groups not managing to cope with something new. A piece of paper and a pencil are manageable” (GP)

”I had no idea that so many PRO questionnaires existed until now!” (GP)

”Great tool for those of us who are used to work with IT—but

what about the elderly?” (Patient)

12

General Practice: Tasks and time consumptionA large part of the general practice staff think that their time consumption has increased in relation to

introduction of the PRO questionnaires to patients (52%). Nevertheless, the GPs, especially, estimate

that the time they spend on some of the tasks is reduced, including e.g. entering data and manual

average calculation, where 71% and 80% think that they spend less time now. 43% and 62% of the GP

staff also think that the time spent on these two tasks has reduced (Figure 7 and 8).

Figure 7: Change in GPs’ daily time consumption

Figure 8: Change in GP staff’s daily time consumption

13

Patients: Usability The patients’ assessment of the usability of the PRO system is generally positive (Figure 9). The

majority of the patients believe that the electronic PRO system is easy to use (92%) and access

(91%). They estimate that most people are able to learn how to use it (83%). More than three out

of four patients (76%) believe that the instructions to the system were easy to understand and that

the layout of the system is user-friendly.

Figure 9: The patients’ assessment of the usability of the system

In total, 288 patients have made elaborating comments on the usability of the system. Several

patients state that the system is “Good”, “Fantastic”, “Easy” and “Smart”. They also appreciate that

the system is digital, meaning the outcomes are stored and do not disappear, so that comparison

over time will be possible and potentially save time and reduce the number of consultations.

14

The evaluation of the use of an electronic PRO system in general practice shows positive results seen from both the GPs and patients’ point of view:

• Saves time The majority of GPs and their staff estimate that they save time and that their tasks in general have gotten easier.

• Increases data quality The majority in general practice estimate that the data quality has improved using electronic PRO.

• Reduces adverse events The majority of GP staff estimate that the risk of adverse events is reduced when using elec-tronic PRO.

• Increases collaboration and patient involvement The majority of patients believe that electronic PRO improves the collaboration with their GP and half of the patients feel more involved in their treatment after being introduced to elec-tronic PRO.

The project’s positive results make for a solid foundation for further deployment of PRO as well as increased realisation of the benefits of digitisation in general practice in Denmark.

Conclusion

15

For the full evaluation report and more information about the project, please visit: https://www.medcom.dk/projekter/pro-i-almen-laegepraksis

16

In collaboration with

Organisation of General Practitioners in Denmark (PLO)

Ministry of Health

Centre for Innovative Medical Technology (CIMT)