The impact of an informational website on behavioural

The impact of an informational website on behavioural

intentions and attitudes about patient engagement in

research: A mixed methods evaluation

by

Donna Plett

A thesis submitted in conformity with the requirements for the degree of Master of Science

Institute of Health Policy, Management and Evaluation University of Toronto

© Copyright by Donna Plett, 2020

ii

The impact of an informational website on behavioural intentions

and attitudes about patient engagement in research: A mixed

methods evaluation

Donna Plett

Master of Science

Institute of Health Policy, Management and Evaluation

University of Toronto

2020

Abstract

Patient engagement in research (PER) is a rapidly emerging practice and resources are needed to

prepare patients and researchers for collaboration. This study asked: What are the perceptions of

patients and researchers about an informational website about PER, and how did it impact

intentions and attitudes toward PER?

A mixed methods approach was used, entailing pre-post surveys, focus groups, and web

analytics. 49 patients and researchers participated. The findings revealed 1) intentions and

attitudes toward PER decreased after viewing the website, however participants reported that the

website did not impact, or positively impacted, their attitudes and intentions, and 2) overall

satisfaction with the website, with variation in preferences for design, and a desire for more PER

guidance, easily accessible information, and detail regarding diversity in engagement and

compensation for participants. These findings reveal considerations for the development of

online resources about PER and understanding divergent findings within mixed methods

research.

iii

Acknowledgments

First, I’d like to thank my thesis supervisor, Susan Law, and committee members Kerry Kuluski

and Judith Versloot for the many hours you’ve dedicated to helping me with this project. Your

feedback and advice have benefitted me in so many ways and will help to inform my decision-

making far beyond this thesis. Thank you also to Julia Ma, who offered fantastic tips and tricks

for analyzing and presenting the quantitative data.

Thank you to all my IHPME professors who not only taught me about research topics and

methods, but also took the time to help me clarify my research interests and aptitudes and

exemplified some of what I hope to accomplish in my career one day.

I’d also like to thank my friends and IHPME classmates who I had the pleasure of working

alongside while completing coursework and group assignments. Your humour and comradery

made the challenging times much more bearable, and your unique skills and interests inspired me

and opened my mind to new areas of inquiry.

Finally, I’d like to thank the two individuals who were nearest and dearest to me throughout this

process. To my cat, Tiki, thank you for curling up next to my keyboard as I write and cheering

me up with your eternal cuteness. And to my partner, Chirag, thank you for keeping me company

throughout COVID-19 social distancing, for listening to me ramble endlessly as I process my

ideas, for challenging and helping me to clarify those ideas, and of course for being a wizard in

all things tech.

iv

Table of Contents

Project Overview .....................................................................................................1

1.1 The Problem .........................................................................................................................1

1.2 Purpose of the Study ............................................................................................................2

1.3 Theoretical Framework ........................................................................................................3

1.4 Importance of the Study .......................................................................................................4

1.5 Scope & Limitations ............................................................................................................4

Literature Review.....................................................................................................6

2.1 What is Patient Engagement in Research?...........................................................................6

2.2 Why Patient Engagement in Research is Important ............................................................7

2.2.1 Defining ‘Patient’.....................................................................................................7

2.2.2 PER in Practice ........................................................................................................8

2.2.3 International Context ...............................................................................................9

2.3 Benefits & Challenges of PER ...........................................................................................10

2.3.1 Benefits for Research Projects ...............................................................................10

2.3.2 Personal Benefits to Research Partners ..................................................................11

2.3.3 Benefits for Communities ......................................................................................11

2.3.4 Logistical Constraints ............................................................................................12

2.3.5 Lack of Receptive Attitudes ...................................................................................13

2.3.6 Representativeness & Recruiting for Diversity .....................................................14

2.3.7 Building Relationships & Avoiding Tokenism......................................................15

2.3.8 Measuring Impact ..................................................................................................17

2.4 Building Capacity for PER: The Resource Landscape ......................................................18

v

2.4.1 The Need for Resources .........................................................................................18

2.4.2 Existing PER Learning Resources .........................................................................18

2.5 Evidence for the Benefits of Using Video Narratives .......................................................20

2.5.1 The Persuasive Power of Narratives ......................................................................20

2.5.2 The DIPEx Model ..................................................................................................21

2.6 Summary and Rationale for Further Evaluation ................................................................23

Methods..................................................................................................................24

3.1 Introduction to Methods .....................................................................................................24

3.1.1 Description of Website ..........................................................................................24

3.1.2 Institutional Context...............................................................................................25

3.1.3 Paradigmatic Perspective .......................................................................................25

3.1.4 Methodological Approach .....................................................................................26

3.2 Study Design ......................................................................................................................28

3.2.1 Overview of Methods ............................................................................................28

3.2.2 Survey Tool Development .....................................................................................31

3.2.3 Web Analytics Strategy .........................................................................................34

3.2.4 Ethical Considerations ...........................................................................................35

3.3 Sampling & Recruitment ...................................................................................................37

3.3.1 Recruitment Process...............................................................................................37

3.4 Data Collection ..................................................................................................................38

3.4.1 Consent Process .....................................................................................................38

3.4.2 Surveys ...................................................................................................................39

3.4.3 Focus Groups .........................................................................................................39

3.4.4 Web Analytics ........................................................................................................40

vi

3.5 Data Analysis .....................................................................................................................40

3.5.1 Survey Data ............................................................................................................40

3.5.2 Qualitative Data .....................................................................................................42

3.5.3 Web Analytics ........................................................................................................43

3.5.4 Integration & Interpretation ...................................................................................43

3.6 Dissemination Plan ............................................................................................................44

Results ....................................................................................................................45

4.1 Overview ............................................................................................................................45

4.2 Study Sample .....................................................................................................................45

4.3 Survey 1 Findings: Baseline Characteristics ......................................................................46

4.3.1 Participant Characteristics .....................................................................................46

4.3.2 Adapted eHIQ Part 1 ..............................................................................................46

4.3.3 Theory of Planned Behaviour at Baseline .............................................................48

4.4 Survey 2 Findings: Impact of Website...............................................................................51

4.4.1 Self-Reported Time on Website.............................................................................51

4.4.2 Adapted eHIQ Part 2 ..............................................................................................51

4.4.3 Theory of Planned Behaviour Post-Intervention ...................................................54

4.5 Website Utilization ............................................................................................................57

4.6 Focus Group Findings ........................................................................................................59

4.6.1 Theme 1: Desire for Practical, Action-Oriented Content ......................................59

4.6.2 Theme 2: Desire for Clarity of Purpose and Definitions .......................................61

4.6.3 Theme 3: Variance in Preferences for Design Aspects .........................................61

4.6.4 Theme 4: Importance Ascribed to Diversity and Compensation in PER ..............63

4.7 Integration of Findings .......................................................................................................64

vii

4.8 Summary of Findings .........................................................................................................68

Discussion ..............................................................................................................69

5.1 Overview ............................................................................................................................69

5.2 Comparison with Related Literature ..................................................................................70

5.3 Interpretation of Divergent and Unanticipated Findings ...................................................71

5.4 Contribution to Academic Knowledge ..............................................................................73

5.5 Implications for Policy & Practise .....................................................................................74

5.6 Opportunities for Future Research .....................................................................................75

5.7 Limitations .........................................................................................................................76

5.8 Conclusion .........................................................................................................................77

References ......................................................................................................................................78

Appendices .....................................................................................................................................86

viii

List of Tables

Table 1: Adapted eHIQ Part 1 results, by domain. ....................................................................... 47

Table 2: Baseline TPB findings by domain – between group comparisons. ................................ 49

Table 3: Barriers and facilitators to engaging in partnerships, selected quotes. ........................... 50

Table 4: Adapted eHIQ Part 2 results, by domain, patients and researchers. ............................... 52

Table 5: Feedback about website in open-ended survey responses, selected quotes by topic, and

number of patients (P) and researchers/research staff (R) who referred to topic in response.* ... 53

Table 6: Pre-post TPB findings by domain, patients and researchers. ......................................... 55

Table 7: Survey feedback about the impact of the website, selected quotes. ............................... 56

Table 8: Summary statistics for evaluation period. ...................................................................... 58

Table 9: Complementary findings. ............................................................................................... 66

Table 10: Divergent findings. ....................................................................................................... 67

Table 11: Original and adapted eHIQ. .......................................................................................... 87

Table 12: Adapted Theory of Planned Behaviour survey questions, by domain. ......................... 89

Table 13: Organizations contacted during recruitment process .................................................... 91

Table 14: Demographics. ............................................................................................................ 112

Table 15: Page views and unique pages views by person featured on website. ......................... 114

Table 16: Number of views by topic page and average length of time spent on page. .............. 115

Table 17: Average self-reported time on site, low and high estimates. ...................................... 116

https://utoronto-my.sharepoint.com/personal/donna_plett_mail_utoronto_ca/Documents/IHPME/Thesis/08%20-%20Thesis%20Drafts/Thesis_12Aug2020_trackedchanges.docx#_Toc48116557




ix

Table 18: Adapted eHIQ Part 1 data, aggregate, patients, and researchers, attitudes toward online

info about PRPs........................................................................................................................... 117

Table 19: Adapted eHIQ Part 1, aggregate, patients, and researchers, attitudes toward sharing

PRP experiences online............................................................................................................... 118

Table 20: Adapted eHIQ Part 2 data, aggregate, patients, and researchers, confidence and

identification, part 1. ................................................................................................................... 119


identification, part 2. ................................................................................................................... 120

Table 22: Adapted eHIQ Part 2 data, aggregate, patients, and researchers, information and

presentation, part 1. ..................................................................................................................... 121


presentation, part 2. ..................................................................................................................... 122

Table 24: Adapted eHIQ Part 2 data, aggregate, patients, and researchers, understanding and

motivation, part 1. ....................................................................................................................... 123


motivation, part 2. ....................................................................................................................... 124

Table 26: Pre-post TPB data, patients and researchers, intentions. ............................................ 125

Table 27: Pre-post TPB data, patients and researchers, attitudes. .............................................. 126

Table 28: Pre-post TPB data, patients and researchers, subjective norms. ................................. 127

Table 29: Pre-post TPB data, patients and researchers, perceived behavioural control. ............ 128

x

List of Figures

Figure 1: Study design. ................................................................................................................. 28

Figure 2: Diagram of methods ...................................................................................................... 31

Figure 3: Theory of Planned Behaviour........................................................................................ 32

Figure 4: Recruitment flowchart. .................................................................................................. 45

Figure 5: Self-reported time on site. ............................................................................................. 51

Figure 6: Daily website page views. ............................................................................................. 57

Figure 7: Image of front page of website. ..................................................................................... 86






xi

List of Appendices

Appendix A Website Layout ........................................................................................86

Appendix B Survey Tool Development .......................................................................87

Appendix C Sample Recruitment Email ......................................................................90

Appendix D Organizations Contacted During Recruitment ........................................91

Appendix E Focus Group Consent Form .....................................................................92

Appendix F Interview Guide for Focus Groups ..........................................................95

Appendix G Pre and Post Surveys ...............................................................................97

Appendix H Demographic Data .................................................................................112

Appendix I Web Analytics Data ...............................................................................114

Appendix J Adapted eHealth Impact Questionnaire Data ........................................117

Appendix K Theory of Planned Behaviour Data .......................................................125

Abbreviations

PER patient engagement in research

PRP patient-researcher partnership

SLT Social Learning Theory

eHIQ eHealth Impact Questionnaire

TPB Theory of Planned Behaviour

PBC perceived behavioural control

Page 1 of 128

Project Overview

1.1 The Problem

The science on the benefits of patient engagement in research (PER) is still young, but there are

many compelling arguments for its widespread adoption: 1) engaging patients in research can

result in improved quality of care because research priorities are better aligned with patient

needs;1 2) PER creates an opportunity to engage marginalized populations, which could lead to

greater generalizability of research findings and more equitable healthcare delivery;1 3)

involving patients empowers them to take a more active role in their healthcare;2 4) research

findings reach a larger population when patients act as ambassadors in their communities;1 5)

PER improves accountability and responsiveness to citizens and contributes to democratic ideals

of transparency and accountability by providing assurance that resources are being used to

further public interests;1,3 and 6) some argue there is a moral imperative to include patients in

research because patients have a fundamental right to involvement in research that will have a

direct impact on them.1,4

It is now commonplace for funding agencies (e.g., Canadian Institutes of Health Research) to

require health researchers to include a plan to engage patients in their study protocol to be

eligible for funding.5,6 However, researchers often do not know how to meaningfully include

patients in their research7 and patients have reported feeling unclear about what their role is in a

partnership with researchers.8 Studies show that poor preparation and including patients just to

‘tick a box’ on a grant application has led to tokenistic patient involvement and ineffective

partnerships4 which are damaging to those involved and reinforces negative attitudes toward

patient engagement in research.9,10

In order to combat these challenges, well-designed resources are needed to demystify research

partnerships by providing information to patients and researchers about how to successfully work

together throughout every stage of the research lifecycle.7 While some educational resources are

already available to provide information about PER, there is ample opportunity for the

development of new resources that tackle the challenges of PER.

Page 2 of 128

1.2 Purpose of the Study

This study evaluates an innovative website, which is designed to build capacity for patient

engagement in research by providing experiential information about research partnerships. The

website features clips and transcripts from video-recorded interviews with patients, caregivers

and researchers describing their experiences of being involved in research partnerships.

To evaluate this website, we invited patients, caregivers and researchers to spend time on the

website and provide feedback in the form of surveys and focus groups. We utilized a convergent

mixed methods approach, in which quantitative data (from surveys and web utilization data) and

qualitative data (from focus groups) were collected and analyzed separately and then combined

during the interpretation of the results.11 A mixed methods approach was appropriate because

very little literature exists about the evaluation of resources for patient engagement in research,

and no literature exists to evaluate a resource that uses video narratives to provide information

about partnerships. Our mixed methods approach facilitated an analysis that was strong in both

breadth and depth – breadth through the inclusion of many survey participants and depth through

the discussions held in focus groups with a segment of the survey participants.

The research question that guided this study was: What are the perceptions of patients and

researchers about a website featuring experiential information about patient engagement in

research, and how did it impact behavioural intentions and attitudes toward PER? The primary

aims of the study were to:

1. Measure attitudes, subjective norms, perceived behavioural control, and intentions related

to PER before and after participants spent time on the website,

2. Analyze how participants navigated the website and which content they engaged with the

most,

3. Learn about the reactions of participants to the website’s design and content and their

perceptions about its usefulness as an informational resource about PER, and

4. Compare and contrast the results that emerged from the different methodologies.

Page 3 of 128

1.3 Theoretical Framework

This evaluation is underpinned by two theoretical frameworks: Social Learning Theory (SLT)

and the Theory of Planned Behaviour (TPB). These theories underlie the design of the website,

the approach to its evaluation and hypotheses about its effectiveness.

Social Learning Theory was developed to account for the role that observing the experiences of

others has in learning. Bandura, the theory’s first proponent, posited that most experiential

learning occurs vicariously through others’ experiences and that the ability to understand others’

experiences symbolically makes it possible to foresee probable consequences of actions without

having to first take the action.12 Other core tenets of SLT include that learning takes place in

social contexts and involves observing others’ behaviours and the consequences of those

behaviours.13

Based on SLT, some common methods of knowledge dissemination may not always be

maximally effective. For example, one common method of knowledge dissemination is to

present de-contextualized facts, statistics and recommendations (e.g., pamphlets, flyers, websites,

etc.). However, when information is presented this way, the learner does not have access to the

individual experiences that inform the aggregated information presented to them. According to

SLT, this could result in the information being less engaging and less likely to influence

behaviour. SLT underlies the design of the website, which includes excerpts from a maximum

variation sample14 of video-taped interviews of patients and researchers describing their

involvement in research partnerships, including lessons learned and recommendations for others.

This design allows website visitors to engage with multiple and varying individual experiences

in a way that is reflective of how people often gain knowledge in their day-to-day lives through

conversations and stories.

The Theory of Planned Behaviour pertains to the determinants of intention toward behavior.

According to the TPB the three determinants of behavioural intention are attitudes toward the

behavior, perceived social pressure to perform the behaviour and the degree of perceived

behavioral control (i.e., beliefs about factors that could facilitate or hinder the behaviour and

perceived ease or difficulty of the behavior).15 The website was developed to build capacity for

Page 4 of 128

patient engagement in research by providing a realistic account of the experiences of research

partners and by offering information and support for partnership. It was the believed that the

website would provide visitors with better insight and understanding of the benefits and

challenges of research partnerships and thereby better equipped to engage in them. It was also

believed that the website’s information would encourage increased participation in research

partnerships. By providing firsthand accounts of different experiences of research partnerships,

we hypothesized that the website would positively influence attitudes and behavioural intentions

toward patient engagement in research and that website visitors would find the experiential

content informative and relatable.

Both SLT and the TPB were built into the evaluation of this study, especially in the design of the

surveys administered to participants and the focus group discussion guide (see Methods).

1.4 Importance of the Study

The value of this study lies largely in its furtherance of knowledge about informational resources

about patient engagement in research, how people learn via online resources, and the usefulness

of using stories and narratives in teaching. It may prove valuable to others who may have found

that traditional models of knowledge dissemination have not been as effective as desired. It will

also add to rapidly emerging literature about the use of the internet for disseminating information

and its usefulness as a medium for influencing knowledge, attitudes and behaviour. This study

will also serve as an important resource for others who are developing tools for patient

engagement in research or other topics where an information resource that uses video narratives

may be useful. By utilizing a variety of methods, this study also offers important methodological

learnings related to mixed methods designs and the usefulness of the survey tools we employed.

1.5 Scope & Limitations

Our mixed methods design enabled an in-depth comparative analysis of the responses of patients

and researchers across Canada but leaves open many opportunities for future research in this

area. For instance, future research may benefit from the inclusion of a control group that was

exposed to a traditionally designed educational resource about PER, under traditional trial

conditions to focus on effectiveness. Also, a larger sample that is more representative of the

Page 5 of 128

Canadian population would aid the generalizability of results. A longer evaluation time period

would allow for more web analytics data collection. Future research may also address the

difficulty in recruiting researchers and patients/caregivers who do not already have a positive

attitude toward patient engagement in research, as those who are already interested in research

partnerships are more likely to be interested in participating in a study about research

partnerships. Unfortunately, those without reliable high-speed internet access (e.g., rural/remote

communities, homeless, and low-income individuals) could not participate in this study. Future

initiatives could encourage utilization of public internet access via libraries and community

centres to ensure these populations are not excluded. Limitations will be discussed further in the

discussion chapter.

Page 6 of 128

Literature Review

2.1 What is Patient Engagement in Research?

Patient engagement in research (PER) is one context for patient engagement, which includes a

very broad range of activities. Patient engagement has applications for every part of the

healthcare system, from direct care, to organizational design and governance to policy making.16

At a basic level, patient engagement takes the form of consultation, such as informing a patient

about their diagnosis, surveying patients about their care experience or conducting focus groups

to ask for opinions about a health care issue. A more involved form of patient engagement could

include asking patients about their preferences, including patients as advisers, or using patient

recommendations in funding decisions. At the highest level of involvement, patient engagement

takes the form of partnership and shared leadership. This includes making treatment decisions

based on patient preferences, patient co-lead committees, co-designing care services and policies,

and equal representation of patients on government agency committees.16

PER is a unique application of patient engagement in that it is not directly related to

policymaking, organizational decision-making, or clinical practice. PER is uniquely applicable to

the process of health research and can occur at specific points within the research process or

throughout the research life cycle. One way to understand PER is through the lens of knowledge

translation (KT). The role of KT is to address the ‘know-do’ gap in health research which occurs

when important findings are not disseminated effectively to the patients, practitioners, or others

who would benefit from the new knowledge.17 According to the Canadian Institutes of Health

Research (CIHR), KT is “a dynamic and iterative process that includes synthesis, dissemination,

exchange and ethically sound application of knowledge to improve the health of Canadians,

provide more effective health services and products and strengthen the health care system.”17

Most commonly KT takes place at the end of the research process, or ‘end-of-grant’ KT.18 In

end-of-grant KT, patients or other knowledge-users may be invited to participate in

dissemination planning but do not contribute to the research itself. A more expanded form of KT

is emerging, called integrated KT (iKT). Integrated KT is an approach that involves patients

contributing in various ways throughout the research process, such as to inform or create the

research question, select appropriate methodology, collect data, choose outcome measures,

Page 7 of 128

interpret findings and support dissemination.17 Depending on the duration and extent of patients’

involvement in research projects, PER can be end-of-grant KT or iKT.

2.2 Why Patient Engagement in Research is Important

The drive for patient engagement in research (PER) is part of a broader goal of patient

engagement in health and healthcare that has gained prominence throughout the world.16,19

Patient engagement at its core is a challenge to the paternalistic models of the past, in which

experts, academics, and healthcare professionals are the purveyors of knowledge and patients

and the lay public are recipients of that knowledge.20 In recent decades it has become clear that

failing to listen to and understand the experiences and perspectives of patients has contributed to

poor quality care, ineffective communication, inefficiencies, and a lack of trust in healthcare

professionals.20

Patients experience frustrations with the healthcare system when they are not adequately

informed about the realities of medical care and their own situation, including the extent of

medical uncertainty, the state of knowledge about effectiveness of their care, and the ways in

which medical practice can vary. Bringing patients into important healthcare conversations can

improve relationships between patients and the healthcare system. The role of patients and

families beyond recipients in healthcare is building momentum as they become more active,

informed and influential.16 PER has a role in improving patient-oriented care by producing

knowledge that is more relevant and focused on patient-centred priorities.

2.2.1 Defining ‘Patient’

The term ‘patient’ is used liberally in the context of patient engagement and clarification is

needed to understand which individuals are included under the label. This paper will use the

definition given by the Strategy for Patient-Oriented Research (SPOR) Patient Engagement

Framework which defines ‘patient’ as: “An overarching term inclusive of individuals with

personal experience of a health issue and informal caregivers, including family and friends.”21 In

PER, the role of the patient is to bring their experiential knowledge to the table, to communicate

their stories and interests and to assist the researchers in various aspects of the research project

Page 8 of 128

including priority setting, governance, KT and conducting research.21 Patients bring added value

to the project by offering a perspective based on their lived experience of the health condition or

service under study, and are not expected to act as a representative for any group.

2.2.2 PER in Practice

There are opportunities to engage with patients during all stages of the research lifecycle –

preparation, planning, study design, data collection, analysis and dissemination – although

engagement is often limited to preliminary activities.22 In early stages of the research lifecycle

(planning and preparation) advisory councils and focus groups can be used to gain an

understanding of the priorities of the patient population and to incorporate their feedback into the

research plan. There is opportunity also for patients to be involved in data collection in various

ways depending on the study design. This can include engagement in conducting interviews,

perhaps as co-researchers, particularly when the patient partner is a member of the community

being studied and can more easily communicate with community members than researchers. The

data analysis stage also creates opportunity, especially within qualitative research, since patient

partners may pick out themes and concepts that researchers would miss or not think were

relevant. During dissemination, consulting with patient partners can ensure that the resulting

product makes sense to lay readers and is sensitive to the concerns of the target population.

PER is not only applicable to formal academic research projects (e.g., clinical trials, health

services research, health policy research) but also to research priority setting and quality

improvement projects. The James Lind Alliance (JLA), initiated in the UK and now with global

spread, has a unique approach to involving patients in research through its Priority Setting

Partnerships (PSPs).23 The JLA believes that patients, caregivers and clinicians should work

together to agree about priorities and that it is important for clinicians to address uncertainties

about the effects of treatments. PSPs bring together patients, caregivers and clinicians that share

expertise about an illness or health experience (e.g., acne, kidney cancer, learning difficulties,

etc.). Through a series of workshops, the group develops a Top 10 list of priorities or research

questions. Following this, the identified priorities are promoted to key groups such as research

funders, researchers, patients, caregivers and the wider research community.23

Page 9 of 128

Experience-based co-design (EBCD) is an intra-organizational form of patient engagement used

in healthcare organizations to inform quality improvement projects.24 The co-design element of

EBCD draws upon participatory design principles and aims to open the boundaries of healthcare

service design to include new stakeholders and forms of expertise.24

2.2.3 International Context

PER has gained traction in many countries including Canada, the US, the UK, Australia, the

Netherlands, and Sweden.25 The UK is recognized as the forerunner in PER6,26 and engaging

patients became a part of its national research governance framework in 2005.6 The non-profit

INVOLVE was founded in 2003 with a mission to put people at the heart of decision-making.

Their projects include citizens’ assemblies on policy issues, priority setting and involving

members of the public in research.27 The JLA is also based in the UK and its research priority

setting techniques have inspired researchers all over the world.

The US is another PER leader. The Patient-Centered Outcomes Research Institute (PCORI) is a

non-profit organization that has funded hundreds of research studies since 2012.28 PCORI aims

to speed up the implementation and use of research evidence, influence other clinical and

healthcare research to be more patient-centered, and support clinical effectiveness research that

engages patients throughout the research process in order to increase the quantity, quality and

timeliness of useful, trustworthy information.28 The International Association for Public

Participation (IAP2) was founded in the US in 1990 with the goal of promoting public

involvement in government and industry decision-making.29 As of 2007 IAP2 had more than a

thousand members from 26 countries.29

Canada is emerging in the field of PER with the creation of CIHR’s SPOR initiative and the 11

provincial SUPPORT Units that act as conveners in their home province or territory.30 SPOR

acts as a catalyst for patient-oriented research by facilitating and directly funding research across

the country, and finding and supporting synergies between various stakeholders.31 Some of the

accomplishments of the Ontario SPOR Support Unit (OSSU) include a project that reduced

rehabilitation length of stay for people with stroke and work that resulted in patient-reported

experience measures being included in funding evaluations.32 Other Canadian organizations that

Page 10 of 128

conduct or support patient engagement in research include the Canadian Foundation for

Healthcare Improvement (CFHI) and the Canadian Common Drug Review Patient Advisors.

Established and emerging organizations have many roles to play, including creating a culture of

public participation, providing research funding for projects that engage patients throughout the

research process, and creating and developing guidelines and methods for engagement. These

organizations engage patients and other members of the public both in the research process and

in informing public and organizational policies. Their work need not all be related to patient

engagement in research, or even healthcare, to support the movement toward patient

engagement. Their broad efforts promote a culture of participation that spreads across sectors.

2.3 Benefits & Challenges of PER

2.3.1 Benefits for Research Projects

The contributions of patients have benefitted research projects in many tangible ways. Patients

help to clarify research questions, extend patient cohorts to include new members, suggest

reasons for low recruitment of study subjects and possible strategies to enhance recruitment,

review transcripts, suggest new outcome measures to include, assist with data analysis including

interpretation of results and identification of themes, and co-lead meetings.8

The most commonly cited quantitatively measurable benefits of PER is better recruitment and

retention of participants in clinical trials.4,33-35 A systematic review by Crocker et al found that

clinical trial studies that included patients in the research team significantly increased the odds of

a patient enrolling in the trial.34 This effect size was even larger when the patients involved in the

recruitment had lived experience of the condition under study, indicating that patients bring

significant added value to research studies as “experts in lived experience.”34 The researchers

noted that for many of the studies, patients were only involved in the recruitment process but if

they had been part of the research study from the beginning there was potential for the effect size

to be even larger.34

In a systematic review of the literature on involvement of patients in surgical studies, Jones et

al33 found two studies that included patients from the beginning and incorporated their

Page 11 of 128

suggestions into research topic identification and the design and conduct of the intervention. The

patients’ input had a direct and immediate effect on the study aim, protocol and outcome and

enhanced the face validity of the study. Some researchers believe that including patients in the

research process helps to ensure high participation rates because their contributions lead to more

salience of the research to the public.33

2.3.2 Personal Benefits to Research Partners

Researchers in the UK conducted a systematic review of the literature about the impact of patient

and public involvement on various stakeholders, including patients, researchers and the

community.25 Most benefits reported by patients are personal benefits, such as feeling listened to,

empowered and valued. Patients have also reported that their confidence increased because of

their research partnership experience and they felt a greater sense of self-worth.25 They also

reported feeling good about having an opportunity to give back to the research community.

Researchers who participated in research partnerships with patients reported that they benefitted

by an improved understanding of the health condition, greater respect for patients, having their

beliefs and attitudes challenged, new areas of research being opened, seeing their efforts

rewarded and gaining new friendships with community members.25

Patients have also reported that through their involvement in research they gained useful

professional skills, such as public speaking, interviewing and working in groups. They reported

that these new skills improved their chances of obtaining future employment.25

2.3.3 Benefits for Communities

The benefits of PER extend beyond the individuals directly involved in the partnership to the

communities that they are members of. These communities are sometimes geographically

defined but can also be broader communities defined by shared experiences of illness or shared

cultural/ethnic background. The benefits of PER to communities include mutual respect,

acceptability and trust of the research, a more research co-operative spirit, increased awareness

of disease, better distribution of information, increased membership for community groups, and a

sense of community ownership.25 Elliott et al summarized the benefits of engagement in research

prioritization among chronic kidney disease (CKD) patients as: increased integration of distinct

Page 12 of 128

stakeholder communities, an enhanced understanding of the CKD lived experience, and a refined

and refocused commitment to patient-centered CKD care. Patients in this study also reported that

their involvement in the research and the relationships with other CKD patients that it enabled

helped to address their perceived lack of community.36

PER has unique potential to benefit marginalized communities, such as Indigenous communities

that historically have had their perspectives and values excluded from decision-making and the

values of the Western medical establishment imposed on them. Research has been conducted to

review the effectiveness of community engagement in cancer control37 and arthritis38 studies

with Indigenous people in Australia, New Zealand, Canada and the US. The researchers found

communities were usually only engaged minimally, even though the most successful outcomes

resulted from engagement that included Indigenous community members throughout the research

process as true partners.37 Many opportunities still remain to build trust and genuine partnerships

with members of Indigenous and other marginalized communities that will facilitate health

research that results in better health outcomes in the community while respecting their unique

values, needs, and interests.

2.3.4 Logistical Constraints

The academic research process is laborious on its own and bringing patients into it introduces

additional logistical challenges for everyone involved. One of the most frequently cited

challenges of PER by both patients and researchers is the time commitment.25 For researchers

who don’t already have existing relationships with patients or patient groups, investing the time

and energy to establish these connections is time-consuming. While many funding agencies now

encourage or in some cases require PER, the amount of funding obtained may not include all that

is required to bring patients into the research process effectively.

Patients face many logistical challenges when joining research teams. In many cases they are not

compensated for their time,39 and they have to find a way to balance their responsibilities to the

research project with their other work or personal responsibilities. In some cases patients have

felt overburdened by the number of tasks required of them, such as reading unfamiliar

documents before meetings.25 Some patients have reported that they found the formal procedures

of research limiting and experienced emotional burdens related to sharing their personal

Page 13 of 128

experiences and feeling responsible for acting as a bridge between the researchers and the

community.25 These factors can limit which patients are able to participate.

There is currently a discussion within the field of engagement about compensating patient

participants, and many patients and researchers believe that engagement-related costs such as

parking, mileage and food should be covered by the research team.40 It is also believed by some

that a lack of compensation creates a power imbalance as it devalues the knowledge and

experiences that patients bring to the table.39 Despite this, the predominant culture in many

contexts resists the idea of compensating patient partners.39 However, some guidance is

emerging from various sources to ensure fairness and transparency in the process of considering

compensation.

2.3.5 Lack of Receptive Attitudes

Researchers are not always motivated to include patients in their research. This can be due to a

perception that patients lack the knowledge to contribute to research or a fear that allowing

people who are untrained in scientific methods to be part of a research team could have a

negative impact on the integrity of research.7 Researchers have reported feelings of skepticism

about the value of partnerships, concern about the value of the contributions that patients could

make, their competence to assist with research and the potential that they might have their own

lobbying agenda.25 In some cases researchers have found it difficult to accept the views of users,

to relinquish control, and to change working practices to accommodate partnerships.25

Researchers’ skeptical attitudes can create a vicious circle, where patients are only included to

fulfill a funder’s requirement, leading to patients feeling undervalued, unimportant and unable to

contribute, ultimately resulting in the partnership not producing results and reinforcing the

researchers’ skepticism.7

For PER to be successfully adopted, researchers need to believe that patients’ knowledge and

potential to be involved is of equal importance as their own and that PER has a meaningful

impact on research outcomes.7 For this to happen a cultural shift is required to lessen the

influence of traditionally positivist epistemology in order for qualitatively reported experiences

to be taken seriously and recognized as a legitimate form of knowledge.7 There are many

Page 14 of 128

variables from both researcher and patient perspectives to consider when engaging in

partnerships – including the capacity, skills, and attitudes that assure a good fit within the team

and enable collaborative working.

2.3.6 Representativeness & Recruiting for Diversity

The difficulty of recruiting a more diverse and representative group of patients is acknowledged

by many researchers40 and the strategy for recruiting vulnerable populations, such as those who

are socioeconomically disadvantaged, frail, or have chronic illness, remains unclear.36 Studies in

the UK emphasize that there is a need for more diverse research partners in terms of age, ability,

gender, class, ethnicity/race, geography, indigeneity, immigration status, religion and sexuality.40

The benefits of including these populations would be significant as their contributions could help

to address the lack of inclusiveness that currently exists in medical research samples, which

would ultimately lead to results that are more applicable to a wider range of patients.1

In a survey of Canadian cardiovascular researchers about PER one of the top concerns raised was

about how to identify and recruit patient partners that reflect the diversity within the patient

population under study.7 There was a sense that there is not an agreed upon criteria for what

constitutes a “good” patient to include as a research partner – should it be the smartest patient,

the sickest patient, the most articulate patient or the patient with the most time? If the patient is

chosen based on convenience or because of their existing knowledge of research or their

articulateness, certain socioeconomic groups and people with low health literacy may be

systematically excluded.7 In fact, there is a growing awareness of the fact that the majority of

those who are included in engagement activities are socioeconomically advantaged women from

ethnic majority groups.41 There is also the concern that if the same individuals are involved in

too many patient engagement activities they run the risk of becoming ‘super-patients.’40 This

occurs when a layperson becomes ‘socialized’ as a researcher and as a result of their formal

training and research experience they lose the ability to represent the voices of patients.40

It is not only logistical and socioeconomic factors that make recruiting for diversity difficult. A

scoping review of PER related to dementia found that there are unique barriers to including

people with dementia but also important reasons to make the effort to do so.42 The barriers that

people with dementia face are associated with cognitive and communication symptoms, but also

Page 15 of 128

stigma because of the societal view that they would not be capable of fully participating in

research. Putting in the work to bring people with dementia, or other conditions that are

stigmatized, into the research process could help to reduce stigma and negative stereotypes as

well as open the door to a much deeper understanding of the lived experiences of these

individuals.42

A systematic review by Bonevski et al found strategies for increasing the number of

socioeconomically disadvantaged populations in research.43 The authors suggest a multipronged

approach including: 1) taking a long-term view and investing the time to develop strong

relationships with communities and community groups, 2) including the additional costs in

funding applications for things like translations services, flexible data collection locations and

times, gifts or incentives, culturally tailored resources and materials, and additional staff training,

and 3) allowing time for authorship negotiations since data ownership may need to be negotiated

with non-academic collaborators.43

2.3.7 Building Relationships & Avoiding Tokenism

Effective, high quality relationships and communication are key components of successful

research partnerships.4 Unfortunately in some cases patients have reported feeling like they were

not being listened to or taken seriously and that health professionals and researchers were

insensitive, which lead to patients feeling low self-esteem and an unease about expressing their

opinions.25 They also reported a lack of feedback from the researchers about the value of their

impact which made them feel less motivated to be involved in future projects.25 This is a

significant problem because feeling part of the team is a strong predictor of whether patients feel

that their involvement made a difference.4,40 Additionally, patients are in a vulnerable position

when they are asked to share their painful, awkward, difficult or intimate experiences in order to

improve health care and contribute to the research. This can be traumatic for patients and

researchers’ insensitivity to it could cause rifts.39

There are a few factors that can contribute to good relationships in research partnerships. For

instance, early involvement of patient partners and including them in roles that are responsive

and managerial rather than oversight roles has been shown to contribute to higher perceived

Page 16 of 128

impact.4 Patients that are only brought into the research after important decisions have already

been made may feel that their role is not as important. Recognizing the circumstances of patient

partners and providing adequate compensation can also contribute to better relationships.

According to a group of patient partners, researchers should be mindful of the power imbalance

that exists if everyone on a research team is compensated except for patient partners.39

Tokenism is a predominant concern in engagement research.8-10 As patient engagement is

increasingly recognized as a priority by funding agencies, researchers are incentivized to include

a plan to engage patients in research protocols in order to be eligible for funding. However, not

all researchers who do this intend to genuinely involve patients in their research and have only

included a plan to engage patients in their protocol to increase their chances of obtaining

funding.6,9

Tokenism occurs within the patient engagement process when patients serve more of a window-

dressing role than as authentic research partners.9 This can be the result of uninformed or overly

simplistic assumptions about the knowledge and abilities that patients have and a belief that their

views don’t need to be taken seriously.8 Tokenism can also occur despite the best intentions of

researchers when the demands and challenges that are involved in building and maintaining

effective relationships with partners become too daunting.9

One of the ways researchers can build effective relationships with patient partners and avoid

tokenism is by ensuring that they have a detailed plan for patient engagement before starting.

The FIRST (facilitate, identify, respect, support, train) framework is a useful tool for creating a

non-tokenistic research partnership.8 To facilitate a partnership, researchers need to make a plan

to address potential barriers that patient partners may face. This could include creating a

reimbursement plan, choosing an accessible location for meetings and determining how patient

partners will be recognized for their contribution (e.g., coauthor on grant application and/or

publication). Researchers must also identify research projects that are conducive to partnerships,

patients who could add value to the project and the roles that those patients will have in the

project based on their skills and background. Any projects that stand to benefit from an

understanding of the lived experiences of patients have potential for patient partnership. Patients

who have in-depth experiential knowledge of the health condition under study (by having

Page 17 of 128

experienced it themselves or having cared for someone with the condition) have much to

contribute to the project. In addition to their lived experiences, patient partners often have other

skills and aptitudes that could make it possible for them to contribute in research activities such

as drafting protocols and questionnaires, analysis of data, and giving presentations. To respect a

patient means to acknowledge the value of their contribution and support means to ensure

ongoing working communication. Training is a key step that is needed to ensure that patient

partners have an adequate understanding of the research process to contribute meaningfully and

it is the responsibility of the researchers to ensure that patients have access to the training

resources that they need.8

Another way for researchers to avoid tokenism is to ensure that the entire research team is

engaged in the patient partnership, rather than having only one team member acting as a liaison

in a way that cordons off the patients from the rest of the team.6

2.3.8 Measuring Impact

PER is far from uniform in its application. It can range from full partnership to minor

consultation and can take place throughout the research lifecycle or only in one phase. These

factors make it difficult to establish a methodology for quantifying its impact. In fact, some

researchers argue that attempting to evaluate PER is fundamentally disingenuous because the

value gained from patient engagement is necessarily context-specific and the knowledge gained

by researchers in the process is experiential and therefore impossible to categorize or quantify.26

However, governments, funding organizations, researchers and patient organizations are

increasingly asking to see evidence for the benefits, value and return on investment of

engagement.44 Effective evaluation of PER could create evidence about the costs, benefits and

risks related to engagement for those who are critical of it and lead to more widespread use.

When the science around PER is strengthened, this knowledge can help to ensure that it becomes

an integral, robustly conducted and well-resourced component of research, and that the particular

gaps in our knowledge can be formally addressed in future research.44

A wide range of evaluation tools for PER have been developed, however most of them focus on

procedural aspects of research partnerships more than outcomes.44 Boivin et al conducted a

systematic review of evaluation tools for engagement in research and health system decision-

Page 18 of 128

making and assessed 27 tools using four criteria: scientific rigour, patient and public perspective,

comprehensiveness, and usability.44 Overall, the tools were lacking in scientific rigour, as

reliability testing was very rare and only a small number of them were informed by a literature

review – stakeholder experience was the most common source of knowledge. More than half of

the tools were developed with patients and members of the public, but their involvement was

usually limited to the pilot stage. Only five tools met all the criteria for comprehensiveness and

two for usability.44 Abelson and colleagues suggest that the fact that most evaluation tools are

questionnaires and surveys is a limitation because there are limits to the depth and breadth of

information gathered without personal interaction. They suggest a mixed methods approach is

best for evaluating PER.45 For example, researchers could administer surveys among research

partners and conduct one-on-one interviews.

2.4 Building Capacity for PER: The Resource Landscape

2.4.1 The Need for Resources

The set-up process for a successful research partnership requires detailed considerations so that

all parties involved are clear about expectations and have the necessary skills and resources to

fulfill those expectations.8,40 Research shows that whether or not researchers have clearly defined

goals and plans for a research partnership (other than to ‘tick a box’ on a grant application) is a

determinant of whether the partnership will have an impact.4 Failure to adequately prepare has

led to patients not feeling like they can participate and as a result, attending less meetings.25 Both

patients and researchers can benefit from resources that inform them about how research

partnerships work, what to expect and how to avoid or mitigate challenges.

2.4.2 Existing PER Learning Resources

A range of resources have been designed to increase knowledge about and build capacity for

PER for healthcare organizations, researchers, patients, families and caregivers. These resources

include webinars, online courses, one-day training courses, video resources, training modules,

and engagement evaluation tools. Many international organizations have developed tools,

particularly in the US and the UK. The Beryl Institute based in Texas provides on-demand

webinars which are led by patient experience leaders with the goal of sharing practises and

Page 19 of 128

strategies that can be implemented in healthcare organizations.46 Washington, DC-based PCORI

provides a series of webinars about patient-centered outcomes research that are designed to

present examples of engagement practises.47 The Warwick Medical School in the UK offers a

free one-day workshop to support patients or members of the public who are thinking about

getting involved in research or are already involved.48 INVOLVE is a national advisory group

established in 1996 by the UK’s National Institutes of Health Research (NIHR). INVOLVE

offers a one-day training course for researchers as well as patients and members of the public

who are already actively involved in research.49

Resource development for PER in Canada is on the rise as funding opportunities from CIHR’s

SPOR initiative increase and provincial SPOR support units provide grants for building capacity

in patient-oriented research. St. Michael’s Hospital in Toronto offers a free webinar course twice

a year called Partners in Research (PiR), designed for patients, caregivers and researchers. The

course consists of 4 live web-based sessions where participants receive personalized learning

through interactive activities and discussions with others who are interested in or already

involved in research.50 The Patient-Oriented Research Curriculum in Child Health (PORCCH)

was developed through the collaboration of various Canadian organizations. PORCCH is an

interactive online curriculum that is designed to build capacity in patient-oriented child health

research and enhance the skills of children, families, trainees, clinicians, scientists, educators and

administrators. It includes interactive tools, video vignettes and evaluation exercises.51 In 2018 a

training tool was developed at Western University in Ontario, called the Patient-Oriented

Research Training and Learning – Primary Health Care (PORTL-PHC) program. PORTL-PHC

is a free, online and self-directed program that focuses on skills development, context-specific

application, methods and examples of patient-oriented research in primary health care.52

The purpose of these supportive resources is to help people prepare for research partnerships and

the examples described show some of the range in format these resources can take. Although

some of these resources feature individual testimonials from people with partnership experience,

there are no resources to date that feature stories of researchers and patients with lived

experiences of partnerships, gathered in a rigorous way. This study evaluates such a resource.

Page 20 of 128

2.5 Evidence for the Benefits of Using Video Narratives

2.5.1 The Persuasive Power of Narratives

Communications literature provides ample evidence for the value of using narratives to influence

knowledge, attitudes and behavior in a wide range of areas. The use of narratives has been

evaluated for its impact on organ donor registration,53 cancer screening,54-56 medical cannabis

use,57 breastfeeding58 and communicating with people with intellectual/developmental

disabilities.59 A 2016 meta-analysis of studies that evaluated the persuasive influence of

narratives found that narratives positively influence message recipients’ beliefs, attitudes,

intentions and behaviors, moving them into closer alignment with the views presented in the

narrative: “Thus, we can categorically conclude that narrative does exert a causal influence on

four of the most common indices of persuasion” (pg. 461)60 (beliefs, attitudes, intentions and

behaviours).

The evidence for the persuasive power of narratives is strengthened further when narratives are

compared with non-narrative formats. In a study to measure the differential impact of messaging

format to promote organ donation compared the impact of video format on 781 study participants

(median age of 29 years, 41% female, 24% non-white). The study included six different videos:

four that included personal narratives, one purely informational and one unrelated to organ

donation (control). The video that had the highest impact on organ donation willingness was a

personal narrative video that featured a pediatric transplant recipient.53 A study that explored the

impact of messaging format for communicating breast cancer screening compared the impact of

video formats on 194 Italian-speaking women ages 18-30 years old. The study included a control

and four experimental formats: narrative video, didactic video, narrative infographic, and

didactic infographic. The results showed that the didactic video had the greatest impact on

awareness and knowledge about screening, while the narrative video had the most positive effect

on attitude and intention toward screening. In all cases the videos outperformed infographics.56

A study comparing narrative versus non-narrative video formats for presenting information about

medical cannabis use found that the narratives positively influenced attitudes, beliefs and

intentions toward medical cannabis use, and this in turn positively influenced attitudes, beliefs

Page 21 of 128

and intentions toward recreational cannabis use among 396 Israeli participants. Exposure to

narratives was particularly influential when the narrative presented the patient using medical

cannabis as not to blame for their illness.57 Authors investigating the differential impact of

informational versus narrative videos about breast cancer among African American women (n =

489; 67% with high school education or less; 77% with annual household income of $20,000 or

less; average age of 61 years) found a significant difference between groups in relation to

counterarguing (i.e., generating thoughts that counter, reject or discount a message), which is

associated with fatalistic beliefs about cancer and higher perceived barriers to mammography.

Those who watched the narrative video had lower counterarguing, higher engagement and talked

with family members more which increased message recall. The authors concluded that the

strength of cognitive and affective responses to narratives effects message processing and

behavioral correlates.61

2.5.2 The DIPEx Model

The internet is a major source of health information and support.62,63 Many online health

resources present “facts and figures” based on scientific evidence, however as social networking

sites continue to gain prominence in the online space, it has been found that many people are

using the internet not only to find technical information but also to find others with similar

experiences.62 A survey of 3,014 adults by the American thinktank Pew Research Center in 2012

found that 26% of adult internet users read or watched someone else’s health experience about

health or medical issues in the past 12 months, and 16% went online for the purpose of finding

others with similar health concerns.64 When people search for health information online they are

drawn to the parts of websites that show others’ experiences.63 Engaging with others’

experiences online can influence decision-making, decrease the sense of isolation and help a

person to adjust to their health condition.62

A unique way of presenting information was developed at the University of Oxford by the

Health Experiences Research Group (HERG) who initiated the Database of Individual Patient

Experiences (DIPEx). Both the HERG and DIPEx were founded by Dr. Ann McPherson, a

general practitioner who wanted to know more about the impact of health conditions on patients’

quality of life. The website that is now called healthtalk.org (formerly dipex.com and

Page 22 of 128

healthtalkonline.com) was created in 2001 to be a repository of evidence about personal

experiences of illness, gathered through rigorous qualitative research, accessible to the public.

Healthtalk.org is managed by the DIPEx charity in collaboration with the HERG, which is

responsible for the research that is displayed on the website.65 The HERG conducts in-depth

qualitative interviews with individuals living with various health conditions using diverse

sampling methods to produce a sample of 40-50 individuals per condition that includes a broad

range of experiences. The interview method combines an unstructured, open-ended narrative

approach with semi-structured prompts, so that the concerns, meanings and priorities of those

being interviewed can be emphasized66 The HERG researchers analyze the interviews to find key

themes and then write a lay summary of each theme which is displayed on healthtalk.org

alongside selected illustrative quotes and audio or video clips from the interviews. To date

healthtalk.org has more than 100 modules, each covering a different health condition.

This UK-based initiative is now a global movement. DIPEx International currently has members

from 14 countries: Australia, Canada, the Czech Republic, Brazil, Germany, Israel, Japan, Korea,

the Netherlands, Norway, Spain, the UK, Switzerland and the US. Each of these member groups

produces their own health experiences website using DIPEx research methodology to learn about

the health experiences of people living in their communities. The groups are all based within

accredited universities or university-affiliated research centers and led by senior scientists.65

With the websites from all of these countries combined there more than 140 health conditions

covered and interviews have been conducted in 11 different languages.65

The usefulness of this kind of internet resource has been subject to a few formal evaluations.

Researchers at Oxford evaluated the relevance of the DIPEx website for the information needs of

cancer patients.55 Before being shown the website, study participants were asked about what they

would like a health website to include and one of the things that they identified as important was

experiential information from other patients. After being shown the website, study participants

said that being able to hear the stories of patients like them would have made them feel less

isolated and afraid if they had access to it during their illness. Study participants also mentioned

that being able to access the emotional experiences of others without having to engage in return,

as would be expected in a support group, was perceived as valuable, especially for those who

experienced embarrassment about seeking help related to their condition. Participants also felt

Page 23 of 128

that the website would have assisted with decision-making and given them the confidence to be

more assertive when communicating their needs to health care professionals. Listening to others

describe experiences like their own created a sense of validation about their experiences and

perspectives.55 An evaluation of a DIPEx module on rheumatoid arthritis found that a majority of

the participants who completed the questionnaire found the website interesting and informative.

A majority also felt the website covered a wide range of information, covered experiences like

their own and would be useful to others.63 The need for further evaluation has been identified as

a priority amongst the DIPEx International members (Susan Law, personal communication).

2.6 Summary and Rationale for Further Evaluation

Patient engagement in research is emerging as an innovative way of involving patients in the

decision-making that impacts their care. It has gained prominence internationally and takes many

forms, some involving patients less and some more. The benefits of PER can include the

production of results that better align with patients’ needs, which can be applied in a way that

improves patient-centered care. Challenges include avoiding tokenism, achieving diverse

representation, and measuring the impact of PER. Several resources to support PER have been

developed in recent years but it is unclear how useful they are for patients and researchers. By

evaluating existing resources, we can gain valuable knowledge about what patients and

researchers need and want, and where the gaps are.

Page 24 of 128

Methods

3.1 Introduction to Methods

3.1.1 Description of Website

The website under evaluation in this study was designed using the Database of Individual Patient

Experiences (DIPEx) methodology.65 The first step in creating the website included recruiting 33

patients, caregivers, and researchers who had experience with patient-researcher partnerships,

using maximum variation sampling. Experienced qualitative researchers then conducted face-to-

face, semi-structured, audio and/or video-recorded, narrative interviews with the participants.65

These interviews took place at the participants’ home or another place that was convenient for

them (e.g., office). Next, the interviews were transcribed and thematically analyzed. Finally, the

website was constructed, with each theme represented as a topic page. Each topic page included

multiple short video/audio clips from the participant interviews and a short lay summary of the

theme. The website also included a page for each participant which included their first name (or

a pseudonym), their photo (if they’ve consented), and a short biography.

The patient-researcher partnership website (https://healthexperiences.ca/patient-researcher-

partnership/patient-researcher-partnerships) is housed within the Canadian DIPEx website

https://healthexperiences.ca/, which also contains other sections (e.g., breast cancer, family

caregiving). For the evaluation, links to the rest of the website were disabled, since the

evaluation was only meant to capture perceptions of the patient-researcher partnership website.

The patient-researcher partnership website contained 17 topic pages at the time of the evaluation,

and included clips from interviews with 33 different patients, caregivers, and researchers. Four of

the 17 topic pages had not yet been created at the time of the evaluation and ‘Coming Soon’ was

added to their title.

Topic pages were listed on the left side and other navigation along the top of the page. See

Appendix A for a screenshot of the website’s landing page.

https://healthexperiences.ca/patient-researcher-partnership/patient-researcher-partnerships

https://healthexperiences.ca/patient-researcher-partnership/patient-researcher-partnerships

https://healthexperiences.ca/

Page 25 of 128

3.1.2 Institutional Context

This evaluation is part of a larger study that received funding from the Ontario Strategy for

Patient-Oriented Research (SPOR) Support Unit (OSSU). It was one of seven projects that

received funding from OSSU through an initiative that aimed to increase capacity for patient

engagement in research across Ontario. The research team for this study was based at the

Institute for Better Health (IBH) at Trillium Health Partners (THP), in collaboration with team

members from The Ottawa Hospital, University of Ottawa, and the University of Toronto. The

advisory team also included two patient/caregiver advisors in addition to other researchers and

educational as well as patient engagement leads. The overall study was led by Susan Law (PI)

and the evaluation component (the focus of this thesis) was led by student PI Donna Plett.

3.1.3 Paradigmatic Perspective

The design of this evaluation is rooted in pragmatism, identified by Peggy Shannon-Baker as one

of four paradigms used in mixed methods research (which also include transformative-

emancipation, dialectics, and critical realism).67 Though there is some disagreement among

researchers about its definition, paradigms in research can be understood as the beliefs a

researcher holds about how knowledge is created.68 These kinds of beliefs are at the root of the

“paradigm wars” which have been a source of antipathy between qualitative and quantitative

researchers for many decades. 67

Pragmatism is distinct from positivist (quantitative) and metaphysical (qualitative) approaches to

research in one key way: the use of abductive reasoning, which moves back and forth between

inductive (qualitative) and deductive (typically quantitative) reasoning.68 Pragmatism is

outcome-oriented, focuses on the product of research, and characterized by an emphasis on

creating practical solutions to social problems.67 By embracing the elements offered by

qualitative and quantitative methods, a pragmatic approach maintains a balance between

subjectivity and objectivity which lends itself to analysis that is both contextually-based and

transferable to other contexts.67

Page 26 of 128

Pragmatism informed the design of this study in a few ways: by the inclusion of qualitative and

quantitative data collection methods, the framing of the study design in multiple theories (i.e.,

Social Learning Theory and Theory of Planned Behaviour), and the blending of different forms

of data in the interpretation of the results. Abductive reasoning was applied throughout the

process, particularly in the interpretation phase, as learnings from qualitative methods were

combined with learnings from quantitative methods. The mixing of methods also served to

increase the practical value of the findings by giving a fuller picture of how the website was used

and how people responded to it in various contexts. In these ways the design of the evaluation

struck a balance between reflexivity and objectivity, with the ultimate goal of creating learnings

that are practically useful and academically sound.

3.1.4 Methodological Approach

Mixed methods research, generally defined as any research that combines qualitative (QL) and

quantitative (QN) methods, has gained prominence in recent decades.69 While some argue that

qualitative and quantitative methods are separate and opposing forces,70 the employment of

mixed methods rejects this dichotomy by using both methods with the belief that together, in

some circumstances, they can provide more robust answers to research questions than either

method alone.

Quantitative methods (e.g., clinical trials, surveys) often use large sample sizes and can be useful

for measuring previously defined entities, processes, and concepts. For example, a mental health

researcher who wishes to understand the impact of depression on a population could administer

surveys across the population to measure how many report pre-defined symptoms of depression.

Qualitative methods (e.g., interviews, field observation) tend to use smaller sample sizes and are

useful for gaining deep and nuanced understandings of new or existing entities, processes, and

concepts. An example of this would be if the mental health researcher chose to conduct

interviews with a selection of individuals to understand what depression means to them and how

it impacts their day-to-day life. The QN approach would provide information with a limited level

of detail about a large proportion of the population (i.e., whether they experience pre-defined

symptoms of depression), and the QL approach would give a more detailed and nuanced

understanding of the lived experience of depression for a smaller proportion of the population. A

Page 27 of 128

mixed methods approach would recognize the value both kinds of data could add to their

research and include both in the design of their study. By combining quantitative and qualitative

findings, the researcher may gain insight into how useful the quantitative measure was for the

target population using qualitative findings. For instance, the qualitative findings may indicate

that some members of the population don’t identify with the terms used in the survey but do

experience symptoms of depression. This could inform a redesign of the survey to match the

terminology used by the population.

This simple example represents one way in which mixed methods could add value to research,

however there are many reasons why researchers may choose to mix methods. According to a

typology developed by Greene et al. there are five rationales for combining methods:

triangulation (for comparison of findings), complementarity (to assess various dimensions of a

phenomenon), development (to design second method based on findings from first), expansion

(to measure different phenomenon), and initiation (to address the goal of divergence).71 Mixed

methods research also takes many forms. Ozawa & Pongpirul (2014) present six prototypes for

mixed methods research: convergent parallel (where QN and QL are collected simultaneously),

explanatory sequential (where QN preceded QL), exploratory sequential (where QL precedes

QN), embedded (where one is contained within the other), transformative (where QN precedes

QL within a framework), and multiphase (where there are at least three phases of data

collection).11,72 The appropriate design in any given circumstance will depend on the research

question, population under study, and available resources.

Mixed methods approaches are importantly different from multi-method approaches. A multi-

method approach employs more than one methodology but does not necessarily integrate

findings across methods. A multi-method study may present findings from different methods

next to each other, but not consider how each may complement or inform the other. Mixing

methods, in contrast, aims to combine the strengths, and compensate for the limitations, of

qualitative and quantitative methods.73

Page 28 of 128

3.2 Study Design

3.2.1 Overview of Methods

This evaluation used a convergent parallel (i.e.,

concurrent) mixed methods approach, in which

qualitative data and quantitative data were

collected and analyzed separately and then

combined during the interpretation of the results.11

The combination of methods in our study was

meant to serve two purposes. First, mixing

methods enabled a comparison of quantitative

findings and qualitative findings (triangulation).

Second, multiple methods enabled

complementarity by using different ways of

capturing how participants perceived and

navigated the website.

The quantitative component of the study used a

one-group quasi-experimental design in that it was designed similarly to an experiment but did

not include a control group or randomized participants.74 Two forms of one-group quasi-

experimental designs were incorporated into the survey design: posttest-only (to obtain direct

feedback about the website) and pretest-posttest design (to obtain baseline and post-intervention

measurements of participants’ intentions and attitudes toward research partnerships).74 This

quantitative methodology was chosen in lieu of a randomized controlled trial because it was the

most feasible and rigorous option given time and budget constraints. The qualitative component

of the study used a qualitative descriptive design, which draws from a naturalistic perspective,

examines phenomena in its natural state, and tends to be less theoretical than other qualitative

approaches.75 This qualitative methodology was chosen because it was the most conducive to

providing a straight description of the phenomena under study (i.e., participants’ reactions to the

website), which was the goal of this study.76

Figure 1: Study design.

Page 29 of 128

We utilized three data collection methods: surveys, focus groups, and website analytics. Surveys

are widely used by social researchers, as they offer a relatively quick way of obtaining

information from a large group of people. By using standardized questions, surveys make it

possible to quantify differences in responses between subgroups, which is useful for developing

an understanding of the needs of different groups.77 Despite the popularity and usefulness of

surveys, they are subject to some limitations. These include response and non-response biases

(e.g., when some groups are more likely to complete the survey than others) and measurement

error (e.g., when questions are poorly written and/or responses are open to misinterpretation).78

Focus groups are a qualitative method of data collection also commonly used by social

researchers. Focus groups typically include smaller groups of participants to enable open

discussion, guided by at least one skilled moderator. Focus groups leverage group interactions

and enable participants to exchange experiences and perspectives with each other.79 These group

dynamics are believed to help participants explore and clarify their views more easily than in a

one-on-one interview.79 Focus groups are also convenient, in that they enable simultaneous data

collection from multiple people. Some challenges or limitations of focus groups include the

possibility for some individuals to dominate the discussion, tendencies toward normative

discourse, and conflicts and arguments between group members.80

Web analytics is a relatively new method of data collection. It is commonly used for business

and market research in order to understand web traffic and to assess and improve a website’s

effectiveness or profitability. Web analytics services, such as Google Analytics, collect detailed

information about how websites are used, including (but not limited to) what visitors click on

within the website, how long visitors stay on the website, and how visitors got to the website.

Web analytics is limited as a research methodology in that it does not capture how visitors

perceive content or why some content is viewed more than other content.

Our study used two surveys, one administered before participants accessed the website and one

after they’d had access for approximately one week. The purpose of the first survey was to

collect demographic information, learn what participants felt about the internet as a resource to

Page 30 of 128

learn about PER, and what their intentions and attitudes toward PER were. The purpose of the

second survey was to learn about their perceptions of the website and whether their intentions

and attitudes toward PER had changed after spending time on the website. This quantitative

component added value by enabling measurement of variables (e.g., participants’ attitudes

toward patient engagement in research) and comparison of differences in responses between

groups (e.g., patients and researchers). It was also helped to give context to the qualitative

findings.

We invited some participants to partake in a focus group a few days after completing the

surveys. The purpose of the focus groups was to gain a richer understanding of participants’

response to the website by allowing them to share their thoughts and feelings in a semi-

structured discussion. This qualitative component added value because very little literature exists

about the evaluation of resources for patient engagement in research, and no literature exists to

evaluate a website that uses video narratives to provide information about partnerships. The

exploratory approach made it possible for study participants to bring new themes to light during

the focus groups.

Web analytics data were collected for the period that participants had access to the website. The

purpose of collecting this data was to gain insight into how participants navigated the website

and which content was the most engaging for participants. Web analytics measured a level of

detail about participants’ real-time utilization of the website in a way that could not be measured

by surveys or in focus groups. Web utilization data also added a unique and essential component

to the research because it is not self-reported, and in that way is not subject to some of the biases

associated with interview or survey data.

Page 31 of 128

By combining the qualitative and quantitative data in the interpretation phase we were able to

uncover convergences and divergences of findings between methods. The overall findings of the

study were likely to be more robust than if only qualitative or only quantitative data had been

gathered. This is because the different

methods compensated for the strengths

and limitations of the others (e.g., by not

being subject to all of the same biases as

the other). Findings that were consistent

between methods were strengthened by

mixing methods, and divergent findings

added value by creating opportunity for

further inquiry.

3.2.2 Survey Tool Development

The surveys used for the evaluation were adapted from two validated survey instruments: the

eHealth Impact Questionnaire (eHIQ) and a Theory of Planned Behaviour (TPB) tool.

The eHIQ is a 2-part, 37-item, 5-point Likert scale tool that was developed and reliability tested

by researchers at the University of Oxford.81 The purpose of the tool is to assess the effects of

online content that includes factual and experiential health information on visitors of the

resource.82 The eHIQ has been used to evaluate other DIPEx (i.e., health experiences) website

modules.83,84 It has not yet achieved widespread use however, so applying it in the context of this

website provided an opportunity to learn more about its usefulness and areas for improvement.

Development of the eHIQ included a literature review and analysis of 99 interviews related to

patient and caregiver health experiences.82 During scale validation an exploratory factor analysis

revealed two distinct item pools, one relating to general views about using the internet in relation

to health and one relating to the results of spending time on a specific health-related website.81

Five subscales were also identified: 1) attitudes towards online health information, 2) attitudes

toward sharing health experiences online, 3) confidence and identification, 4) information and

presentation, and 5) understanding and motivation (see Appendix B, Table 11 for survey items in

each subscale). The first two subscales are Part 1 of the eHIQ and were administered before

Figure 2: Diagram of methods

Page 32 of 128

participants accessed the website. The remaining three are Part 2 and were administered after

participants had spent time on the website.

The TPB survey used for this study was developed using two resources: a TPB tool developed

and tested by Boyko et al85 to evaluate research use in policymaking as well as the manual used

by Boyko et al to develop their tool.86 The manual was designed to assist health services

researchers in developing TPB tools for a range of purposes. The TPB is a widely used

psychological model of behaviour

change and enables researchers to

investigate attitudes and beliefs

that underly health-related

behaviour.86 TPB tools measure

behavioural intention based on

three predictors: attitudes,

subjective norms, and perceived

behavioural control (Figure 3).

3.2.2.1 Adaptation Process

To adapt the eHIQ for this study the research team met to review each item and determine

whether it was appropriate for use in this context of patient-researcher partnership resources. The

main consideration was whether or not each survey item was uniquely relevant to the

experiences of a person managing a health condition (e.g., have to make decisions about

treatment options), or also applicable in the context of a person making decisions related to

research partnerships. After multiple discussions the following items were excluded because they

were considered to be relevant to a website about an illness or disease, but not to a website about

patient-researcher partnerships: “The internet is useful if you don’t want to tell people around

you (for example, your family or people at work) how you feel,” “It can be reassuring to know

that I can access health-related websites at any time of the day or night,” and “I would use the

internet to check that the doctor is giving me appropriate advice.” All remaining items were

retained but adapted for relevance to the website (see Appendix B, Table 11 for comparison of

Figure 3: Theory of Planned Behaviour.

Page 33 of 128

original and adapted versions). As in the original version, all survey prompts were presented in a

5-point Likert agree/disagree format.

The TPB is a conceptual model that researchers adapt for their purposes, and there is not on

original survey tool. Instead, the TPB tool used for this study was developing using a guidebook

written for health researchers.86 The TPB tool was adapted to be relevant to the behaviour the

website was believed to inspire – participation in patient-researcher partnerships (see Appendix

B, Table 12 for survey items by domain). To determine the website’s influence, we used a pre

and post model for this tool. All survey participants responded to the TPB questions twice, once

before they were given access to the website, and once after they had had approximately one

week to review the website content. This measure was used to separate pre-existing attitudes and

intentions related to patient engagement in research from any changes resulting from their

experience engaging with the website content. As in the guidance materials used,85,86 all

questions were presented in a 7-point Likert format. Alternative formats are acceptable for some

constructs but for the sake of simplicity and consistency we chose to use the 7-point Likert

throughout.

Additional questions were added to the survey to capture participant details, including age,

gender, country of birth, self-described ethnic origin/cultural background, city and province of

residence, education, employment status, and occupation. Participants were also asked to identify

what their role in a patient-researcher partnership would be (researcher, patient/caregiver, either,

or other), their amount of prior experience as a research partner, and whether they had received

prior training in research partnerships. These questions were included to enable a descriptive

account of participants as well as options for stratification of responses to other survey questions

by participants partnership role (e.g., patient or researcher) (see Appendix G for full surveys).

The survey was reviewed by multiple members of the research team throughout the development

process. This was to ensure that questions were appropriate and worded clearly and in a way that

both patients and researchers would understand.

Page 34 of 128

3.2.2.2 Integration with Online Survey Software

We chose to administer the surveys online, rather than in person or by telephone, for a few

reasons: 1) it would be convenient for participants because they could complete the survey at

their location and time of choice, 2) it would enable us to broaden our recruitment strategy to

include participants across Canada, since involvement in the study could be completed entirely

online, and 3) since the subject of our evaluation was a website, we could only recruit

participants with internet access anyway (because non-internet users could not use the website),

so there was not a concern about excluding non-internet users from the evaluation.

We considered a variety of online survey platforms, including Google Forms, Qualtrics,

SurveyMonkey, and REDCap. Our main considerations were data privacy, user-friendliness,

cost, and the software’s ability to accommodate a pre and post survey design. Our first choice

was REDCap, as it offered the highest level of data privacy and functionality, was the most

likely to receive REB approval, and THP had an existing account that could be used for this

study at no additional cost. The surveys were imported into REDCap and tested by the research

team.

It was later learned that THP’s REDCap account only allowed for surveys to be completed by

those connected to THP’s intranet. This would have made it impossible for participants to

complete the surveys at home, so it was necessary to move the surveys to another platform and

submit a REB amendment before proceeding with data collection. We chose to move the surveys

to SurveyMonkey because of its user-friendly interface and because THP had an account that

could be used at no additional cost. This change caused a delay to data collection, but both REBs

(THP and University of Toronto) approved the change.

3.2.3 Web Analytics Strategy

In addition to obtaining direct feedback from participants through surveys and focus groups we

wanted to learn how website visitors navigated the website, which pages were clicked on the

most, how much time was spent on each page, and how much time was spent overall per session.

To capture this, and additional web utilization data we used Google Analytics. We created a

dashboard on Google Analytics to track visitor behaviour on the section of the website under

Page 35 of 128

evaluation (the patient-researcher partnership section), separating out traffic to other sections of

the website.

Since a large portion of the website content was video, we also accessed the YouTube account

where the videos embedded on the website were housed. YouTube provides highly detailed

information about video views, including frequency and timing of views, viewer location, and

referral source. This data would enable us to know which videos were most popular, whether

viewers watched videos in their entirety or only in part, and whether there was a difference in

views of audio-only and video clips. It was later learned that the way most videos were posted on

the website (with autoplay set to ‘on’) caused YouTube not to count their views or track any

statistics. As a result, we could not access reliable data about users’ interaction with video

content on the website.

The focus of our web analytics strategy was user behaviour (i.e., how users navigated the

website) rather than acquisition (i.e., how users got to the website) since at the time of the

evaluation the website was only accessible through a private link and there had been no formal

promotion through social media or otherwise. Therefore, we did not expect organic traffic (e.g.,

via web search) and knew that the only visitors to the website would be those we provided access

to (i.e., participants). We also did not use demographic data, such as age and gender, collected by

Google Analytics because 1) this data was only available for a small percentage of website

visitors, 2) it is unclear how reliable these data are, and 3) we collected demographic information

about participants in our surveys.

3.2.4 Ethical Considerations

Since the evaluation included human subjects, approval from the ethics boards of the institutions

where the research would be conducted was required. The ethical considerations we addressed

included potential conflicts of interest, recruitment methods, possible risks to participants, and

the protection of patients’ confidentiality and privacy.

Only one potential conflict of interest was identified, and that was prior relationships between

potential participants and the study PI (SL). This risk was minimized by clarifying that the

student PI (DP) would extend the invitations to participate.

Page 36 of 128

Ethical considerations related to recruitment were addressed by clarifying our target population

and our intention to include a diverse sample with respect to age, gender, ethnic background,

type of illness, and type/amount of previous engagement or health research experience.

The only potential risk to participants identified was the possibility the focus group participants

would disclose personal health information. This risk was mitigated by the inclusion of a highly

experienced facilitator (SL) who ensured that participants were comfortable and aware of their

rights as study participants, as well as the duty of all focus group participants to respect the

confidentiality of information shared during the session.

Confidentiality and privacy of participant information were protected in multiple ways. First, the

surveys did not request any identifying information (e.g., name, email address, birth date) so

survey responses could not be linked to any individual. A unique identifier was assigned to each

participant in SurveyMonkey (the online survey software used for this study) so that the research

team could link pre and post survey responses without recording identifying information.

Second, identifying participant information that was collected during recruitment (e.g., names,

email addresses) was stored securely on THP’s intranet, separate from survey responses and

other data. Third, after data collection was completed the files containing identifying participant

information linked to unique identifiers were permanently deleted and the email addresses used

to send survey invitations were removed from the SurveyMonkey account.

This evaluation received REB approval from the University of Toronto in September 2019. An

earlier version of the evaluation that only included focus groups had been approved by THP’s

REB as part of the larger research project, but an amendment was needed to approve the changes

made to the design of the evaluation. Approval for the amendment was received in October

2019. A further amendment was required from both REBs in April 2020 when it was learned that

the originally proposed survey software (REDCap) would not enable participants to complete the

surveys at home and we were required to switch to a different software (SurveyMonkey). Both

REBs approved this amendment in April 2020.

Page 37 of 128

3.3 Sampling & Recruitment

For the focus group and survey components of the study we aimed to recruit an equal number of

Canadian health researchers and patients/caregivers (individuals who have current or past lived

experience with, or caring for someone with, illness). We used a combination of convenience,

snowball and purposive maximum variation sampling in order to achieve a diverse sample with

respect to age, gender, ethnic background, type of illness and type/amount of previous

engagement or health research experience (no prior engagement experience was required).

Inclusion criteria for participants included: 1) minimum 18 years of age; 2) able to read and

express themselves in English; 3) able to participate independently in focus groups and/or filling

out surveys; 4) willing to be audio recorded (for focus group participants only) and, 5) have

access to the internet. Exclusion criteria included: 1) unable to communicate in English; and 2)

less than 18 years of age. We recruited health researchers and patients/caregivers to complete the

surveys and participate in focus groups.

With assistance from JV (thesis committee member) and JM, a THP biostatistician, a statistical

power analysis was conducted to estimate the sample size needed for statistical significance.

Assuming a moderate effect size, as recommended by the THP manual,86,87 a minimum of 34-44

participants was required. Our target sample size was 44, so that if some participants were lost to

follow-up our final number would still fall within the required range.

3.3.1 Recruitment Process

Our primary recruitment strategy was to reach out to organizations with existing databases of

patient advisors and/or health researchers and ask them to distribute an invitation to participate to

their membership via electronic mailing list and/or web posting (see Appendix D for list of

organizations we reached out to). We chose to target existing patient advisors in our recruitment

strategy for two reasons: 1) we believed that patients with prior experience with patient

engagement would be more likely to be familiar with patient engagement in research and be

interested in participating in our project, and 2) distributing our invitation through existing

networks enabled a broader reach than what would have been possible if we had built our own

Page 38 of 128

network for the purposes of this project. We included all forms of health researchers (e.g.,

clinical, biomedical, health services) in order to represent a range of researcher perspectives.

We provided organizations that expressed interest in sharing our invitation with details about the

study, including the project title, purpose, inclusion criteria, time commitment, and our contact

information (see Appendix C for sample recruitment email). Interested individuals who received

the invitation through their network were invited to contact us by email indicating their interest.

We informed interested individuals (i.e., those that contacted us by email) that a link to the

surveys and website under evaluation would be sent to them by email. In addition to the strategy

described above, we distributed the invitation among health researchers within the professional

networks of the research team.

Focus Group Recruitment. We contacted two patient and researcher groups to inquire about

group member’s interest in participating in the study by completing online surveys and

participating in a focus group. One group was a hospital-based research and innovation working

group located in Mississauga, whose membership included patient and caregiver advisors. The

other group was a university-based patient engagement working group located in Toronto, whose

membership included patient and caregiver advisors as well as health researchers.

3.4 Data Collection

3.4.1 Consent Process

A separate consent process was used for surveys and focus groups, as ethical considerations were

different for these methods. Participants indicated their consent to participate in each survey by

selecting ‘agree’ on the first page of the survey, which detailed the study, their involvement, and

any risks (see Appendix G for survey consent form). Consent for focus groups was obtained

through email. Participants were sent a copy of the focus group consent form (see Appendix E

for focus group consent form) and given time to read the forms and invited to ask any questions

they had. When they felt comfortable with the information, they could indicate their consent by

email, by signing the consent form and emailing it to us prior to the focus group, or verbally at

the beginning of the focus group after their questions had been answered. All participants were

Page 39 of 128

invited to share questions and concerns about consent at the start of the focus group. Permission

to audio record the session was requested both in the consent form and verbally at the start of the

focus group.

3.4.2 Surveys

Before being introduced to the website, participants were invited by email to complete the first of

two surveys, which included sociodemographic questions, Part 1 of the adapted eHIQ, the TPB

questions, as well as open-ended questions to enable participants to add additional comments and

clarification of their answers. Following the completion of this survey, participants were sent a

link to the website and invited to use the website at their convenience over a period of

approximately one week (participants could request an additional few days if required). A week

after completing the first survey, participants were sent a link to the final survey, which included

Part 2 of the adapted eHIQ, the TPB questions, and open-ended questions to enable participants

to add feedback about the website in their own words (see Appendix G for full surveys). All

surveys were administered online using SurveyMonkey online survey software.

3.4.3 Focus Groups

Focus groups were originally scheduled to take place at IBH and the University of Toronto but

were conducted online via Zoom because of COVID-19 social distancing guidelines. Participants

were contacted by email to ask if they could accommodate this change, and whether they

required any assistance with the videoconferencing technology.

The focus groups lasted one to two hours and took place after participants had completed both

pre and post surveys. This was to ensure that their experience during the focus group did not

influence their survey responses. Focus group discussions were led with the aid of an interview

guide designed according to similar themes as in the surveys. Questions were open-ended and

exploratory to allow for participants to direct the discussion toward what was most important to

them.14,88 The interview guide probed: (i) reactions to the website design and layout, (ii)

reactions to the website content and understanding of the information presented, (iii) perceptions

about the potential usefulness of the website, and (iv) the influence of the website on knowledge,

attitudes and intentions related to patient engagement in research (see Appendix F for interview

Page 40 of 128

guide). The discussion was co-led by an experienced moderator (SL, thesis supervisor) and the

student PI (DP). SL welcomed participants, introduced the discussions, fielded questions about

the consent form, and confirmed consent for audio-recording. DP provided details about the

study and led the discussion about the website, which included asking questions from the

interview guide and probing on topics brought up by participants during the discussions.14,88

At the end of the focus group participants were offered a small honorarium for their time. Since

the focus groups took place online and participants joined from home, no travel or parking costs

were incurred.

3.4.4 Web Analytics

No specific action was required to collect web utilization data since Google Analytics

automatically tracks user behaviour.

3.5 Data Analysis

3.5.1 Survey Data

Survey data was exported directly from SurveyMonkey into Microsoft Excel. All responses from

both surveys (which contained no identifying information) were compiled into a single

spreadsheet with a row for each respondent and a column for each survey item. Each participant

was assigned a unique identifier that included a number and ‘P’ for patients and caregivers, “R’

for researchers, ‘B’ for those who identified as both, and “O” for others (e.g., research staff).

Participants who identified as both were grouped into either ‘P’ or ‘R’ for the quantitative

analyses, based on which group they chose to identify as for the evaluation. Unique codes were

also created for each survey item. We linked each participants’ responses to Survey 1 and 2

using a unique identifier in SurveyMonkey. This made it possible to protect the privacy of

respondents by separating all identifiers (e.g., email addresses, IP addresses) from survey

responses while enabling us to link pre and post answers.

After importing into Excel, DP checked the data for missing, unusual, or inconsistent responses

(e.g., responses outside of the Likert response range or conflicting responses to similar questions

Page 41 of 128

by the same respondent). Open-ended responses were moved into a separate document, which

was sorted by survey question and each response was recorded verbatim alongside the

respondent’s unique identifier. To facilitate statistical analysis, all responses to Likert-type

questions and a selection of demographic information for stratification (i.e., role in partnership)

were copied from the spreadsheet containing all responses into a separate spreadsheet.

Statistical analysis was conducted in MS Excel and RStudio by DP, with guidance from JV and

JM, a Trillium Health Partner’s biostatistician. Individual survey responses from the adapted

eHIQ and TPB tool were grouped according to their source literature.81,86 The score for each

eHIQ subscale was calculated using the scoring algorithm used by the original authors, which

transforms the score to a 0-100 metric (where 0 indicates low perceived value and 100 indicated

high perceived value).81 Cronbach’s Alpha tests were run for all five sections of the adapted

eHIQ to find if adaptations to the survey tool reduced internal consistency of responses within

each grouping. A score of 0.70 or higher was considered acceptable (see Table 1 and Table 4 for

scores).

While the eHIQ has been formally validated81 conducting a test for internal consistency was

required to ensure that our revisions to the wording of questions and omissions of some

questions did not result in poor correlations. More robust forms of validation (e.g., factor

analysis) were determined not to be feasible for this study due to time constraints and our small

sample size. The four groupings of the TPB were organized according to the design manual86

with the score for each group defined as the average of per-item scores (see Appendix B, Table

12 for TPB items by domain).

Individual scores from the TPB survey tool were evaluated to determine changes before and after

interaction with the website using Wilcoxon Signed Rank tests. Mann-Whitney U tests were used

to compare differences in responses between groups (i.e., patients and researchers) for both eHIQ

and TPB data.89 These non-parametric tests were chosen because our sample size was small and

our data were not normally distributed, which made it unsuitable for parametric tests.

Demographic variables and responses to individual survey items were presented in counts and

percentages. Means and standard deviations were calculated for survey domains/subscales (e.g.,

Page 42 of 128

behavioural intention, attitudes). An alpha level of 0.05 was used to assess significance for the

pre-post analysis and between group comparisons.

3.5.2 Qualitative Data

Audio recordings from focus groups were transcribed verbatim (by DP) but excluded utterances

that did not contribute to the topic (e.g., ‘um,’ ‘uh’).90 Responses to open-ended survey questions

were compiled into a word document and sorted by question and type of participant (i.e.,

patient/caregiver, researcher, or other). To develop a detailed understanding of the information

provided by focus group participants we conducted a hybrid content analysis of the focus group

data,88 informed inductively and deductively by themes from the interview guides as well as

themes that emerged during the focus groups.

Analysis of data from the focus groups followed Braun and Clarke’s six step method.91 Before

coding began, DP checked the transcripts for accuracy and re-read to increase familiarity.91

Initial codes (i.e., recurrent topics or ideas) were created by DP to systematically categorize the

content.91 A separate document was created by DP with each code as a heading and data from

transcripts were copied into the document under the relevant code, alongside the participant’s

unique identifier. This was an iterative process that involved multiple readings of the transcripts.

After the initial codes were created, they were collated into categories (i.e., topics of discussion,

e.g., the website’s navigation) and a codebook was created. The codebook identified each

category and its codes and included a short definition of each code. Next, SL (who co-facilitated

the focus groups) and KK (thesis committee member) reviewed the codebook created by DP to

ensure accuracy and completeness. Using feedback from SL and KK, DP revised the coding

framework to include additional codes and created a concept map to explore relationships

between codes. The codes were then arranged into themes (i.e., meanings derived from the text),

which were reviewed again by KK and SL. Once agreement about the themes was reached, DP

wrote summaries of each theme, including selected quotes.

Responses to open-ended survey questions were analyzed separately and presented alongside the

quantitative survey findings to add context and explanation for the quantitative survey findings.

Page 43 of 128

3.5.3 Web Analytics

Website analytics data was exported from Google Analytics to MS Excel. We only analyzed data

collected within the evaluation period, defined as the period between the date the first participant

was given access to the website and the date when the final participant completed the final

survey. The exported data included the number and type (i.e., new or unique) of users, number

and average length of sessions, number and types of pages viewed, average length of time spent

on each page, number of pages viewed per day, and device types (desktop, mobile, or tablet).

Page view data was then organized by type of page (e.g., topic) and ordered by number of views

per page to show the most and least popular content within each category. A timeline was

created to illustrate daily page views over time (including the two weeks before and two weeks

after the evaluation period for context).

3.5.4 Integration & Interpretation

After analyzing the quantitative and qualitative data separately, the findings derived from each

analysis were compared to uncover divergence and convergence. This integrative technique is

one of four integrative processes identified by Creswell and Plano Clark (2011),11 in which only

findings are merged, not the data itself.92 The study was designed so that both components of the

research question (impact and overall impression of the website) were probed using both

quantitative and qualitative methods: impact was measured quantitatively through the pre-post

analysis of participants’ attitudes and intentions related to research partnerships, and qualitatively

through the inclusion of open-ended questions that asked directly about participants’ perceptions

of the website’s impact, and; overall impression was measured quantitatively through the eHIQ

survey questions, and qualitatively through open-ended survey questions and the focus groups.

By including this duplication, we aimed to gain a fuller and more accurate understanding of

participants’ overall perception of the website and its impact. Furthermore, duplication enabled

both qualitative and quantitative answers to the research question, which could then be compared

and contrasted to uncover complementarity and/or conflict among them.

Some integration also occurred during the writing phase, which involved transformation of

qualitative data from open-ended survey questions into quantitative data (“quantitizing”)93 to

Page 44 of 128

facilitate the creation of mixed data tables. The tables included direct quotes from participants

(qualitative data) as well as counts (quantitative data) that represented the proportion of

participants who gave a similar kind of response (e.g. agreed with a statement) or referred to a

similar topic in their answer.

3.6 Dissemination Plan

Upon completion of the research a mixed methods paper will be prepared for publication in an

academic journal (e.g., Patient Experience Journal), as well as a slide-deck and/or poster for

presentation at an academic conference (e.g., CAHSPR Academic Conference). These materials

will include a joint display of quantitative and qualitative findings to showcase convergent

and/or divergent results and illustrate the benefits of applying a mixed methods approach.

A plain language summary of the results will be created and disseminated to the advisory group

for the overarching study and the evaluation participants who indicated interest in seeing the

results. It will also be made available on the website (healthexperiences.ca). We will also reach

out to other teams that have developed or are developing resources for patient engagement in

research to inform them of our findings, refer them to the webpage where the evaluation findings

are summarized and extend an open invitation for further discussion or collaboration.

http://www.healthexperiences.ca/

Page 45 of 128

Results

4.1 Overview

This chapter begins with the recruitment results (i.e., the number of participants recruited). Next,

results from both surveys are reported, including demographic details, analysis of responses to

the adapted eHealth Impact Questionnaire and the Theory of Planned Behaviour tool, and

presentation of responses to open-ended questions. Next, web utilization is reported, followed by

findings from the focus groups. Finally, convergences and divergences within the findings are

summarized and an overall summary of findings is provided.

4.2 Study Sample

Thirteen of the 27 organizations that we reached out to distributed project information to their

members. Most invitations were sent via electronic mailing lists and one was posted on the

organization’s website. The total number of people

that the invitation reached is unknown as the

organizations did not provide us with details about

their membership or the number of people on their

mailing lists. We estimate that the total was well over

500 based on our knowledge of the size of the

mailing lists of some of the organizations we reached

out to.

A total of 57 people responded to the invitation by

email, expressing interest in participating in the

study. The invitations they responded to came from

11 different sources. Of the 57 who expressed

interest, 49 completed the first survey, 6 did not

respond to follow-up emails, and 2 requested to be

removed from the study. Of the 49 who completed

the first survey, 41 completed the final survey, 7 were unresponsive to email reminders and one

requested to complete the survey after the window for data collection had closed.

Figure 4: Recruitment flowchart.

Page 46 of 128

Of the 41 who completed both surveys, 11 agreed to also participate in a focus group (one later

opted out). Two focus groups were held with five participants attending each.

4.3 Survey 1 Findings: Baseline Characteristics

4.3.1 Participant Characteristics

Demographic details were collected in the first survey, so we obtained participant characteristics

from all 49 participants who completed the first survey. The largest proportion of respondents

were patients or caregivers (51%), followed by researchers (39%), and research staff (e.g.,

patient engagement coordinator) (10%). Five participants indicated that they had experience both

as a researcher and as a patient or caregiver – three chose to take the perspective of a patient for

the purposes of this study and two chose a researcher perspective.

A large proportion of participants held postgraduate degrees (73%), were born in Canada (69%),

were female (67%), resided in Ontario (65%), were over the age of 54 (53%), and held full-time

employment (47%). Many had experience as a patient or researcher partner in three or more

projects (43%), while less than half had received formal training related to research partnerships

(37%). Of the 19 researchers, 11 identified as health services researchers.

Patient participants differed from researcher participants in age and employment status, as most

patients were 65+ (56%) and retired (52%), while almost no researchers identified as 65+ (5%)

or retired (5%). Patients and caregivers had experience with a wide range of health conditions

including cancer, diabetes, mental health conditions, dementia, allergies, osteoarthritis, chronic

pain, fibromyalgia, respiratory illness, kidney disease, and spina bifida. See Appendix H for full

demographic details.

4.3.2 Adapted eHIQ Part 1

Prior to being given access to the website, participants answered survey questions about their

attitudes toward online information about patient-researcher partnerships and the value of

experience-based information related to partnerships. Measuring these attitudes before

Page 47 of 128

participants were given access to the website helped to give context to their responses to

questions in Survey 2.

All 49 respondents completed all 8 survey questions in this section (no missing data). The survey

question that received the highest positive response (41 of 49 respondents selected 4 or 5

(agree/strongly agree)) was “It would be helpful to see other people’s partnership-related

experiences on the internet.” The least positive response (24 of 49 respondents selected 4 or 5)

was “The internet is a good way of finding other people who are facing patient-researcher

partnership-related decisions I may also face” (see Appendix J).

Mann-Whitney U tests were run to determine whether patients and researchers responded

differently to these questions and no significant differences were found (see Table 1).

Table 1: Adapted eHIQ Part 1 results, by domain.

Patients Researchers P vs R

n M±SD n M±SD p Cronbach’s a

Attitudes toward online info

about PRPs

25 71.25±18.84 19 68.09±18.03 0.56 0.87

Attitudes toward experience-

based info about PRPs

25 68.75±22.39 19 68.75±18.04 0.95 0.82

PRP = patient-researcher partnership

In the survey, respondents were given the opportunity to answer open ended questions which

aimed to capture any other thoughts they had about web-based information and PRPs. Comments

provided related to concerns about internet accessibility for certain populations (e.g., special

needs):

“[I]t’s still critically important to understand that there are still many people

who do not have the technology or knowledge accessibility to the internet so

there needs to be additional resources for these people.” -B26

Reliability and trustworthiness of information on the internet was also referenced:

“There is a lot of bogus information on the internet. Don't see why patient-

researcher partnerships would be any different.” -P07

Page 48 of 128

“The tool (internet) is great. The trustworthiness of its content is not.” -O31

Some expressed preference for in-person conversation as a source of information:

“The Internet is a good place to start but talking to people is much better for

the feeling of what researchers and patients can have in common.” -P32

Others noted the difficulty of portraying the complexity and context-dependant nature of

partnerships online:

“Its a great starting point, but not sure one can understand the complexities

involved without actually doing it!” -R35

“The internet can be helpful, but most patient-researcher partnerships are so

unique that it can be hard to draw on others experience.” -R03

4.3.3 Theory of Planned Behaviour at Baseline

The first survey also contained the TPB questions. Measuring participants’ attitudes, intentions,

subjective norms, and perceived behavioural control related to PRPs at baseline served two

purposes: 1) to enable a comparison between responses before and after visiting the website, and

2) to provide a deeper understanding of the participants recruited for this study, which informed

the interpretation of our findings.

All 49 participants responded to the TPB questions. Overall, only 2 of 588 possible responses

(49 participants x 12 survey items) were missing – a >99.9% response rate.

Results from the TPB tool measured at baseline showed high levels of intention to engage in

partnerships and positive attitudes about partnerships among both patients and researchers. The

median response to both intention questions (“I want to engage in PRPs” and “I intend to engage

in PRPs”) was 7, which is the highest level of agreement allowed on the 7-point scale.

Researchers reported feeling a higher degree of pressure from their peers (‘Subjective Norms’) to

engage in partnerships than patients (p = <0.01). Also, patients felt a moderately higher degree of

control over their decisions (‘Perceived Behavioural Control’) about whether to engage in

partnerships than researchers (p = 0.04) (see Table 2).

Page 49 of 128

Table 2: Baseline TPB findings by domain – between group comparisons.


n M±SD n M±SD p

Behavioural Intentions 25 6.36±0.94 19 6.68±0.61 0.17

Attitudes 25 6.07±0.84 19 6.44±0.45 0.20

Subjective Norms 25 2.43±1.28 19 4.98±1.17 <0.01*

Perceived Behavioural Control 25 5.77±0.92 19 5.18±1.06 0.04*

*indicates a significant difference (p = <0.05)

The two survey items that received the greatest different in responses between patients and

researchers were “It is expected of me that I engage in patient-researcher partnerships” and

“People who are important to me want me to engage in patient-researcher partnerships.” Patients

strongly disagreed with these statements, while researchers strongly agreed (see Table 28).

The survey included questions about what participants perceived as barriers and facilitators to

engaging in PRPs (i.e., what made them ‘easy’ and what made them ‘difficult’) (see Table 3).

Prior experience was cited by many patients and researchers as a facilitator. Researchers tended

to also cite their relationships with colleagues and patients tended to cite their personal

motivations. The challenges reported by researchers often related to logistics and time

commitments as well as finding patients to work with. Patients cited a lack of knowledge or

adequate information as well as feeling unheard or disrespected by researchers.

The survey section on subjective norms included questions about whether participants felt

pressure or expectations from people who are important to them to participate in partnerships.

After responding to those questions, respondents were asked to indicate which people they had in

mind while answering those questions. There was a large difference in responses between

researchers and patients – researchers almost exclusively identified colleagues as the source of

pressure and most patients named family as the source.

The survey section on perceived behavioural control included an open-ended question where

respondents could explain why they felt participating in partnerships was not in their control.

Page 50 of 128

Table 3: Barriers and facilitators to engaging in partnerships, selected quotes.

What makes PRPs easy for you? What makes PRPs difficult for you?

Patients &

Caregivers

Experience

“I have experience in patient-oriented

research and research and am really

interested in positive change.” -P33

“i have done it few times and i believe i

can make important contributions” -P20

Personal Motivation

“What I do believe is in my own heart,

that would be great if we all could give

our personal experiences and journeys to

people that are making changes in

treatment through research for people

coming after us.” -P32

“I have motivation to pursue medicine

and this is one way I can be involved and

learn a little bit about the field.” -P46

Feeling Unheard or Disrespected

“feeling left out, not understanding, not

being listened to, fighting for a voice.” -

B26

“if the research team doesn't value

patient oriented research or doesn't

respect the opinions of the patient

partner” -P20

“Not being taken seriously” -P22

Medical Jargon & Knowledge Gap

“The jargon in the field of the research.”

-P13

“not knowing enough about specific

topics, medicine” -P29

“Lack of information about the process”

-P23

Researchers

& Research

Staff

Experience

“My lived experience with both "hats" -

patient and researcher and my experience

with facilitating collaboration more

generally.” -B49

“having a background in a profession

with a strong history and professional

models related to client-centred care and

partnership in care.” -R43

Relationships & Network

“What makes it easy is the people around

me who have a lot of experience in

patient-researcher partnerships.” -R34

“having clinical research collaborators

who can easily identify suitable and

interested patients from their practices” -

R04

Logistics & Time

“Having adequate time and schedule

availability of myself and other research

members to develop relationships and

make sure engagement is meaningful.” -

R43

Finding Patients

“Finding patient representatives when

not working in a hospital or clinical

setting” -R25

Establishing Relationships

“The difficult part would be the initial

stage of creating new partnerships with

people I don't know well” -R34

“Patients and researchers may enter into

the partnership with differing

expectations. It can be a challenge to

"meet in the middle".” -R15

This table does not represent all participants’ responses – it shows some of the recurrent response

types. While these are just select examples it demonstrates the range of responses provided. PRPs =

patient-researcher partnerships

Page 51 of 128

Researchers cited requirements from funding agencies:

“It is becoming a requirement for grants.” -R25

“[PRPs] may not always be suitable, yet CIHR and CIHR peer reviewers

require that all studies include patient research partners.” -R04

Patients cited the need for researchers to invite them to participate or give final approval:

“final decisions are made by project or researcher not me.” -P28

“I can express my intention to participate and provide background

information, but the final participant selection decision is not up to me.” -P06

4.4 Survey 2 Findings: Impact of Website

4.4.1 Self-Reported Time on Website

Participants were asked in Survey 2 to estimate the

amount of time they spent on the website (less than

30 minutes, half to 1 hour, 1 to 3 hours, or more than

3 hours). Most reported having spent half to one hour

(n = 19). A calculation of the number of self-reported

minutes spent on the website revealed an estimated

average between 49 to 108 minutes (see Table 17

for calculation).

4.4.2 Adapted eHIQ Part 2

Respondents were asked to complete the second part of the adapted eHIQ after they had access

to the website for at least one week. While the first part of the eHIQ was about the internet in

general, the second part only included questions about the website under evaluation. A total of

41 participants responded to these questions. There were some missing values, however no

survey item had more than 2 missing responses and no participant’s responses were missing for

6

19

12

4

0 - 0.5 hrs

0.5 - 1 hr

1 - 2 hrs

3+ hrs

Figure 5: Self-reported time on site.

Page 52 of 128

more than 3 of the 26 survey items in this section. Overall, only 10 of 1,066 possible responses

(41 participants x 26 survey items) were missing – a 99.0% response rate.

The questions that received the highest positive response (largest proportion answered ‘4’ or ‘5’)

were: “The website has a positive outlook” (80.5% agreed), “I can easily understand the

information on the website” (80.5% agreed), “I trust the information on the website” (78.0%

agreed), “I value the advice given on the website” (78.0% agreed), and “The language on the

website made it easy to understand” (75.6% agreed).

The questions that received the lowest positive response (smallest proportion answered ‘4’ or

‘5’) among positively phrased questions were “I would consult the website if I had to make a

decision about whether or not to be part of a PRP” (47.5% agreed), “I feel a sense of solidarity

with people on the website” (53.7% agreed), “I feel like I have a lot in common with people on

the website” (53.7% agreed), “The website makes me more confident to discuss my partnership

experience with the people around me” (55.0% agreed), and “The website encourages me to take

actions to become a PRP” (56.1% agreed).

Two negatively phrased questions (“I found the stories or other content on the website

distressing or worrisome” and “The information left me feeling confused”) received high

disagreement (95.1% and 67.5% answered ‘1’ or ‘2’) (see Appendix J).

Mann-Whitney U tests were run to determine if there were any differences in responses to these

questions between patients and researchers and no significant differences were found (Table 4).

Table 4: Adapted eHIQ Part 2 results, by domain, patients and researchers.


n M±SD n M±SD p Cronbach’s a

Confidence & Identification 23 66.24±19.00 15 67.52±15.30 0.79 0.91

Information & Presentation 23 73.04±16.89 15 76.70±10.94 0.50 0.79

Understanding & Motivation 23 65.66±21.09 15 66.35±16.96 1.00 0.91

An open-ended survey question followed the adapted eHIQ questions inviting participants to

share any other comments they had about the website and/or the use of video narratives. Of the

41 who completed Survey 2, 32 responded to this question (see Table 5). Much of the feedback

Page 53 of 128

Table 5: Feedback about website in open-ended survey responses, selected quotes by topic, and

number of patients (P) and researchers/research staff (R) who referred to topic in response.* # Selected Quotes

Desire for inspiring or practical content

P: 3

R: 5

“Where was the "razzmatazz" to want me to pursue being in a patient-researcher partnership?”

-P08

“There wasn't really a "how-to" guide/PDF for researchers to click on to get guidance on how

to get going with this type of partnership.” -R30

“The site should inform and inspire. […] Give me some key learning about the value and

reinforce with success stories.” -P14

Appreciation for video narratives

P: 2

R: 3

“Really appreciated how the video narratives punctuated, illustrated and underscored the

narrative sections.” -P36

“I like the use of video narratives - much more powerful than just reading.” -R38

Overwhelmed by amount of content

P: 9

R: 1

“I found the videos overwhelm the message. I got tired of watching after the first three.” -P27

“There are a LOT of interviews. A LOT of data to go through. It may overwhelm some as

there is a lot in print, interviews etc.” -P18

“I was overwhelmed at first when I realized the amount of information on one page.” -P29

“I felt like there were too much information and too many videos.” -P28

“There is a great deal of good information on the site, but the vastness is somewhat daunting.” -

P17

Importance of diversity

P: 3

R: 0

“[P]atient partners are often wealthy and white says the research: how to diversify so others

feel welcome?” -P33

“How can we get other participants that may help us reflect the general population? I

understand that there are "characteristics" that are required e.g. relevant sickness or experiences

etc. but we need all parts of society involved to get a good picture.” -P23

Purpose of website unclear

P: 3

R: 0

“What do you want the audience to feel when they first land on the site? Is this a new

approach? Innovation? Is it about reconstructing teams or accessing the voice of the patient?

[…] Again, to whom are we talking (target group) and what are we trying to get them to do?

What's the purpose and benefit?” -P14

“Do not yet understand purpose behind website.” -P07

Challenges navigating the website

P: 3

R: 2

“There were too many separate videos. That makes it hard to access info in an organized

manner.” -P22

“The site is very text heavy, with overly long pages where the content is not particularly well

structured, so it's easy to miss text appearing below the last video.” -P21

*This table shows selected quotes from 32 survey responses. Total patient responses = 18; total

researcher/research staff responses = 14.

Page 54 of 128

related to the design and layout of the website as well as the amount of content. Feedback about

the video content was mixed, with some stating that it was engaging and helpful, and others

expressing dissatisfaction with the way the videos were presented. Of the 18 patients who

responded to the survey prompt, 9 characterized the amount of content (e.g., number of videos,

length of pages, amount of text) as excessive or overwhelming. Of the 12 researchers who

responded to the survey prompt, five expressed desire for a form of summarized content (e.g.,

checklists, how-to guide, highlighted main points of videos). A few patients (n=3) referenced

diversity among patient partners in research as a point of concern and something they wished to

see represented more on the website and in practise. Other topics that were brought up by

participants include appreciation for video narratives, lack of clarity about the website’s purpose,

and challenges navigating the website.

4.4.3 Theory of Planned Behaviour Post-Intervention

The same TPB questions that were asked in the first survey were asked again in the second

survey. This was to detect if spending time on the website changed how participants responded

to these questions. Forty-one participants responded to these questions. There were some missing

values, but no survey item had more than 2 missing responses and no participant’s responses

were missing for more than one of the 12 survey items in this section. Overall, only 2 of 492

possible responses (41 participants x 12 survey items) were missing – a >99.9% response rate.

Wilcoxon signed rank tests were conducted to measure differences in pre and post responses to

TPB questions among patients and researchers. No significant differences were found between

pre and post responses for subjective norms or perceived behavioural control. Some significant

differences were found for intentions to participate in research partnerships and attitudes toward

them. A slight decrease in intention was observed among patients (p=0.01). A slight decrease in

attitudes was observed among researchers (p=0.02), but not among patients (see Table 6).

Page 55 of 128

Table 6: Pre-post TPB findings by domain, patients and researchers.

Patients Researchers

n

Pre

M±SD

Post

M±SD p n

Pre

M±SD

Post

M±SD p

Behavioural Intentions 23 6.48±0.65 5.98±1.02 0.01* 15 6.60±0.66 6.03±1.26 0.06

Attitudes 23 6.19±0.75 6.12±0.72 0.51 15 6.42±0.48 5.96±0.78 0.02*

Subjective Norms 23 2.39±1.31 2.30±1.27 0.75 15 4.93±1.29 5.29±1.12 0.23

Perceived Behavioural

Control

23 5.83±0.92 5.89±0.91 0.89 15 5.08±0.96 4.83±1.14 0.46

*indicates a significant difference (p = <0.05)

The TPB section of the survey included a series of open-ended questions that allowed

participants to explain and expand on their responses to the Likert-style questions. The open-

ended questions probed directly on whether participants felt the website impacted their attitudes

and intentions related to partnerships. Most responses indicated that the website did not change

how respondents felt about partnerships or their intentions to participate in them in the future.

Several participants did note, however, that the website brought attention to issues and

challenges which may influence how they approach future partnerships (see Table 7).

Page 56 of 128

Table 7: Survey feedback about the impact of the website, selected quotes.

Category Response Type # Selected Quotes

Desire to

Engage in

PRPs

No impact

Positive impact

Negative impact

Unclear

Total Responses

17

12

0

7

36

“[T]he website positively influenced my willingness to engage in

patient-researcher partnerships because seeing and hearing the

individual accounts from the videos added the personal touch that

text alone can't provide. It further motivated me to be more

engaged to hear the personal accounts.” -R41

“While I found some of the comments quite discouraging, I have

sufficient past experience of the value of research partnerships

that I can choose not to allow it to influence me.” -P21

“Funders require us to do it, so I am compelled to do it regardless

of the web site.” -R04

Intent to

Engage in

PRPs

No impact

Positive impact

Negative impact

Unclear

Total Responses

17

6

0

10

33

“It did not influence me with respect to whether I think I will

engage I patient-researcher partnerships in future. I was already

open to continuing that kind of engagement as opportunities

present themselves. However, it has influenced how I will assess

those opportunities.” -P36

“Yes, it provides high evidence that this is needed for the success

of future research.” -B10

“The website supported my incoming believes but also created

some concern about how to sign up and the lack of a consistent

process both in volunteering and training and education.” -P14

Attitudes

about

Engaging

in PRPs

No impact

Positive impact

Negative impact

Unclear

Total Responses

18

10

0

5

33

“The website did not change how I feel about such partnerships

per se, as I always felt strongly that patient-research partnerships

are very important and extremely beneficial. That being said,

however, the website was amazing in that the videos (particularly)

provided first-hand accounts of the benefits and challenges of

these partnerships which was very insightful and helpful.” -R15

“I was already a believer but the website confirms areas to be

aware of.” -O24

“It made me more comfortable to see/hear what other people are

doing.” -P29

Confidence

to Engage

in PRPs

No impact

Positive impact

Negative impact

Unclear

Total responses

15

9

0

6

30

“Seeing people similar to me re-assures me that I can do it too.” -

P17

“I think I'm more confident in engaging in partnerships since the

website gave me information about some of the challenges to

patient-researcher partnerships which help me understand

considerations prior to beginning projects.” -R39

“It did not change how confident I feel, but it provides much to

further equip me.” -P36

Each row represents responses to one open-ended survey question. PRP = patient-researcher partnership

Page 57 of 128

4.5 Website Utilization

Website utilization statistics were exported from Google Analytics to MS Excel and analyzed for

the evaluation period, defined as the period between the date that the first participant finished the

first survey (2020-04-17) and the date that the final participant finished the final survey (2020-

05-09). The website had some traffic prior to this period since the research team and the 33

researchers and patients featured on the website had access to the site. The link to the site was

private however, and only those given the link could access it. Study participants gained access

to the website through a link that was sent to them by email after they completed the first survey.

They were given instructions to visit the website at their convenience over a period of one week

and to navigate it according to their own interests.

Figure 6: Daily website page views.

Box indicates evaluation period (2020-04-17 – 2020-05-09).

During the evaluation period there were a total of 1,695 page views within 88 non-bounce

sessions (when a visitor views at least two pages before exiting the website), the majority of

which were topic pages (n=893). Of the 88 non-bounce sessions, 67 were by unique visitors,

which means that 21 sessions were by returning visitors. A total of 52 visitors accessed the

website for the first time during the evaluation period, so 15 (67 minus 52) of the visitors had

access to the website prior to the evaluation period (e.g., research team and those featured on the

website). The number of new visitors corresponds closely with the number of individuals who

completed the first survey (n=49) and were sent a link to the website. The source of the

0

50

100

150

200

250

300

20

20

-04

-03

20

20

-04

-05

20

20

-04

-07

20

20

-04

-09

20

20

-04

-11

20

20

-04

-13

20

20

-04

-15

20

20

-04

-17

20

20

-04

-19

20

20

-04

-21

20

20

-04

-23

20

20

-04

-25

20

20

-04

-27

20

20

-04

-29

20

20

-05

-01

20

20

-05

-03

20

20

-05

-05

20

20

-05

-07

20

20

-05

-09

20

20

-05

-11

20

20

-05

-13

20

20

-05

-15

20

20

-05

-17

20

20

-05

-19

20

20

-05

-21

20

20

-05

-23

Page 58 of 128

additional three new visitors is unknown but could have resulted from participants sharing the

website with others or accessing the website from multiple devices, each of which would be

counted as a unique visitor.

Table 8: Summary statistics for evaluation period.

Metric Definition Value

Total page views Total number of visits to all website pages combined, including repeated

visits to same page

1695

Unique page

views

Total number of visits to all website pages combined, not including

repeated visits to same page during a single session

979

Sessions Periods of time where a user is actively using the website (e.g., clicking

on links within the website) within the selected date range

88

Unique users Number of distinct individuals accessing the website within the selected

date range

67

New users Number of first-time visitors to the website within the selected date

range

52

Average session

length

Average time spent on website per session (mm:ss) 18:11

Average time per

page

Average time spent per website page (mm:ss) 1:34

Bounce sessions (where only one page is viewed before exiting website) are excluded.

The page that was most commonly selected first (in 33 of 88 sessions) after landing on the home

page was “Getting Started - Path to Involvement,” located at the top of the list of topic pages on

the left hand side of the website, then “People,” located in the top navigation bar (in 19 of 88

sessions) (see Appendix A for layout of website home page). The people featured on the website

whose content received the most views were Maureen (n=42), John (n=36), and Maxime (n=30),

all of whom were patient partners (see Table 15).

The topic pages with the highest number of views were those located at the top of the list of

topics (Path to Involvement (n=156), Developing Partnerships (n=109), and Motivations (n=77)),

suggesting that many navigated through the website in a systematic way starting at the top of the

list of topics. However, page views for some topic pages did not follow this pattern, such as

Balancing Time and Commitments which was 13th on the list of topics but received the 4th

highest number of views (n=73) (see Table 16).

Page 59 of 128

A large majority of page views were via desktop (91.7%). A very small amount of page views

were via tablet (5.4%) and mobile (2.9%).

See Appendix I for more details about website utilization.

4.6 Focus Group Findings

A total of two focus groups took place, and five participants took part in each. The majority of

the 10 participants were patients or caregivers (n=7), two were researchers, and one identified as

both a patient and researcher. All had prior experience with patient-researcher partnerships. One

group was composed of only patients and caregivers and the other was composed of two

researchers, two patients/caregivers and one who identified as both.

A thematic analysis of the focus group data uncovered four overarching themes: 1) a desire for

the website to include more practical, action-oriented content, 2) desire for clarity of purpose and

definitions, 3) variance in preferences for design aspects, and 4) importance ascribed to diversity

and compensation in PER.

4.6.1 Theme 1: Desire for Practical, Action-Oriented Content

Participants offered perspectives about what purposes they felt a website about PER should

serve. For instance, many patients felt that its purpose should be to draw potential patient

partners in and inspire interest and enthusiasm about participating in partnerships:

“I think that there’s got to be a little bit of pizzazz on how helping people to

get motivated, what are the benefits, how am I going to be a help in this

situation?” -P08

Patients also expressed disappointment in some of the aspects of the website that they felt didn’t

inspire interest in partnerships, such as negative phrasing and the inclusion of negative

engagement experiences:

“They all have their own take on positive, negative and neutral, but it’s the

way the headings are set up. It puts me off from even wanting to participate if I

Page 60 of 128

didn’t know anything about it. Because all I see is it’s a lot of work, I’ve got to

juggle my personal life with this, why am I doing this?” -P06

In addition to a desire for the website to inspire interest in partnerships, patients wished that the

website would provide direct or indirect links to patient engagement opportunities, so that if

someone visiting was inspired to participate in a research partnership they could be linked to an

opportunity to do so:

“That is one of the problems I had, if you had a frequently asked questions,

like how do I get involved? Or who do I contact if I want more information?” -

P13

“Ok, here’s something. I go through, I’m interested, I’ve gone in through

enough. What’s the next step then? I apply? Or, am I redirected somewhere?

But what is the next step after I’ve gone through. I’ve listened, whatever, what

happens next?” -P06

Researchers also expressed a desire for the website to include action-oriented content, however

what they felt was lacking related to guidance for decision-making and summarized messages

and tip sheets:

“For me too I think just being able to find information faster so say I’m a

researcher I want to be involved in patient engagement, but how do I do that?

How do you send me to a tip sheet? Something to summarize the information in

a more quick digestible way.” -R25

“Today, just so you know, I’ve been struggling between a research team that

has no money and a patient partner who wants to be compensated and I’ve

been the go-between back and forth between them, right. And as I looked

through the content on the site I want to know like, where’s the right answer?”

-B26

Page 61 of 128

4.6.2 Theme 2: Desire for Clarity of Purpose and Definitions

The website was designed to be an informational resource about patient-researcher partnerships

that displayed the experiences and perspectives of a broad range of patients and researchers with

PER. However, this purpose was met with confusion by many focus group participants:

“I guess one question is, and maybe I’ve just clued out, what is the objective of

the site? And, who are you hoping to get to the site and what their takeaway is

from the site?” -P17

“I don’t know if what your goal is to have that too much information and give

perspective. But then if it is, is it guidance on how to do patient partnerships?”

-B26

The variety of perspectives represented on the website were not perceived as valuable for other

participants who wished to see more concrete definitions on the website:

“If you’re looking to draw people in, they’re giving their own personal

experience and that’s great except if I just want to know in general what a

patient partner relationship is I’m looking more for an overall definition and

these videos could show a more personal take on it.” -P06

“I think we could also sort of describe in more detail what the partnership

really did and how it benefits the research, because I think even that gets lost

in the information that we have already.” -P13

4.6.3 Theme 3: Variance in Preferences for Design Aspects

Content on the website was organized by topics, with multiple, short video clips on each topic

page, along with accompanying transcripts and short descriptive text. Participants could also

navigate the website by viewing the profiles of patients and researchers featured on the site. This

formatting was met with an assortment of sometimes conflicting critiques and criticisms. Some

patients reacted strongly to the amount of content that the website contained, expressing that it

was overwhelming:

Page 62 of 128

“I found the number of videos to be excessive and also as [name] pointed out,

the script that was presented in the videos it was an awful lot to go through.” -

P08

In contrast, some patients and researchers felt that the depth of information as well as the

representation of multiple views and perspectives was valuable:

“I was quite impressed by it. I thought that there was quite a depth of

information provided.” -P02

Many participants offered suggestions for how the website could maintain its richness and depth

of information and representation of various perspectives, while also being attractive and easy to

navigate for some of the specific groups that might visit the website:

“Maybe there’s a box that says “new patient interested in partnerships” or

“new researcher” or maybe it just says “patient” and “researcher”, even if

behind those click boxes the information is identical, I would immediately think

that it’s tailored to me. You can kind of play with my mind a little bit, right.” -

R03

Participants expressed appreciation for some of the website’s design elements, such as the use of

videos and the inclusion of accompanying transcripts:

“Sometimes I have time to sit and like to review and certainly they’re so

different, the videos just brings it to life, but sometimes I’ve just got to skim a

transcript, I only have time to skim the transcript.” -B26

“I like both. Often what I would do is I would watch the video but if it’s too

long I wouldn’t watch at all but it gave me the sense of the voice that I was

then going to read in the transcript.” -P02

Page 63 of 128

4.6.4 Theme 4: Importance Ascribed to Diversity and Compensation in

PER

Participants initiated conversations about the page titled ‘Valuing Contributions’ in both focus

groups, without prompting from the facilitators. Some of the conversation centered around the

perceived focus of the page on compensation instead of the value of patients’ contributions to

research projects or non-monetary ways to value their contributions:

“Even the lead paragraph is all about how can we value it and then it’s

guidelines on compensation, honorarias, the whole thing speaks to

compensation rather than the value of the contribution the right person can

make.” -P06

“As soon as I started reading I thought, oh my god, this section is about money

when I thought it was going to be about so many other – how to really value

contributions.” -O12

Even though this page generated some negative reaction, it was also appreciated for its

publicizing of an important and challenging topic:

“The fact that you have done, the information about the whole payment, not

just payment, but compensation, I think is really good. You know, that’s a topic

that does not ever get attacked in any other website that I’ve seen. It’s really

important, so I think that was good to put that out there at the forefront.” -R03

The topic of diversity was also highlighted by participants, who recognized it as an ongoing

challenge in PER, and wished it was addressed in a better way on the website:

“I really felt that the diversity issue was not as visible as I might have thought

it needed to be. One of the observations I have around patient engagement and

it is changing rapidly which is a great thing, but I’m finding the diversity isn’t

perhaps as well as represented as it needs to be given who the users of the

systems are.” -P02

Page 64 of 128

“How do you go about trying to ensure you do have diverse representation?

It’s something that I’m always aware of [in the work] I do in spinal cord

injury. It’s just trying to ensure that you have diversity in the voices and

making sure you’re not getting the same type of individual that’s involved in

the partnerships.” -R25

4.7 Integration of Findings

Findings from different methods were compared to examine convergences and divergences.

While each method captured some unique, non-overlapping elements of participants’ use of and

perceptions about the website, some findings overlapped in complementary and conflicting

ways.

Three findings were consistent between quantitative and qualitative data collection methods.

Two related to how participants navigated the website and the pages that received relatively high

views. It was not anticipated that the four topics pages that included ‘Coming Soon’ in their title

and contained no content would receive much, if any, traffic. Three of these four pages received

the lowest number of views of all topic pages, as expected. ‘Impact on Research,’ however,

received higher traffic than the other three empty pages. It also received more views than two

completed topic pages (‘Diversity’ and ‘Looking Forward’). The unexpected traffic to this

section matched with qualitative findings in which participants referred to impact on research as

something of interest:

“And of course Impact o[n] Research is something everyone wants to see so

that will be an interesting section once it's written and produced.” -P33

“I think when the "Impact/Making a Difference" section is complete, it will

impact views about the value/helpfulness of patient-researcher partnerships.”

-P17

“the website probably would have had even greater influence if the

Impact/Making a difference content was available.” -R41

Page 65 of 128

Another topic page that received relatively high views (higher than expected based on its order in

the list of topic pages) was ‘Advice to Others,’ which was 15th on the list of topic pages, but

received the 9th highest number of views (n=47). This corresponds with focus group findings in

which participants expressed desire for guidance:

“There’s not a lot of guidance for researchers in this topic [of compensation]

and whenever you have the conversation it always makes people feel

uncomfortable so I think having it out there and having more guidance I do

think it’s helpful to provide guidance for having that discourse and not feeling

uncomfortable with bringing it up and talking about it.” -R25

“Can you do both? Can you provide a wealth of information that shows the

different perspectives and provide a path forward that guides people in their

practical, day-to-day research life?” -B26

Responses to the TPB tool showed a significant difference in responses between patients and

researchers to questions related to perceived behavioural control (‘PBC’) and influence of

pressure from others (‘Subjective Norms’) related to engaging in partnerships. Patients reported

feeling a higher level of PBC than researchers and lower level of pressure from others than

researchers when deciding whether to be part of a partnerships. The differences in responses

corresponds with the differences in factors patient and researcher participants described as

challenges, barriers, and facilitators to engagement in partnerships in responses to open-ended

survey questions. For instance, many patients referenced their personal or internal motivation as

a facilitator for their participation in partnerships (see Table 3), which corresponds with the

finding that patients feel little pressure from others to engage in partnerships as well as the

finding that patients feel the decision to engage is mostly up to them. Patients largely identified

their family as the source of pressure to engage in partnerships, but that this pressure was not

strong. Researchers, who reported a much higher sense of pressure from others, identified their

colleagues and funders as the source of that pressure.

Page 66 of 128

Table 9: Complementary findings.

Quantitative Findings Qualitative Findings

Relatively high number of visits to ‘Impact

on Research – Coming Soon’ topic page,

despite no content on page

↔ Comments in survey open-ended responses about

desire to see content about the impact of PER on

research

Relatively high number of visits to ‘Advice

to Others’ topic page in relation to order on

topic page list

↔ Feedback in focus groups about desire for

guidance and advice on how to successfully

participate in research partnerships

Significant differences in perceived

behavioural control (PBC) and influence of

pressure from others (Norms) between

patients and researchers

↔ Difference between the facilitators and sources of

pressure to engagement in partnerships described

by patients and researchers in survey open-ended

responses and focus groups

Findings from the pre-post analysis of responses to TPB survey questions conflicted with

findings from focus groups and survey open-ended responses. Decreases in both intention and

attitude scores were observed after participants spent time on the website (see Table 6), however

almost no participants reported that the website had this influence on them in the focus groups or

survey open-ended questions. When participants were asked to respond to the website’s impact

on their attitudes and intention in the survey, their responses mostly indicated that they felt the

website had no impact. Those who felt it had some impact described a positive impact, not a

negative one (see Table 7).

Responses to Part 2 of the adapted eHIQ, which asked about the website’s design and content,

were largely positive (see Appendix J). These responses suggest that participants hold a highly

favorable perception of the website, however the comments provided in survey open-ended

fields and focus groups were largely negative, particularly with respect to the amount of content,

the way it was organized, the framing of content (perceived as negative), and the lack of

actionable content (e.g., tip sheets, links to engagement opportunities). Some responses to survey

items appeared to contradict the qualitative data. For instance, despite comments about negative

framing, 80.5% (n=33) of respondents agreed (i.e., answered ‘4’ or ‘5’) with the statement “The

website has a positive outlook.” Also, despite many comments about there being too many

videos, 61.0% (n=25) agreed with the statement “Videos were used appropriately on the

website.” Additionally, despite comments about a desire that the website include more guidance

and tips, 68.3% (n=28) agreed that “The website includes useful tips on how to be a better

patient-researcher partner.”

Page 67 of 128

Responses to the adapted eHIQ also appeared to contradict findings from other parts of the same

survey. Over half (57.5%, n=23) agreed with the statement “I feel more inclined to participate in

a PRP after visiting the website.” In contrast, this positive inclination was only reported in 6 out

of 33 responses (18.2%) to an open-ended survey question (“Did the website influence whether

or not you think you are going to engage in a patient-researcher partnership in the future? Why

or why not?”) about whether the website impacted participants’ intention to participate in

partnerships (Table 7). Furthermore, 58.5% (n=24) agreed with the statement “The website gives

me the confidence that I am able to participate in a PRP.” However, increased confidence was

only reported in 9 out of 30 responses (30.0%) to an open-ended survey question about whether

the website impacted confidence to engage (Table 7).

Another divergent finding was related to diversity among the people displayed on the website

and among the patients and researchers who participate in research partnerships. Participants

referred to a lack of diversity on multiple occasions in the surveys and focus groups without

prompting, suggesting that this issue is top-of-mind for many. However, the topic page on the

website dedicated to this subject received the second to fewest views (n=25) among the

completed pages.

Table 10: Divergent findings.

Quantitative Findings Qualitative Findings

Decrease in intention to participate in

partnerships among patients, according to pre-

post analysis of TPB survey responses

≠ No reports of decreased intention to participate

in partnerships in survey open-ended responses

or focus groups

Worsened researchers’ attitudes toward

partnerships, according to pre-post analysis of

TPB survey responses

≠ No reports of worsened attitudes toward

partnerships in survey open-ended responses or

focus groups

Majority positive responses to the website’s

design, according to responses to the second

part of adapted eHealth Impact Questionnaire

≠ Widespread reports of dissatisfaction with

website design expressed in survey open-ended

responses and focus groups

High proportion of participants who agreed

with statements from the adapted eHIQ about

the website’s positive impact on their

confidence and inclination to participate in

partnerships

≠ Low proportion of participants who reported

that the website positively impacted their

confidence, desire, or intention to participate in

partnerships in survey open-ended responses

Relatively low number of visits to ‘Diversity’

topic page in relation to order on topic page list

≠ Interest in the topic of diversity expressed by

many participants in focus groups and survey

open-ended responses

Page 68 of 128

4.8 Summary of Findings

This study found that 1) attitudes toward research partnerships and intentions to participate in

them were very high among those who participated in the study, 2) patients report significantly

lower pressure from others to participate in research partnerships and higher perceived

behavioural control than researchers, 3) perceptions about the website’s design and content were

conflicting, with highly positive perceptions reported in the quantitative survey questions and

low perceptions reported in focus groups and open-ended survey questions, 4) participants

reported lower (but still high) intentions to participate in research partnerships after spending

time on the website, according to the pre-post analysis of survey responses, but reported no

impact or a positive impact on intentions in responses to open-ended questions about the impact

of the website on intentions, 5) researchers reported lower (but still high) attitudes toward

research partnerships after spending time on the website, according to the pre-post analysis of

survey responses, but reported no impact or a positive impact on attitudes in responses to an

open-ended survey question, 6) participants expressed a desire for action-oriented content about

research partnerships, including guidance for researchers seeking decision-making support and

links to engagement opportunities for patients seeking to be part of a research partnership, 7)

participants had mixed feelings about how information was presented on the website – some

(mostly researchers) appreciated the rich content and presentation of varied perspectives, and

others (mostly patients) were overwhelmed by the amount of content and wished there were

more clear definitions, and 8) according to website analytics, participants tended to navigate the

website by starting at the top of the list of topic pages and moving down.

Page 69 of 128

Discussion

5.1 Overview

The research question that guided this study asked: What are the perceptions of patients and

researchers about a website featuring experiential information about patient engagement in

research, and how did it impact behavioural intentions and attitudes toward PER? We employed

a mixed methods approach that utilized surveys, focus groups, and web analytics. Our findings

indicated that participants had highly positive attitudes toward research partnerships and high

intentions to participate in them both before and after using the website. Our findings also

revealed a significant different between patients and researchers in the amounts and types of

pressures to be part of research partnerships. Integration of the findings between methods

revealed some divergence. In responses to open-ended Survey 2 questions participants reported

that the website had no impact or a positive impact on their attitudes and intentions. However, in

the same survey participants recorded lower agreement with the statements “I want to engage in

PER” and “I intend to engage in PER” than they did in Survey 1 (before visiting the website).

Furthermore, in Survey 2 most participants agreed with almost all positively phrased questions

about the website. However, in the same survey, when asked if they have “Any other comments

you would like to add about the website and/or the use of video narratives?” their responses

largely contained criticisms and negative critiques of the website.

Unfortunately, divergence between findings from different methods make it difficult to make

conclusive statements about either of these factors (i.e., overall perception of the website and its

impact on intentions and attitudes). Despite this, our findings contribute to dialogues on several

related topics. For instance, the discovery that patients and researchers report significantly

different amounts and sources of pressure to engage in research partnerships contributes to an

understanding of the factors that lead to the decision to engage in partnerships. Also, the

discovery that participants are interested in action-oriented information about research

partnerships, including guidance and links to opportunities, adds to an understanding of the kind

of content that future developers of online content about research partnerships may want to

include. Our findings also contribute to discussions about evaluation methods, including the use

Page 70 of 128

of mixed methods and the usefulness of the eHealth Impact Questionnaire and Theory of Planned

Behaviour tools as evaluative tools.

5.2 Comparison with Related Literature

Our findings can be compared with literature in a few fields, including research about the

usefulness of video narratives. The meta-analysis by Braddock & Dillard60 on the persuasive

effect of narratives found that exposure to narratives can result in movement of the audience’s

perceptions to closer alignment with those portrayed in the narratives. Since the website included

the narratives of 33 different people expressing a variety of perspectives and experiences it’s

difficult to determine the overall impact of the narratives. However, it’s possible that hearing

stories from others that included their challenging experiences shifted the perspective of website

visitors from a more optimistic and idealistic point of view to a view that included some less

pleasant and challenging aspects of research partnerships. This would support our finding that

participants felt they had been made aware of key issues in research partnerships.

While few published studies used the eHIQ, it is worth considering how our findings compare

with the findings of others who also used it. A 2019 study by Heath and colleagues83 used the

eHIQ to evaluate a similarly designed website (includes patient narratives) about burn-injury in

children. Parents of burn-injured children (n=9) and health care providers (n=22) participated in

the evaluation. Their findings for the five eHIQ subscales were slightly higher than what we

found, as nearly all were above 70 (out of 100, using the eHIQ subscale formula81) and most of

ours were slightly below 70. Their qualitative findings had some similarities to ours in that

participants commented about the amount of content, the need for diverse representation on the

website, and the use of technical terminology. A 2020 study by Talboom-Kamp and colleagues84

used the eHIQ to evaluate an online resource for communicating laboratory results. Findings

from the two eHIQ subscales they employed (Information and Presentation, and Motivation and

Confidence to Act) were slightly lower than what we found. However, given the different

purpose their website aimed to serve it is difficult to draw a comparison with our results.

Our findings can also be compared with other studies that evaluation websites designed using the

DIPEx methodology. Newman and colleagues (2009)63 evaluated a website about rheumatoid

Page 71 of 128

arthritis. One theme that came out in their evaluation was that some (22%) participants felt the

content was too negative and may be depressing for people newly diagnosed with RA. This

matches with our finding that some participants felt the content didn’t represent research

partnerships in a good light and may discourage some from wanting to participate. An evaluation

of a DIPEx website in 2004 about prostrate and breast cancer found that participants found it

highly valuable.55 There were few correlation between their findings and ours, but both studies

suggest there is an interest in websites that includes both experiential and reliable health

information (in our case, information about research partnerships).

Our findings align with PER literature about diversity and compensation. Diversity has been

cited by many as an area in need of improvement,40 and our findings confirm that this is an area

of concern as it was brought up by multiple participants without prompting. The issue of

compensation is an ongoing debate within PER,39 and our study hosted a microcosm of this

debate. Participants expressed a range of passionate, and often conflicting, opinions about this

subject, which reflects the lack of consensus about compensation that still exists and is a source

of confusion and stress for researchers and patient partners.

5.3 Interpretation of Divergent and Unanticipated Findings

The divergent finding that intention scores were lower after spending time on the website and

that no participants indicated decreased intention when asked in the survey open-ended questions

invites a discussion about factors that could have contributed to this. One factor to consider is in

the details respondents provided in their responses to open-ended survey questions, where

multiple participants indicated that the website increased their awareness of challenges and

issues to consider when engaging in partnerships or choosing which opportunities to pursue

(Table 7). It is possible that the website’s presentation of challenging aspects of research

partnerships dampened the enthusiasm that participants held before engaging with the website

content. According to Fading Affect Bias (FAB), which is the tendency of positive memories to

fade more slowly than negative memories,94 participants’ memories of past negative experiences

with research partnerships may have been weak or faded when filling out the first survey. Then,

the website’s portrayal of challenging aspects of research partnerships refreshed their memory,

which impacted the way in which they responded to the TPB prompts in the second survey.

Page 72 of 128

According to this interpretation, the website may cause participants to think more critically about

partnerships and consider factors that can make them successful or unsuccessful, thereby slightly

dampening their desire and intention to engage.

Another bias may explain the divergence between the pre-post analysis of intention scores and

participants reporting of the website’s impact in open-ended survey questions. Response Shift

Bias (RSB) occurs when a person’s frame of reference toward the construct being measured

changes as the result of an intervention. Some researchers argue that the influence of changed

frames of reference can compromise evaluations of educational interventions that use pre and

post measurements.95 This is because what was measured in the first test is fundamentally

different from what is being measured in the second test, and therefore not comparable. The

constructs we aimed to measure in this case was participants’ attitudes and intention to engage in

research partnerships and the intervention was spending time on the website. The website may

have changed participants’ frame of reference by changing their mental image/memory of

research partnerships. The initial frame of reference may have been an idealized memory or

understanding of research partnerships, and the frame of reference at the second measurement

was a more grounded and realistic understanding of partnerships, resulting from spending time

on the website.

To account for the confounding influence of RSB, some researchers have employed a

retrospective pre-test method design, in which both pre and post perceptions are collected at the

same time, after the intervention.95 In this design participants are asked to recall their prior

perceptions to compare with their post-intervention perceptions.95 In this way the participant uses

the same frame of reference for their assessment of their current and past perceptions. Our study

approximated this model with the inclusion of open-ended questions in the second survey that

directly asked respondents if they believed that the website impacted their desires, intentions,

feelings, and confidence related to research partnerships (Table 7). By asking about impact, the

questions required participants to reflect on their prior perceptions and contrast them with their

current perceptions, like what would occur in a retrospective pre-test method design. Responses

to these questions contradicted the findings from the pre-post analysis, by not including any

reports of decreased intention and by including some reports of positive impact. This

contradiction in findings may be explained by RSB. According to this interpretation, the pre-post

Page 73 of 128

analysis is less reliable than self-reported perceptions of impact in the open-ended survey

questions, where most reported that the website had no impact. Following from this, the website

did not change participants’ intentions to engage in research partnerships, but rather influenced

their frame of reference for considering research partnerships.

Another discrepancy that deserves further discussion is the majority positive responses to nearly

all Likert-scale survey questions about the website in contrast to the largely negative statements

about the website made by participants in open-ended survey questions and focus groups. There

are a few response biases known to impact responses to questions using Likert scales (e.g., social

desirability, acquiescence).96 Confirmation bias, which is the tendency for respondents to

activate beliefs that are consistent with the way an item is stated,97 holds promise as an

interpretation of our divergent findings. Since the website contained a broad range of content, it

would be possible for participants to recall how an aspect of the website aligned, even if only in

a small way, with each positively phrased survey items. For example, the highly positive

response to the survey item “Videos were used appropriately on the site” conflicts with the

feedback from many participants that the number of videos was excessive and overwhelming.

Participants may have read the prompt and thought of some ways in which the videos were

appropriate and selected ‘agree,’ even though they felt there were too many of them.

An alternative interpretation of this divergent finding is that participants did hold an overall

positive perception of the website, as indicated in responses to Likert-style survey questions, and

the critical feedback found in the qualitative data skewed negative because participants chose to

highlight potential areas for improvement instead of discussing their overall impression.

These interpretations do not provide a conclusive answer to the divergence within our data, but

they do introduce interesting lines of inquiry to be taken up in future research.

5.4 Contribution to Academic Knowledge

This study contributes to knowledge about the use of mixed methods, specifically how it can

uncover conflicts within the data which may necessitate further consideration of the influence of

biases. If this study had only utilized one method, the findings may have been more conclusive,

however those results may have been less robust due to uninterrogated biases. The study also

Page 74 of 128

contributes to knowledge about the kinds of online resources that patients and researchers desire.

Our findings suggest that there is a strong desire for practical guidance and links to engagement

opportunities. Our findings also uncovered differences in the levels and amounts of pressure to

engage in partnerships experienced by patients and researchers. This is an area that has not yet

been explored in the context of research partnerships and may be interesting for other researchers

to investigate further.

The study also adds to literature about informational or educational interventions, and cases

where there may not be clarity in the intervention’s objectives with respect to ‘informing,’

‘inspiring,’ and ‘advising.’ Our findings revealed conflicts about the purpose of the website in

these three domains. It was designed to be informative, yet patients expressed a desire for it to

also be inspiring and researchers that it also provide guidance. This adds an interesting nuance to

discussions about the purpose of educational interventions, and the need to consider potential

unintended consequences when developing these interventions (e.g., if providing more

information about challenges associated with an activity may result in reduced enthusiasm to

participate in that activity).

5.5 Implications for Policy & Practise

Our findings are particularly useful for the website development team, providing feedback to

enables revisions that better match the preferences and needs of patients and researchers as

potential users. The findings could also prove useful to other teams who are considering creating

online resources about research partnerships.

The topic of diversity and achieving representation in patient engagement in research was raised

by multiple participants and deserves further attention. PER laudably aims to move healthcare

toward a post-paternalistic future by giving voice to the lived experiences of patients. However,

if the patients whose voices are heard are not representative, PER risks inadvertently stalling

healthcare’s progress by perpetuating the tired tradition of over-representing the voices of

wealthy, white, urban-dwelling, highly educated individuals. If achieving diverse representation

in PER is currently a nice-to-have, decision-makers may wish to consider taking steps to make it

a requirement. For instance, research ethics boards could require researchers who propose to

Page 75 of 128

include patient partners in their study to ensure that they include more traditionally underserved

populations (e.g., ethnic minorities, low income populations) than traditionally overserved

populations.

This study may also prove useful for the Database of Individual Patient Experiences (DIPEx)

community and those who maintain each member countries’ websites. Our findings reveal that

participants struggled to navigate the website content, and many felt overwhelmed by the volume

of material. Since all DIPEx websites use a similar format it may be worthwhile to investigate

whether other websites are also perceived this way and whether guidelines for formatting of

content should be revisited. It may also invite a conversation about the usefulness of applying the

DIPEx model outside of the direct context of patient experiences, which is what it is designed

for. Research partnerships are not a health experience so much as they are an experience of a

relationship or collaboration, and they may be better represented using a different method.

5.6 Opportunities for Future Research

Our finding that patients report very low pressure from others (subjective norms) to participate in

research partnerships, but at the same time report extremely high intention to engage in research

partnerships, suggests there may be limitations to the Theory of Planned Behavior. Our findings

suggest that many patients’ motivations are internal and personal. However, the TPB describes

subjective norms and a sense of pressure from others as a key predictor of behavioural intention.

This invites further research into the role of internal motivations as a factor in behavioural

intention in the context of research partnerships and otherwise.

Another aspect of the Theory of Planned Behaviour that deserves further inquiry relates to

perceived behavioural control. When participants were asked about factors that make them feel

engaging in partnerships is outside of their control, researchers mostly identified factors that

would require them to participate in partnerships (e.g., funding agencies) and patients identified

factors that could exclude them from participating (e.g., the need for the research team to

approve their involvement). Both kinds of factors contribute to decreased perceived control, but

one could increase intention to participate (requirements from funders) and the other could

decrease it (feeling that researchers make the final decision). According to the Theory of

Page 76 of 128

Planned Behaviour, high perceptions of control should correlate with high behavioural intention.

However, in the case of researchers, low perceptions of control resulting from funder

expectations may correlate with high behavioural intention (which is what PER advocates may

wish for). This invites further research into cases where decreases in perceived behavioural

control can result in higher behavioural intention. It also invites a discussion about whether

pressure from funders may contribute to insincere or tokenistic PER, and whether there are better

ways to incentivise researchers to participate in PER.

5.7 Limitations

One of the limitations to this study is the small sample size, which limits the generalizability of

our results. Another is that, despite our goal to recruit a diverse sample, our sample over-

represented white, Canadian-born, Ontario-dwelling, highly educated women over the age of 55

who had prior experience with PER. This limits the generalizability of our findings to large

segments of the Canadian population, including ethnic minorities, young people, people without

university degrees, non-Ontarian Canadians, and those not familiar with PER. This lack of

diversity in our sample is reflective of the lack of diversity in PER, and likely results from many

of the same factors (e.g., challenges researchers face in establishing relationships with some

population groups).

Additional limitations include the fact that the website was incomplete, which may have

influenced participants’ perceptions of the website, as may have the short period of time

participants were given access to the website. Also, the inability to separate the research team’s

web traffic from the participants’ web traffic may have skewed the web analytics data, which

limits the reliability of this data. Furthermore, the fact that we could not access analytics data

about participants’ engagement with video content caused us to rely on self-reported

information, which is subject to memory and other biases. These limitations were recognized in

the interpretation of our findings, as we chose not to overemphasize the findings from less

reliable data.

Some other limitations arose due to COVID-19 restrictions. Some technical issues arose during

the focus groups, including audio and video connectivity problems. This resulted in some

Page 77 of 128

members being asked to repeat statements because their audio cut out and some members were

unable to join by video at all, and instead called in to the meeting. Videoconferencing does not

facilitate some of the norms of in-person meetings, such as eye contact with the speaker and the

informal interactions that occur before and after meetings begin. The full extent to which this

impacted the focus groups is unknown, but participants expressed some frustration about the

videoconferencing format during the groups. Another limitation related to COVID-19 was its

impact on the recruitment process. We had not yet reached out to all the organizations we had

planned to when travel restrictions and work-from-home orders came into place in March 2020.

With consideration for the need for organizations to commit extra time and resources to adapting

to this change, we chose to stop reaching out to organizations to distribute our invitation to

participate during this time. This limited both the number of participants we recruited as well as

their representativeness, as we had not yet reached out to organizations in provinces east of

Ontario.

5.8 Conclusion

While this study did not provide a definitive or succinct answer to the question of whether the

website was perceived as useful or whether it had a positive impact on attitudes or intentions

related to research partnerships, it opened many interesting dialogues. It demonstrated the value

of mixed methods in uncovering the role of biases and potential weaknesses in some methods

and survey tools, as well as challenges in interpreting and integrating the results. It elucidated

some of the desires patients and researchers have for online content about research partnerships.

It uncovered some of the differences between patients and researchers in their motivations to

engage in research partnerships. It introduced avenues of inquiry related to the Theory of

Planned Behaviour and its relevance for behaviours that are internally motivated and cases where

higher perceived behavioural control may result in lower behavioural intention. It added to

knowledge about the desire to learn from research partnerships that feature diversity amongst

patient partners, as well as for decisive guidelines and supports for compensating patient partners

in a way that demonstrates respect for their contributions and consideration of their economic

circumstances.

Page 78 of 128

References

1. Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research:

moving from theory to practice. J. 2015;4:133-45.

2. Mullins CD. Integrating Patients' Voices in Study Design Elements with a Focus on

Hard-to-reach Populations. Washington, DC, USA: Patient-Centered Outcomes

Research Institute (PCORI),;2012.

3. Abelson J, Forest PG, Eyles J, Smith P, Martin E, Gauvin FP. Deliberations about

deliberative methods: issues in the design and evaluation of public participation

processes. Social Science & Medicine. 2003;57:239-51.

4. Dudley L, Gamble C, Preston J, et al. What difference does patient and public

involvement make and what are its pathways to impact? Qualitative study of patients and

researchers from a cohort of randomised clinical trials. PLoS ONE. 2015;10:e0128817.

5. Crocker JC, Boylan AM, Bostock J, Locock L. Is it worth it? Patient and public views on

the impact of their involvement in health research and its assessment: a UK-based

qualitative interview study. Health Expect. 2017;20:519-28.

6. Wilson P, Mathie E, Poland F, et al. How embedded is public involvement in mainstream

health research in England a decade after policy implementation? A realist evaluation.

Journal of Health Services Research & Policy. 2018;23:98-106.

7. Carroll SL, Embuldeniya G, Abelson J, McGillion M, Berkesse A, Healey JS.

Questioning patient engagement: research scientists' perceptions of the challenges of

patient engagement in a cardiovascular research network. PATIENT PREFERENCE AND

ADHERENCE. 2017;11:1573-83.

8. Hewlett S, De Wit M, Richards P, et al. Patients and professionals as research partners:

Challenges, practicalities, and benefits. Arthritis Care and Research. 2006;55:676-80.

9. Belisle-Pipon JC, Rouleau G, Birko S. Early-career researchers' views on ethical

dimensions of patient engagement in research. BMC MEDICAL ETHICS. 2018;19:21-10.

10. Price A, Albarqouni L, Kirkpatrick J, et al. Patient and public involvement in the design

of clinical trials: An overview of systematic reviews. Journal of Evaluation in Clinical

Practice. 2018;24:240-53.

11. Creswell JW, Plano CVL. Designing and conducting mixed methods research. Third ed:

SAGE Publications, Inc.; 2018.

12. Bandura A. Social Learning Theory: General Learning Corporation; 1971.

13. Grusec J. Social Learning Theory and developmental psychology: The legacies of Robert

Sears and Albert Bandura. Developmental Psychology. 1992;28:776-86.

Page 79 of 128

14. Patton MQ. Qualitative research and evaluation methods. Thousand Oaks, Calif: Sage

Publications; 2002.

15. Ajzen I. The theory of planned behavior. Organizational Behavior and Human Decision

Processes. 1991;50:179-211.

16. Carman KL, Dardess P, Maurer M, et al. Patient and family engagement: a framework for

understanding the elements and developing interventions and policies. Health Aff

(Millwood). 2013;32:223-31.

17. Canadian Institutes of Health Research (CIHR). Guide to Knowledge Translation

Planning at CIHR: Integrated and End-of-Grant Approaches. Ottawa, CA 2012.

18. Parry D, Salsberg J, Macaulay AC, Participatory Research at McGill (PRAM). A Guide

to Researcher and Knowledge-User Collaboration in Health Research. 2015.

19. Domecq JP, Prutsky G, Elraiyah T, et al. Patient engagement in research: a systematic

review. BMC Health Serv Res. 2014;14:89.

20. Richards T, Montori VM, Godlee F, Lapsley P, Paul D. Let the patient revolution begin:

Patients can improve healthcare: it's time to take partnership seriously. BMJ: British

Medical Journal. 2013;346:7-.

21. Canadian Institutes of Health Research (CIHR). Strategy for Patient-Oriented Research -

Patient Engagement Framework 2014; http://www.cihr-irsc.gc.ca/e/48413.html. Accessed

April 13, 2019.

22. Kirwan JR, de Wit M, Frank L, et al. Emerging Guidelines for Patient Engagement in

Research. Value Health. 2017;20:481-6.

23. James Lind Alliance (JLA). The James Lind Alliance. 2019; http://www.jla.nihr.ac.uk/.

Accessed April 13, 2019.

24. Donetto S, Pierri P, Tsianakas V, Robert G. Experience-based Co-design and Healthcare

Improvement: Realizing Participatory Design in the Public Sector. The Design Journal.

2015;18:227-48.

25. Brett J, Staniszewska S, Mockford C, et al. A Systematic Review of the Impact of Patient

and Public Involvement on Service Users, Researchers and Communities. The Patient -

Patient-Centered Outcomes Research. 2014;7:387-95.

26. Staley K. 'Is it worth doing?' Measuring the impact of patient and public involvement in

research. Research involvement and engagement. 2015;1:6.

27. INVOLVE. Our Projects. 2019; https://www.involve.org.uk/our-work/our-projects.


http://www.cihr-irsc.gc.ca/e/48413.html

http://www.jla.nihr.ac.uk/

https://www.involve.org.uk/our-work/our-projects

Page 80 of 128

28. Patient-Centered Outcomes Research Institute (PCORI). Our Story. 2017;

https://www.pcori.org/about-us/our-story. Accessed April 15, 2019.

29. International Association for Public Participation (IAP2). About IAP2. 2019;

https://www.iap2.org/page/A3. Accessed April 15, 2019.

30. Canadian Institutes of Health Research (CIHR). Strategy for Patient-Oriented Research:

Patient Engagement Framework. 2014; http://www.cihr-irsc.gc.ca/e/48413.html.


31. Canadian Institutes of Health Research. About SPOR. 2018; http://www.cihr-

irsc.gc.ca/e/51036.html. Accessed April 28, 2019.

32. Ontario SPOR Support Unit (OSSU). Our Impact. 2019; https://ossu.ca/our-impact/.


33. Jones EL, Williams-Yesson BA, Hackett RC, Staniszewska SH, Evans D, Francis NK.

Quality of Reporting on Patient and Public Involvement Within Surgical Research: A

Systematic Review. Annals of Surgery. 2015;261:243-50.

34. Crocker JC, Ricci-Cabello I, Parker A, et al. Impact of patient and public involvement on

enrolment and retention in clinical trials: systematic review and meta-analysis. BMJ-

BRITISH MEDICAL JOURNAL. 2018;363:k4738.

35. Evans D, Bird E, Gibson A, et al. Extent, quality and impact of patient and public

involvement in antimicrobial drug development research: A systematic review. Health

Expect. 2018;21:75-81.

36. Elliott MJ, Goodarzi Z, Sale JEM, et al. Perceived Significance of Engagement in

Research Prioritization Among Chronic Kidney Disease Patients, Caregivers, and Health

Care Professionals: A Qualitative Study. Canadian Journal of Kidney Health and

Disease. 2018;5:205435811880748.

37. Miller J, Knott VE, Wilson C, Roder D. A review of community engagement in cancer

control studies among Indigenous people of Australia, New Zealand, Canada and the

USA. European Journal of Cancer Care. 2012;21:283-95.

38. Chu Yang L, Loyola-Sanchez A, Hurd K, et al. Characterization of indigenous

community engagement in arthritis studies conducted in Canada, United States of

America, Australia and New Zealand. Seminars in Arthritis and Rheumatism. 2018.

39. Richards DP, Jordan I, Strain K, Press Z. Patient partner compensation in research and

health care: the patient perspective on why and how. Patient Experience Journal. 2018;5.

40. Black A, Strain K, Wallsworth C, et al. What constitutes meaningful engagement for

patients and families as partners on research teams? Journal of Health Services Research

& Policy. 2018;23:158-67.

https://www.pcori.org/about-us/our-story

https://www.iap2.org/page/A3




https://ossu.ca/our-impact/

Page 81 of 128

41. Pii KH, Schou LH, Piil K, Jarden M. Current trends in patient and public involvement in

cancer research: A systematic review. Health Expect. 2019;22:3-20.

42. Bethell J, Commisso E, Rostad HM, et al. Patient engagement in research related to

dementia: A scoping review. Dementia. 2018;17:944-75.

43. Bonevski B, Randell M, Paul C, et al. Reaching the hard-to-reach: A systematic review of

strategies for improving health and medical research with socially disadvantaged groups.

BMC Medical Research Methodology. 2014;14:42-.

44. Boivin A, L'Esperance A, Gauvin FP, et al. Patient and public engagement in research

and health system decision making: A systematic review of evaluation tools. Health

Expect. 2018;21:1075-84.

45. Abelson J, Humphrey A, Syrowatka A, Bidonde J, Judd M. Evaluating Patient, Family

and Public Engagement in Health Services Improvement and System Redesign.

Healthcare Quarterly. 2018;21:31.

46. The Beryl Institute. Improving the Patient Experience. http://www.theberylinstitute.org/.

Accessed Feb 23, 2019.

47. Patient-Centered Outcomes Research Institute (PCORI). Engagement.

https://www.pcori.org/engagement. Accessed Feb 23, 2019.

48. Warwick Medical School. Patient Public Involvement – Training. 2018;

https://warwick.ac.uk/fac/sci/med/research/ctu/ppitraining/. Accessed April 27, 2019.

49. INVOLVE. Training case study four: Introduction to patient and public involvement in

research. 2012; https://www.invo.org.uk/posttyperesource/planning-training-general-

overview/training-case-study-four/. Accessed April 27, 2019.

50. St. Michael's Hospital. Patient Engagement: Tools & Resources.

http://stmichaelshospitalresearch.ca/patient-engagement-resource/. Accessed Feb 23,

2019.

51. Canadian Child Health Clinician Scientist Program (CCHCSP). New Partnership to

Advance Building Capacity in Patient-Oriented Research. https://www.child-

bright.ca/new-blog/2017/10/11/training-partnership. Accessed Feb 23, 2019.

52. Terry M. New Patient Engagement in Primary Health Care Research online training

program launched. [Monthly eNewsletter]. 2018;

https://www.schulich.uwo.ca/familymedicine/about_us/newsletter/2018/december/new_p

atient_engagement_in_primary_health_care_research_online_training_program_launche

d.html. Accessed April 27, 2019.

53. Rodrigue JR, Fleishman A, Vishnevsky T, Fitzpatrick S, Boger M. Organ donation video

messaging: differential appeal, emotional valence, and behavioral intention. Clinical

transplantation. 2014;28:1184-92.

http://www.theberylinstitute.org/

https://www.pcori.org/engagement

https://warwick.ac.uk/fac/sci/med/research/ctu/ppitraining/

https://www.invo.org.uk/posttyperesource/planning-training-general-overview/training-case-study-four/

https://www.invo.org.uk/posttyperesource/planning-training-general-overview/training-case-study-four/

http://stmichaelshospitalresearch.ca/patient-engagement-resource/

https://www.child-bright.ca/new-blog/2017/10/11/training-partnership

https://www.child-bright.ca/new-blog/2017/10/11/training-partnership

https://www.schulich.uwo.ca/familymedicine/about_us/newsletter/2018/december/new_patient_engagement_in_primary_health_care_research_online_training_program_launched.html



Page 82 of 128

54. Occa A, Suggs LS. Communicating Breast Cancer Screening With Young Women: An

Experimental Test of Didactic and Narrative Messages Using Video and Infographics.

Journal of Health Communication. 2016;21:1-11.

55. Rozmovits L, Ziebland S. What do patients with prostate or breast cancer want from an

Internet site? A qualitative study of information needs. Patient Educ Couns. 2004;53:57-

64.

56. Shaw BR, Han JY, Baker T, et al. How women with breast cancer learn using interactive

cancer communication systems. Health Education Research. 2007;22:108-19.

57. Sznitman SR, Lewis N. Examining effects of medical cannabis narratives on beliefs,

attitudes, and intentions related to recreational cannabis: A web-based randomized

experiment. Drug and Alcohol Dependence. 2018;185:219-25.

58. Taylor AM, Hutchings M. Using video narratives of women's lived experience of

breastfeeding in midwifery education: Exploring its impact on midwives' attitudes to

breastfeeding. Maternal and Child Nutrition. 2012;8:88-102.

59. Coret A, Boyd K, Hobbs K, Zazulak J, McConnell M. Patient Narratives as a Teaching

Tool: A Pilot Study of First-Year Medical Students and Patient Educators Affected by

Intellectual/Developmental Disabilities. Teaching and Learning in Medicine.

2018;30:317-27.

60. Braddock K, Dillard JP. Meta-analytic evidence for the persuasive effect of narratives on

beliefs, attitudes, intentions, and behaviors. Communication Monographs. 2016;83:446-

67.

61. McQueen A, Kreuter MW, Kalesan B, Alcaraz KI. Understanding Narrative Effects: The

Impact of Breast Cancer Survivor Stories on Message Processing, Attitudes, and Beliefs

Among African American Women. Health Psychology. 2011;30:674-82.

62. Ziebland S, Wyke S. Health and Illness in a Connected World: How Might Sharing

Experiences on the Internet Affect People's Health? Milbank Quarterly. 2012;90:219-49.

63. Newman MA, Ziebland S, Barker KL. Patients' views of a multimedia resource featuring

experiences of rheumatoid arthritis: pilot evaluation of www.healthtalkonline.org. Health

Inform J. 2009;15:147-59.

64. Fox S. The social life of health information. FactTank: New in the Numbers 2014;

https://www.pewresearch.org/fact-tank/2014/01/15/the-social-life-of-health-information/.

65. DIPEx International. Research. 2019; http://www.dipexinternational.org/research.


66. University of Oxford. Health Experiences Research Group. 2019;

https://www.phc.ox.ac.uk/research/health-experiences. Accessed April 11, 2019.

www.healthtalkonline.org

https://www.pewresearch.org/fact-tank/2014/01/15/the-social-life-of-health-information/

http://www.dipexinternational.org/research

https://www.phc.ox.ac.uk/research/health-experiences

Page 83 of 128

67. Shannon-Baker P. Making Paradigms Meaningful in Mixed Methods Research. Journal

of Mixed Methods Research. 2016;10:319-34.

68. Morgan DL. Paradigms Lost and Pragmatism Regained: Methodological Implications of

Combining Qualitative and Quantitative Methods. Journal of Mixed Methods Research.

2007;1:48-76.

69. Sandelowski M. Unmixing Mixed‐Methods Research. Research in Nursing & Health.

2014;37:3-8.

70. Pope C, Mays N. Reaching The Parts Other Methods Cannot Reach: An Introduction To

Qualitative Methods In Health And Health Services Research. BMJ: British Medical

Journal. 1995;311:42-5.

71. Greene J, C. , Caracelli V, J. , Graham W, F. . Toward a Conceptual Framework for

Mixed-Method Evaluation Designs. Educational Evaluation and Policy Analysis.

1989;11:255-74.

72. Ozawa S, Pongpirul K. 10 best resources on … mixed methods research in health

systems. Health Policy and Planning. 2014;29:323-7.

73. Pluye P, Hong QN. Combining the power of stories and the power of numbers: Mixed

methods research and mixed studies reviews. Annual Review of Public Health.

2014;35:29-45.

74. Privitera GJ, Ahlgrim-Delzell L. Research Methods for Education: SAGE Publications;

2018.

75. Kim H, Sefcik JS, Bradway C. Characteristics of Qualitative Descriptive Studies: A

Systematic Review. Research in nursing & health. 2017;40:23-42.

76. Sandelowski M. Whatever happened to qualitative description? Res Nurs Health.

2000;23:334-40.

77. Groves RM, Fowler Jr FJ, Couper MP, Lepkowski JM, Singer E, Tourangeau R. Survey

methodology. Vol 561: John Wiley & Sons; 2011.

78. Coughlan M, Cronin P, Ryan F. Survey research: Process and limitations. International

Journal of Therapy and Rehabilitation. 2009;16:9-15.

79. Kitzinger J. Introducing Focus Groups. BMJ: British Medical Journal. 1995;311:299-

302.

80. Smithson J. Using and analysing focus groups: Limitations and possibilities.

International Journal of Social Research Methodology. 2000;3:103-19.

Page 84 of 128

81. Kelly L, Ziebland S, Jenkinson C. Measuring the effects of online health information:

Scale validation for the e-Health Impact Questionnaire. Patient Education and

Counselling. 2015;98:1418-24.

82. Kelly L, Jenkinson C, Ziebland S. Measuring the effects of online health information for

patients: Item generation for an e-health impact questionnaire. Patient Education and

Counseling. 2013;93:433-8.

83. Heath J, Williamson H, Williams L, Harcourt D. Supporting children with burns:

Developing a UK parent-focused peer-informed website to support families of burn-

injured children. Patient Educ Couns. 2019;102:1730-5.

84. Talboom-Kamp E, Tossaint-Schoenmakers R, Goedhart A, Versluis A, Kasteleyn M.

Evaluating an online patient portal for communicating laboratory test results: A real-

world study into patients' attitude using the eHealth Impact Questionnaire (eHIQ). JMIR

Form Res. 2020;04:04.

85. Boyko JA, Lavis JN, Dobbins M, Souza NM. Reliability of a tool for measuring theory of

planned behaviour constructs for use in evaluating research use in policymaking. Health

Research Policy and Systems. 2011;9:29-.

86. Francis J, Eccles MP, Johnston M, et al. Constructing questionnaires based on the theory

of planned behaviour: A manual for health services researchers. Newcastle upon Tyne,

UK: Centre for Health Services Research, University of Newcastle upon Tyne; 2004.

87. Cohen DJ. Statistical power analysis (2nd ed.). Hillsdale, NJ.: Erlbaum.; 1988.

88. Morgan DL, Hoffman K. "Focus Groups". In: Flick U, ed. The SAGE Handbook of

Qualitative Data Collection. 55 City Road: SAGE Publications Ltd; 2018:250-63.

89. Field A. Discovering Statistics Using SPSS. Third ed: SAGE Publications Ltd; 2009.

90. Roulston K. "Analysing Interviews". In: Flick U, ed. The SAGE Handbook of Qualitative

Data Analysis. 55 City Road: SAGE Publications, Inc.; 2014:297-312.

91. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in

Psychology. 2006;3:77-101.

92. Vogl S. Integrating and Consolidating Data in Mixed Methods Data Analysis: Examples

From Focus Group Data With Children. Journal of mixed methods research.

2018;13:536-54.

93. Sandelowski M, Voils CI, Knafl G. On Quantitizing. Journal of mixed methods research.

2009;3:208-22.

94. Skowronski JJ, Walker WR, Henderson DX, Bond GD. Chapter Three - The Fading

Affect Bias: Its History, Its Implications, and Its Future. In: Olson JM, Zanna MP, eds.

Advances in Experimental Social Psychology. Vol 49: Academic Press; 2014:163-218.

Page 85 of 128

95. Drennan J, Hyde A. Controlling response shift bias: the use of the retrospective pre-test

design in the evaluation of a master's programme. Assessment & Evaluation in Higher

Education. 2008;33:699-709.

96. Kreitchmann RS, Abad FJ, Ponsoda V, Nieto MD, Morillo D. Controlling for Response

Biases in Self-Report Scales: Forced-Choice vs. Psychometric Modeling of Likert Items.

Frontiers in Psychology. 2019;10.

97. Davies MF. Confirmatory Bias in the Evaluation of Personality Descriptions: Positive

Test Strategies and Output Interference. Journal of Personality and Social Psychology.

2003;85:736-44.

Page 86 of 128

Appendices

Appendix A Website Layout

Figure 7: Image of front page of website.

Page 87 of 128

Appendix B Survey Tool Development

Table 11: Original and adapted eHIQ.

Original eHIQ82 Adapted eHIQ

1.1 Attitudes towards online health information

I would use the internet if I needed help to make a

decision about my health (for example,

whether I should see a doctor, take medication

or seek other types of treatment).

I would use the internet if I needed help to make a

decision about whether to be part of a patient-

researcher partnership.

The internet can be useful to help people decide if

their symptoms are important enough to go to

see a doctor.

The internet can be useful to help people decide if

they are equipped to be part of a patient-


I would use the internet to check that the doctor is

giving me appropriate advice.

(removed)

The internet can help the public to know what it is

like to live with a health problem.

The internet can help the public to know what it is

like to be part of a patient-researcher

partnership.

The internet is a reliable resource to help me

understand what a doctor tells me.

The internet is a reliable module to help me

understand patient-researcher partnerships.

1.2 Attitudes towards sharing health experiences online

The internet is a good way of finding other people

who are experiencing similar health problems.


who have similar patient-researcher

partnership experiences.

It can be helpful to see other people’s health-

related experiences on the internet.

It would be helpful to see other people's

partnership-related experiences on the internet.


who are facing health-related decisions I may

also face.


who are facing patient-researcher partnership-

related decisions I may also face.

Looking at health websites reassures me that I am

not alone with my health concerns.

Looking at websites reassures me that I am not

alone in my concerns about my patient-

researcher partnership experience.

The internet is useful if you don’t want to tell

people around you (for example, your family

or people at work) how you feel.

(removed)

It can be reassuring to know that I can access

health-related websites at any time of the day

or night.

(removed)

2.1 Confidence and identification

I feel I have a sense of solidarity with other

people using the website.

I feel I have a sense of solidarity with people on

the website.

I can identify with other people using the website. I can identify with people on the website.

I feel I have a lot in common with other people

using the website.

I feel like I have a lot in common with people on

the website.

The website gives me the confidence to explain

my health concerns to others.

The website gives me the confidence to explain

my involvement in research to others.

The website gives me confidence that I am able to

manage my health.

The website gives me the confidence that I am

able to participate in a patient-researcher

partnership.

Page 88 of 128

The website makes me more confident to discuss

my health with the people around me (for

example, my family or people at work).

The website makes me more confident to discuss

my partnership experience with the people

around me (for example, my family or people

at work).

I value the advice given on the website. I value the advice given on the website.

The people who have contributed to the website

understand what is important to me.

The people who have contributed to the website

understand what is important to me.

The website prepares me for what might happen

to my health.

The website prepares me for what might happen

in a patient-researcher partnership.

2.2 Information and presentation

The website is easy to use. The website is easy to use.

The language on the website made it easy to

understand.

The language on the website made it easy to

understand.

I can easily understand the information on the

website.

I can easily understand the information on the

website.

I trust the information on the website. I trust the information on the website.

The information on the website left me feeling

confused.

The information on the website left me feeling

confused.

The website provides a wide range of information. The website provides a wide range of information.

Photographs and other images were used

appropriately on the website.

Videos were used appropriately on the website.

I found the images on the website distressing. I found the stories or other content on the website

distressing or worrisome.

2.3 Understanding and motivation

The website encourages me to take actions that

could be beneficial to my health.

The website encourages me to take actions to

become a patient-researcher partner.

The website includes useful tips on how to make

life better.

The website includes useful tips on how to be a

better patient-researcher partner.

The website encourages me to play a more active

role in my healthcare.

The website encourages me to play a more active

role as a patient-researcher partner.

The website helps me to have a better

understanding of my personal health

The website helps me to have a better

understanding of patient-researcher

partnerships.

I feel more inclined to look after myself after

visiting the website.

I feel more inclined to participate in a patient-

researcher partnership after visiting the

website.

I have learnt something new from the website. I have learnt something new from the website.

On the whole, I find the website reassuring. On the whole, I find the website reassuring.

The website has a positive outlook. The website has a positive outlook.

I would consult the website if I had to make a

decision about my health.

I would consult the website if I had to make a

decision about whether or not to be part of a

patient-researcher partnership.

Page 89 of 128

Table 12: Adapted Theory of Planned Behaviour survey questions, by domain.

Construct Survey Question

Behavioural

Intention

I want to engage in patient-researcher partnerships.

strongly disagree 1 2 3 4 5 6 7 strongly agree

I intend to engage in patient-researcher partnerships.


Attitude Patient-researcher partnerships are...

very harmful 1 2 3 4 5 6 7 very beneficial

Patient-researcher partnerships are, or would be...

very unpleasant 1 2 3 4 5 6 7 very pleasant

Patient-researcher partnerships are...

very unhelpful 1 2 3 4 5 6 7 very helpful

Subjective

Norms

It is expected of me that I engage in patient-researcher partnerships.


I feel under social pressure to engage in patient-researcher partnerships.


People who are important to me want me to engage in patient-researcher partnerships.


Perceived

Behavioural

Control

I am confident that I could engage in patient-researcher partnerships.


For me to engage in patient-researcher partnerships is, or would be…

very difficult 1 2 3 4 5 6 7 very easy

The decision to engage in patient-researcher partnerships is beyond my control.


Whether or not I engage in patient-researcher partnerships is entirely up to me.


Page 90 of 128

Appendix C Sample Recruitment Email

Dear <organization/group>,

I am a graduate student at IHPME and as part of my thesis I’m looking for researchers to help

with the evaluation of a website about patient engagement in research. Your participation would

be highly valued!

Project Details

Short Title: Evaluation of a Website about Patient-Researcher Partnerships

Purpose of the project: To get feedback about the helpfulness of the website and to find out if

spending time on it influences perceptions about patient engagement in research.

Who we’re looking for: Patients/caregivers (anyone with lived experience of illness) and health

researchers (includes clinical, biomedical, health services, and

social/cultural/environmental/population researchers) who have experience with, or want to learn

more about, patient engagement in research.

Description and time commitment: Involvement is entirely online and will include completing

an initial survey, browsing the website at your convenience over a period of 1-2 weeks (we

estimate a total time of no more than 2-3 hours on the website, though there is no time limitation)

and then completing a final survey.

If interested: Forward this email to Donna at [email protected] or Susan Law at

[email protected]. We will email you links to the surveys and website.

Kind regards,

Donna Plett, MSc(c)

Institute of Health Policy, Management and Evaluation | University of Toronto

[email protected]

https://ihpme.utoronto.ca/

mailto:[email protected]



https://ihpme.utoronto.ca/

Page 91 of 128

Appendix D Organizations Contacted During Recruitment

Table 13: Organizations contacted during recruitment process

Organization Jurisdiction Invitation

Type

Alberta SPOR SUPPORT Unit Alberta Surveys only

BC SUPPORT Unit British Columbia Surveys only

Canadian Foundation for Healthcare Improvement (CFHI) Canada Surveys only

Canadian Patient Safety Institute (CPSI) Canada Surveys only

Centre of Excellence on Partnership with Patients and the Public

(CEPPP)

Quebec Surveys only

Change Foundation Ontario Surveys only

Faculty, Department of Psychiatry, University of Toronto Ontario Surveys only

Faculty, Institute of Health Policy, Management and Evaluation,

University of Toronto

Ontario Surveys only

George & Fay Yee Centre for Healthcare Innovation Manitoba Surveys only

Institute of Health Policy, Management, and Evaluation (IHPME)

Patient Engagement Working Group

Toronto, ON Surveys and

focus group

Manitoba Institute for Patient Safety (MIPS) Manitoba Surveys only

Michael Smith Foundation for Health Research (MFSR) British Columbia Surveys only

Patient and Community Engagement Research (PaCER) Innovates Alberta Surveys only

Patient and Family Advisors, North York General Toronto, Ontario Surveys only

Patient and Family Advisory Council, Central LHIN Toronto, Ontario Surveys only

Patient and Family Advisory Councils (PFAC), Trillium Health

Partners (THP)

Peel Region,

Ontario

Surveys only

Patient and Family Advisory Group, Alberta Health Services Alberta Surveys only

Patient and Family-Centred Care (PFCC) Community Ontario Surveys only

Patient Voices Network British Columbia Surveys only

Patient, Family, and Public Advisors Network, Health Quality

Ontario

Ontario Surveys only

Patient-Oriented Research and Innovation (PORI) Working Group,

Trillium Health Partners

Peel Region,

Ontario

Surveys and

focus group

Patients as Partners, BC Government British Columbia Surveys only

Patients Canada Canada Surveys only

Public & Patient Engagement, Winnipeg Regional Health

Authority (WRHA)

Winnipeg,

Manitoba

Surveys only

Public Advisory Council, ICES Ontario Surveys only

Research Manitoba (RM) Manitoba Surveys only

Saskatchewan Centre for Patient-Oriented Research Saskatchewan Surveys only

Saskatchewan Health Quality Council Saskatchewan Surveys only

Saskatchewan Health Research Foundation (SHRF) Saskatchewan Surveys only

Strategic Clinical Networks PERG, Alberta Health Services Alberta Surveys only

Note: The recruitment process was interrupted due to COVID-19 restrictions. We planned to

reach out to more academic institutions and patient organizations in Quebec and the East Coast.

https://spor.albertainnovates.ca/the-alberta-spor-support-unit/patient-engagement-platform/

https://ceppp.ca/en/

https://chimb.ca/

https://mips.ca/contact-us.html

https://www.msfhr.org/our-work/patient-oriented-research

https://pacerinnovates.ca/who-we-are

https://www.albertahealthservices.ca/info/Page6620.aspx

https://patientvoicesbc.ca/

https://www2.gov.bc.ca/gov/content/health/about-bc-s-health-care-system/partners/patients

https://patientscanada.ca/get-involved/join/

https://www.wrha.mb.ca/about/engagement/index.php

https://www.wrha.mb.ca/about/engagement/index.php

https://researchmanitoba.ca/

https://www.scpor.ca/

https://hqc.sk.ca/

https://www.shrf.ca/

Page 92 of 128

Appendix E Focus Group Consent Form

Participant Information and Consent Form for Focus Groups

Study: Experiences of patient/caregiver-researcher partnerships in integrated knowledge

translation (iKT): A study to develop online training modules for patient-oriented research

Principal Investigator: Susan Law, PhD Student PI: Donna Plett, MSc Candidate

Research Team: David K Wright, PhD (co-PI); Ian Graham, PhD (co-PI); Kerry Kuluski, PhD

(Collaborator); Sharon Straus, PhD (Collaborator); Gwen Barton (Manager of Patient Experience

for the Patient and Family Advisory Council); Linda Murphy (Patient/Caregiver Advisor);

Patricia Pottie (Patient/Caregiver Advisor); Michelle Marcinow, PhD (Research Associate); Ilia

Ormel, PhD (Research Associate)

Sponsors: The Strategy for Patient-Oriented Research (SPOR) is an initiative of Canada’s

federal research granting agency for health – the Canadian Institutes for Health Research (CIHR)

– that focuses on the translation of research results to improve health outcomes for Canadians.

The Ontario SPOR Support Unit (OSSU) is mandated to carry out activities in Ontario in support

of CIHR’s Strategy for Patient- Oriented Research. This OSSU Capacity Building award

opportunity was introduced to fund projects that help to build capacity for patients and families,

healthcare providers, decision makers and researchers to conduct and use patient-oriented

research in Ontario.

What is the purpose of this study?

The aim of this study is to create two online training modules to better understand people’s

experiences of patient-oriented research and improve partnerships between researchers and

patients/caregivers.

What will happen if I take part?

If you choose to participate, you will take part in an approximately 60-90 minute audio recorded

focus group with 5-7 other people. You will review the online training modules that we have co-

designed and provide us with feedback about the modules. A member of the study team will

facilitate the focus group and you will have the opportunity to provide any comments about ways

to improve the design and content of the modules.

The focus group will be transcribed and a summary of the key themes and suggestions will be

sent back to you to review. The feedback you provide will be used to improve the usability of the

online training modules.

Do I have to take part? What if I change my mind?

No, it is entirely up to you to decide whether or not you would like to take part in this study. If

you change your mind and wish to withdraw prior to the focus group, no information will be

collected. If you wish to withdraw after the focus group, your identifying information will be

destroyed, but we will be unable to remove your statements from the audio recording and

transcripts as these are not attributed to individual speakers.

What are the risks and benefits to participating in this study?

Page 93 of 128

Taking part in the focus groups will take 60-90 minutes. This may cause you to be tired.

However, your contributions to this study will benefit those who view the online training

modules in the future.

Who has reviewed the study?

This study has been reviewed and approved by the Research Ethics Board at Trillium Health

Partners (Study #880).

Who is organising and funding the research?

The Institute for Better Health, based at Trillium Health Partners, is collaborating with the

University of Ottawa and St. Mary’s Research Centre to complete this research. The project

“Experiences of patient/caregiver-researcher partnerships in integrated knowledge translation: a

study to develop online training modules for patient-oriented research” is financially supported

by the Ontario SPOR Support Unit’s Patient-Oriented Research Capacity Building Initiatives

Awards.

Contact for further information

If you have any questions at all about the project or wish to discuss anything, please contact Dr.

Susan Law by telephoning (416) 270-6643, or emailing at [email protected].

If you want to talk to someone not connected with the study about your rights as a study

participant, or if you have any complaints about the research, you can call Trillium Health

Partners’ Research Ethics Board at (905) 848 7580 x1682 or the University of Toronto Review

Ethics Board at (416) 946-3273 or [email protected].


Page 94 of 128

DOCUMENTATION OF INFORMED CONSENT

Study Title: Experiences of patient/caregiver-researcher partnerships in integrated knowledge

translation: A study to develop online training modules for patient-oriented research

Participant:

By signing this form, I confirm that:

• I have read each page of this form and this study has been fully explained to me.

• I understand the research being done, what is required to participate in this study, and what is

expected of me.

• I have had a chance to have all of my questions answered to my satisfaction.

• If I choose not to participate, I can hand back an unsigned copy of the consent form to the

research team member.

• I understand that your consent does not waive any of your legal rights, nor relieve the research

team from any legal responsibility.

• I agree to participate in this study.

I wish to receive general information in the future about the health experiences research (e.g.,

newsletter, announcements regarding new modules, etc.) YES/NO

I can be contacted about future research on health experiences. YES/NO

Preferred phone number: __________________________

Preferred email: ________________________________

_______________________________ ________________________ _______________

Name of Participant (print) Signature Date

Statement of Research Team Member:

I will respect the rights and wishes of the participant as described in this informed consent

document, and conduct this study according to all applicable laws, regulations, and guidelines

relating to the ethical and legal conduct of research. I have explained the purpose of the study to

the participant and have answered all of his or her questions.

_______________________________ ________________________ _______________

Name of Person Obtaining Consent Signature Date

Page 95 of 128

Appendix F Interview Guide for Focus Groups

Introduction, Purpose and Consent (20 minutes)

Thank everyone for attending.

Hello, my name is Susan Law and I am working with a research team investigating patient-

researcher partnerships and the potential value of an online learning module that features the

experiences of patients/caregivers and researchers as partners in research, using video/audio

excerpts from individual interviews. The purpose of this group session is to discuss your

perceptions of patient-researcher partnerships and the online resource that you have had access to

over the last few weeks. We will aim to be finished in no more than 2 hours today.

Roundtable introductions – name, role, experience of research and partnerships

Consent form

• Review consent form

• Request permission to record the session for purposes of analysis

• Participants will be requested not to disclose statements made by other participants

during the session

Brief Overview of the Project and Results to Date (20 minutes)

Susan, Donna – present overview of purpose, methods, results to date.

Focus group questions – for discussion (60 minutes):

A. Initial Reflections about the Online Resource and Survey Results to Date

1. Tell me about your overall impression of the use and value of the website?

2. Would you use the website? How/why? Examples? Who else might find it useful?

3. Any suggestions for improving the design or layout of the website?

4. Any content that might be missing or removed?

5. Any concerns/risks regarding the information presented?

B. KNOWLEDGE: Existing understanding about partnerships

1. In what sorts of ways was the website helpful in your understanding of partnerships –

what it is and how it works? Examples of what you learned, what was new, what was

surprising?

Page 96 of 128

2. Do you think this information might change people’s attitudes towards partnerships?

Why? What about yours – change or no change?

3. To what extent do you feel that (patients, caregivers, researchers) receive adequate

information about patient-researcher partnerships?

a. positive and negative examples?

b. examples of how

4. Tell me about particular information gaps that should be addressed? Examples?

a. Does the website address any of these gaps?

C. ATTITUDES: Perceptions about value of partnerships

1. What do you believe are the advantages of patient-researcher partnerships?

a. For yourself and in general

2. What do you believe are the disadvantages of patient-researcher partnerships?

a. For yourself and in general

3. How, if at all, did the website influence your attitude toward patient-researcher

partnerships?

D. BARRIERS & FACILITATORS: Motivations to participate in partnerships

1. What factors or circumstances would enable you to participate in patient-researcher

partnerships?

2. What factors or circumstances might make it difficult or impossible for you to participate

in patient-researcher partnerships?

3. Are there any other issues that come to mind when you think about participating in

patient-researcher partnerships?

4. In what ways did the website influence how prepared you would feel to participate in a

research partnership? More or less motivated to participate in partnership?

Wrap-up and Next Steps (20 minutes)

Susan to sum up key messages

Participants to comment – final comments or additional ideas

Explain what we will do with the results – inform any changes to online module, contribution to

evaluation results.

Thank you for participating in the focus group and sharing your feedback!

Page 97 of 128

Appendix G Pre and Post Surveys

PRE SURVEY

Title: Evaluation of a Website about Patient-Researcher Partnerships: Survey 1

Consent Form

Principal Investigator: Susan Law, PhD, Director - Research, Institute for Better Health -


Student Investigator: Donna Plett, MSc Student - Health Services Research, IHPME, University

of Toronto

Hello! You are invited to participate in this web-based survey to help evaluate a website about

engagement between patients and researchers as partners in research, featuring video/audio

excerpts from interviews about their experiences.

This evaluation is in two parts:

1) Before reviewing the website, we will ask about you and your interests in patient-researcher

partnerships;

2) Then, after providing some time (approximately 1 week) to review the website and explore the

various themes and topics featured there, we will ask you to complete a follow-up survey to

obtain your views about the value and potential usefulness of this website and thoughts about

patient-researcher partnerships.

It should take a maximum of 10-15 minutes to complete each survey (pre and post).

PARTICIPATION: Your participation in this survey is voluntary. You may refuse to take part in

the research or exit the survey at any time without penalty. You are free to decline to answer any

particular question you do not wish to answer for any reason. If you would like to exclude your

responses from the study please contact us within 2 weeks and we will delete them.

BENEFITS: You will receive no direct benefits from participating in this research study.

However, your responses may help us learn more about a website about patient-researcher

partnerships.

RISKS: There are no foreseeable risks involved in participating in this study.

CONFIDENTIALITY: The email address that you have already provided to the research team

will only be accessed by the principal investigator (PI) and student PI. It will only be used for the

purposes of follow-up communication, after which your e-mail will be deleted. No names or

identifying information will be included in any publications or presentations based on these data,

and your responses to this survey will remain confidential. Our records of participants' email

addresses will be stored securely on Trillium Health Partners servers (accessible only to the

research team) and will remain separate from your survey responses.

Page 98 of 128

IP address tracking will be turned off on the survey tool. Survey responses from SurveyMonkey

are sent over a secure, SSL encrypted connection. Since SurveyMonkey is a company located in

the United States, USA authorities under the provisions of the Patriot Act may access the survey

data in rare instances.

CONTACT: If you have questions at any time about the study or the procedures, you may

contact the PI, Susan Law, via phone at (905) 848-7580 x1659 or email at [email protected].

If you feel you have not been treated according to the descriptions in this form, or that your

rights as a participant in research have not been honored during the course of this project, or you

have any questions, concerns, or complaints that you wish to address to someone other than the

investigator, you may contact either the Research Ethics Board at Trillium Health Partners at

(905) 848-7580 ext.1682 or [email protected], OR the University of Toronto Review Ethics Board

at (416) 946-3273 or [email protected].

ELECTRONIC CONSENT: Please select your choice below. You may print a copy of this

consent form for your records.

Clicking on the "Agree" button indicates that:

• You have read the above information and fully understand the research that is being done and

what is being asked of you

• You voluntarily agree to participate in this study

• You understand that you may withdraw from the study at any time and your withdrawal does

not waive any of your legal rights, nor relieve the research team from any legal responsibility

• You are 18 years of age or older

Agree

Disagree

Part 1

Please provide the following details about yourself.

Note: For the purposes of this study, we are defining patient-researcher partnerships as

activities where patients and/or caregivers and researchers are working together in any or all

aspects of planning studies, reviewing literature, collecting and analyzing data/information

and/or disseminating the results of research.

Page 99 of 128

1.1 Age (circle):

18-24 25-34 35-44 45-54 55-64 65+

1.2 Gender (circle)

Female Male Other: ____________________

1.3 Country of Birth: __________________________________

1.4 Self-Identified Ethnic Origin and/or Cultural Background:

_____________________________________________

1.5 Town/City of Residence: _____________________

1.6 Province of Residence: ______________________

1.7 Highest Level of Education Attained (circle one):

Less than high school High school College Undergraduate Postgraduate

1.8 Employment Status (circle all that apply):

Full-time Part-time Retired Student Unemployed

Other:_______________________

1.9 Occupation: _______________________________

1.10 My primary role in patient-researcher partnerships is or would be as a… (circle one)

Patient or Caregiver (includes anyone with lived experience of illness)

Researcher (with formal training and/or academic appointment)

Could be either (role varies depending on the project)

Other: ____________________________

1.11 FOR ‘COULD BE EITHER’: Which perspective would you like to use for this study?

Patient/Caregiver

Researcher

Page 100 of 128

1.12 FOR PATIENTS/CAREGIVERS: What illness or problem(s) did you or a loved one

experience?

__________________________________________________________________________

1.13 FOR RESEARCHERS: What is your primary field of research?

Health Services & Policy Population & Public Health Basic Biomedical Clinical Other:

_______________

1.14 I have been part of (circle)… 0 1 2 3+ …research projects that included patient-

researcher partnerships.

1.15 Have you received formal training in research partnerships? (circle) Yes No

Part 2

This section is about the internet as a place to learn about patient-researcher partnerships. Please

answer as best you can even if you have no prior experience with patient-researcher partnerships.

2.1 The internet is a reliable module to help me understand patient-researcher partnerships.

Strongly Disagree 1 2 3 4 5 Strongly Agree

2.2 The internet can help the public to know what it is like to be part of a patient-researcher

partnership.


2.3 The internet can be useful to help people decide if they are equipped to be part of a patient-



2.4 I would use the internet if I needed help to make a decision about whether to be part of a



Page 101 of 128

2.5 The internet is a good way of finding other people who have similar patient-researcher



2.6 It would be helpful to see other people’s partnership-related experiences on the internet.


2.7 The internet is a good way of finding other people who are facing patient-researcher

partnership-related decision I may also face.


2.8 Looking at websites reassures me that I am not alone in my concerns about my patient-



2.9 Do you have any other comments about patient-researcher partnerships and web-based

information?

___________________________________________________________________________

___________________________________________________________________________

Part 3

This section is about how you think and feel about patient-researcher partnerships. Please give

your best answer, even if you have no prior experience with patient-researcher partnerships.

Note: For the purposes of this study, we are defining patient-researcher partnerships as activities

where patients and/or caregivers and researchers are working together in any or all aspects of

planning studies, reviewing literature, collecting and analyzing data/information and/or

disseminating the results of research.

3.1 I want to engage in patient-researcher partnerships.

Strongly Disagree 1 2 3 4 5 6 7 Strongly Agree

Page 102 of 128

3.2 I intend to engage in patient-researcher partnerships.


3.3 Patient-researcher partnerships are, or would be…

Very Harmful 1 2 3 4 5 6 7 Very Beneficial

Very Unpleasant 1 2 3 4 5 6 7 Very Pleasant

(for me) (for me)

Very Unhelpful 1 2 3 4 5 6 7 Very Helpful

3.4 Do you have any comments to add regarding your expectations, intentions or thoughts about

engaging in patient-researcher partnerships?

___________________________________________________________________________

___________________________________________________________________________

3.5 It is expected of me that I engage in patient-researcher partnerships.


3.6 I feel under social pressure to engage in patient-researcher partnerships.


3.7 People who are important to me want me to engage in patient-researcher partnerships.


3.8 Which types of people did you have in mind while answering the last three questions (e.g.,

family, friends, colleagues, work-related superiors, etc.)?

___________________________________________________________________________

___________________________________________________________________________

3.9 I am confident that I could engage in patient-researcher partnerships.


Page 103 of 128

3.10 For me to engage in patient-researcher partnerships is, or would be…

Very Difficult 1 2 3 4 5 6 7 Very Easy

3.11 What makes engaging in patient-researcher partnerships easy for you?

___________________________________________________________________________

___________________________________________________________________________

3.12 What makes engaging in patient-researcher partnerships difficult for you?

___________________________________________________________________________

___________________________________________________________________________

3.13 The decision to engage in patient-researcher partnerships is beyond my control.


3.14 Whether or not I engage in patient-researcher partnerships is entirely up to me.


3.15 If you feel it is not in your control, why not?

___________________________________________________________________________

___________________________________________________________________________

3.16 Do you have any further comments about patient-researcher partnerships and your

involvement?

___________________________________________________________________________

___________________________________________________________________________

Part 4

Next Steps:

1. Go to the website (the link will be emailed to you when you press 'submit'). Please note that

this website is still being developed and the final version will include some additional content.

Page 104 of 128

2. Spend time on the website at your convenience over the next week. The website consists of

many different themes, topics and pages related to patient-researcher partnerships. We ask that

you navigate through the website and engage with whichever content is interesting or relevant to

you.

3. Complete final survey in approximately one week (you will receive a link by email)

If you have any questions please don't hesitate to contact Donna at [email protected]

or Susan at [email protected].

This information will be sent to your email address when you press 'submit.'

Thank you for filling out this survey!

Page 105 of 128

POST SURVEY

Title: Evaluation of a Website about Patient-Researcher Partnerships: Survey 2

Consent Form

Principal Investigator: Susan Law, PhD, Director - Research, Institute for Better Health -


Student Investigator: Donna Plett, MSc Student - Health Services Research, IHPME, University

of Toronto

Hello! You have reached the second and final stage of this study. Please complete this follow-up

survey so we can obtain your views about the value and potential usefulness of the website and

your thoughts about patient-researcher partnerships.

It should take a maximum of 10-15 minutes to complete this survey.

PARTICIPATION: Your participation in this survey is voluntary. You may refuse to take part in

the research or exit the survey at any time without penalty. You are free to decline to answer any

particular question you do not wish to answer for any reason. If you would like to exclude your

responses from the study please contact us within 2 weeks and we will delete them.

BENEFITS: You will receive no direct benefits from participating in this research study.

However, your responses may help us learn more about a website about patient-researcher

partnerships.

RISKS: There are no foreseeable risks involved in participating in this study.

CONFIDENTIALITY: The email address that you have already provided to the research team

will only be accessed by the principal investigator (PI) and student PI. It will only be used for the

purposes of follow-up communication, after which your e-mail will be deleted. No names or

identifying information will be included in any publications or presentations based on these data,

and your responses to this survey will remain confidential. Our records of participants' email

addresses will be stored securely on Trillium Health Partners servers (accessible only to the

research team) and will remain separate from your survey responses.

IP address tracking will be turned off on the survey tool. Survey responses from SurveyMonkey

are sent over a secure, SSL encrypted connection. Since SurveyMonkey is a company located in

the United States, USA authorities under the provisions of the Patriot Act may access the survey

data in rare instances.

CONTACT: If you have questions at any time about the study or the procedures, you may

contact the PI, Susan Law, via phone at (905) 848-7580 x1659 or email at [email protected].

If you feel you have not been treated according to the descriptions in this form, or that your

rights as a participant in research have not been honored during the course of this project, or you

have any questions, concerns, or complaints that you wish to address to someone other than the

investigator, you may contact either the Research Ethics Board at Trillium Health Partners at

Page 106 of 128

(905) 848-7580 ext.1682 or [email protected], OR the University of Toronto Review Ethics Board

at (416) 946-3273 or [email protected].

ELECTRONIC CONSENT: Please select your choice below. You may print a copy of this

consent form for your records.

Clicking on the "Agree" button indicates that:

• You have read the above information and fully understand the research that is being done and

what is being asked of you

• You voluntarily agree to participate in this study

• You understand that you may withdraw from the study at any time and your withdrawal does

not waive any of your legal rights, nor relieve the research team from any legal responsibility

• You are 18 years of age or older

Please indicate if you agree or disagree.

Agree

Disagree

Part 1

Approximately how much time did you spend on the website in total?

Less than 30 minutes

30 minutes to 1 hour

1 to 3 hours

3+ hours

Part 2

This section is about your thoughts about the website. Please answer as best you can even if you

have no prior experience with patient-researcher partnerships.





Page 107 of 128

2.10 The website encourages me to take actions to become a patient-researcher partner.


2.11 The website has a positive outlook.


2.12 The information on the website left me feeling confused.


2.13 The website includes useful tips on how to be a better patient-researcher partner.


2.14 The website provides a wide range of information.


2.15 The language on the website made it easy to understand.


2.16 I feel more inclined to participate in a patient-researcher partnership after visiting the

website.


2.17 I have learnt something new from the website.


2.18 I can easily understand the information on the website.


2.19 The website prepares me for what might happen in a patient-researcher partnership.


2.20 The people who have contributed to the website understand what is important to me.


Page 108 of 128

2.21 I trust the information on the website.


2.22 I would consult the website if I had to make a decision about whether or not to be part of a



2.23 I feel I have a sense of solidarity with people on the website.


2.24 I can identify with people on the website.


2.25 On the whole, I find the website reassuring.


2.26 I value the advice given on the website.


2.27 The website gives me the confidence that I am able to participate in a patient-researcher

partnership.


2.28 I feel like I have a lot in common with people on the website.


2.29 The website gives me the confidence to explain my involvement in research to others.


2.30 The website helps me to have a better understanding of patient-researcher partnerships.


Page 109 of 128

2.31 The website encourages me to play a more active role as a patient-researcher partner.


2.32 The website makes me more confident to discuss my partnership experience with the

people around me (for example, my family or people at work).


2.33 Videos were used appropriately on the website.


2.34 I found the stories or other content on the website distressing or worrisome.


2.35 The website is easy to use.


2.36 Any other comments you would like to add about the website and/or the use of video

narratives?

___________________________________________________________________________

___________________________________________________________________________

Part 3

This section asks about your thoughts and attitudes toward patient-researcher partnerships after

spending time on the website (note: many questions are the same as in the last survey). Please

answer as best you can even if you have no prior experience with patient-researcher partnerships.





3.17 I want to engage in patient-researcher partnerships.


Page 110 of 128

3.18 I intend to engage in patient-researcher partnerships.


3.19 Did the website influence whether or not you want to engage in a patient-researcher

partnership in the future? Why or why not?

___________________________________________________________________________

___________________________________________________________________________

3.20 Did the website influence whether or not you think you are going to engage in a patient-

researcher partnership in the future? Why or why not?

___________________________________________________________________________

___________________________________________________________________________

3.21 Patient-researcher partnerships are, or would be…

Very Harmful 1 2 3 4 5 6 7 Very Beneficial

Very Unpleasant 1 2 3 4 5 6 7 Very Pleasant

(for me) (for me)

Very Unhelpful 1 2 3 4 5 6 7 Very Helpful

3.22 Did the website change how you feel about patient-researcher partnerships? Why or why

not? If so, how?

___________________________________________________________________________

___________________________________________________________________________

3.23 It is expected of me that I engage in patient-researcher partnerships.


3.24 I feel under social pressure to engage in patient-researcher partnerships.


Page 111 of 128

3.25 People who are important to me want me to engage in patient-researcher partnerships.


3.26 I am confident that I could engage in patient-researcher partnerships.


3.27 For me to engage in patient-researcher partnerships is, or would be…

Very Difficult 1 2 3 4 5 6 7 Very Easy

3.28 Did the website change how confident you feel about engaging in partnerships? If so, how?

___________________________________________________________________________

___________________________________________________________________________

3.29 The decision to engage in patient-researcher partnerships is beyond my control.


3.30 Whether or not I engage in patient-researcher partnerships is entirely up to me.


3.31 Any other comments you would like to add?

___________________________________________________________________________

___________________________________________________________________________

Thank you for participating in this research!

Page 112 of 128

Appendix H Demographic Data

Table 14: Demographics.

Total

n (%)

Patients

n (%)

Researchers

n (%)

Role in Partnership

Patient/Caregiver

Researcher

Research Staff

25 (51)

19 (39)

5 (10)

Researcher Type

Health Services Research

Social, Cultural, Environmental,

and Population Health

Research

Clinical Research

11 (22)

7 (14)

1 (2)

-

-

-

11 (58)

7 (37)

1 (5)

Age

18-24

25-34

35-44

45-54

55-64

65+

4 (8)

7 (14)

7 (14)

5 (10)

11 (22)

15 (31)

3 (12)

-

2 (8)

1 (4)

5 (20)

14 (56)

1 (5)

5 (26)

5 (26)

4 (21)

3 (16)

1 (5)

Gender

Female

Male

33 (67)

16 (33)

14 (56)

11 (44)

15 (79)

4 (21)

Country of Birth

Canada

India

UK

USA

England

Hong Kong

China

The Netherlands

Ireland

34 (69)

4 (8)

2 (4)

2 (4)

1 (2)

1 (2)

1 (2)

1 (2)

1 (2)

17 (68)

4 (16)

-

1 (4)

-

1 (4)

-

1 (4)

1 (4)

13 (68)

-

2 (11)

1 (5)

1 (5)

-

1 (5)

-

-

Province

Ontario

Alberta

Quebec

Manitoba

British Columbia

Saskatchewan

32 (65)

8 (16)

3 (6)

2 (4)

2 (4)

1 (2)

15 (60)

7 (28)

-

-

2 (8)

1 (4)

12 (63)

1 (5)

3 (16)

2 (11)

-

-

Highest Education

Postgraduate

Undergraduate

College

High school

36 (73)

6 (12)

4 (8)

2 (4)

15 (60)

5 (20)

3 (12)

2 (8)

18 (95)

-

-

-

Page 113 of 128

Employment

Full-time

Retired

Part-time

Student

Unemployed

23 (47)

14 (29)

8 (16)

5 (10)

1 (2)

4 (16)

13 (52)

4 (16)

4 (16)

1 (4)

15 (79)

1 (5)

3 (16)

1 (5)

-

Prior Partnership Experience

None

1 project

2 projects

3+ projects

8 (16)

6 (12)

14 (29)

21 (43)

6 (24)

5 (20)

7 (28)

7 (28)

1 (5)

-

7 (37)

11 (58)

Prior Training

Yes

No

18 (37)

31 (63)

8 (32)

17 (68)

9 (47)

10 (53)

Page 114 of 128

Appendix I Web Analytics Data

Table 15: Page views and unique pages views by person featured on website.

Values include combination of visits to person’s profile page as well as pages with one of

their individual video clips.

Name/Alias Role Page

Views

Unique Page

Views

Maureen Patient 42 27

John Patient 36 24

Maxime Patient 30 18

Nicole Researcher 23 17

Annette Researcher 21 8

Wendy Researcher 21 14

Julie Researcher 15 13

Carol Patient 13 9

Karen Patient 12 10

Laurie Patient 12 7

Franco Researcher 10 5

Cathy Patient 9 7

Linda Patient 9 7

Suzanne Patient 6 6

Amy Patient 4 4

Emma Researcher 4 4

Nicolas Both 4 4

Zarah Researcher 4 4

Claire Patient 3 3

Dawn Researcher 3 3

Esther Researcher 3 3

Frank Patient 3 3

Ian Researcher 3 3

Janet Researcher 3 3

Katie Researcher 3 3

Manda Caregiver 2 2

Marc Caregiver 2 2

Martin Researcher 2 2

Mona Researcher 2 2

Gillian Researcher 1 1

Page 115 of 128

Table 16: Number of views by topic page and average length of time spent on page.

Section Page List

Order on

Website

Order

by

Views

Page

Views

Unique

Page

Views

Avg.

Time on

Page

Getting Started Path to Involvement 1 1 156 79 02:45

Getting Started Developing Partnerships 2 2 109 56 01:24

Getting Started Motivations 3 3 77 35 01:46

Debates and

Challenges

Balancing Time and

Commitments

13 4 73 32 00:28

Being Involved Role Determination 5 5 71 46 02:29

Getting Started Defining Partnerships 4 6 61 35 01:22

Being Involved Valuing Contributions 8 7 59 34 02:16

Being Involved Skills for Partnerships 7 8 51 37 01:32

Advice to Others Advice to Others 15 9 47 34 01:18

Thoughts About the

Future

Supports Needed 16 10 43 30 01:38

Being Involved Relationship Building 6 11 37 27 01:20

Impact/Making a

Difference

Impact on Research -

Coming Soon

9 12 28 24 00:07

Debates and

Challenges

Diversity 14 13 25 18 02:04

Thoughts about the

Future

Looking Forward 17 14 23 17 01:04

Impact/Making a

Difference

Improving Patient Care and

Experience - Coming

Soon

10 15 18 12 00:05

Impact/Making a

Difference

Learning from Others and

Making Connections -

Coming Soon

12 16 12 9 00:07

Impact/Making a

Difference

Measuring Impact of

Partnership - Coming

Soon

11 17 3 2 00:05

Page 116 of 128

Table 17: Average self-reported time on site, low and high estimates.

Survey Response

Low

Estimate

(minutes)

High

Estimate

(minutes)

Participants

(#)

# Participants x

Low Estimate

(minutes)

# Participants x

High Estimate

(minutes)

Less than 30 minutes 0 30 6 0 180

30 minutes to 1 hour 30 60 19 570 1,140

1 hour to 3 hours 60 180 12 720 2,160

3+ hours 180 240 4 720 960

Total minutes ÷ 41 participants = 49 108

Page 117 of 128

Appendix J Adapted eHealth Impact Questionnaire Data

Table 18: Adapted eHIQ Part 1 data, aggregate, patients, and researchers, attitudes toward

online info about PRPs.

n Likert Scale


The internet is a reliable module to help me understand patient-researcher partnerships.

All 49 0% 10% 31% 43% 16%

Patients 25 0% 8% 28% 48% 16%

Researchers 19 0% 16% 37% 32% 16%

The internet can help the public to know what it is like to be part of a patient-researcher

partnership.

All 49 0% 4% 22% 55% 18%

Patients 25 0% 8% 12% 60% 20%

Researchers 19 0% 0% 37% 42% 21%

The internet can be useful to help people decide if they are equipped to be part of a patient-


All 49 0% 4% 27% 47% 22%

Patients 25 0% 8% 16% 48% 28%

Researchers 19 0% 0% 42% 37% 21%

I would use the internet if I needed to help make a decision about whether to be part of a


All 49 0% 8% 31% 39% 22%

Patients 25 0% 8% 32% 32% 28%

Researchers 19 0% 5% 32% 42% 21%

Page 118 of 128

Table 19: Adapted eHIQ Part 1, aggregate, patients, and researchers, attitudes toward sharing

PRP experiences online.

n Likert Scale


The internet is a good way of finding other people who have similar patient-researcher


All 49 0% 20% 20% 33% 27%

Patients 25 0% 28% 20% 20% 32%

Researchers 19 0% 16% 21% 42% 21%

It would be helpful to see other people's partnership-related experiences on the internet.

All 49 0% 2% 14% 39% 45%

Patients 25 0% 4% 16% 32% 48%

Researchers 19 0% 0% 11% 47% 42%

The internet is a good way of finding other people who are facing patient-researcher

partnership-related decisions I may also face.

All 49 0% 14% 37% 24% 24%

Patients 25 0% 20% 32% 20% 28%

Researchers 19 0% 11% 42% 21% 26%

Looking at websites reassures me that I am not alone in my concerns about my patient-


All 49 2% 10% 39% 31% 18%

Patients 25 0% 12% 40% 20% 28%

Researchers 19 5% 11% 37% 37% 11%

Page 119 of 128


identification, part 1.

n Likert Scale


The website prepares me for what might happen in a patient-researcher partnership.

All 41 2% 7% 22% 51% 17%

Patients 23 4% 13% 13% 48% 22%

Researchers 15 0% 0% 40% 53% 7%

The people who have contributed to the website understand what is important to me.

All 39 0% 8% 23% 46% 23%

Patients 21 0% 10% 24% 43% 24%

Researchers 15 0% 7% 27% 40% 27%

I feel I have a sense of solidarity with people on the website.

All 41 2% 5% 39% 39% 15%

Patients 23 0% 0% 52% 30% 17%

Researchers 15 7% 13% 20% 53% 7%

I can identify with people on the website.

All 41 0% 2% 34% 41% 22%

Patients 23 0% 0% 48% 22% 30%

Researchers 15 0% 7% 20% 67% 7%

I value the advice given on the website.

All 41 2% 2% 17% 49% 29%

Patients 23 4% 4% 22% 43% 26%

Researchers 15 0% 0% 13% 53% 33%

Page 120 of 128


identification, part 2.

n Likert Scale


The website gives me the confidence that I am able to participate in a patient-researcher

partnership.

All 41 0% 5% 37% 37% 22%

Patients 23 0% 9% 30% 39% 22%

Researchers 15 0% 0% 47% 33% 20%

I feel like I have a lot in common with people on the website.

All 41 0% 10% 37% 39% 15%

Patients 23 0% 17% 35% 30% 17%

Researchers 15 0% 0% 40% 53% 7%

The website gives me the confidence to explain my involvement in research to others.

All 41 0% 22% 20% 44% 15%

Patients 23 0% 26% 17% 39% 17%

Researchers 15 0% 20% 27% 47% 7%

The website makes me more confident to discuss my partnership experience with the people

around me (for example, my family or people at work).

All 40 1% 18% 28% 38% 18%

Patients 23 0% 22% 30% 30% 17%

Researchers 14 0% 14% 29% 43% 14%

Page 121 of 128


presentation, part 1.

n Likert Scale


The information on the website left me feeling confused. *

All 40 0% 20% 13% 43% 25%

Patients 23 0% 35% 4% 30% 30%

Researchers 15 0% 0% 27% 53% 20%

The website provides a wide range of information.

All 39 3% 10% 18% 36% 33%

Patients 22 5% 14% 14% 23% 45%

Researchers 15 0% 7% 27% 60% 7%

The language on the website made it easy to understand.

All 41 2% 2% 20% 39% 37%

Patients 23 4% 4% 26% 35% 30%

Researchers 15 0% 0% 13% 40% 47%

I can easily understand the information on the website.

All 41 0% 2% 17% 49% 32%

Patients 23 0% 4% 17% 43% 35%

Researchers 15 0% 0% 13% 53% 33%

*item has been reverse scored

Page 122 of 128


presentation, part 2.

n Likert Scale


I trust the information on the website.

All 41 0% 0% 22% 39% 39%

Patients 23 0% 0% 30% 26% 43%

Researchers 15 0% 0% 13% 53% 33%

Videos were used appropriately on the website.

All 41 0% 20% 20% 41% 20%

Patients 23 0% 17% 26% 39% 17%

Researchers 15 0% 20% 13% 47% 20%

I found the stories or other content on the website distressing or worrisome. *

All 41 0% 0% 5% 32% 63%

Patients 23 0% 0% 9% 35% 57%

Researchers 15 0% 0% 0% 33% 67%

The website is easy to use.

All 40 3% 5% 25% 43% 25%

Patients 23 4% 4% 26% 39% 26%

Researchers 14 0% 7% 21% 43% 29%

*item is reverse scored

Page 123 of 128


motivation, part 1.

n Likert Scale


The website encourages me to take actions to become a patient-researcher partner.

All 41 0% 17% 27% 39% 17%

Patients 23 0% 13% 26% 43% 17%

Researchers 15 0% 27% 27% 33% 13%

The website has a positive outlook.

All 41 2% 5% 12% 41% 39%

Patients 23 0% 9% 17% 30% 43%

Researchers 15 7% 0% 7% 53% 33%

The website includes useful tips on how to be a better patient-researcher partner.

All 41 2% 7% 22% 46% 22%

Patients 23 4% 9% 26% 39% 22%

Researchers 15 0% 7% 20% 67% 7%

I feel more inclined to participate in a patient-researcher partnership after visiting the website.

All 40 0% 20% 23% 43% 15%

Patients 22 0% 27% 14% 45% 14%

Researchers 15 0% 13% 40% 33% 13%

I have learnt something new from the website.

All 41 2% 10% 29% 34% 24%

Patients 23 4% 9% 26% 35% 26%

Researchers 15 0% 13% 33% 33% 20%

Page 124 of 128


motivation, part 2.

n Likert Scale


I would consult the website if I had to make a decision about whether or not to be part of a


All 40 8% 15% 30% 28% 20%

Patients 23 4% 22% 26% 30% 17%

Researchers 15 13% 7% 40% 20% 20%

On the whole, I find the website reassuring.

All 41 0% 20% 24% 44% 12%

Patients 23 0% 30% 22% 39% 9%

Researchers 15 0% 7% 33% 47% 13%

The website helps me to have a better understanding of patient-researcher partnerships.

All 40 0% 15% 13% 40% 33%

Patients 23 0% 22% 9% 35% 35%

Researchers 14 0% 7% 21% 43% 29%

The website encourages me to play a more active role as a patient-researcher partner.

All 40 0% 13% 25% 43% 20%

Patients 23 0% 17% 26% 35% 22%

Researchers 14 0% 7% 21% 57% 14%

Page 125 of 128

Appendix K Theory of Planned Behaviour Data

Table 26: Pre-post TPB data, patients and researchers, intentions.

n Likert Scale

Strongly Disagree 1 2 3 4 5 6 7 Strongly agree

I want to engage in patient engagement in research.

Patients 23 pre* 0% 0% 0% 0% 7% 32% 61%

23 post 0% 0% 2% 5% 20% 32% 41%

Researchers 15 pre* 0% 0% 0% 0% 7% 20% 73%

15 post 0% 0% 7% 0% 20% 27% 47%

I intend to engage in patient engagement in research.

Patients 22 pre* 0% 0% 0% 3% 13% 30% 55%

23 post 0% 2% 0% 7% 24% 24% 41%

Researchers 15 pre* 0% 0% 0% 0% 13% 20% 67%

15 post 0% 7% 0% 0% 20% 27% 47%

*only includes responses from participants who completed both Survey 1 and 2

Page 126 of 128

Table 27: Pre-post TPB data, patients and researchers, attitudes.

n

Likert Scale


Patient-researcher partnerships are, or would be, beneficial. **

Patients 23 pre* 0% 0% 0% 4% 9% 30% 57%

22 post 0% 0% 0% 5% 0% 32% 64%

Researchers 15 pre* 0% 0% 0% 0% 13% 33% 53%

14 post 0% 0% 0% 7% 7% 50% 36%

Patient-researcher partnerships are, or would be, very pleasant (for me). **

Patients 23 pre* 0% 0% 0% 9% 13% 52% 26%

23 post 0% 0% 0% 13% 13% 57% 17%

Researchers 15 pre* 0% 0% 0% 0% 13% 47% 40%

15 post 0% 0% 0% 20% 13% 53% 13%

Patient-researcher partnerships are, or would be, very helpful. **

Patients 23 pre* 0% 0% 0% 9% 4% 43% 43%

23 post 0% 0% 0% 13% 4% 43% 39%

Researchers 15 pre* 0% 0% 0% 0% 0% 40% 60%

15 post 0% 0% 0% 7% 13% 40% 40%


**Question phrasing adapted for table. See Table 12 for actual phrasing.

Page 127 of 128

Table 28: Pre-post TPB data, patients and researchers, subjective norms.

n Likert Scale


It is expected of me that I engage in patient-researcher partnerships.

Patients 23 pre* 35% 30% 4% 13% 0% 4% 13%

23 post 48% 26% 0% 17% 4% 0% 4%

Researchers 15 pre* 0% 0% 7% 27% 13% 20% 33%

15 post 0% 0% 7% 13% 13% 33% 33%

I feel under social pressure to engage in patient-researcher partnerships.

Patients 23 pre* 65% 13% 9% 4% 4% 4% 0%

23 post 43% 35% 9% 4% 4% 0% 4%

Researchers 15 pre* 7% 7% 7% 47% 13% 7% 13%

15 post 0% 7% 20% 20% 33% 13% 7%

People who are important to me want me to engage in patient-researcher partnerships.

Patients 23 pre* 35% 22% 13% 22% 4% 0% 4%

23 post 35% 22% 13% 13% 13% 4% 0%

Researchers 15 pre* 0% 7% 13% 20% 13% 20% 27%

15 post 0% 0% 13% 7% 20% 20% 40%


Page 128 of 128

Table 29: Pre-post TPB data, patients and researchers, perceived behavioural control.

n Likert Scale


I am confident that I could engage in patient-researcher partnerships.

Patients 22 pre* 0% 0% 0% 0% 14% 45% 41%

23 post 0% 0% 0% 0% 13% 35% 52%

Researchers 15 pre* 0% 0% 0% 7% 20% 40% 33%

15 post 0% 0% 0% 13% 13% 27% 47%

For me to engage in patient-researcher partnerships is, or would be, very easy.

Patients 23 pre* 0% 0% 9% 13% 30% 26% 22%

23 post 0% 0% 4% 9% 30% 39% 17%

Researchers 15 pre* 0% 0% 7% 7% 73% 13% 0%

15 post 0% 0% 13% 20% 47% 20% 0%

The decision to engage in patient-researcher partnerships is beyond my control. **

Patients 23 pre* 9% 0% 0% 9% 13% 13% 57%

23 post 4% 4% 0% 9% 0% 22% 61%

Researchers 15 pre* 0% 7% 7% 7% 27% 40% 13%

15 post 7% 13% 0% 13% 27% 20% 20%

Whether or not I engage in patient-researcher partnerships is entirely up to me.

Patients 23 pre* 9% 0% 4% 0% 17% 9% 61%

23 post 13% 0% 4% 4% 4% 26% 48%

Researchers 15 pre* 7% 20% 20% 13% 7% 13% 20%

15 post 7% 33% 7% 20% 7% 20% 7%


**item is reverse scored