Supporting cancer survivors’ participation in peer review:perspectives from NCI’s CARRA program

Melissa B. Gilkey

Received: 22 July 2013 /Accepted: 4 October 2013 /Published online: 9 November 2013# Springer Science+Business Media New York 2013

AbstractPurpose Including cancer survivors in the peer review ofcancer-related research is increasingly valued as a strategyfor bringing the “patient perspective” to discussions of re-search merit and human subject protection. Because integrat-ing lay stakeholders into peer review poses challenges, thisqualitative study explored the perspectives of experiencedpatient advocates to identify programmatic supports for sur-vivors’ participation.Methods Semi-structured telephone interviews were conduct-ed with a purposive sample of 19 cancer survivors and 6administrators involved in the National Cancer Institute’sConsumer Advocates in Research and Related Activities pro-gram. Audio-recorded interviews were transcribed verbatimand analyzed via thematic content analysis. Participants werehighly educated and included survivors of breast, prostate, andblood-related cancers.Results Interviewees emphasized the importance of adequate-ly preparing survivors to serve as advocates. Given the intel-lectual challenge of peer review, interviewees noted the needfor intensive and ongoing training on how to review pro-posals, and they identified mock reviews and peer mentoringas effective strategies to complement didactic instruction.Participants also stressed the need to address social challengesinherent in advocate–scientist encounters. In addition to train-ing for both advocates and scientists, participants reported thatopportunities for informal social interaction were importantfor facilitating collaboration. Finally, participants recommend-ed structuring advocates’ role so as to give them a voice viaequal voting privileges and protected opportunities to speak.

Conclusions Programs that seek to include cancer survivorsin peer review can prepare advocates for the intellectual andsocial challenges of working with scientists through carefulattention to training, networking, and programmatic design.Implications for Cancer Survivors Cancer survivors havebeen leaders in developing a role for patient advocates in thepeer review of research. As the concept of patient-centeredoutcomes continues to gain currency, lessons learned fromearly programs for patient inclusion in peer review can help toinform future efforts aimed at giving patients a voice inshaping agendas for health-related research.

Keywords Cancer . Survivor . Consumer participation .

Patient advocacy . Peer review . Research

Introduction

Patient-centered outcomes research, which seeks to evaluatehealth care from the perspectives of patients themselves, isguided by the principle that patients should have a voice inall aspects of research, including the allocation of researchfunding [1]. The recent establishment of the Patient-CenteredOutcomes Research Institute (PCORI) has spurred renewedinterest in understanding how patients, as nonscientists, can beincorporated most effectively into scientific processes such aspeer review [2, 3]. Although many scientists would agree intheory that the patient perspective is relevant to the appraisal ofresearch merit and human subject protection, the actual prac-tice of giving patient advocates a meaningful role in peerreview is far from straightforward given the highly technicalnature of those discussions [3]. As patient inclusion in peerreview becomes more common, fledgling programs may ben-efit from lessons learned by early adopters of this practice.

Cancer-related research is one area in which the inclusionof patient advocates in peer review has a fairly long tradition,

M. B. Gilkey (*)University of North Carolina Lineberger Comprehensive CancerCenter, CB 7440, Rosenau Hall, 3rd Floor, Chapel Hill,NC 27599-7440, USAe-mail: gilkey@email.unc.edu

J Cancer Surviv (2014) 8:114–120DOI 10.1007/s11764-013-0318-2

due in large part to the success of the breast cancer advocacymovement. In 1991, the National Breast Cancer Coalition leda grassroots campaign that resulted in a congressional alloca-tion of more than $200 million for the establishment of theBreast Cancer Research Program (BCRP) within the U.S.Department of Defense [4]. In addition to securing researchfunding, the coalition instituted a policy for the inclusion ofbreast cancer survivors in the peer review of research pro-posals funded by BCRP [4]. In this capacity, survivors weremeant to bring the perspective of breast cancer patients to thereview process in order to ensure that funded studies werefeasible, ethically sound, and in keeping with the priorities ofpatient communities [4–6]. Although initially controversial,the program was well received by scientists and advocates,and to date, breast cancer survivors have served as votingmembers on all peer review panels [5, 6].

Based on the success of BCRP, the practice of patientinclusion in peer review has spread in various forms to othercancer research funding organizations. Nonprofit organiza-tions, including Susan G. Komen for the Cure, have incorpo-rated cancer survivors in peer review [7]. Perhaps most im-portantly, the practice has also been adopted by the nation’slargest funder of cancer-related research, the National CancerInstitute (NCI). Since 2001, NCI has recruited and trainedcancer survivors through its Consumer Advocates in Researchand Related Activities (CARRA) Program, an initiative whichserves as the programmatic context for this study [8]. Al-though patient involvement in peer review remains unusualwith regard to some diseases, in the area of cancer-relatedresearch, the practice is relatively well established.

This qualitative study investigated the practice of patientinclusion in peer review from the perspective of cancer survi-vors participating in NCI’s CARRA program. More specifi-cally, this study aimed to explore programmatic supports thatsurvivors identified as facilitating their work on peer reviewpanels. By investigating the practice of patient inclusion in thecontext of a well-established program, this research aims toinform future efforts to expand the voice that survivors ofcancer, and many other diseases, have in guiding health-related research.

Methods

Programmatic context

Established in 2001, CARRA maintains a pool of over 100advocates drawn from across the USA. To select advocates,CARRA staff members first solicit nominations from canceradvocacy organizations, and nominated individuals are theninvited to submit an application. Staff members select advo-cates from among these applications with an eye towardrepresenting different advocacy communities in terms of

cancer type, racial/ethnic background, and geographic loca-tion. Unlike programs such as BCRP, CARRA does notrequire advocates to be cancer survivors in the sense of havinghad a personal cancer diagnosis. Instead, eligibility is extend-ed also to those who have experience as a caretaker of a personwith cancer.

Before joining peer review, CARRA advocates participatein a multiday workshop. Advocates are instructed in the“anatomy” of a grant proposal, and they are trained in howto critique and score proposals. Other topics include humansubject protection, the inclusion of underrepresented popula-tions, and basic research design concepts such as randomiza-tion. The role of the advocate is set forth as the provision of“the patient/public perspective in the assessment of scientificexcellence.” Advocates are discouraged from presenting a“personal or political agenda,” but rather are asked to “repre-sent the collective views of people affected by and at risk forcancer.” The workshop, which was developed with input frompracticing advocates, culminates in a mock review.

CARRA advocates are invited to participate in the reviewof clinical and translational research, including research pro-ject (R01) grants, small project (R03) grants, research pro-gram project (P01) grants, and cancer center (P20/30/50)grants. The process of matching trained advocates to peerreview panels begins with the scientific review officer man-aging the panel in question. The officer requests advocateparticipation and may specify a preference for advocates ofcertain backgrounds (e.g., experience with a certain type ofcancer). CARRA staff members then select and contact advo-cates to invite their participation.

Once matched to a review panel, advocates receive grantproposals to review off-site. Although the procedure varies,advocates typically submit a written critique of each proposaland then travel to face-to-face meetings of the entire reviewpanel. Advocates are invited to comment on each proposalafter the primary and secondary scientific reviewers havemade their statements. Advocates participate in peer reviewpanels as voting members, and their scores carry the sameweight as those of scientist members.

Participants

The primary study sample consisted of CARRA advocateswho had participated in peer review in the previous 4 years.Eligible advocates included cancer survivors who had experi-enced a personal cancer diagnosis as well as those who hadbeen the caregiver of a family member or friend with cancer.Using the program’s volunteer registry, a CARRA staff mem-ber recruited advocates using a recruitment script sent via e-mail. With the goal of achieving maximum variation, thisstudy employed a purposive sampling strategy so as to includeparticipants of four different cancer advocacy interests: breast,prostate, blood-related, and other cancers. Of 43 advocates

J Cancer Surviv (2014) 8:114–120 115

who were initially contacted, 29 indicated interest in the study,21 received invitations to participate, and 19 ultimately com-pleted interviews. The final sample size was determined bythematic saturation [9], which occurred in this study when thefinal four interviews yielded no information that substantivelychanged the analysis.

To contextualize advocates’ perspectives, a secondarystudy sample consisted of CARRA program administrators.Administrators were interviewed to provide additional infor-mation about program history, policies, and procedures aswell as an organizational viewpoint on facilitators and barriersto advocates’ participation in peer review. These participantsincluded current and former CARRA staff members as well asconsultants who had contributed to the development of theprogram. Using a snowball sampling strategy, 11 programadministrators were identified, 6 received invitations to par-ticipate, and all completed interviews.

Data collection

In 2010–2011, participants were interviewed by phone using asemi-structured interview guide. Open-ended questions ex-plored how participants became involved in CARRA, theirexperience of cancer and cancer-related advocacy, and theirparticipation in peer review. Of greatest relevance to thepresent study was a section of the interview guide that askedparticipants to reflect on CARRA training as well as otherprogrammatic factors that facilitated or hindered their partic-ipation in peer review. Participants also provided demographicand other background information including educational at-tainment, race/ethnicity, and survivorship status (i.e., personalcancer diagnosis versus caretaker of a person with cancer).

Prior to beginning the interview, participants reviewed astudy fact sheet and orally indicated informed consent. Inter-views ranged from 20 to 90 min, with the average sessionlasting approximately 45min. Interviews were audio-recordedwith permission and transcribed “clean verbatim” (i.e., with-out nonverbal utterances such as “um”). The InstitutionalReview Board of the Johns Hopkins Bloomberg School ofPublic Health approved the study protocol.

Data analysis

Data were analyzed in iterative cycles of content analysis inthe manner described by Patton [9]. In the first, inductive phaseof analysis, initial transcripts were read multiple times in theirentirety and analyzed through a process of open coding aimedat identifying broad topics of discussion. As codes were iden-tified, subsequent interviews were structured so as to exploreareas of interest in greater depth. Over time, codes were refinedas they were applied to additional transcripts and throughconsultation with other investigators. Finalized codes wereorganized into a codebook and then applied to the data

systematically using ATLAS.ti Version 6.0 (ATLAS.ti Scientif-ic Software Development GmbH, Berlin, Germany).

In the second, deductive phase of analysis, data were con-sidered code by code to identify areas of convergence anddivergence [10]. Data were compared within codes by partici-pants’ area of cancer interest as well as by role (i.e., advocateversus administrator). Emerging patterns were described the-matically, and quotations were selected to be representative ofthematic descriptions. Transcripts were then reread to test iden-tified themes and to check for dissenting views, or exceptions tooverall themes, that might require additional analysis or discus-sion. Unless otherwise stated, quotations presented in the find-ings are from CARRA advocates.

Results

Sample characteristics

The primary sample of CARRA advocates consisted of 10 menand 9 women, all of whom had experience in peer review(Table 1). Participants were highly educated with all but oneholding a graduate degree. In terms of survivorship status, 11participants had personally received a cancer diagnosis, whilethe remaining 8 had served as a primary caretaker of a familymember or friend with cancer. In terms of racial/ethnic back-ground, 12 participants described themselves as white or Cau-casian, 4 were Black or African-American, and 3 indicated

Table 1 Sample characteristics

n (%)

Sex

Male 10 (53)

Female 9 (47)

Educational attainment

High school degree 1 (5)

College degree 0 (0)

Graduate degree 18 (95)

Survivorship status

Personal cancer diagnosis 11 (58)

Caregiver 8 (42)

Race/ethnicitya

White/Caucasian 12 (63)

Black/African-American 4 (21)

Other 2 (11)

Primary cancer advocacy interest

Breast 5 (26)

Prostate 4 (21)

Blood-related 4 (21)

Other 6 (32)

a One participant chose not to specify race/ethnicity

116 J Cancer Surviv (2014) 8:114–120

another background or chose not to answer. When asked tospecify their primary advocacy interest by cancer type, partic-ipants most often reported breast (n =5), prostate (n =4), andblood-related (n =4) cancers.

The secondary sample of CARRA administrators consistedof the program’s current and former staff members (n =3) aswell as consultants or others involved in developing theCARRA program (n =3). Demographic characteristics of thesecondary sample are not reported to protect participants’identities.

Training and mentorship

Theme 1: Comprehensive training to address both information-al and social aspects of peer review is crucial for advocates’success. All interviewees emphasized the importance of train-ing advocates to meet the challenges of participating in peerreview. They noted the importance of introducing advocates tothe structure of grant proposals, the basic principles of re-search design, and the procedures of peer review. Severaladvocates also mentioned the value of participating in a mockreview session, which gave them an opportunity to apply whatthey had learned.

For some, training on the social aspects of peer review waseven more important than the procedural and informationalorientation, and interviewees identified two competing chal-lenges in this regard. First, interviewees noted that, given thehighly technical language of peer review and the scientificexpertise of other reviewers, advocates could easily becomeintimidated. For this reason, one goal of training should be tofoster confidence:

[The CARRA training includes] a whole session on howto participate in peer review, and the purpose of that is toget people accustomed to speaking up. Because our[societal] training has always been “the scientist knowsbest.”Well, the scientist may know best about 99 percent[of the proposal], but our 1 percent, we really know best.

At the same time, some advocates and all administratorsstressed the importance of teaching advocates how to presenttheir critiques in a “fashion that’s acceptable to the panel.” Inthis regard, the goal was a professional tone achieved bypresenting critiques in a brief, nonconfrontational, and emo-tionally neutral way and by avoiding the discussion of expe-riences or viewpoints deemed to be personal. Thus, one per-ceived purpose of training was to foster within advocates acareful balance between assertiveness and restraint that wouldallow them to contribute to peer review without alienatingscientist panel members.

Theme 2. Peer mentoring provides an opportunity to extendadvocates’ formal training. Advocates who had participatedin CARRA training uniformly praised the program and had

relatively few suggestions for how the sessions might beimproved. Two advocates who had participated in other peerreview programs in addition to CARRA, however, suggestedtrainees could benefit from peer mentoring. Specifically,based on their experiences in other programs, these inter-viewees found that opportunities to shadow an experiencedadvocate provided a useful transition into the program.

Theme 3. Researchers also need training so as to betterunderstand advocates’ role. Several interviewees noted thatresearchers also needed orientation:

You need to train the researchers how to treat the patientadvocates and what to expect and what they’ll bring tothe table.

Although CARRA provided online resources for re-searchers interested in learning more about working withadvocates, CARRA administrators related that researchers’training often happened informally. For example, scientificreview officers were perceived as valuable intermediarieswho could facilitate advocates’ involvement by introducingthe chair and other panel members to their role.

Programmatic structure

Theme 1. Advocates need ongoing support and a clearlydefined role to effectively manage the burden of participatingin peer review. In addition to training, interviewees noted theimportance of ongoing supports for advocates as they facedthe daunting task of reviewing lengthy grant proposals. Forexample, advocates appreciated that they could contactCARRA staff members with questions:

They’re always available by phone when it comes to the“fun” part of wading through the mounds and moundsand mounds of paper.

In addition to this ongoing technical assistance, advocatesfound CARRA’s periodic conference calls to be a good way toconnect with other advocates and to stay abreast of currentissues.

Other supports included tools for conducting the reviews.For example, some advocates found it helpful to have an“assignment,” or a list of panel-specific prompts that focusedtheir attention on certain aspects of the proposal, such as theprotection of human subjects or the recruitment of underrep-resented populations. For these interviewees, such supportsmade the review more manageable by reducing the amount ofmaterial to be reviewed.

Interviews with other advocates, however, suggested thatdefining advocates’ role was far from straightforward. Whilesome advocates endorsed a fairly narrow role for themselvesas human subject reviewers, others hoped to speak more

J Cancer Surviv (2014) 8:114–120 117

broadly about how meaningful the proposed research wouldbe to improving patients’ quality of life. These advocatestended to feel an obligation to review the grant moreholistically:

I like to read the entire proposal. They keep telling methat it’s not necessary, that I should just read the confi-dentiality statement and the recruitment statement andthat’s all I need. I don’t believe that’s true. I need to seethis thing for me.

In this way, interviews with advocates reflected some de-gree of disagreement about the nature of their role and, in turn,their interpretation of supports to guide that role. Guidelinesthat were helpful to some seemed to others to risk co-opting or“pigeonholing” advocates’ voice.

Theme 2. Advocates’ participation should be structured sothat their voice is equal to that of researchers. Advocatesnoted several aspects of CARRA’s organization that facilitatedtheir role. Of particular importance was their status as votingpanel members:

Your vote [as an advocate] is equal to their vote [as aresearcher]. You are just as important to the peer reviewpanel as anMD or a PhD is. That helped me gain quite abit of confidence in terms of participating in thediscussion.

In addition to voting privileges, advocates also appreciatedthat they were always assigned to be one of several primaryreviewers and, therefore, had a formalized opportunity towrite and present critiques to the panel. Thus, several aspectsof the program’s structure were important in helping advo-cates gain entrée into panel discussions.

Theme 3. Including multiple advocates on each review panelenhances their ability to participate effectively. CARRA ad-vocates typically served on peer review panels in pairs, butsome advocates had also served alone. A few noted that thisexperience was less than ideal:

I really think there should never be less than two [advo-cates on a panel] because you need the support. In [an-other peer review advocacy program], there will be sev-eral advocates mixed in with the scientists and that helps.

In addition to easing the perceived power differential be-tween advocates and scientists, several interviewees noted thathaving multiple advocates on a panel was useful for introduc-ing a multiplicity of advocacy perspectives.

Theme 4. Matching advocates to opportunities ischallenging. When reflecting on whom they wished to repre-sent in peer review, advocates expressed the view that theyhoped to speak for all cancer survivors. Nevertheless, some

felt that they could make special contributions with regard tothe types of cancer and populations they knew best. Severaladvocates and all of the administrators noted that matchingadvocates’ expertise, interests, and peer review experience tothe needs of the panel was difficult. One administratorexplained:

It’s really hard to turn somebody into fields in a databasethat can be searched and used to match [advocates]appropriately [to peer review panels]…. Being anadvocate—it’s sort of a challenge because the commu-nity is so diverse.… To assess someone’s skill set andthen determine whether they’re a good fit for [a panel]can be very challenging.

Program administrators noted that finding advocates withexperience with particularly stigmatized or deadly cancer,such as lung cancer, was especially hard. These intervieweesemphasized that ongoing recruitment was crucial formaintaining a diverse volunteer pool and for introducing“new blood” into the program.

Social interaction and networking

Theme 1. Opportunities to engage with researchers informallyhelp advocates establish relationships and legitimacy. Aspreviously noted, many advocates described the social envi-ronment of peer review as challenging. Researchers were notonly highly trained scientists but they also shared a language,research interests, and sometimes a history of collaborationthat bound them together. Some interviewees reported thatsocial events, such as group dinners or even cab rides, offereda valuable opportunity to connect with researchers outside ofthe rigid confines of the panel meeting:

With the [panels] that are two days or three days, gen-erally everybody goes out to dinner, and I think that’s sovaluable…. It allowed me to relax with people. [Re-searchers] have a lot to talk about with each other, butwe don’t. But that loosens up, and some of them areextremely interested in the advocacy work we do.

For advocates, such interactions were important not onlyfor “becoming comfortable with each other” but also forhelping researchers to understand advocates’ backgroundsand, in turn, how they could contribute to the research process.Several advocates observed that panel meetings held via con-ference call were difficult precisely because the disembodiednature of the conversation reduced the opportunity for person-al connection and exchange.

Theme 2. Opportunities for advocates to engage with oneanother strengthen their practice by broadening theirperspectives. Several advocates also noted the value of

118 J Cancer Surviv (2014) 8:114–120

interacting with one another. Because they aimed to representthe patient perspective most broadly, these advocates foundinteractions with other CARRA members important forgaining exposure to issues pertinent to other cancer advocacycommunities:

By serving on these different panels, I come in contactwith persons who have different specialties as it relatesto cancer. Therefore, I’m able to learn a lot more aboutthe needs of cancer survivors, caregivers, and otherpersons with different forms of cancer other than theone that I may be focusing my attention on.

In addition to serving on peer review panels together,advocates reported connecting with each other throughCARRA conference calls and by arranging “meet-ups” atprofessional conferences. Table 2 summarizes programmaticrecommendations drawn from interviews.

Discussion

The findings of this qualitative study suggest that, giventhe challenges cancer survivors face as nonscientist partic-ipants in peer review, successful programs for patient in-clusion require careful attention to training, programmaticstructure, and networking. While interviewees emphasizedadvocates’ informational needs, they were also adamant

about preparing advocates for the social environment ofpeer review through strategies such as peer mentoring andthe provision of opportunities for social interaction. Thesefindings support prior research that suggests that patientinclusion, while valuable, is also difficult; lay participantsneed supports as they learn to navigate the unfamiliarprofessional culture of peer review [4, 11–13].

CARRA advocates described their experience with theprogram’s multiday training in highly positive terms. Thisfinding is encouraging and suggests that CARRA trainingmaterials may serve as a useful guide for new programs forpatient inclusion. In addition to their initial orientation,interviewees noted the importance of providing ongoingtraining opportunities via conference calls or meetings.Overall, findings indicate that, to be successful, programplanners should expect to devote significant time and re-sources to the goal of training.

This study suggests that one challenge to planning pro-grams for patient inclusion may be the difficulty in uni-formly defining advocates’ role in peer review. Althoughsome CARRA advocates appreciated programmatic effortsto structure their critiques of research proposals, othersfound these tools to be overly restrictive because theyenvisioned their role as extending beyond any single sec-tion of the proposal. Given different expectations for par-ticipation, advocates’ role in peer review is likely a topicthat merits ongoing discussion among program adminis-trators and advocates themselves. Although advocatesmust tailor their role to the needs of the funding organiza-tion as well as their own constituencies, recently devel-oped tools to guide patient inclusion in peer review mayprove useful by suggesting a more holistic approach tocritiquing proposals [14].

The findings of this research should be interpreted inlight of several limitations. First, this study privileged theperspectives of CARRA advocates so as to understand thepractice of survivor participation in peer review from thestandpoint of those who practice it. Although CARRAprogram administrators were interviewed to contextualizeadvocates’ comments, topics that were discussed only byadministrators (e.g., program evaluation), but not by advo-cates, were not included in this report. Second, the studysample was limited to a relatively small number of partic-ipants from a single advocacy program. This approachfacilitated an in-depth exploration of a prominent andwell-established program that draws from a number ofcancer advocacy communities, but findings may be lesstransferrable to patient inclusion efforts in disease areasother than cancer. Finally, this study was limited to activeCARRA advocates; additional insights could no doubt begained from former advocates, particularly if a lack ofperceived programmatic support motivated their departurefrom the program.

Table 2 Summary of programmatic recommendations

Training

• Hold mock reviews during training to allow advocates to applydidactic instruction

• Arrange for newly trained advocates to shadow a more experiencedpeer mentor

• Host periodic conference calls to provide training updates andopportunities for networking

• Engage the panel chair or other administrators in orienting scientistpanel members to advocates’ role

Programmatic structure

• Define advocates’ role as clearly as possible

• Provide advocates with guidelines for reviewing proposals whileavoiding narrow or prescriptive rules

• Give advocates equal voting rights and protected opportunities tospeak during panel meetings

• Assign multiple advocates to each panel

Networking and social support

• Organize meals or other opportunities for advocates to interactinformally with scientists

• Prioritize face-to-face panel meetings over conference calls

• Hold “meet-ups” at conferences or other venues to encourageadvocates to interact with each other

J Cancer Surviv (2014) 8:114–120 119

In conclusion, cancer survivors have helped lead the way indeveloping a role for patient advocates in peer review. Asinnovative programs have matured and become institutional-ized, the time is ripe to assess the benefits and challenges thatattend this work. This study aimed to describe programmaticsupports for patient inclusion in peer review, including thoserelated to training, program structure, and networking. Theseand other “lessons learned” may help new programs devel-oped by PCORI and other funding organizations as they workto expand the opportunities patients have to shape agendas forhealth-related research.

Acknowledgments This research was supported by the Cancer ControlEducation Program at UNC Lineberger Comprehensive Cancer Center(R25 CA57726).

Conflict of interest The authors have no conflicts of interest to report.

References

1. Patient-Centered Outcomes Research Institute. Patient-centered out-comes research. 2013. http://www.pcori.org/research-we-support/pcor/. Accessed 10 Jul 2013.

2. Methodology Committee of the Patient-Centered OutcomesResearch Institute. Methodological standards and patient-centeredness in comparative effectiveness research: the PCORI per-spective. JAMA. 2012;307:1636–40.

3. Fleurence R, Selby JV, Odom-Walker K, et al. How the Patient-Centered Outcomes Institute is engaging patients and others in shap-ing its research agenda. Health Aff. 2013;32:393–400.

4. Rich IM, Andejeski Y, Alciati MH, et al. Perspective from theDepartment of Defense Breast Cancer Research Program. BreastDis. 1998;10:33–45.

5. Andejeski Y, Bisceglio IT, Dickersin K, et al. Quantitative impactof including consumers in the scientific review of breast cancerresearch proposals. J Womens Health Gen Based Med. 2002;11:379–88.

6. Andejeski Y, Breslau ES, Hart E, et al. Benefits and drawbacksof including consumer reviewers in the scientific merit review ofbreast cancer research. J Womens Health Gen Based Med.2002;11:119–36.

7. Susan G. Komen for the cure. Advocates in science. 2013. http://ww5.komen.org/ResearchGrants/KomenAdvocatesinScience.html.Accessed 10 Jul 2013.

8. National Cancer Institute. Office of Advocacy Relations. 2013. http://carra.cancer.gov. Accessed 10 Jul 2013.

9. Patton MQ. Qualitative research and evaluation methods. ThousandOaks: Sage; 2002.

10. Guba EG. Toward a methodology of naturalistic inquiry in educationalevaluation. Los Angeles: Center for the Study of Evaluation; 1978.

11. Hubbard G, Kidd L, Donaghy E, et al. A review of literature aboutinvolving people affected by cancer in research, policy and planningand practice. Patient Educ Couns. 2007;65:21–33.

12. Saunders C, Girgis A, Butow P, et al. From inclusion to indepen-dence—training consumers to review research. Health Res PolicySyst. 2008;6:3.

13. Brody J, Brown P, et al. Public involvement in breast cancer research:an analysis and model for future research. Int J Health Serv. 2004;34:625–46.

14. Saunders C, Girgis A, Butow P, et al. Beyond scientific rigour: fundingcancer research of public value. Health Policy. 2007;84:234–42.

120 J Cancer Surviv (2014) 8:114–120