Evaluation and Program Planning 35 (2012) 518–522

Collaborative Evaluation within a framework of stakeholder-orientedevaluation approaches

Rita G. O’Sullivan *

Evaluation, Assessment, & Policy Connections, School of Education CB #3500, University of North Carolina, Chapel Hill, NC 27599, United States

A R T I C L E I N F O

Article history:

Available online 2 February 2012

Keywords:

Collaborative Evaluation

Participatory evaluation

Empowerment evaluation

Utilization-focused evaluation

A B S T R A C T

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation

planning and implementation. Unlike ‘‘distanced’’ evaluation approaches, which reject stakeholder

participation as evaluation team members, Collaborative Evaluation assumes that active, on-going

engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data

collection and analysis, and results that stakeholder understand and use. Among similar ‘‘participant-

oriented’’ evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation

distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different

program evaluations will experience different levels of collaborative activity. The sliding scale is applied

as the evaluator considers each program’s evaluation needs, readiness, and resources. While

Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often

used interchangeably with participatory and/or empowerment evaluation, the terms can be used to

mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation

Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself

from the other participatory evaluation approaches.

� 2012 Elsevier Ltd. All rights reserved.

Contents lists available at SciVerse ScienceDirect

Evaluation and Program Planning

journa l homepage: www.e lsev ier .com/ locate /eva lprogplan

1. Introduction

Collaborative Evaluation systematically invites and engagesstakeholders in program evaluation planning and implementation.Unlike ‘‘distanced’’ evaluation approaches, which reject stakehold-er participation as evaluation team members, CollaborativeEvaluation assumes that active, on-going engagement betweenevaluators and program staff, result in stronger evaluation designs,enhanced data collection and analysis, and results that stakeholderunderstand and use.

As with any of the many evaluation approaches (Fitzpatricket al., 2011), collaborative evaluation consistently follows typicalprogram evaluation processes, whereby a client/supervisor/fund-er/program staff member is interested in asking questions about aprogram, which will require the systematic collection of informa-tion to answer those questions. In conducting the evaluation, acompetent evaluator is expected to follow appropriate profession-al guidelines (see for example the Guiding Principles for Evaluators

of the American Evaluation Association, Evaluation Guidelines fromthe International Program Evaluation Network, or the UnitedNations’ Standards for Evaluation), which help to assure that theevaluation is of high quality. The collaborative aspect of the

* Tel.: +1 919 843 7878; fax: +1 919 843 6781.

E-mail address: ritao@unc.edu.

0149-7189/$ – see front matter � 2012 Elsevier Ltd. All rights reserved.

doi:10.1016/j.evalprogplan.2011.12.005

evaluation is found in how the evaluators goes about conductingthe evaluation, which includes a variety of efforts to engageprogram stakeholders in the evaluation.

Among similar ‘‘participant-oriented’’ evaluation approaches(Fitzpatrick et al., 2011), Collaborative Evaluation distinguishes itselfin that it uses a sliding scale for levels of collaboration. This meansthat different program evaluations will experience different levels ofcollaborative activity. The sliding scale is applied as the evaluatorconsiders each program’s evaluation needs, readiness, and resources.

Rodriguez-Campos, another Collaborative Evaluation propo-nent wrote, ‘‘Collaborative Evaluation is an evaluation in whichthere is a significant degree of collaboration between the evaluatorand stakeholders in the evaluation process’’ (2005, p. 1). Thus, aCollaborative Evaluation stance requires evaluators to enhanceevaluation activities by creating environments that invite andallow stakeholder involvement. Additionally collaborative evalua-tors need to understand and assess barriers to collaboration andcreate opportunities to overcome them. All this must be done,cognizant of the evaluation being conducted and sensitive to theorganizational context of the program.

2. Theoretical perspectives

While Collaborative Evaluation is a term widely used inevaluation, its meaning varies considerably. Often used inter-changeably with participatory and/or empowerment evaluation,

R.G. O’Sullivan / Evaluation and Program Planning 35 (2012) 518–522 519

the terms can be used to mean different things, which can beconfusing. The Topical Interest Group, representing evaluatorsfollowing this ‘‘participant-oriented’’ approach in the AmericanEvaluation Association, have entitled themselves, ‘‘Collaborative,Participatory, and Empowerment Evaluation.’’ Building on theexisting literature contributed by members of this TopicalInterest Group, Collaborative Evaluation is defined as anapproach that actively engages program stakeholders asmembers of the evaluation team to the extent that they areable and willing. The work of O’Sullivan (2004) and Rodriguez-Campos (2005) best represent how such evaluation are plannedand implemented.

From a broad, theoretical perspective, Collaborative Evaluationbelongs on the use branch of evaluation theory as described byAlkin (2004) in Evaluation Roots, sharing much in common withparticipatory evaluation (Cousins & Earl, 1995; King, 2007;Whitmore, 1998), utilization-focused evaluation (Alkin, 2004;Patton, 2008), and empowerment evaluation (Fetterman &Wandersman, 2005). Consistent among these approaches are astrong appreciation for stakeholder involvement in evaluation anda desire for the evaluation results to be useful.

3. Aspects of evaluation

Because there is much overlap among the approaches, an effortto distinguish similarities and differences requires dimensions forcomparison. For the purposes of this presentation these dimen-sions are referred to as ‘‘Aspects of Evaluation.’’ This sectionestablishes the foundation for the inclusion of 11 aspects by whichto compare the four approaches.

Essential aspects of evaluation include those componentssurrounding implementation. Because program evaluation pre-sents an incredibly complex set of considerations for evaluators,beginning evaluators initially focus on the essential aspects of itthat comprise the cyclical steps of conducting an evaluation – fromthe initial request for an evaluation to clarification of what isneeded to designing and implementing the evaluation tosummarizing information and reporting the results. Ideally thisprocess repeats regularly so that programs benefit from theevaluation findings. While the number of steps differ amongevaluation approaches and evaluators, there is almost universalagreement about this process sequence.

In addition to the nuts and bolts of evaluation, however, aremore nuanced aspects of the effort that are concomitant with aprogram evaluation endeavor. Evaluative data may be collectedfrom a distanced or engaged stance by internal and/or externalevaluators. Evaluators may or may not consider the potential toenhance the capacity of program staff members to consume andconduct evaluation. Similarly, evaluators may or may not choose toconsider the systemic role of the program in a broader context andits policy implications. Further, the importance of culturalcompetence to the endeavor is considered important. These andother aspects of evaluation are often what distinguish oneevaluation approach from another.

In the mid 1990s evaluators who shared a common desire towork with and consider program stakeholders created the‘‘Collaborative, Participatory, and Empowerment Evaluation’’(CPE) topical interest group within the American EvaluationAssociation. During that time these evaluators began to expandcurrent thinking around this evaluation approach, which had beenpioneered by Stake (1967, 1983) and had a growing following ofproponents. As of 2010, the CPE group was the third largest TIGwith the organization. In 1994, Fetterman introduced Empower-ment Evaluation, and in 1995, Cousins and Earl edited a book onParticipatory Evaluation in education. The group was formalizingits components.

An important discussion was begun by Cousins and Earl (1995)and expanded by Cousins, Donohue, and Bloom (1996) when theycontrasted different aspects of stakeholder-based evaluation alongthree dimensions: stakeholder selection for participation, controlof evaluation technical decision-making, and depth of participa-tion. Cousins and Whitmore (1998) then used these threedimensions to distinguish types of participatory evaluation alongwith a host of other collaborative inquiry methods (both evaluativeand applied research). They also raised questions and issuesaround the aspects of power, ethics, participant selection, technicalquality, cross cultural issues, training, and enabling conditions,saying that they hoped responses would emerge from sustainedpractice and reflection.

Evaluation as a field has expanded and matured in the lastdecade to include new ideas and refined concepts with slightlydifferent terms and connotations. For example the ‘‘cross-cultural’’aspect of evaluation identified by Cousins and Whitmore (1998)would now more commonly be referred to as ‘‘culturallyresponsive evaluation.’’ Their issue labeled, ‘‘training’’ would beakin today to ‘‘evaluation capacity building.’’

In the past 10 years, the discussion around participant-orientedevaluations also has evolved from where it was in the 1990s.Proponents of these approaches have encountered new evaluationsituations and from their experiences have refined their views,sharing the results of their practice thorough books, articles, andpresentations (see for example: Fetterman, 2001; Fetterman &Wandersman, 2005, 2007; King, 2007; O’Sullivan, 2004; Rodri-guez-Campos, 2005).

One can see the essential aspects of evaluation (i.e., evaluationdesign, implementation, analysis and reporting) and the subtleraspects of evaluation (e.g., engagement of stakeholders, policyimplications, power of decision-making, etc.) as a framework bywhich to compare and contract Collaborative Evaluation to otherparticipant-oriented approaches. This article identifies 11 aspectsof evaluation, essential to Collaborative Evaluation, and thencompares them with Participatory Evaluation, EmpowermentEvaluation, and Utilization-Focused Evaluation. The hope is thatfrom this theoretical perspective, the reader may understand howCollaborative Evaluation distinguishes itself from the otherparticipant-oriented evaluation approaches.

4. Essential aspects of Collaborative Evaluation

The 12 essential aspects of Collaborative Evaluation used forthis article are:

1. P

rimary Evaluation Focus; 2. E valuation Decision-Making; 3. S takeholder Roles; 4. E valuator Roles, 5. P re-Evaluation Clarification Activities; 6. E valuation Design Orientation; 7. T ype(s) of Data Collection Used; 8. T ype(s) of Data Reporting; 9. E valuation Capacity Building; and

10. C

ultural Responsiveness and 11. S ystems/Networking Considerations 12. A ctive Stakeholder Engagement in Evaluation Implementation;

These aspects were selected based on the literature, discussionsand presentations at the annual meeting of the AmericanEvaluation Association, and experience gained from more than60 Collaborative Evaluation projects conducted during the last 10years. These aspects seemed to best distinguish how CollaborativeEvaluation might contrast with the other participant-oriented

R.G. O’Sullivan / Evaluation and Program Planning 35 (2012) 518–522520

approaches. The first 11 address the more nuanced aspects ofevaluation; the last one speaks to the degree to which stakeholdersare engaged in the essential evaluation implementation states.

Each of the aspects includes the author’s best attempt atcapturing the current level of practice for each of the fourapproaches: Collaborative Evaluation, Participatory Evaluation,Empowerment Evaluation, and Utilization-Focused Evaluation.Touchstones for these definitions have come from the work ofevaluators identified with each of these approaches; however, asmany different evaluators practice within the paradigms somedegree of inaccuracy is expected. Rather than as a static picture ofwhat is, the framework is presented as a working model to helpemerging evaluators better understand the choices amongparticipant-oriented evaluation approaches and stimulate discus-sion among those of us with longer perspectives in the field.

1. P

rimary Evaluation Focus – While there is much overlap amongall participant-oriented approaches, each of its proponents willidentify a primary purpose for their evaluation stance.Collaborative Evaluation wants to promote participationthroughout the evaluation as a mechanism for achieving theultimate goal of improving programs that serve people.Participatory Evaluation defines itself by some stakeholderengagement but does not clearly specify an amount or type ofparticipation. Empowerment Evaluation ‘‘aims to increase thelikelihood that programs will achieve results by increasing thecapacity of program stakeholders to plan, implement, andevaluate their own programs’’ (Wandersman, 2005, p.27).Finally Utilization Focused Evaluation is about promoting theuse of evaluation findings, as a step toward program learning.

2. E

valuation Decision-Making – This aspect of evaluations isextremely important and indispensable to comparing the fourapproaches because the vary greatly among themselves but byvirtue of the fact that decision-making power differentiates theparticipant-oriented approaches from the more distancedones; all four include some degree of shared power arounddecisions made about the evaluation. In Collaborative Evalua-tion and Utilization-Focused Evaluation the level of decision-making is negotiated. With Participatory Evaluation, evalua-tors and participants share the responsibility (Cousins &Whitmore, 1998; Whitmore, 1998). In Empowerment Evalua-tion, program participants are the key decision makers.

3. S

takeholder Roles – All four approaches allow for stakeholdersto participate in the evaluation process. Collaborative Evalua-tion talks about stakeholders as clients, partners, evaluationassistants and data sources. Participatory recognizes stake-holders most often as clients or data sources. EmpowermentEvaluation consistently refers to stakeholders as in charge of orpartners in the process. Utilization-Focused evaluation refersto collaborating with key stakeholders.

4. E

valuator Roles – Collaborative Evaluation sees the evaluatoras the team leader in the evaluation and as the collaboratorwith the program staff. Participatory Evaluation allows for anumber of evaluators roles that may vary greatly fromparticipant observer to team leader. In Empowerment Evalua-tion, the evaluator is seen as a guide, facilitator, or ‘‘criticalfriend’’. Utilization-Focused Evaluation speaks of active,reactive, interactive, and adaptive roles for the evaluator.

5. P

re-Evaluation Clarification Activities – This aspect addressesthe degree to which the evaluator works interactively withclients prior to formal implementation of the evaluation in theprocess of designing the evaluation. Collaborative Evaluationpromotes working with clients and stakeholders to probeprogram dynamics, evaluation purposes, and potentialresources for the evaluation. Participatory Evaluation doesnot really speak to this type of activity, while EmpowermentEvaluation addresses these aspects more in the conduct of the

evaluation. Utilization-Focused Evaluation promotes extensiveactivities prior to beginning the evaluation.

6. D

esign – This aspect refers to the rigor or technical quality ofthe evaluation research, as described in Guiding Principles for

Evaluators (American Evaluation Association, 2011). TheGuiding Principles include Systematic Inquiry as one of thekey precepts. These principles encourage evaluators to conductsystematic, data-based inquiries. They further say, ‘‘To ensurethe accuracy and credibility of the evaluative information theyproduce, evaluators should adhere to the highest technicalstandards appropriate to the methods they use’’. Proponents ofthe four approaches of course would endorse this principle butthe importance of it in practice is somewhat varied.Collaborative Evaluation would argue for designs that are asrigorous as possible in the context. Participatory Evaluations,depending on the roles assumed by the evaluator, can varygreatly in their degree of rigor. Empowerment Evaluationcultivates a culture of evidence and attempts to be as rigorousas possible, but recognizes that the rigor of the evaluationdesign is dictated by the context, because it is more participantcentered. Utilization-Focused Evaluation would probablyshade the need for technical quality by it appropriateness tothe evaluation.

7. T

ype(s) of Data Collection – All four approaches allow for bothqualitative and quantitative data to be collected, as isappropriate to the evaluation questions and design.

8. T

ype(s) of Data Reporting – Data reporting choices are many.Collaborative Evaluation negotiates reporting options withclients individually. Participatory Evaluation does not reallyspecify a stance about reporting. Empowerment Evaluationfocuses on reporting processes, results, and outcomes within acontext that is meaningful to participants and sponsors alike.Utilization-Focused Evaluation emphasizes the importance ofproviding data throughout the evaluation as it is available.

9. E

valuation Capacity Building – This aspect of evaluation speaksto the extent to which the conduct of evaluation enhancesstakeholders’ ability to commission, consume, and conductevaluation. It is an educative component of evaluation that isreceiving growing attention, as evaluators recognize the powerinherent in such activities. Collaborative Evaluation, Empow-erment Evaluation, and Utilization-Focused Evaluation allassume this aspect in the conduct of their work. This is lessclear with Participatory Evaluation.

10. C

ultural Responsiveness – This aspect of evaluation addressesthe need for evaluators to be culturally competent and torecognize and adapt to the cultural context of a program. Onceagain Collaborative Evaluation, Empowerment Evaluation, andUtilization-Focused Evaluation would argue that they havesystems in place to promote this aspect, while this is unknownfor Participatory Evaluation.

11. S

ystems/Networking Considerations – This aspect of evalua-tion encompasses those activities that include looking beyonda program to address its impact on the larger social system.Currently more commonly couched in evaluation discussionsof collaborations with other organizations, program sustain-ability and tracking policy indicators, it is a more recentconsideration that is just gaining momentum (O’Sullivan,2007). Both Collaborative Evaluation and Utilization-FocusedEvaluation directly accommodate these types of needs. Theprinciples of Empowerment Evaluation (Fetterman & Wan-dersman, 2005) address this with its focus on social justice(principle number 5). Participatory Evaluation does notexplicitly address this consideration.

12. A

ctive Stakeholder Engagement in Evaluation Implementation– The participation of stakeholders in the implementation ofthe evaluation is the final aspect for comparison. Stakeholders

Table 1Collaborative Evaluation framework comparisons with other participant-oriented evaluation approaches.

Aspects of evaluation Collaborative Evaluation Participatory evaluation Empowerment evaluationa Utilization-focused evaluation

1. Primary Evaluation Focus Promote ParticipationThroughout

Engage Some Stakeholders Stakeholders use evaluation

tools to achieve results

Promote the Use Evaluation

Findings

2. Evaluation Decision-Making Negotiated Evaluator & Participants Participants Negotiated3. Stakeholder Roles Clients, Partners,

Assistants,Data Sources

Clients, Data Sources In Charge of or Partners Key Stakeholders collaborate

4. Evaluator Roles Team Leader,Collaborator

From Participant

Observer to

Team Leader

Guide/Facilitator/Critical Friend Active-Reactive-Interactive-

Adaptive

5. Pre-Evaluation Clarification

Activities

Probe Program,Purposes,& Resources

Unknown Addressed in Conduct of

Evaluation

Extensive

6. Design As Rigorousas Possible

Varies with

evaluator role

Participant-Centered As Rigorous as Appropriate

7. Type(s) of Data Collection Qualitative& Quantitative

Qualitative &Quantitative

Qualitative & Qualtiative Qualitative & Quantitative

8. Type(s) of Data Reporting: As Agreed Upon Unknown Process, Results, & Outcomes On-going as data available

9. Evaluation Capacity Building YES Unknown YES YES10. Cultural Responsiveness YES Unknown YES YES11. Systems/Networking

Considerations

YES NO YES YES

12. Implementation-Stakeholders as:

Instrument Developers YES NO YES NO

Data Collectors YES NO YES NO

Data Analyzers YES NO YES NO

Data Interpreters YES YES YES YESData Reporters YES NO YES NO

a Reviewed by Abe Wandersman & David Fetterman.

R.G. O’Sullivan / Evaluation and Program Planning 35 (2012) 518–522 521

may participate as, Instrument Developers, Data Collectors,Data Analyzers, Data Interpreters, and Data Reporters.Collaborative Evaluation promotes using stakeholders in allthese roles; it particularly encourages data reporting throughEvaluation Fairs or Learning Communities, where stakeholderspresent the findings of their evaluation to peers, participants,and/or funders. Similarly, Empowerment Evaluation hasstakeholders active in all aspects of implementation of theevaluation. Participatory Evaluation and Utilization-FocusedEvaluation often use stakeholders for data interpretation butdo not speak to the other functions.

5. Comparison of four approaches across CollaborativeEvaluation aspects

The 11 Collaborative Evaluation Aspects are listed in Table 1.,indicating how each of the four participant-oriented approachesaddresses each aspect. To contrast the approaches, CollaborativeEvaluation practices are underlined and commonalities betweenCollaborative Evaluation and the other three approaches areindicated with underlining in their particular column. This contrastreveals commonalities but also some unique qualities of Collabo-rative Evaluation.

Common to all four approaches is the fact that they use bothqualitative and quantitative data and employ stakeholders to helpinterpret data. For evaluation capacity building and culturallyresponse evaluation aspects, all but Participatory Evaluation speakdirectly inclusion of these two aspects in their evaluation process.Both Collaborative Evaluation and Utilization-Focused Evaluationnegotiate decision-making around the evaluation process andalong with Empowerment Evaluation directly address the issues ofnetworking and systems change. Collaborative Evaluation andParticipatory Evaluation share the recognition of ‘‘clients’’ amongstakeholder roles; Collaborative Evaluation and EmpowermentEvaluation speak about stakeholders as ‘‘partners’’ and employstakeholders in all aspects of evaluation implementation.

The comparisons across the framework in Table 1 revealmarked differences between Collaborative Evaluation and theother three approaches. While these differences decidedly do notseparate Collaborative Evaluation from the other three in terms ofits participant-oriented approach, the emphasis and frameworkCollaborative Evaluation uses to plan and implement evaluationsappears unique. Most notably the pre-implementation activitiesemphasize somewhat different practices. Collaborative Evalua-tion’s engagement of stakeholders as members of the evaluationteam to design instruments, collect and analyze data, and sharefindings is only shared with Empowerment Evaluation. Similarly,Collaborative Evaluation’s inclusion of systems/networking con-siderations is only shared with Empowerment Evaluation andUtilization-Focused Evaluation. Thus identifying 11 aspects ofevaluation essential to Collaborative Evaluation, and then com-pares them with Participatory Evaluation, Empowerment Evalua-tion, and Utilization-Focused Evaluation has demonstrated howCollaborative Evaluation distinguishes itself from the otherparticipant-oriented evaluation approaches.

References

American Evaluation Association. (2011). Guiding Principles for Evaluators. Retrievedfrom http://www.eval.org/Publications/GuidingPrinciples.asp

Alkin, M. (2004). Evaluation roots. Thousand Oaks, CA: Sage.Cousins, J. B., Donohue, J. J., & Bloom, G. A. (1996). Collaborative evaluation in North

America: Evaluators’ self-reported opinions, practices, and consequences. Evalua-tion Practice, 17(3), 207–226.

Cousins, J. B., & Earl, L. M. (Eds.). (1995). Participatory evaluation in education: Studies ofevaluation use and organizational learning. London: Falmer.

Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. New Directionsfor Program Evaluation, 80, 5–23.

Fetterman, D. M. (1994). Empowerment evaluation. Evaluation Practice, 15, 1–15.Fetterman, D. M. (2001). Foundation of empowerment evaluation. Thousand Oaks, CA:

Sage.Fetterman, D. M., & Wandersman, A. (2005). Empowerment evaluation principles in

practice. New York: Guildord Press.Fetterman, D. M., & Wandersman, A. (2007). Empowerment evaluation: Yesterday,

today, and tomorrow. American Journal of Evaluation, 28(2), 179–198.Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative

approaches and practical guidelines (4th edition). Boston: Allyn & Bacon.

R.G. O’Sullivan / Evaluation and Program Planning 35 (2012) 518–522522

King, J. A. (2007). Making sense of participatory evaluation. New Directions for Evalua-tion, 114, 83–86.

O’Sullivan, R. G. (2004). Practicing evaluation: A collaborative approach. Thousand Oaks,CA: Sage.

O’Sullivan, R. G. (2007). Upping the accountability odds: Toward systems-focusededucational program evaluation. Paper presented at the annual meeting of the 5thAnnual International Conference on Education.

Patton, M. Q. (2008). Utilization-focused evaluation. Thousand Oaks, CA: Sage.Rodriguez-Campos, L. (2005). Collaborative evaluations: A step-by-step model for the

evaluator. Tamarac, FL: Llumina Press.Stake, R. E. (1967). The countenance of educational evaluation. Teachers College Record,

68, 523–540.

Stake, R. E. (1983). Program evaluation, particularly responsive evaluation. In G. F.Madaus, M. Scriven, & D. L. Stufflebeam (Eds.), Evaluation models (pp. 287–310).Boston: Kluwer-Nihoff.

Wandersman, A. (2005). The principles of empowerment evaluation. In D. M. Fetter-man & A. Wandersman (Eds.), Empowerment evaluation principles in practice. NewYork, NY: Guilford Press.

Whitmore, E. (1998). Understanding and practicing participatory evaluation. NewDirections for Evaluation, 80.

Dr. Rita O’Sullivan is Associate Professor and Director of Evaluation, Assessment, &Policy Connections within the School of Education at the University of North Carolina.