Evaluation and Program Planning 35 (2012) 558–563

Commentary—Collaborative evaluation within the larger evaluation context

Jody L. Fitzpatrick

University of Colorado Denver, ,

Contents lists available at SciVerse ScienceDirect

Evaluation and Program Planning

journa l homepage: www.e lsev ier .com/ locate /eva lprogplan

A R T I C L E I N F O

Article history:

Available online 19 February 2012

Keywords:

Collaborative evaluation

Evaluation approaches

Participatory evaluation

Empowerment evaluation

E-mail address: Jody.Fitzpatrick@UCDenver.edu.

0149-7189/$ – see front matter � 2012 Elsevier Ltd. All rights reserved.

doi:10.1016/j.evalprogplan.2011.12.012

Many date the formal beginnings of evaluation in the UnitedStates with Congress’ evaluation requirement for each grantfunded by the Elementary and Secondary Education Act in 1965. Atthat time, this and other evaluations were primarily concernedwith providing information to federal decision makers (Stuffle-beam, Maduus, & Kellaghan, 2000). Today, along with manyinitiatives for greater citizen participation in government, evalua-tors are concerned with developing ways to involve many otherstakeholders in evaluation. This volume adds to that dialogue.

While Robert Stake’s responsive evaluation is generally, andappropriately, identified as the forerunner of modern-day partici-patory evaluation, even Ralph Tyler, traditionally associated witheducational objectives and objectives-oriented evaluations,worked closely with his stakeholders. In his prime evaluationyears, in the 1940s and 1950s, he collaborated with teachers in theclassroom to insure that the evaluation work would be ofassistance to them in improving and guiding their work (Goodlad,1979; Madaus & Stufflebeam, 1989). So, ways for involving clientsand other stakeholders in evaluation, in efforts to improve validityor use, have been around for some time.

I introduce this history to emphasize that evaluators have longbeen concerned with involving others in evaluation and, in sodoing, to collaborate with clients and others on programdevelopment and improvement and on the evaluation itself. Thisvolume adds to our knowledge of ways to involve stakeholdersthrough its presentation of collaborative models, discussions ofthem, and illustrations of collaborative evaluation. Today, asO’Sullivan and Rodriguez-Campos note, we have many models ofparticipatory or collaborative evaluation. The variety of models isdemonstrated through the naming of the American EvaluationAssociation Topical Interest Group, Collaborative, Participatory,

and Empowerment Evaluation. The array of different modelsprompted Cousins and Whitmore (1998) to research differentmodels of evaluation and applied research. Their article wasrecognized in 2007 as one of the four most influential articles in thetwenty-year history of New Directions for Evaluation (King, 2007a).Cousins and Whitmore identified three factors to distinguish suchmodels: control of technical decisions, which types of stakeholdersare involved, and the depth of participation of those stakeholders.As O’Sullivan notes, since that article, evaluators have moved intomore nuanced discussions of ways to involve stakeholders, at leastpartly in consideration of these factors. In this volume, O’Sullivanposits twelve aspects of evaluation to distinguish different types ofparticipatory models. Interestingly, she does not use Cousins andWhitmore’s original factors. The types of stakeholders involvedand their depth of participation may be subsumed underO’Sullivan’s factor of stakeholder roles, though she discusses rolesmore in terms of the functions the stakeholder may play. Sincemodels do differ, and evaluators do make conscious or unconsciouschoices about which stakeholders they involve and the depth oftheir involvement, O’Sullivan and Rodriguez-Campos mightexpand their factor of stakeholder roles to consider these well-known concepts and important choices. Among the usefulconcepts O’Sullivan adds to Cousins and Whitmore’s originalfactors are capacity building, cultural responsiveness, and systems/networking considerations, all concepts that have emerged indialogues of evaluators since 1998.

Capacity building has emerged as an important issue inevaluation as funders and foundations have called upon organiza-tions to play a role and, in some cases, to conduct their ownevaluations of their programs. (See Hendricks, Plantz, & Pritchard,2008, for a discussion of the successes and challenges United Wayorganizations have faced as they attempt to measure theiroutcomes.) Many have recognized the opportunities such require-ments provide for educating employees and administrators ofprograms in not only evaluation methods, but in evaluative ways ofthinking. Compton, Braizerman, and Stockdill (2002), Jean King(2007b), and others have provided helpful discussions andexamples of evaluation capacity building. More recently, Preskilland Boyle (2008) have developed a useful model for organizationallearning and capacity building.

Cultural responsiveness, or attention to the cultures oforganizations, clients, and political actors has been an on-goingthread in evaluation but has received renewed attention as theUnited States, and European countries, become more diverse.Revisions of both the Joint Committee’s Program EvaluationStandards (2010) and the Guiding Principles (2004) of the

J.L. Fitzpatrick / Evaluation and Program Planning 35 (2012) 558–563 559

American Evaluation Association have both given attention tocultural responsiveness. The 2004 Guiding Principles emphasizecultural competence in reference to diversity and stress the needfor evaluators to attain cultural competency by reflecting on theirown culturally-based assumptions and those of culturally-differ-ent participants and stakeholders. A primary change in the newProgram Evaluation Standards is its consideration of culture. TheJoint Committee on Standards for Educational Evaluation approvedfive new standards which touch on many aspects of culture: thecultural viability of the evaluation and of formal agreements andconclusions and decisions, the need to be explicit about values thatunder pin the evaluation, and the need for evaluators to havecredibility in the context in which they are working. Recentdiscussions of the influence of context on evaluations have alsohelped us understand that the culture in which an evaluation isembedded includes many aspects from the norms of organiza-tions; the disciplinary training of program deliverers, adminis-trators, and evaluators; to the norms of the political arena in whichthe program is situated.

Systems and networks represent the newest influence onevaluation practice. Social network analysis is being used in manydisciplines and programs to learn more about how to buildeffective systems and work together to achieve complex goals. (SeeDurland and Fredericks, 2005, for an introduction to social networkanalysis and examples of how it has been used in evaluation.)Woodward’s chapter in this volume illustrates how evaluatorsshould attend to inter- and intra-agency networks, and evaluatethem, to help organizations examine their external environmentand take greater advantage of it. Thus, each of last three evaluationaspects identified by O’Sullivan in her comparison of models –capacity building, cultural responsiveness, and system/networkingconsiderations should be considered by evaluators in the contextof each particular evaluation.

1. Differentiating models or enhancing evaluation practice

The question is what are our goals in discussing new methods toconsider or implement to improve evaluations through stakehold-er involvement? O’Sullivan and Rodriguez-Campos provide manyuseful ideas and concepts to consider in involving stakeholders andcollaborating with them to improve programs and evaluations.(And, I will elaborate on those later.) But, I would argue,distinguishing among different models of evaluation may not bean appropriate goal at this stage of evaluation. Evaluation capacitybuilding, mainstreaming evaluation, attention to context and otherrecent issues which have drawn the attention of evaluators havebeen offered not as new models, to replace old, but, rather, asissues that any evaluator should consider and use, as appropriate,to the situation of the evaluation. Just as Michael Patton, CarolWeiss, and others inspired us some years ago to think about whatwe mean by ‘‘use’’ and to consider ways to increase use ofevaluation results and Robert Stake encouraged us to be responsiveand adaptive to stakeholders and context, the issues raised byvarious models of participatory evaluation can raise the awarenessand increase the skills of all evaluators concerning various meansto collaborate, empower, and enjoin stakeholders to participate inevaluation and gain useful skills.

While evaluators parse the differences between models,evaluation research (Christie, 2003; Cullen, 2009) suggests thatmany evaluation practitioners, often juggling other jobs as well,are not aware of the principles, much less the practices, ofstakeholder involvement. My own interviews with well-knownevaluators on individual evaluations they have conducted showthat these evaluators involve stakeholders in many ways. Christie(2003), too, found that evaluation theorists, even those on differentends of the spectrum of models, argued for involving stakeholders.

Why the discrepancy between the work of many of those whoconduct evaluations and exemplars who conduct evaluations anddevelop theories? Somehow, we as evaluators are failing tocommunicate with practitioners about practical means forstakeholder involvement. Datta (2006), an old hand at evaluation,has written of the need for us to focus on training and educatingthose who are conducting evaluations, but are new to the concepts.As such, I would like to discuss O’Sullivan’s and Rodriguez-Campos’work and volume on collaborative evaluation, not as a separateapproach or model, but in terms of the concepts and ideas it offersto the practice of evaluation for all. The recent edition ofFitzpatrick, Sanders, and Worthen (2010) condensed its chapterson evaluation approaches to combine models and discuss broaderconcepts, ones that might resonate with the newcomer toevaluation and the practitioners who do not think much abouttheory. Similarly, I would like to use this chapter to highlight someof the useful techniques or activities that these authors articulateunder the umbrella of collaborative evaluation.

2. Concepts and techniques of use to practicing evaluators:reasons for participation, evaluation decision-making,evaluator and stakeholder roles

In introducing collaborative evaluation, O’Sullivan distin-guishes between the ‘‘nuts and bolts’’ or the ‘‘essential aspects’’of evaluation, e.g., clarifying the initial request, planning the study,collecting and analyzing data, and reporting conclusions, and themore ‘‘nuanced aspects.’’ O’Sullivan is correct in calling ourattention to these issues. Many people who conduct evaluationsare not aware of these nuanced aspects, but, these issues are, infact, ones that can often make or break a successful evaluation, onethat is used or has an impact in some way. While these issues arecentral to many professional evaluators’ discussions and writings,many of those who come to evaluation have skills in some aspect ofresearch and, as such, apply those skills to an evaluation. They areunaware of the importance of gaining knowledge of the culture andcontext for the evaluation and considering its implications for thenuts and bolts of the evaluation: What designs and types of datawill be credible to different stakeholders? What cultural factorsshould be considered in data collection and analysis? Whatreporting styles and formats fit best to the culture and context andare most likely to lead to use? The ‘‘nuances’’, then, concerncollaboration and participation and O’Sullivan highlights some ofthese in her list of the essential aspects of evaluation: With whomshould evaluators collaborate? What should be the role of theevaluator? The stakeholder(s)? What pre-evaluation clarificationactivities are necessary? What networks might be used toencourage collaboration and use?

O’Sullivan’s list of twelve aspects of evaluation include bothnuts and bolts factors (design, data collection, and reporting) andnuances. With the first factor, primary evaluation focus, she isconcerned not with issues of formative or summative evaluation ora focus on needs assessment, process, or outcomes, but rather onthe reason for involving or collaborating with stakeholders.Identifying these reasons is central to successful participatoryevaluation. Clarifying these reasons helps the evaluator considerwhich stakeholders to involve and in what manner, in whatactivities and choices. If participation and collaboration are soughtto provide stakeholders with evaluation skills or evaluative ways ofthinking for future organizational development, managers andstaff of the organization are likely stakeholders. With non-profitorganizations, such stakeholders might include members ofboards. In organizations or departments with heavy citizen inputor involvement, stakeholders would likely include representativesof these citizens. If, instead, collaboration is sought to facilitateacceptance and use of the evaluation, of course the stakeholders

J.L. Fitzpatrick / Evaluation and Program Planning 35 (2012) 558–563560

involved would be the primary intended users. In summativeevaluations, such users might include funders, policy makers,legislative staff, or high-level appointed or elected officials. (Note:These are not stakeholders one would involve for organizationaldevelopment, since they are not employed by the organization ofinterest in the evaluation.) If collaboration or stakeholderparticipation is intended to increase the validity of the datacollected, evaluators may collaborate with clients for the programor residents of the neighborhood or community in which theprogram is delivered. Collaborating with these stakeholders toobtain their suggestions, input, and reactions to proposed methodsof data collection and, then, to involve them in data collection oranalysis can improve the information collected and add insight toanalyses and interpretations.

O’Sullivan’s recognition of the need to clarify the purpose of thecollaboration, and placing that first on her list, makes us aware ofthe obligation of the evaluator to consider those purposes and toprioritize them. This consideration can help evaluators makeappropriate choices concerning potential collaborators, their roles,and the role of the evaluators. In this volume, for example, Corn,Byrom, Knestis, Matzen, and Thrift are using collaboration andstakeholder involvement for purposes of capacity building, i.e., tohelp schools plan and conduct the evaluation of their particularIMPACT program. In contrast, although some capacity buildingamong students is involved in the O’Sullivan and O’Sullivan studyin the south Sudan, collaboration and participation are usedprimarily to enhance the utility of the data base and the validity ofthe data collected.

O’Sullivan’s next aspect, evaluation decision-making, concernswho will maintain responsibility for key evaluation decisions, inparticular, technical ones. Several authors in the volume refer tothe need, in most situations, for evaluators to maintain thisresponsibility. O’Sullivan sees the evaluator as the team leader andsuch appears the case in many of the other chapters. In a session atthe American Evaluation Conference in 2010, O’Sullivan andRodriguez-Campos spoke of the need for evaluators to retain anduse their evaluation expertise in collaborative evaluation whilesharing that expertise with their collaborators. Similarly, thosewith whom they are collaborating (program administrators, staff,clients) use and share their expertise in the collaboration. Theirexpertise is in the knowledge of the program (as planners,deliverers, or recipients) and their personal experiences with it andwith the community environment. As O’Sullivan eloquently notedin that AEA session, such collaborations are like a symphony witheach person an expert in his or her own instrument.

Several authors in this volume illustrate the role of theevaluator in providing his or her technical expertise in theevaluation: Kuzmin or another training leader is the clear leader ofhis Participatory Evaluation Training Method in forming theevaluation questions and acting as the facilitator. As externalevaluators, Orsini and other evaluators with EvAP are clearly usingtheir technical expertise and past experience to develop the design,methods, and data collection strategies to meet the needs andexpectations of Tanglewood Research and the funding agency.Corn and her colleagues are the technical evaluation experts inhelping a number of schools to conduct their formative evalua-tions.

Other important aspects of participant-oriented evaluationidentified by O’Sullivan include defining the roles of both thestakeholder and the evaluator. As noted, in the examples in thischapter, the evaluator is the primary technical decision-maker butseeks input from other stakeholders. In the case described byOrsini and his colleagues, the roles of the two collaborators areclearly defined: EvAP is an external evaluator bringing anindependent view and desired rigorous design to the project.Tanglewood Research is the program developer. The two partners

collaborate, however, in seeking schools to participate and intraining teachers to deliver the program and to collect evaluationdata. The evaluators make useful suggestions for recruitment ofschools, retention of data, and for revisions in the ways in whichthe program is delivered. Kuzmin, as the trainer/evaluator definesthe trainees’ roles in the evaluation, i.e., collecting data throughinterviews, analyzing, reporting, and reflecting on the results.Trainees carry out these proscribed steps with his facilitation. Ofcourse, the results the trainees obtain from the interviews givethem experience in the evaluation process, greater knowledge andunderstanding of the results, and a feeling of ownership in them.Corn et al. note that although they initially assumed all schools,their partners in the collaboration, would have similar needs, thatwas not the case. Some already knew evaluation and used it. Othersdid not. So, in one of their many useful lessons and suggestions,they observe that ‘‘one size does not fit all.’’ Evaluators need toadapt their technical assistance activities to provide ‘‘develop-mentally appropriate support’’ to stakeholders in their activities inthe evaluation in order to achieve the desired goals of participa-tion.

3. Concepts and techniques of use to evaluators: pre-evaluationclarification activities and the planning stage

Pre-evaluation clarification activities, O’Sullivan’s next aspectof evaluation, are recommended in most evaluation textbooks(Fitzpatrick, Sanders, & Worthen, 2009; Rossi, Lipsey, & Freeman,2004; Stufflebeam & Shinkfield, 2007) and are used by mosteffective evaluators to learn about the program; its theory ormodel and its activities, clients, and context in order toappropriately position the evaluation for understanding and use.Rodriguez-Campos elaborates on pre-evaluation clarificationactivities in her presentation of two collaborative models. Thefirst model, O’Sullivan’s Annual Collaborative Evaluation Cycle,presents four major stages of the collaborative evaluation cycle.The first stage concerns reviewing the program status andunderstanding the context, asking for available evidence, andprobing the evaluation culture. The more comprehensive model forcollaborative evaluations (MCE) presented by Rodriguez-Camposincludes three components that can take place during the planningstage, though unlike the first model, this model does not followthrough with clear stages of implementing the evaluation andsharing findings. Instead, its focus is more on the aspects ofcollaboration that should occur at every stage, i.e., ensure opencommunication, encourage best practices, follow specific guide-lines.

In regard to pre-evaluation clarification activities, severalchapters in this volume helpfully illustrate the concepts presentedby O’Sullivan and Rodriguez-Campos in action: Corn and her fellowevaluators learn about IMPACT, the program designed by the statedepartment of education, to be implemented in the schools; theylearn of the schools’ needs and experiences with evaluation; andthey research change processes to help them in preparing theschools for change. One of the outcomes of this phase is theirrecognition that helping teachers and administrators learn how touse data is more important than learning how to do evaluation.O’Sullivan and O’Sullivan’s case takes place in a quite differentculture, Southern Sudan. In this setting, the authors learn muchabout the culture of agricultural markets in Sudan through pastresearch. Their role is not so much evaluation as developing a datacollection tool, a trader profile, to allow the agricultural market towork more effectively by providing a data base of traders. Althoughthere does not appear to be a specific program for them to study, asthe data base was being developed, they talked with what theycalled ‘‘visiting ‘knowledgeables’’’ to learn more about what shouldgo in the data base and how such information might be used. Orsini

J.L. Fitzpatrick / Evaluation and Program Planning 35 (2012) 558–563 561

and colleagues’ pre-evaluation clarification activities are partiallyconcerned with learning the evaluation expectations of theprogram developers at Tanglewood Research and the SubstanceAbuse and Mental Health Services Administration (SAMHSA). Inorder for Tanglewood’s program to appear in SAMHSA’s nationalregistry the evaluation must follow specific rigorous criteria,including examining outcomes, intervention fidelity, and potentialconfounding variables. So, in this case, the evaluation is largelyformed by these expectations identified during the pre-evaluationclarification stage. Appropriately in this evaluation, the pre-clarification evaluation activities are not as concerned withreviewing the program status as with learning about theevaluation culture and its expectations. These tasks appear tomirror the first stage of the Annual Collaborative Evaluation Cyclein probing the evaluation culture and understanding the context,but not as much with the first stages of Rodriguez-Campos’ Modelfor Collaborative Evaluation.

The chapter by Askew, Beverly, and Jay is an essay about aspectsthat should be considered by evaluators using collaborativeprocesses, rather than cases. But, their points are relevant here,at the pre-evaluation clarification or planning stage. Askew and herco-authors’ chapter comparing collaborative and culturally-responsive evaluation models helps evaluators consider how theycan use culturally-responsive methods during the early planningstages of an evaluation, and, then, throughout the evaluation to besensitive to cultural issues. They note that the value of theculturally-responsive model (Frierson, Hood, & Hughes, 2002) is toencourage evaluators to give explicit attention to ‘‘the voices andconcerns of marginalized or underserved populations.’’ In com-paring the two models, they find considerable overlap, but identifyways they can use elements of both to produce successfulevaluations. Their chapter, however, reveals some of my concernswith our focus on the development of different models and, insome cases, a concern with serving as an acolyte or advocate for amodel. For example, they refer to their perception that they aretrying to ‘‘straddle two approaches.’’ Although Rodriguez-Camposin her discussion of the model for collaborative evaluationindicates that the first stage of identifying the situation should‘‘ensure that it [collaborative evaluation] is appropriate for thecurrent situation’’ and Askew and her co-authors note thatresource constraints, time and money, can limit collaboration,Askew et al. suggest that, in practice, they, and other evaluators,are torn between different models. As their comparison concludes,I have concerns that the focus on different models, and theirdifferences, may discourage evaluators from being sufficientlyflexible to adapt their methods to the nuances of the evaluation,the context, and the resources available. As Askew and her co-authors’ discussion of straddling two models implies, someevaluators do struggle to retain each concept of a model in spiteof the fact that model developers, like O’Sullivan and Rodriguez-Campos, encourage evaluators to consider whether a model isappropriate for a particular evaluation and its context and to beflexible in applying the methods of a model.

One last comment on the planning stage and the modelspresented: The early evaluation activities listed in the two modelspresented by Rodriguez-Campos include important, even criticalelements: understanding the context and evaluation culture,developing logic models (Fig. 1), and, in the comprehensive modelfor collaborative evaluation (Fig. 2), identifying stakeholders, theevaluation scope, and critical evaluation activities; clarifying roles,evaluand criteria and standards, the evaluation plan and thebudget, and steps for establishing a shared commitment, i.e.,ground rules for behavior, conflict resolution, and decision-making. But, I am struck in this chapter and others with thefrequent omission of defining the evaluation questions to beanswered. This step is critical and should be conducted in

collaboration with selected stakeholders, but it should be anexplicit step. Evaluations that fail to clearly define their purposes(formative, summative, organizational development, accountabil-ity) and to identify the specific questions they hope to answer candrift and, ultimately, provide information that may not meet theneeds of stakeholders or the organization. Michael Patton hascalled this a Type III error, answering the wrong questions. Theplanning phase is designed for evaluators to learn about theprogram and its culture and context, to understand its rationale(logic model), to consider which stakeholders should be involvedand how, but, ultimately, the evaluation must be grounded inevaluation questions to be answered. Perhaps the specification ofevaluation questions is so basic that in the discussion of ways tocollaborate, this critical step is overlooked. But, particularly whenmany groups are collaborating, the evaluation questions providean anchor to keep the evaluation on target or to make an explicitdecision to change the evaluation questions to address newunderstandings that have emerged through collaboration orconditions discovered in the program. The evaluation questionsidentified at the planning stage, however, provide the initialfoundation and focus for the evaluation and for collaboration.

4. Concepts and techniques of use for evaluators: datacollection, analysis, interpretation, and reporting

These aspects of evaluation – data collection, analysis,interpretation, and reporting – are primarily the ‘‘nuts and bolts’’of evaluation to which O’Sullivan refers. Nevertheless, this volumeprovides some important suggestions to evaluators for thesephases. As O’Sullivan’s Annual Collaborative Evaluation Cycle,presented by Rodriguez-Campos demonstrates, evaluators caninclude stakeholders in developing or reviewing drafts of instru-ments and checking and analyzing data. O’Sullivan and O’Sullivanin developing the trader profile data base involve stakeholdersknowledgeable in the community around data collection. Recentlygraduated students, from Southern Sudan, play an active role inidentifying needed information, developing interview questions,and developing the data base. Kuzmin, with his ParticipatoryEvaluation Training Method, describes how he involves trainees incollecting, analyzing, and reporting data. Orsini and his fellowevaluators, working with their client Tanglewood Research, trainteachers in not only methods of delivering the substance abuseprevention program in the classroom, but in ways to administerevaluation measures. Corn and her colleagues learn through theteachers and administrators collecting data that teachers can bebetter involved by learning how to use data and understandingwhat they are doing and why, at least partly because teachers’ timeis limited. This outcome demonstrates the two-way arrow ofcollaboration learning; evaluators learned from clients about theirneeds in regard to evaluation.

Although O’Sullivan’s Annual Collaborative Evaluation Cycleshows the final stage as sharing evaluation findings, includingevaluation fairs, and Askew et al. cite evaluation fairs as one ofcollaborative evaluation’s four key techniques, this volume doesnot provide much information on this collaborative/participatorymeans of sharing evaluation results. (See O’Sullivan, 2004 for moreinformation on evaluation fairs.) Kuzmin provides the most detailon reporting with his quick, but innovative, methods for havingsmall groups of trainees interview each other on evaluator-definedevaluation questions, analyze and interpret what they havelearned and make short presentations to each other seeking inputand asking questions. The evaluator/facilitator also becomesinvolved in asking questions during the presentation of resultsto help arrive at some conclusions for the evaluation. It would behelpful to learn more about how other authors in this volumeshared results.

J.L. Fitzpatrick / Evaluation and Program Planning 35 (2012) 558–563562

5. Strengths and concerns about collaborative/participatoryapproaches

This volume, through its aspects of evaluation, models, essays,and case studies, provides many useful concepts and methods forpracticing evaluators to consider and adopt for particularevaluations. As I have indicated, I think that differentiatingbetween models or working strictly within one model can preventus from adapting the large variety of participatory and collabora-tive methods and techniques that can be used to improveevaluations and, more importantly, to communicate participativeor collaborative methods to the many people conducting evalua-tions as only part of their job or profession. Perhaps models help usin sorting and categorizing many different participatory andcollaborative methods, but, I argue that we simply need to havemore collaboration and participation, drawing from methodsintroduced in many models – utilization-focused evaluation,empowerment evaluation, practical participatory evaluation,and collaborative evaluation – to make choices about participationand collaboration that are appropriate to each evaluation settingand purpose. Developers of models, like O’Sullivan and Rodriguez-Campos and others, recognize the need for flexibility in use ofmodels and their associated methods, adapting to the purposes ofthe evaluation and the program context. But, often evaluators newto the field do not, and the existence of apparently competingmodels may lead them to make inappropriate choices. Instead,professional evaluators and those who write in the field may needto consider their means for conveying the many viable participa-tory and collaborative methods to educate new evaluators,researchers who enter the field, those who perform evaluationas only part of their work, and others who may not have the time tokeep up with the literature on participation and collaboration, butwho can benefit from using participatory and collaborativemethods in their evaluation work.

Finally, we need to be aware of concerns about participatoryand collaborative methods. Fitzpatrick et al. (2010) identify threeareas of concern with participative models: feasibility, credibility,and competence. Will it be feasible to involve intended stake-holders to the extent necessary to achieve the participative goals?Do they have the time, resources, and interest? Will participationor collaboration between stakeholders and evaluators threaten thecredibility of the evaluation to important outside audiences? And,finally, will stakeholders involved in technical aspects of theevaluation have the competence to conduct these activities in avalid and reliable manner?

Corn and her colleagues and Askew and her co-authors bothaddress the problems of feasibility for conducting participatory orcollaborative evaluations. Corn et al. conclude that it is lessimportant to involve educators in data collection and trainteachers to do evaluations than to help them learn the purposesof the evaluation and how to make use of the data. Their primarypurpose in collaboration was building organizational capacity forevaluation, but they learned that teachers’ roles in evaluation willbe primarily concerned with using it to improve teaching, notconducting evaluations. Stakeholders may not want to develop theevaluative capacities evaluators have; they have their owncapacities and responsibilities. And, collaborative evaluation doesnot advocate that stakeholders become evaluators. Recall O’Sulli-van’s comments on each person knowing his or her owninstrument and the collaborative result is a symphony. Corn andher co-authors recommend this: ‘‘[A]lways keep in mind that thefocus of the effort should be on them [educators or employees inother settings] and their projects, not on you, and not on learninghow to do evaluation.’’

Orsini and his colleagues’ presentation of their case illustratessome of the concerns about credibility. The National Institutes of

Health reacts positively and strongly to the existence of anindependent evaluation team and the use of rigorous methods suchas a randomized control trial. In this case, Orsini and EvAP are ableto work together collaboratively with their client, TanglewoodResearch, but this is largely because Tanglewood shares the NIHconcerns for independence and rigor. The evaluation team’s‘‘independence’’ from the program is seen as an advantage in thisevaluation and in others where summative decisions are to bemade about program funding and continuation. Funders areconcerned that bias may enter in when evaluators are closelyinvolved with the program. As Orsini and his colleagues report,NIH’s feedback on the proposal stated: ‘‘Perhaps most importantly,the investigators have chosen to engage an external evaluationteam to independently evaluate the success of the program, whichwill ensure that bias does not affect program measurement.’’ I donot think such independence guarantees a lack of ‘‘bias.’’Nevertheless, it reveals NIH’s views. None of the ten points offeedback from NIH concerns the advantages of collaboration.However, a major theme of Orsini and his colleagues’ chapter isthat they were ‘‘blending a prescriptive research design withmethods of collaborative evaluation.’’ They demonstrate thatcombinations of collaboration and rigorous design are possible,and, indeed O’Sullivan’s comparison of designs across differentparticipative models indicates that evaluations using collaborativeevaluation should use designs that are as rigorous as possible. But,the comments of NIH reveal the value that many agencies ascribeto what they see as ‘‘independent’’ evaluations and illustrate theconcerns that others may have about collaborating with programstaff or administrators and/or having them participate in anevaluation. Evaluators know the many advantages that can comefrom participatory or collaborative elements, but, we should beaware of the challenge presented by perceptions of bias andconcerns about independence. These concerns are particularlystrong when the evaluation’s purpose is summative, to makedecisions about program continuation or expansion, and evalua-tive judgments made in the study are about ultimate programquality.

Finally, competence can be a concern in those participativemodels where evaluators are not in charge of technical aspects ofthe study. The collaborative model maintains the evaluators’leadership role on technical issues as team leader, althoughO’Sullivan in discussing evaluation decision-making says the levelof decision-making is negotiated. A critic of some participatoryelements, Peter Dahler-Larsen bemoans the ‘‘expectation that laypersons will become good evaluators based on a short introductionand rudimentary training in methodology’’ (2006, p. 145). And,external consultants working to help organizations apply theUnited Way guidelines to evaluate their programs have ultimatelyexpressed concerns about the capacity of program staff to conductsuch evaluations (Hendricks et al., 2008).

These concerns are legitimate and should be recognized andconsidered in each evaluation as potential reasons to conduct a lessparticipatory or collaborative evaluation. But, as Orsini and hiscolleagues demonstrated maintaining credibility can occur jointlywith collaborative evaluation in some situations. Feasibility, theskills, time, and interest of intended stakeholders deserveattention as Corn and her co-authors advise. Finally, collaborativeevaluation generally advocates that evaluators’ maintain technicalcontrol, their key skills. The knowledge and skills which otherstakeholders are called upon to provide should be ones that arealready within their areas of expertise (knowledge of program,clients, staff, context, networks, systems) or skills they can readilygain.

Meanwhile, participatory methods have much to offer.Evaluators have learned, since the 1970s, that evaluationsgenerally find a place on a bookshelf, or, more recently, a hard

J.L. Fitzpatrick / Evaluation and Program Planning 35 (2012) 558–563 563

drive or web site, if intended users are not engaged and involved inthe evaluation. Further, collaboration, each group contributing andlearning information relative to the evaluation from other groups’areas of expertise, can produce evaluations that are more valid andresponsive to the context and culture of the program andcommunity. Finally, participative and collaborative methods helpbuild organization’s capacity, not just for conducting traditionalprogram evaluations, but for mainstreaming evaluation, makingevaluative ways of thinking – questioning, considering evidence,deliberating – part of daily operations in an organization. Thisvolume contributes to our understanding of some of thesemethods and provides useful examples of their application.

References

American Evaluation Association. (2004). Guiding principles for evaluators. RetrievedJanuary 16, 2011 from http://www.eval.org/Publications/GuidingPrinciples.asp

Christie, C. A. (2003). What guides evaluation? A study of how evaluation practice mapsonto evaluation theory. In (Series Ed.) & & C. A. Christie (Vol. Eds.), New directionsfor evaluation: . The practice-theory relationship in evaluation (pp. 7–36). .

D. W. Compton, M. Braizerman, & S. H. Stockdill (Vol. Eds.), (2002). The art, craft, and

science of evaluation capacity building (No. 93). New directions forevaluationSan Francisco, CA: Jossey-Bass.

Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. In (Series Ed.) && E. Whitmore (Vol. Eds.), New directions for evaluation: . Understanding andpracticing participatory evaluation (pp. 5–23). San Francisco: Jossey-Bass.

Cullen, A. (2009). The politics and consequences of stakeholder participation on interna-tional development evaluation. Unpublished doctoral dissertation, Western Michi-gan University.

Datta, L. (2006). The practice of evaluation: Challenges and new directions. In I. F. Shaw,J. C. Greene, & M. M. Mark (Eds.), The Sage handbook of evaluation. Thousand Oaks,CA: Sage.

M. M. Durland, & K. A. Fredericks (Vol. Eds.), (2005). Social network analysis in

program evaluation (No. 107). New Directions for EvaluationSan Fran-cisco: Jossey-Bass.

Fitzpatrick, J. L., Christie, C. A., & Mark, M. M. (2009). Evaluation in action. ThousandOaks, CA: Sage.

Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2010). Program evaluation: Alternativeapproaches and practical guidelines. Boston: Pearson.

Frierson, H., Hood, S., & Hughes, S. (2002). Strategies that address culturally responsiveevaluation. In J. Frechtling (Ed.), The 2002 user-friendly handbook for projectevaluation. Arlington, VA: National Science Foundation.

Goodlad, J. (1979). What schools are for. Bloomington, IN: Phi Delta Kappa EducationalFoundation.

Hendricks, M., Plantz, M. C., & Pritchard, K. J. (2008). Measuring outcomes of unitedway-funded programs: Expectations and reality. In (Series Ed.) & J. G. Carman, & K.A. Fredericks (Vol. Eds.), New directions for evaluation: . Nonprofits and evaluation(pp. 13–35). San Francisco: Jossey-Bass.

Joint Committee on Standards for Educational Evaluation. (2010). The program evalua-tion standards (3rd ed.). Thousand Oaks, CA: Sage.

King, J. A. (2007a). Making sense of participatory evaluation. In (Series Ed.) & & S.Mathison (Vol. Eds.), New directions for evaluation: . Enduring issues in evaluation:The 20th anniversary of the collaboration between NDE and AEA (pp. –).San Francisco,CA: Jossey-Bass.

King, J. A. (2007b). Developing evaluation capacity through process use. In (Series Ed.)& & J. B. Cousins (Vol. Eds.), New directions for evaluation: . Process use in theory,research, and practice (pp. 45–59). San Francisco: Jossey-Bass.

Madaus, G. F., & Stufflebeam, D. L. (Eds.). (1989). Educational evaluation: Classic works ofRalph W. Tyler. Boston: Kluwer Academic.

O’Sullivan, R. G. (2004). Practicing evaluation: A collaborative approach. Thousand Oaks,CA: Sage.

Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacitybuilding. American Journal of Evaluation, 28, 127–138.

Rossi, P. H., Lipsey, M. E., & Freeman, H. E. (2004). Evaluation: A systematic approach (7thed.). Thousand Oaks, CA: Sage.

Stufflebeam, D. L., Madaus, G., & Kellaghan, T. (Eds.). (2000). Evaluation models. Boston:Kluwer Academic.

Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, and applications.San Francisco: Jossey-Bass.

Jody Fitzpatrick is a faculty member with the School of Public Affairs at University ofColorado Denver. She is the author of several books and many articles on evaluationand has conducted evaluations for 35 years. She is president-elect of the AmericanEvaluation Association.