Evaluating Communities of Practice and Knowledge Networks ... · Scopus, Web of Science, Medline, EMBASE, CINAHL, ERIC, EconLit, and Business Review—captured frame-works from a

Evaluating Communities of Practice and KnowledgeNetworks: A Systematic Scoping Review of EvaluationFrameworks

Kaileah A. McKellar,1,2 Kristen B. Pitzul,1 Juliana Y. Yi,1 and Donald C. Cole2,3

1Institute of Health Policy, Management and Evaluation, Faculty of Medicine, University of Toronto, 155 College Street, Suite 425, Toronto, ON M5T

3M6, Canada2Community of Practice in Ecosystems Approaches to Health, 155 College St, Toronto, ON, Canada3Dalla Lana School of Public Health, University of Toronto, 155 College St, Toronto, ON, Canada

Abstract: Communities of Practice (CoPs) are increasingly considered a part of ecohealth and other sectors

such as health care, education, and business. However, there is little agreement on approaches to evaluate the

influence and effectiveness of CoPs. The purpose of this review was to understand what frameworks and

methods have been proposed or used to evaluate CoPs and/or knowledge networks. The review searched

electronic databases in interdisciplinary, health, education, and business fields, and further collected references

and forward citations from relevant articles. Nineteen articles with 16 frameworks were included in the

synthesis. The purposes of the evaluation frameworks varied; while some focused on assessing the performance

of CoPs, several frameworks sought to learn about CoPs and their critical success factors. Nine of the

frameworks had been applied or tested in some way, most frequently to guide a case study. With limited

applications of the frameworks, strong claims about generalizability could not be made. The review results can

inform the development of tailored frameworks. However, there is a need for more detailed and targeted CoP

evaluation frameworks, as many imperative CoP evaluation needs would be unmet by the available frame-

works.

Keywords: Communities of practice, Knowledge networks, Evaluation, Scoping review

INTRODUCTION

Ecosystem approaches to health (ecohealth) is an emerging

field producing ongoing debates about environmental and

health issues, and about the development of the field itself

(Webb et al. 2010; Charron 2012; Parkes 2011). In its early

stages, the ecohealth field recognized the value of sup-

porting communities of practice (CoPs) through net-

working and capacity building activities (e.g., meetings on

specific issues such as malaria in urban areas of Africa or

regional training workshops) (De Plaen and Kilelu 2004).

CoPs are groups of people who interact on an ongoing

basis to share knowledge and expertise about common

practices, problems, or topics (Wenger et al. 2002). A

knowledge network (KN) refers to a set of connections

Electronic supplementary material: The online version of this article (doi:10.1007/

s10393-014-0958-3) contains supplementary material, which is available to autho-

rized users.

Published online: July 15, 2014

Correspondence to: Kaileah A. McKellar, e-mail: [email protected]

EcoHealth 11, 383–399, 2014DOI: 10.1007/s10393-014-0958-3

Review

� 2014 International Association for Ecology and Health

http://dx.doi.org/10.1007/s10393-014-0958-3

http://dx.doi.org/10.1007/s10393-014-0958-3

among people who provide resources to solve problems,

share knowledge, and make further connections (Wenger

et al. 2011). CoPs and KNs may occur naturally, but they

are increasingly being employed deliberately as knowledge

(KM) management tools (Wenger 2004; Cox 2005; Li et al.

2009a). Foundational to ecohealth is that ‘‘health and well-

being are the result of complex and dynamic interactions

between determinants, and between people, social and

economic conditions, and ecosystems’’ (Charron 2012).

Structures that promote alliances among multiple disci-

plines and bring in new ideas can help people understand

the interrelationships between health, ecology, and society,

and apply these understandings in practice.

The CoP model can provide opportunities to break

down professional and organizational barriers (Ranmu-

thugala et al. 2011b) and to support the learning of new-

comers to the field (Lave and Wenger 1991). It can offer a

way to translate and share tacit knowledge or ‘know how,’

which can be a valuable resource for capacity building and

the implementation of evidence-based practices (Wenger

1998; Barwick et al. 2009). CoPs offer new ways of struc-

turing collaboration in response to the challenges of com-

plex systems. With the current emphasis on partnership,

collaboration, and networks, and with the momentum of

the CoP model, new concepts, methodologies, and tech-

niques to understand the mechanisms and the potential

value of these networks are necessary. However, despite the

growing interest in, and implementation of, CoPs, little

agreement exists on approaches to evaluate their influence

(Li et al. 2009b) and limited evaluation of CoPs and

knowledge networks has occured (Li et al. 2009b; Bertone

et al. 2013). As a result, empirical evidence supporting the

effectiveness of CoPs remains limited, and even fewer

analyses investigate the mechanisms that determine effec-

tiveness (Li et al. 2009b).

Communities of practice in ecosystem approaches to

health (CoPEHs) are an application of CoPs within the

field of ecohealth. CoPEHs grew from a desire for

researchers and practitioners to share knowledge and

experience. They were developed with the central purposes

of improving research on specific ecohealth themes and

teaching others how to conduct ecohealth research (Parkes

et al. 2012). Five CoPEHs have emerged around the globe

over the last decade in Latin America and the Caribbean

(CoPEH-LAC), the Middle East and North Africa (CoPEH-

MENA), West Africa (CoPEH-AOC), South and South East

Asia (CoPEH-SSEA), and Canada (CoPEH-Canada) (see

listing and additional information on CoPEHs in Supple-

mental Material). Other partnerships are also emerging

around ecohealth and One Health. Recent additions of

regionalized ecohealth efforts include initiatives in Africa

and Oceania focused on regional conferences with potential

to catalyze ‘regional chapters’ of the International Associ-

ation for Ecology & Health. CoPs and other forms of col-

laborative networks are gaining momentum in the

ecohealth field (Parkes et al. 2012) as they provide oppor-

tunities to define and advance the field of ecohealth

through knowledge creation, enhanced learning, identity

building, and professional development. Although there is

potential for CoP and knowledge network structures to

promote learning and knowledge mobilization, limited

research exists to understand and evaluate how they work.

This review drew from Wenger and colleagues (2002,

2011) in defining CoPs and KNs; however, CoP and KN

terminology is applied differently throughout the literature.

Some authors suggest that the difference between networks

and CoPs is merely a matter of terminology (Creech et al.

2012), and definitions of KNs often align closely to those of

CoPs (Stein et al. 2001; Serrat 2010). However, despite

performing many of the same functions, other authors

distinguish basic differences, such as identity (McDermott

1999; Verburg and Andriessen 2006) or visibility (Botkin

1999). The definitions of CoP and KN used in this review

align with CoPEHs’ focus on capacity building and learn-

ing. Wenger et al.’s (2011) definition of knowledge net-

works highlights linkages that allow information flow,

knowledge creation, and learning leveraged by an individ-

ual. The original goals of CoPEHs—‘‘to foster a decen-

tralized communication mechanism to deliver key services

to its membership’’ (Flynn-Dapaah, 2003)—align with the

concept of knowledge networks. Hence, this review in-

cluded both concepts/terms, thus avoiding exclusion of

relevant evaluation frameworks and capturing an appro-

priate breadth of evaluation frameworks that would be

applicable to a variety of collaborative structures in dif-

ferent fields, including ecohealth CoPs and networks.

Several ecohealth CoPs and networks are currently

engaged in evaluation; CoPEH-LAC and CoPEH-Can are

involved in self-evaluation of their networks, informed by

social network analysis (SNA) and longitudinal qualitative

analysis (Parkes et al. 2012); Spiegel et al. (2011) developed

a framework to guide CoP activities and assess their out-

comes and impacts. Despite efforts to evaluate and

understand CoPs in Ecohealth, many questions remain,

particularly around health and social outcomes of CoPs

(Spiegel et al. 2011). Parkes et al. (2012) highlight evalua-

384 Kaileah A. McKellar et al.

tion summative questions, such as, ‘‘Would we do differ-

ently or not as well in the absence of collaboration?’’

Additionally, addressing developmental questions, such as

‘What factors contribute to successful knowledge ex-

change?’ will promote more effective practice in commu-

nities and networks. These questions can be challenging to

answer as CoPEHs exhibit elements of complex interven-

tions and have been described as dynamic and complex

learning systems (Barragan-Ocana et al. 2012). Evaluation

frameworks can help in answering these, and other, eval-

uation questions. They can guide evaluations by outlining

the approach or design to be applied, and by providing the

scheme or analytical tool for the process of assessment.

Ecohealth community members are calling for more sys-

tematic monitoring of ecohealth research and evaluation of

collaborative structures (Charron 2012; Parkes et al. 2012).

Hence, the purpose of this systematic scoping review

was to understand what frameworks and methods have

been proposed or used to evaluate CoPs and/or knowledge

networks. A scoping review approach (Arksey and O’Mal-

ley 2005; Davis et al. 2009; Levac et al. 2010) was deter-

mined to be appropriate for this review because it allowed

for a summary of the breadth and depth of the evaluation

of CoP and KNs. This review aims to provide practical

information for evaluators and network practitioners

regarding the current state of literature and to increase their

awareness of the frameworks available for adaptation to

their own ecohealth CoP and network evaluations.

METHODS

Overall Approach

As the underlying concept of CoP and/or knowledge net-

work is not well operationalized, the search terms and

databases were chosen to be inclusive of CoPs in multiple

fields and reflective of ecohealth CoPs and networks. Al-

though overlap with other types of networks, such as epi-

stemic communities (Haas 1992), practice-based networks,

and service delivery networks may occur, this review does

not include these other types because they neither fit with

the capacity building nature of CoPs nor the nature and

membership of CoPEHs and ecohealth networks. This re-

view sought to find frameworks from multiple disciplines

to facilitate learning across disciplines. For example,

frameworks for CoPs and/or knowledge networks from

fields outside of ecohealth remain relevant to ecohealth.

Hence, this review included both networks that occur

within the bounds of an organization and those that link

individuals across organizations.

Search Strategy

To identify existing evaluation frameworks for CoPs and

KNs, the review used two strategies to find studies and

reports: (1) searching eight electronic databases from

multiple disciplines and (2) snowball sampling of refer-

ences to, and citations of, relevant articles. The database

search took place in January 2013 and the eight databases—

Scopus, Web of Science, Medline, EMBASE, CINAHL,

ERIC, EconLit, and Business Review—captured frame-

works from a variety of disciplines. The search terms were

based on a preliminary scoping review and on other CoP-

related systematic reviews (Li et al. 2009b; Ranmuthugala

et al. 2011b). The terms included ‘communit* of practice’,

‘knowledge network*’ ‘network* of practice’, ‘situated

learning’, and ‘communit* of interest,’ as well as ‘evalua-

tion framework’, ‘program evaluation’, ‘evaluation meth-

ods’, and ‘logic model’. To capture the complex

intervention nature of CoPs, the review included ‘‘theor* of

change’’ and ‘realist* evaluation*’. In addition to searching

popular databases, the review examined the references and

forward citations of relevant articles. Citing articles were

found using Google Scholar and Scopus, or Web of Science,

if not found in Scopus. The ‘snowball sampling’ occurred

in October 2013.

Two reviewers (KM, KP) independently screened a

sample (4%) of article titles to pilot the screening criteria;

then one reviewer applied the criteria to the entire set

(KM). The title screening stage excluded articles with

clearly unrelated topics (e.g., artificial intelligence, clinical

trials), captured initially due to the broad nature of the

search terms. Two reviewers (KM, KP) then screened three

random samples of 100 articles (totalling 8%) to ensure the

screening criteria were clear and that they were consistently

applied (full inclusion and exclusion criteria are set out in

Supplemental Material). One reviewer (KM) then reviewed

the remaining titles and abstracts. When it was not possible

to exclude based on title and abstract, two of three

reviewers (KM, KP, JY) assessed each full text indepen-

dently. Discrepancies were discussed through to agreement;

if agreement was not achieved during initial discussion, a

third reviewer joined the process. Once the reviewers had

established relevant articles from the database search, one

reviewer (KM) screened titles and abstracts from their

references and citations, to find additional frameworks that

Evaluating Communities of Practice and Knowledge Networks 385

met the inclusion criteria. Each of the selected articles was

full text screened by two reviewers (KM, KP, JY) for rele-

vance.

English language peer-reviewed articles, conference

proceedings, and reports published after 1991 were in-

cluded. Existing reviews related to communities of practice

(Li et al. 2009b; Ranmuthugala et al. 2011b) or evaluation

frameworks (Van Eerd et al. 2011) informed the inclusion

and exclusion criteria. Articles were included based on

relevance—whether the article presented a framework de-

signed to evaluate a CoP or knowledge network. Articles

did not need to use the terms ‘CoP’, or ‘knowledge net-

work’ or ‘evaluation framework’ to describe the model and/

or methods that they presented. The screening process did

not exclude frameworks based on study quality, as the goal

was not to assess quality, but to describe frameworks that

have been proposed.

Data Extraction

Three reviewers (KM, KP, JY) iteratively tested data

extraction templates with three articles. Based on the

consistency of the results and discussions by the reviewers,

modifications for clarity were made, and additional fields

were added after each of the first two articles. One reviewer

then applied the template to the remaining articles, and two

reviewers checked 25% of the completed extractions for

accuracy and consistency. The extraction template included

multiple spreadsheets collecting data about the articles, the

types of networks, specific network information (if appli-

cable), the evaluation frameworks, and the application or

testing of the frameworks (if applicable).

RESULTS

The database search yielded 5348 original articles (Fig. 1).

After the screening process, nine articles were selected from

this set. A further ten articles were located through refer-

ences and snowball searching. The 19 articles included in

the review had 16 unique frameworks that were variously

described as, for example, ‘systematic framework for ana-

lyzing…’, ‘guidelines for assessment’, ‘diagnosis frame-

work’, or ‘conceptual framework’.

The frameworks’ publication dates ranged from 2002

to 2013, and the publications originated from a variety of

countries, including Australia, Canada, Italy, Korea, and

the Netherlands. The majority of frameworks (13 out of 16)

used CoP terminology to describe their units of analysis,

while others used the terms ‘‘knowledge network,’’

‘‘working group,’’ or a combination of these terms. In those

that focused on CoPs, the work of Wenger et al. (2002) was

commonly used or referenced to define CoPs. Most

frameworks (11 out of 16) focused on evaluating CoPs that

were set within organizations; these frameworks came from

management literature, but were not limited to a specific

sub-sector (e.g., manufacturing technology). Other frame-

works came from the fields of health (n = 3), education

(n = 1), international development (n = 1), and interdis-

ciplinary research (n = 1). The majority of the articles

provided limited descriptions of the member populations.

The review assumed that CoPs/KNs members within an

organization were employees; their disciplinary back-

grounds included finance, software development, and

engineering. Researchers, health care practitioners, and

transnational actors in health policy were included as

members of CoPs/KNs that crossed organizational

boundaries.

Purpose and Main Components of the Frameworks

Table 1 outlines the multiple aims and approaches among

the frameworks. Generally, the frameworks served one or

several of the following purposes: to assess CoPs/KNs, to

understand CoPs/KNs, and/or to promote the value of

CoPs/KNs. The majority of the frameworks were focused

on understanding CoPs/KNs rather than on assessing them

against benchmarks or specific goals. Some frameworks

addressed multiple aims; for example, to understand the

goals, strategies, or mechanism of CoPs and to measure

performance. The structures and dimensions of the

frameworks also varied. The majority used categories of

measures as bases for their models, although two studies

also included a framework that outlined the evaluation

process (Ranmuthugala et al. 2011a; Kim et al. 2012). The

right column of Table 1 outlines the main components

present in each framework.

The frameworks varied in extensiveness. Some were

quite limited in their scope while others attempt to trace

pathways between the dimensions that lead to CoP success,

and to provide a series of indicators and suggestions for

data collection and analysis (Meessen and Bertone 2012). In

some cases, authors provided limited methodological

information, making it unclear how one might operation-

alize the framework (Loyarte and Rivera 2007). Other

frameworks were quite specific and offered tools that could


be applied in the study of CoPs and/or knowledge networks

(Verburg and Andriessen 2006; Wenger et al. 2011). Several

of the articles were analysis oriented and provided detailed

information about analysis methods and associated for-

mulae (Chu and Khosla 2009; Lee et al. 2010; Chu et al.

2012).

Dimensions Included in the Frameworks

Table 2 outlines common measurements or considerations

that are part of the frameworks. This overview provides a

sense of the types of evaluation that the frameworks pro-

pose (e.g., process or summative). This analysis explored

whether the frameworks take into consideration the inter-

nal characteristics, the external context, and the process and

outcomes of the group.

This analysis examined whether each framework con-

siders the network’s stage of development (e.g., coalescing,

stewardship) or changes over time and whether it considers

the specific goals or scope of the network or whether

incorporates measures of structure. Three frameworks

considered the stage of the CoP, each with a different ap-

proach. The framework developed by Lee et al. (2010)

provides a method for determining the stage of a CoP (i.e.,

building, growth, adaptive, close), while Zhang et al. (2011)

reported on one network’s progress through various stages.

Finally, Creech et al. (2012) drew attention to the life cycle

of the network and the importance of measuring change

over time. Only four of the frameworks incorporated the

goals or scope of the CoP/KN by including a network-level

exploration of goals/scope (Loyarte and Rivera 2007; Cre-

ech et al. 2012) or measuring progress against network

goals (Lee et al. 2010; Kim et al. 2012). Approximately half

of the frameworks included some measure of the network

structure. Earlier frameworks used descriptive measures of

structure, while frameworks that are more recent suggested

incorporating SNA into the evaluation.

Frameworks were considered to have met this criterion

if they incorporated context as defined by the frameworks’

authors. Interpretations of context varied; for example, the

Verburg and Andriessen (2006) model includes measures of

the basic structural elements of the network (e.g., size of

network, existence of subgroups) as well as organizational

support, while Scarso and colleagues (2008, 2009) consid-

ered context as the host organization’s knowledge strategy

and business environment. Context was most commonly

considered the external environment of the CoP/KN, al-

though several articles noted that the community itself is an

important context when considering the identity of mem-

bers (Verburg and Andriessen 2006; Loyarte and Rivera

2007).

The way in which context was factored into the

framework also varied. The most comprehensive inclusion

of context comes from Grootveld and Helms (2008), who

presented three categories of contextual factors (people,

organization, and system), and their fit in the organiza-

tional culture. In Conklin and Stolee’s (2008; Stolee and

Conklin, n.d.) framework, context is a knowledge exchange

dimension, and they suggested that context should be

measured at each network dimension (i.e., for the network,

network component, and implementation site). In the

frameworks proposed by Verburg and Andriessen (2006),

Scarso and colleagues (2008, 2009), and Barragan-Ocana

Figure 1. Literature search strategy: article identification and

screening for relevance


Tab

le1.

Co

Pev

alu

atio

nfr

amew

ork

sin

chro

no

logi

cal

ord

er

Au

tho

rs(y

ear)

Net

wo

rkty

pe

Sect

or/

sett

ing

Aim

of

fram

ewo

rkD

escr

ipti

on

/mai

nco

mp

on

ents

McD

erm

ott

(200

2)C

oP

Man

agem

ent/

wit

hin

org

aniz

atio

ns

To

mak

eco

nn

ecti

on

sb

etw

een

acti

viti

esan

dre

-

sult

sin

ari

goro

us,

use

ful,

and

con

vin

cin

gw

ay

Fra

mew

ork

incl

ud

es(1

)ac

tivi

ties

;(2

)

ou

tpu

ts(p

erso

nal

kno

wle

dge

,

stre

ngt

ho

fre

lati

on

ship

,an

dac

cess

to

info

rmat

ion

);(3

)va

lue

(in

div

idu

al,

team

/pro

ject

,o

rgan

izat

ion

al);

(4)

bu

sin

ess

resu

lts

Ver

bu

rgan

dA

nd

ries

sen

(200

6)

Co

P

Man

agem

ent/

wit

hin

org

aniz

atio

ns

To

get

syst

emat

icin

sigh

tsin

toth

ech

arac

teri

stic

s

and

per

form

ance

so

fC

oP

san

do

fth

eo

pin

ion

s

of

Co

Pm

emb

ers,

and

top

rovi

de

feed

bac

ko

n

the

per

form

ance

of

Co

Ps

atth

ein

div

idu

al,

gro

up

,an

do

rgan

izat

ion

alle

vel

Fra

mew

ork

stru

ctu

res

item

sin

to(1

)

char

acte

rist

ics

(fo

rms,

goal

s,ro

les)

,

(2)

pro

cess

es(a

ctiv

itie

s,co

mm

un

i-

cati

on

s,co

ord

inat

ion

and

ICT

use

),

and

(3)

ou

tco

mes

(in

div

idu

alre

-

war

ds,

gro

up

vita

lity

and

org

aniz

a-

tio

nal

pro

du

cts)

Lo

yart

ean

dR

iver

a(2

007)

Co

P

Man

agem

ent/

wit

hin

org

aniz

atio

ns

To

anal

yze

the

coh

eren

ceb

etw

een

org

aniz

atio

nal

ob

ject

ives

and

the

cult

ivat

ing

pro

cess

of

Co

Ps

(McD

erm

ott

’s(1

999)

thre

ed

imen

sio

ns)

and

to

anal

yze

the

coh

eren

ceb

etw

een

the

inte

grat

ion

of

Co

Ps

ino

rgan

izat

ion

san

dth

eir

resu

lts

(Wen

ger’

s(1

998)

bas

icco

mp

on

ents

of

Co

Ps)

Fra

mew

ork

incl

ud

es:

(1)

ob

ject

ives

(org

aniz

atio

nal

,in

div

idu

al),

(2)

pro

cess

of

cult

ivat

ion

(in

cl.

the

kin

d

of

kno

wle

dge

the

com

mu

nit

y

shar

es);

the

deg

ree

of

con

nec

tio

nan

d

iden

tity

amo

ng

mem

ber

s,an

dh

ow

clo

sely

inte

grat

edsh

arin

gkn

ow

led

ge

isw

ith

peo

ple

’sev

eryd

ayw

ork

:(3

)

inte

grat

ion

char

acte

rist

ics:

and

(4)

resu

lts

(po

siti

ve,

neg

ativ

e)

Scar

soan

dB

oli

san

i

(200

8),

Scar

soet

al.

(200

9)

Co

P

Man

agem

ent/

wit

hin

org

aniz

atio

ns

To

iden

tify

and

inte

grat

eth

em

ain

dim

ensi

on

s

shap

ing

the

crea

tio

nan

dm

anag

emen

to

fC

oP

s

Fra

mew

ork

incl

ud

esfo

ur

inte

rnal

pil

-

lars

/mai

nd

imen

sio

ns

that

gro

un

da

Co

P:

(1)

org

aniz

atio

nal

,(2

)co

gni-

tive

,(3

)ec

on

om

ic,

(4)

tech

no

logi

cal.

Itin

clu

des

two

exte

rnal

envi

ron

-

men

tal

elem

ents

:b

usi

nes

sco

nte

xt

and

kno

wle

dge

stra

tegy


Tab

le1.

con

tin

ued

Au

tho

rs(y

ear)

Net

wo

rkty

pe

Sect

or/

sett

ing

Aim

of

fram

ewo

rkD

escr

ipti

on

/mai

nco

mp

on

ents

Co

nkl

inan

dSt

ole

e(2

008)

,

Sto

lee

and

Co

nkl

in(n

.d.)

Kn

ow

led

gen

etw

ork

Hea

lth

care

/mu

ltip

leo

rgan

izat

ion

s

To

asse

sskn

ow

led

geex

chan

geac

tivi

tyin

rela

tio

n

toth

ed

imen

sio

ns

of

evid

ence

,co

nte

xt,

faci

li-

tati

on

,an

dre

sult

s,o

nm

ult

iple

scal

eso

fn

et-

wo

rkfu

nct

ion

ing

Fra

mew

ork

isre

pre

sen

ted

by

atw

o-

dim

ensi

on

alru

bri

cw

ith

net

wo

rk

leve

ls(n

etw

ork

wid

e,n

etw

ork

com

-

po

nen

t,im

ple

men

tati

on

site

)an

d

kno

wle

dge

exch

ange

(evi

den

ce,

con

-

text

,fa

cili

tati

on

,re

sult

s)

Gro

otv

eld

and

Hel

ms

(200

8)C

oP

Ind

ust

ry/w

ith

ino

rgan

izat

ion

s

To

iden

tify

con

text

fact

ors

that

infl

uen

ceC

oP

succ

ess,

and

top

rop

ose

ap

reli

min

ary

asse

ss-

men

tto

ol

tod

iagn

ose

Co

Pco

nte

xtfa

cto

rs

wit

hin

ano

rgan

izat

ion

Fra

mew

ork

iden

tifi

es11

con

text

fact

ors

wit

hin

thre

egr

ou

ps:

(1)

peo

ple

(mem

ber

valu

e,le

ader

,tr

ust

,id

en-

tity

)(2

)o

rgan

izat

ion

(man

agem

ent

sup

po

rt,

lau

nch

stra

tegy

,K

Mst

rat-

egy,

rew

ard

syst

em),

and

(3)

syst

em

thro

ugh

wh

ich

kno

wle

dge

shar

ing

occ

urs

(med

iach

oic

e,u

sab

ilit

y).

Fi-

nal

ly,

thes

eo

ccu

rw

ith

inth

eo

rgan

i-

zati

on

’scu

ltu

re,

pre

sen

ted

asan

add

itio

nal

fact

or

Ch

uan

dK

ho

sla,

(200

9),

Ch

uet

al.

(201

2)

Co

P

Ind

ust

ry/w

ith

ino

rgan

izat

ion

s

To

anal

yze

vari

ou

sin

dex

pri

ori

ties

and

stra

tegy

pre

fere

nce

so

fC

oP

sto

un

der

stan

dth

eir

KM

cult

ure

,al

loca

tere

sou

rces

tow

ard

sth

ech

ose

n

bu

sin

ess

stra

tegy

and

mea

sure

KM

per

for-

man

cein

the

con

text

of

the

cho

sen

bu

sin

ess

stra

tegy

Fra

mew

ork

incl

ud

esfo

ur

dim

ensi

on

s:

(1)

locu

so

fle

ader

ship

,(2

)in

cen

tive

mec

han

ism

,(3

)m

emb

erin

tera

ctio

n,

(4)

com

ple

men

tary

asse

t.E

ach

has

fou

rcr

iter

ia(e

.g.,

ho

mo

gen

eity

of

mem

ber

s)w

ith

fou

ras

soci

ated

bu

si-

nes

sst

rate

gies

Lee

etal

.(2

010)

Co

P

Man

agem

ent/

wit

hin

org

aniz

atio

ns

To

asse

ssth

eco

nd

itio

no

fth

eC

oP

ind

irec

tly

and

toes

tab

lish

stag

eo

fm

atu

rity

and

aro

adm

ap

for

mo

vin

gb

etw

een

stag

es

Fra

mew

ork

iden

tifi

esa

six-

step

pro

cess

that

iden

tifi

esth

est

age

atw

hic

hth

e

Co

Pis

and

the

asso

ciat

edcr

itic

al

succ

ess

fact

ors

.In

clu

des

fou

rm

atu

-

rity

leve

ls(b

uil

din

g,gr

ow

th,

adap

-

tive

,an

dcl

ose

),11

crit

ical

succ

ess

fact

ors

,an

d28

corr

esp

on

din

gC

oP

acti

viti

es


Tab

le1.

con

tin

ued

Au

tho

rs(y

ear)

Net

wo

rkty

pe

Sect

or/

sett

ing

Aim

of

fram

ewo

rkD

escr

ipti

on

/mai

nco

mp

on

ents

Ran

mu

thu

gala

etal

.(2

011a

,b

)C

oP

Hea

lth

care

/mu

ltip

lese

ttin

gs

To

iden

tify

con

text

-mec

han

ism

-ou

tco

me

con

fig-

ura

tio

ns

that

wil

lex

pla

inth

ero

leo

fC

oP

sin

imp

rovi

ng

hea

lth

care

pra

ctic

e

Stu

dy

pro

toco

lfo

ra

real

ist

eval

uat

ion

ou

tlin

ein

fou

rst

ages

(1)

theo

ry—

CM

Oco

nfi

gura

tio

ns

gen

erat

ion

,(2

)

hyp

oth

eses

gen

erat

ion

/ref

ram

ing,

(3)

ob

serv

atio

n—

hyp

oth

eses

test

ing,

w/

surv

ey,

(4)

pro

gram

spec

ifica

tio

n—

revi

ewth

eori

esin

ligh

to

ffi

nd

ings

Wen

ger

etal

.(2

011)

Co

P

Ed

uca

tio

n/w

ith

ino

rgan

izat

ion

s

To

pro

vid

eth

efo

un

dat

ion

for

anev

alu

atio

n

pro

cess

that

can

inte

grat

eh

eter

oge

neo

us

sou

rces

and

typ

eso

fd

ata

tocr

eate

aco

mp

elli

ng

pic

ture

of

ho

wC

oP

san

dn

etw

ork

scr

eate

valu

e

for

thei

rm

emb

ers,

for

ho

stin

go

rgan

izat

ion

s,

and

for

spo

nso

rs.

Fra

mew

ork

incl

ud

esb

oth

ase

to

fre

le-

van

tin

dic

ato

rsfo

rd

ata

coll

ecti

on

and

ap

roce

ssfo

rin

tegr

atin

gth

ese

ind

icat

ors

into

am

ean

ingf

ul

acco

un

t

of

valu

ecr

eati

on

.It

incl

ud

es5

cycl

es:

(1)

imm

edia

teva

lue—

acti

viti

esan

d

inte

ract

ion

s,(2

)p

ote

nti

alva

lue—

kno

wle

dge

cap

ital

,(3

)ap

pli

edva

-

lue—

chan

ges

inp

ract

ice,

(4)

real

ized

valu

e—p

erfo

rman

ceim

pro

vem

ent,

and

(5)

refr

amin

gva

lue—

red

efin

ing

succ

ess

Zh

ang

etal

.(2

011)

Wo

rkin

ggr

ou

p

Inte

rdis

cip

lin

ary

rese

arch

/mu

lti-

na-

tio

nal

To

dem

on

stra

tep

rogr

ess

of

net

wo

rks

and

exam

ine

chal

len

ges

and

mec

han

ism

sto

add

ress

chal

len

ges

Fra

mew

ork

incl

ud

esst

ruct

ure

(net

wo

rk

size

,d

ensi

ty,

cen

tral

izat

ion

)an

d

pro

cess

des

crip

tio

nb

ased

on

stag

es

of

dev

elo

pm

ent

(in

itia

tin

g,fo

rmu

la-

tio

nag

reem

ent,

adva

nci

ng

rese

arch

,

sust

ain

ing

rese

arch

gro

up

)

Bar

raga

n-O

can

aet

al.

(201

2)C

oP

Ind

ust

ry/w

ith

ino

rgan

izat

ion

s

To

exp

lore

stru

ctu

ral

mec

han

ism

sin

use

wit

hth

e

Co

Ps,

par

ticu

larl

yth

ose

that

guar

ante

eo

rga-

niz

atio

nal

viab

ilit

yth

rou

ghth

efu

nct

ion

they

carr

yo

ut

Use

sa

viab

lesy

stem

mo

del

focu

sed

on

org

aniz

atio

ns

inte

ract

ing

wit

hth

eir

envi

ron

men

t.In

tera

ctio

ns

incl

ud

ean

op

erat

ive

elem

ent,

wh

ich

carr

ies

ou

t

the

org

aniz

atio

n’s

fun

dam

enta

l

op

erat

ion

s,an

da

met

a-sy

stem

,

wh

ich

guar

ante

esth

eo

rgan

izat

ion

’s

fun

ctio

ns

asa

‘wh

ole

’


Tab

le1.

con

tin

ued

Au

tho

rs(y

ear)

Net

wo

rkty

pe

Sect

or/

sett

ing

Aim

of

fram

ewo

rkD

escr

ipti

on

/mai

nco

mp

on

ents

Cre

ech

etal

.(2

012)

Net

wo

rks

and

/or

Co

Ps

Inte

rnat

ion

ald

evel

op

men

t

To

stre

ngt

hen

per

form

ance

asse

ssm

ent

and

imp

rove

men

to

fn

etw

ork

s,an

dto

iden

tify

star

tin

gp

oin

tsin

layi

ng

ou

ta

mo

red

etai

led

rese

arch

agen

da

for

futu

rew

ork

Fra

mew

ork

incl

ud

esfo

ur

po

ints

(1)

focu

san

dex

ten

sive

nes

so

fth

en

et-

wo

rk,

(2)

evo

luti

on

of

the

stru

ctu

re

of

the

net

wo

rko

ver

tim

e(3

)so

cial

cap

ital

wit

hin

the

net

wo

rk,

and

(4)

acti

viti

esan

do

utc

om

eso

fth

en

et-

wo

rk,a

nd

the

shar

edva

lue

crea

ted

by

the

net

wo

rk

Kim

etal

.(2

012)

Co

P

Man

agem

ent/

wit

hin

org

aniz

atio

ns

To

iden

tify

curr

ent

kno

wle

dge

-sh

arin

gac

tivi

ty

stat

us

ina

Co

P

Fra

mew

ork

ou

tlin

esse

vera

lco

nse

cuti

ve

stag

eso

f(1

)le

arn

ing

abo

ut

the

net

-

wo

rk(S

NA

):cl

assi

fyin

gm

emb

ers

(bal

ance

dp

laye

r,eg

ois

tic

pro

pag

ato

r,

ego

isti

cre

ceiv

eran

dkn

ow

led

geis

o-

lato

r),

(2)

clas

sify

ing

the

net

wo

rk

(act

ive

com

mu

nit

y,sp

read

ing

com

-

mu

nit

y,le

arn

ing

com

mu

nit

y,in

ac-

tive

com

mu

nit

y),

and

then

(3)

alig

nin

git

wit

ha

set

of

Co

Pst

rate

gies

(e.g

.,in

crea

seaw

aren

ess

of

mem

ber

s’

kno

wle

dge

)

Mee

ssen

and

Ber

ton

e(2

012)

Co

P

Hea

lth

po

licy

/tra

nsn

atio

nal

To

anal

yze

the

role

that

kno

wle

dge

(wh

ich

is

pro

du

ced

,cr

eate

dan

dm

anag

edth

rou

ghth

e

Co

Ps)

pla

ysin

the

pro

cess

of

sele

ctin

gan

d

imp

lem

enti

ng

hea

lth

po

lici

es,

wh

ich

intu

rn,

may

hav

eim

pac

tso

nh

ealt

ho

utc

om

esva

lued

by

citi

zen

s

Fra

mew

ork

retr

aces

the

cau

sali

typ

ath

bet

wee

na

seri

eso

fd

imen

sio

ns

that

mem

ber

sac

tiva

teto

ensu

reth

eC

oP

fun

ctio

ns

wel

l.T

he

six

dim

ensi

on

s

(ava

ilab

lere

sou

rces

,st

rate

gies

to

mo

bil

ize

reso

urc

es,

kno

wle

dge

man

-

agem

ent

pro

cess

es,

exp

ansi

on

of

kno

wle

dge

,kn

ow

led

ge-b

ased

po

lici

es

and

pra

ctic

es,

bet

ter

hea

lth

and

wel

-

fare

ou

tco

mes

)co

nta

ins

elem

ents

and

/or

sub

-ele

men

tsan

dan

asso

ci-

ated

set

of

ind

icat

ors


et al. (2012), context/environment is one of three or four

major components of their model. Loyarte and Rivera

(2007), Chu and colleagues (2009, 2012), and Wenger et al.

(2011) gave mention to the importance of context, but do

not state specific measures.

Understanding the processes and activities of a net-

work is a key step in understanding the factors that make it

effective and the ways through which it achieves outcomes.

Frameworks that propose documentation of a CoP/KNs

activities or their process of development were included.

Most frameworks measured some dimension of network

activity. Several of these frameworks (Scarso and Bolisani

2008; Ranmuthugala et al. 2011a; Meessen and Bertone

2012) focused on the mechanisms that create effective

networks. Eight of the frameworks included measures of

CoP/KN outcomes. Of the frameworks that included out-

comes, most focused only on the positive effects of net-

works; two frameworks (Loyarte and Rivera 2007; Scarso

and Bolisani 2008) included both positive outcomes and

negative outcomes or costs of the networks. The level of

impact refers to whether outcomes are measured by mul-

tiple scales; seven of the frameworks do. Four focused on

individual levels and group and/or organizational levels. Set

in the healthcare environment, Conklin and Stolee (2008)

and Stolee and Conklin (n.d.) included outcomes at the

implementation site, at the network-component (CoP) le-

vel and at the wider network level. Wenger et al. (2011) had

a different interpretation of level of outcome; they include

levels of immediate value, potential value, applied value,

and realized value. Creech et al. (2012) suggested that

assessment took into consideration the level of value cre-

ation outlined by Wenger et al. (2011).

Application/Testing of Evaluation Frameworks

Nine of the studies applied or tested their frameworks in

some form (Table 3). Seven of these frameworks were fo-

cused on CoPs within organizations; two were focused on

groups that crossed organizational boundaries. The level of

reporting with respect to the application or testing process

was varied. What was learned about CoPs/KNs were more

commonly reported than what was learned about the

framework or the evaluation process. Only two of the

articles reported any modifications to the frameworks after

they were piloted (Verburg and Andriessen 2006; Grootveld

and Helms 2008). The majority of the articles framed the

testing or application as a case study. Three of the studies

tested or pilot tested their frameworks; this was possibleTab

le1.

con

tin

ued

Au

tho

rs(y

ear)

Net

wo

rkty

pe

Sect

or/

sett

ing

Aim

of

fram

ewo

rkD

escr

ipti

on

/mai

nco

mp

on

ents

Yap

and

Ro

bb

en(2

012)

Co

P

Ind

ust

ry/w

ith

ino

rgan

izat

ion

s

To

asse

ssth

ead

ded

valu

eo

fn

etw

ork

edle

arn

ing

thro

ugh

on

lin

eco

mm

un

itie

san

dn

etw

ork

sfo

r

bu

sin

esse

s

Th

em

od

elli

nks

retu

rn-o

n-i

nve

stm

ent

for

net

wo

rks

of

pra

ctic

ete

chn

olo

gy

toth

ere

turn

-on

-in

vest

men

tfo

rth

e

reso

urc

esan

dti

me.

Spec

ific

com

po

-

nen

tsar

eo

nli

ne

dia

log

(so

cial

-cap

i-

tal,

lear

nin

g);

nar

rati

vecy

cles

of

mea

sure

men

t(i

mm

edia

te,

po

ten

tial

,

app

lied

,re

aliz

ed,

and

refr

amin

g);

and

mea

sure

so

fle

arn

ing

(sh

are-

ho

lder

valu

e,cu

sto

mer

enga

gem

ent,

and

emp

loye

eex

per

ien

ce)


because each framework was, or incorporated, an analytical

tool. Two of the case study articles provided limited

information about how the framework was applied, making

it difficult to understand how the framework and evalua-

tion process could be applied or transferred to another

setting (Loyarte and Rivera 2007; Scarso et al. 2009).

Many of the frameworks provided general guides and

approaches to evaluate CoPs/KNs; therefore, determining

their test validities was not relevant. Four of the frame-

works provided quantitative analytical tools; only Verburg

and Andriessen (2006) reported on the process (i.e., per-

forming scale analysis and reliability checks) and the results

of determining the tools’ validity. One study made claims

about the reliability of the framework without demon-

strating how this was determined (Chu et al. 2012). The

majority of studies suggested that their frameworks would

benefit from additional empirical investigation to validate

and refine them.

Articles also discussed the generalizability of the

frameworks, making suggestions about where else and how

the framework could be used; however, due to lack of

testing, there was limited evidence of their applicability.

Four articles suggested that the frameworks were applicable

outside of the contexts in which they were tested. For

example, Kim et al. (2012) suggested that, although their

framework focused on CoP activities, the lessons learned

would be applicable to any knowledge-sharing activities

within an organization. Five of the articles concluded that

further application of the frameworks would be needed to

understand their applicability to other settings. Some of the

frameworks’ authors highlighted the need to test within

different types of communities or settings.

All articles concluded that the frameworks were useful

in assessing or understanding dimensions of CoP/KNs.

While many authors acknowledged the limitations of their

studies in demonstrating validity, several studies high-

Table 2. Overview of CoP evaluation framework components

Authors (year) Goals/scope Stage/time Context Structure Process/activities Outcomes Level of impact

McDermott (2002) 4 4

Verburg and Andriessen (2006) 4 4 4 4

Loyarte and Rivera (2007) 4 4 4 4 (+/-)

Scarso and Bolisani (2008),

Scarso et al. (2009)

4 4 4 4 (+/-)

Conklin and Stolee (2008), Stolee and

Conklin (n.d.)

4 4 4 4

Grootveld and Helms (2008) 4

Chu and Khosla (2009),

Chu et al. (2012)

4 4

Lee et al. (2010) 4 4

Ranmuthugala et al. (2011a) 4 4 (SNA)

Wenger et al. (2011) 4 4 (SNA) 4 4 4

Zhang et al. (2011) 4 4 (SNA) 4

Barragan-Ocana et al. (2012) 4 4 4

Creech et al. (2012) 4 4 4 (SNA) 4 4 4

Kim et al. (2012) 4 4(SNA)

Meessen and Bertone (2012) 4 4 4

Yap and Robben (2012) 4 4 4

Stage/time: considered the stage of development of the network (e.g., coalescing, stewardship), or changes in the network over time. Goals/scope: considered

the specific goals or scope of the network. Context: incorporated context in the framework (context as defined by authors). Structure: incorporated measures

of structure in the framework (e.g., size of network). SNA: used social network analysis to measure the structural dimension of the network. Process/activities:

included measures of network activities or the processes of development. Outcomes: incorporated measures of outcomes in the framework. -/+: considered

both positive and negative outcomes and/or costs of the network. Level of impact: incorporated outcome measures and different scales (e.g., individual,

network, organization)


Tab

le3.

Eva

luat

ion

fram

ewo

rkap

pli

cati

on

/tes

tin

gap

pro

ach

es

Au

tho

rs(y

ear)

Ho

wfr

amew

ork

was

app

lied

or

test

ed

Mem

ber

s’d

isci

pli

nar

yb

ackg

rou

nd

Met

ho

ds

Th

efr

amew

ork

s’au

tho

rs‘

con

clu

sio

nfr

om

app

lica

tio

n

Ver

bu

rgan

dA

nd

ries

sen

(200

6)

Pil

ot

test

edto

ol

on

7C

oP

sw

ith

ina

larg

eo

rgan

izat

ion

Soft

war

ed

evel

op

men

t

Dat

aco

llec

tio

nco

nsi

sted

of

thre

ep

arts

:(1

)

mem

ber

s-o

nli

ne

qu

esti

on

nai

reo

fm

emb

ers)

;

(2)

coo

rdin

ato

r-in

terv

iew

wit

hC

oP

coo

rdin

a-

tor;

(3)

con

text

-ch

eckl

ist

for

hig

hle

vel

key

info

rman

tsfr

om

org

aniz

atio

n

Refi

nem

ents

mad

eto

item

sin

the

too

l.T

he

asse

ssm

ent

too

lp

rovi

ded

au

sefu

l

met

ho

dfo

rq

uan

tita

tive

lyan

alyz

ing

Co

P

effe

ctiv

enes

s

Lo

yart

ean

dR

iver

a(2

007)

Use

dfr

amew

ork

tost

ud

yth

eex

per

i-

ence

so

f15

org

aniz

atio

ns

cult

ivat

ing

Co

Ps

Mu

ltip

led

isci

pli

nes

(e.g

.,b

iosc

ien

ce,

edu

cati

on

,fi

nan

ce)

Do

cum

ent

revi

ewan

do

bse

rvat

ion

wer

eu

sed

to

coll

ect

info

rmat

ion

.A

uth

ors

con

tras

ted

theo

ry

of

Co

Ps

wit

hex

per

ien

ces,

and

anal

yzed

chal

-

len

ges

inb

uil

din

gC

oP

s

Th

eau

tho

rsd

rew

no

con

clu

sio

ns

abo

ut

the

fram

ewo

rkit

self

.B

ased

on

the

re-

sult

so

fth

est

ud

y,th

eycr

eate

da

cult

i-

vati

on

mo

del

togu

ide

thin

kin

gw

ith

in

org

aniz

atio

ns

Scar

soet

al.

(200

9)C

ase

stu

dy

of

Co

Ps

wit

hin

ala

rge

mu

ltin

atio

nal

org

aniz

atio

n

Exp

lora

tio

nan

dp

rod

uct

ion

ino

il

com

pan

y

Qu

alit

ativ

ean

dex

plo

rato

ryca

sest

ud

yw

ith

dir

ect

invo

lvem

ent

of

staf

f

Th

ere

sult

sw

ere

enco

ura

gin

gfo

rth

e

fram

ewo

rks

use

fuln

ess

for

aso

un

dan

d

syst

emat

icin

vest

igat

ion

of

the

fun

c-

tio

nin

gan

dcr

itic

alsu

cces

sfa

cto

rso

f

exis

tin

gC

oP

s.H

ow

ever

,fu

rth

erin

ves-

tiga

tio

nis

nee

ded

tore

fin

ean

dva

lid

ate

the

fram

ewo

rk

Co

nkl

inan

dSt

ole

e(2

008)

Cas

est

ud

yo

fa

kno

wle

dge

exch

ange

acti

vity

that

incl

ud

eda

Co

P

Hea

lth

care

(sen

iors

’h

ealt

h)

Co

llec

tio

nm

eth

od

sin

clu

de

do

cum

ent

revi

ew,

tele

ph

on

ein

terv

iew

wit

hkn

ow

led

geb

roke

r,

and

emai

lsu

rvey

of

acti

vity

par

tici

pan

ts

Th

eau

tho

rsco

ncl

ud

edu

sin

gel

emen

tso

f

the

PA

RiH

Sfr

amew

ork

(Kit

son

etal

.

1998

)an

dco

nsi

der

ing

pro

cess

and

im-

pac

tsin

term

so

fle

vel

of

op

erat

ion

tob

e

valu

able

.A

dd

itio

nal

test

ing

and

revi

ew

wo

uld

ben

efit

the

eval

uat

ion

mo

del

Gro

otv

eld

and

Hel

ms

(200

8)

Cas

est

ud

yo

fC

oP

sw

ith

ina

larg

e

mu

ltin

atio

nal

org

aniz

atio

n

En

gin

eeri

ng

Sem

i-st

ruct

ure

din

terv

iew

wit

hC

oP

mem

ber

s,

each

fact

or

fro

mfr

amew

ork

info

rmed

an

inte

rvie

wq

ues

tio

n

Th

efr

amew

ork

cou

ldb

eu

sed

toas

sess

the

con

text

ual

fact

ors

that

con

trib

ute

to

Co

Psu

cces

sw

ith

inan

org

aniz

atio

n.

Fra

mew

ork

was

use

ful

inp

ract

ice,

bu

ta

sin

gle

case

was

no

tsu

ffici

ent

tova

lid

ate


Tab

le3.

con

tin

ued

Au

tho

rs(y

ear)

Ho

wfr

amew

ork

was

app

lied

or

test

ed

Mem

ber

s’d

isci

pli

nar

yb

ackg

rou

nd

Met

ho

ds

Th

efr

amew

ork

s’au

tho

rs‘

con

clu

sio

nfr

om

app

lica

tio

n

Ch

uan

dK

ho

sla

(200

9),

Ch

uet

al.

2012

Sin

gle

(200

9)an

dco

mp

arat

ive

(201

2)

case

stu

dy

of

Co

Ps

wit

hin

larg

e

org

aniz

atio

ns

Tec

hn

olo

gyre

sear

ch

Qu

esti

on

nai

red

istr

ibu

ted

tosp

ecifi

cu

nit

sw

ith

in

two

org

aniz

atio

ns

Th

ere

sult

sge

ner

ated

wer

eco

nsi

sten

tw

ith

goal

san

dst

rate

gies

of

bo

thth

esu

rvey

ed

org

aniz

atio

ns.

Th

ism

od

elw

asth

eref

ore

‘‘re

liab

le,

pra

ctic

al,

and

suit

able

for

ado

pti

on

by

mu

ltin

atio

nal

kno

wle

dge

-

bas

edo

rgan

izat

ion

sin

gen

eral

.’’

Lee

etal

.(2

010)

Pil

ote

dm

od

elw

ith

thre

eC

oP

sw

ith

in

on

eco

mp

any

Fin

ance

Co

Pm

emb

ersu

rvey

toid

enti

fycu

rren

tst

atu

s

amo

ng

pre

-defi

ned

mat

uri

tyst

ages

and

an

anal

ytic

alh

iera

rch

yp

roce

ssto

det

erm

ine

rela

-

tive

imp

ort

ance

of

crit

ical

succ

ess

fact

ors

Fra

mew

ork

is‘‘

ho

list

ic,

syst

emat

ic,

com

-

pre

hen

sive

wit

hm

atu

rity

stag

esfo

r

nav

igat

ing

Co

Ps’

’h

ow

ever

veri

fica

tio

n

thro

ugh

add

itio

nal

app

lica

tio

nis

nee

ded

Zh

ang

etal

.(2

011)

Sin

gle

case

stu

dy

wit

hSN

A(e

valu

atio

n

bu

ilt

aro

un

dsp

ecifi

cn

eed

so

fn

et-

wo

rk)

Mu

ltip

led

isci

pli

nes

rela

ted

tod

igit

al

gove

rnm

ent

rese

arch

Self

-ad

min

iste

red

soci

alm

atri

xq

ues

tio

nn

aire

,

and

syst

emat

ico

bse

rvat

ion

san

dre

flec

tio

ns

of

the

exp

erie

nce

mad

eb

yth

ete

amm

emb

ers

Th

est

ud

yw

asab

leto

iden

tify

chal

len

ges

and

key

enab

lin

gfa

cto

rso

fth

eev

alu

ated

gro

up

;h

ow

ever

,th

ege

ner

aliz

abil

ity

wil

l

req

uir

ead

dit

ion

alca

sest

ud

ies

fro

m

vari

ou

sse

ttin

gs

Kim

etal

.(2

012)

Ap

pli

edm

eth

od

olo

gyto

59C

oP

sin

a

com

pan

yw

ell

kno

wn

for

kno

wle

dge

man

agem

ent

Man

ufa

ctu

rin

g

Tra

cked

on

lin

elo

go

fC

oP

acti

vity

(po

sts

to

web

site

),u

sed

SNA

Th

efr

amew

ork

cou

ldb

eap

pli

edin

oth

er

org

aniz

atio

ns

and

the

less

on

sle

arn

ed

fro

mre

sult

sco

uld

be

use

ful

toan

alyz

e

kno

wle

dge

-sh

arin

go

rkn

ow

led

getr

ans-

fer

acti

viti

esin

any

area

of

ano

rgan

i-

zati

on


lighted the utility of the frameworks when drawing con-

clusions about their value. Authors provided examples of

how their case studies were able to inform strategic action

in the CoPs (Grootveld and Helms 2008) or shared the

results that came from using their frameworks; for example,

after implementing framework-based recommendations,

CoPs became more productive in their contributions to

online forums (Kim et al. 2012).

DISCUSSION

The body of literature pertaining to CoPs is growing;

however, relatively few studies examine the evaluation of

these structures. This study demonstrates the variety of

purposes, scopes, and methodological approaches taken in

evaluating CoPs and/or KNs. A key observation is that,

despite finding approximately half of the frameworks

through references and citations, there have been limited

adaptations and adoptions from previous studies. Fur-

thermore, reports on applications of the frameworks have

been restricted to the framework developers.

Addressing Common Challenges in Evaluation

CoPs can, and CoPEHs do, exhibit elements of a complex

intervention, bringing about certain evaluation challenges.

A complex intervention has characteristics of nonlinearity,

emergence, adaptation, uncertainty, dynamic interactions,

and co-evolution (Patton 2011). Evaluators often catego-

rize interventions as simple, complicated, complex, or

chaotic (Rogers 2008; Patton 2011). There are common

challenges in evaluating CoPEHs and other complex

interventions: the emergent nature of activities and out-

comes, multiple scales of impact, long timelines of impact,

hard-to-measure/intangible outcomes, and challenges of

attribution/contribution. The evaluation frameworks re-

viewed here dealt with the challenges in different ways, with

some authors incorporating flexibility and suggesting

exploratory approaches.

CoPs are dynamic structures, which poses challenges

for evaluation. Networks change, and with new members

taking part, the foci of the CoPs can shift, not only in terms

of the practices, but also in terms of forms of engagement

and the structures of the networks themself. Similarly,

CoPEHs have changing membership and shifting priorities.

Some of the frameworks reviewed are likely too rigid to fit

the dynamic nature of the CoPEHs. More adaptable

frameworks could support evaluations of emergent net-

works (e.g., Conklin and Stolee 2008; Scarso and Bolisani

2008; Creech et al. 2012; Meessen and Bertone 2012).

Theory-based approaches and realist evaluations (Ranmu-

thugala et al. 2011a) would also be appropriate. It is nearly

impossible to isolate the effects of a CoP, and to determine

causality for a specific program. Designs solely focused on

attribution of CoP effects or employing counterfactuals are

not suitable for evaluating CoPs. More plausible are pre-

post designs, such as that suggested by McDermott (2002),

to understand the value of CoPs within organizations.

CoPs can foster conditions for innovation (Lesser and

Storck 2001; Verburg and Andriessen 2006; Chu and

Khosla 2009). Innovation, however, is difficult to measure

given its tendency to occur in quantum leaps (Perrin 2002).

Trust, social capital, communication of tacit knowledge,

and learning are other examples of important CoP out-

comes that are challenging to measure. While some

frameworks do not address the intangible or hard-to-

measure aspects of CoPs, Wenger et al. (2011) suggested

using member narratives. Another challenge for evaluation

is the multiple scales on which CoPs operate and can expect

to see outcomes. Several of the frameworks did incorporate

measures at multiple levels of impact, for example,

exploring value at the individual member level, the network

level, and the organizational level. Nevertheless, the Co-

PEHs work at additional scales and multiple timelines of

impact. Some outcomes will occur in the short-term, such

as connections developed among researchers, whereas

ecological, health, and/or social impacts are likely to hap-

pen in the longer term.

Application to Ecohealth CoPs and Networks

Multiple forms of information from the frameworks can

provide guidance for the evaluation of Ecohealth CoPs and

Networks. Several frameworks proposed detailed sets of

indicators (Wenger et al. 2011; Meessen and Bertone 2012),

which might be helpful for ecohealth researchers and

practitioners evaluating their CoPs, particularly for

understanding value creation and influence on policy and

practice. Verburg and Andriessen (2006) and Wenger et al.

(2011) provide tools that with some adaption could guide

data collection. Other frameworks are less detailed, but

highlight key considerations that would be useful in eval-

uating CoPs; for example, Creech suggests the importance

of CoP goals and stage of development. In addition to the

guidance for CoP and KN assessments provided by the


frameworks, evaluators can learn from the results of the

evaluations. Several of the articles and reports included

potential mechanisms of how CoPs and/or KNs work and

some suggested critical success factors that could inform

the development of a CoP. The level of evidence supporting

the mechanisms and success factors varies, in some cases,

information was based on case studies, in others a list was

generated from a literature review. One article did not

specify the source or methods used in determining success

factors. Examples of success factors include financial re-

source, developing trust, opportunities for face-to-face

meeting, and self-selected membership (as cited by Zhang

et al. 2011; Ranmuthugala et al. 2011a). Keeping in mind

that mechanism can produce different results depending on

the context, a realist informed review would be useful in

understanding which are applicable to CoPs and networks

working within the field of ecohealth.

It is worth highlighting several frameworks that would

be most relevant to Ecohealth CoPs and networks. As

mentioned previously, Verburg and Andriessen (2006)

provide a tool and member questionnaire that could be

useful in collecting relevant information with respect to

CoP activities and value. Given that knowledge-to-action is

one of the principles of Ecohealth, the Conklin and Stolee

(2008) framework includes relevant structure for evalua-

tion questions. Furthermore, the multiple levels of analysis

could accommodate the nodal structure and heterogeneous

membership of the CoPEHs. Exploring impacts at the

implementation site acknowledges that knowledge flow

within a network can have impacts beyond the members or

their organization. Attention to context is stressed by the

Grootveld and Helms (2008) framework, though not all

contextual factors would be relevant to CoPs and KNs

working within the field of ecohealth due to the within

organization context. While the framework by Chu and

colleagues (2009, 2012) is situated in a business context, the

focus on strategies for CoP improvement could inform an

approach to evaluate Ecohealth CoPs and networks. The

Wenger et al. (2011) framework would be particularly

useful in understanding the value created for members and

could be adapted to include value-creation outside of the

CoP/KN, thereby promoting a greater understanding of the

social, health, and environmental impacts. Meessen and

Bertone’s (2012) knowledge-focused framework brings in

potentially relevant long-term goals of influencing knowl-

edge-based policy decisions and practices and better health

outcomes.

While the reviewed frameworks provide useful starting

places, the specific needs of ecohealth evaluations cannot be

met by simply applying one of the frameworks. For

example, the members of the CoPEHs hold a number of

roles; these include researcher, trainee/student, ecohealth

practitioner, and funder. Many of the frameworks in this

review did not include approaches for heterogeneous

membership. This is not surprising as homogeneity has

been used as a characteristic to describe CoPs (Fischer

2001), and many of the frameworks came from within

organizations with less heterogeneity. Notably, transdis-

ciplinarity was not broached in the reviewed frameworks.

One would therefore need to look elsewhere to answer

questions regarding the measurement of transdisciplinarity

outcomes, such as the cross-fertilization of ideas.

Acknowledging the diversity of CoPs and KNs implies

acknowledging a required equal diversity in approaches to

their evaluation. While some of the frameworks reviewed

here provided enough detail (including data collection

tools), it would be inappropriate to apply any tool without

considering community type (e.g., as outlined by Dube

et al. (2006) in their typology of virtual CoPs), the learning

needs of the community, the key evaluation questions, and

the desired application of evaluation processes and results.

Future research should explore how evaluation frame-

works, methods, and approaches can complement partic-

ular types of CoPs and their objectives, thereby informing

insights on how best to evaluate a particular CoP or KN.

An exploration that matches frameworks to CoPs could be

useful in answering questions such as, ‘Are there key out-

comes that should be measured in the early stages (e.g.,

relationship building) that are less relevant after a CoP is

well established?’ or ‘What approaches are best for dis-

persed multinational, multi-linguistic networks, such as

several of the CoPEHs?’

Strengths and Limitations

This review builds on existing literature and can inform

practices of the CoPEHs, as unique CoPs. A recent article

by Bertone et al. (2013) reports findings from a scoping

review of frameworks for assessing CoP performance that

would inform a transnational health policy CoP evaluation

model. The reviews differ in the methods used; Bertone

et al.’s search methods were not systematic or exhaustive,

they based their criteria for inclusion on relevance to a

research question that was defined post hoc. In addition to


assessment frameworks, Bertone et al. (2013) also included

literature on limitations and success factors of CoPs. While

there is some overlap with frameworks in this review, our

review examined a broader scope of evaluation frameworks

from multiple disciplines and explicitly dealt with testing of

evaluation frameworks. Two other systematic reviews of

CoP-related topics exist. Li et al. (2009b) examined the

definition and use of CoPs in the health and business

sectors and the effectiveness of CoPs in the health care

sectors; however, the authors were not able to find studies

that met their criteria for determining CoP effectiveness.

Ranmuthugala et al. (2011b) looked at how and why CoPs

are established in the health care sector; however, this re-

view did not focus on evaluation, but did conclude with a

call for use of evaluation to assess the role of CoPs in health

care.

The strengths of this review are in the systematic nat-

ure of the search, data extraction, and assessment of the

utility of the results as a starting point for future evalua-

tions of CoPs in ecosystem health. This review is limited, as

not all potentially useful frameworks or measures could be

included. Lessons could be gained from frameworks that

focus on networks of organizations rather than individuals

(Creech and Ramji 2004) or that focus on other forms of

collaborative partnerships, such as research networks

(Fenton et al. 2007). Furthermore, books are also potential

sources of guidance in evaluating CoPs and KNs; for

example, Stein et al. (2001) highlight important evaluation

questions for knowledge networks to consider.

Conclusion

This scoping review synthesized current evidence that can

inform the evaluation practices of any CoPs or knowledge

networks. The systematic scoping review methodology al-

lowed for a breadth of framework exploration. The review

provides a base of conceptual frameworks, data collection

tools, indicators, analytical approaches, and key consider-

ations. Although application of the frameworks to the

COPEHs will vary, one can glean insights from each. With

the current emphasis on networked knowledge exchange

and capacity building, it is important to understand the

potential costs and benefits of CoPEHs as learning net-

works, and the ways in which benefits can be maximized.

By understanding the value of CoPs and the ways in which

they work, ecohealth researchers and practitioners will be

able to improve knowledge exchange and mutual learning,

towards the shared goal of societal benefit.

REFERENCES

Arksey H, O’Malley L (2005) Scoping studies: towards a meth-odological framework. International journal of social researchmethodology 8:19–32

Barragan-Ocana A, Quijano-Solıs A, Vega-Dıaz G, Sanchez-Lara B(2012) Communities of practice: A focus from complex systems.International Journal of General Systems 41:741–755

Barwick MA, Peters J, Boydell K (2009) Getting to uptake: Docommunities of practice support the implementation of evi-dence-based practice? Journal of the Canadian Academy of Childand Adolescent Psychiatry 18:16

Bertone MP, Meessen B, Clarysse G, Hercot D, Kelley A, KafandoY, Lange I, Pfaffmann J, Ridde V, Sieleunou I (2013) Assessingcommunities of practice in health policy: A conceptual frame-work as a first step towards empirical research. Health ResearchPolicy and Systems 11:39

Botkin JW (1999) Smart business: How knowledge communities canrevolutionize your company, New York: The Free Press

Charron DF (2012) Ecohealth Research in Practice: InnovativeApplications of an Ecosystem Approach to Health, Insight andInnovation in Development, New York: Springer and Ottawa,Canada: International Development Research Centre

Chu M-T, Khosla R (2009) Index evaluations and businessstrategies on communities of practice. Expert Systems withApplications 36:1549–1558

Chu M-T, Khosla R, Nishida T (2012) Communities of practicemodel driven knowledge management in multinational knowl-edge based enterprises. Journal of Intelligent Manufacturing23:1707–1720

Conklin J, Stolee P (2008) A model for evaluating knowledgeexchange in a network context. CJNR (Canadian Journal ofNursing Research) 40:116–124

Cox A (2005) What are communities of practice? A comparativereview of four seminal works Journal of Information Science31:527–540

Creech H, Ramji A (2004) Knowledge networks: Guidelines forassessment. International Institute for Sustainable Develop-ment. http://www.iisd.org/pdf/2004/networks_guidelines_for_assessment.pdf. Accessed 7 Oct, 2013.

Creech H, Laurie M, Paas L, Parry J-E (2012) Performanceimprovement and assessment of collaboration: Starting pointsfor networks and communities of practice. International Insti-tute for Sustainable Development. http://www.iisd.org/pdf/2012/performance_improvement_networks.pdf. Accessed 7 Oct2013.

Davis K, Drey N, Gould D (2009) What are scoping studies? Areview of the nursing literature International journal of nursingstudies 46:1386–1400

De Plaen R, Kilelu C (2004) From multiple voices to a commonlanguage: Ecosystem approaches to human health as anemerging paradigm. EcoHealth 1:SU8–SU15

Dube L, Bourhis A, Jacob R (2006) Towards a typology of virtualcommunities of practice. Interdisciplinary Journal of Informa-tion, Knowledge, and Management 1:69–93

Fenton E, Harvey J, Sturt J (2007) Evaluating primary care re-search networks. Health Services Management Research 20:162–173

Fischer G (2001) Communities of interest: Learning through theinteraction of multiple knowledge systems. In Proceedings of the24th IRIS Conference.


http://www.iisd.org/pdf/2004/networks_guidelines_for_assessment.pdf

http://www.iisd.org/pdf/2004/networks_guidelines_for_assessment.pdf

http://www.iisd.org/pdf/2012/performance_improvement_networks.pdf

http://www.iisd.org/pdf/2012/performance_improvement_networks.pdf

Flynn-Dapaah K (2003) Ecosystem approaches to human healthglobal community of practice: Report on the design phase consul-tations, Ottawa: International Development Research Centre

Grootveld A, Helms R (2008) Development and application of afactor framework to diagnose possible failure of communities ofpractice. In The 9th European Conference on Knowledge Man-agement: Eckm 2008.

Haas PM (1992) Introduction: epistemic communities andinternational policy coordination. International organization46(1):1–35

Kim S, Hong J, Suh E (2012) A diagnosis framework for identi-fying the current knowledge sharing activity status in a com-munity of practice. Expert Systems with Applications 39:130893

Kitson A, Harvey G, McCormack B (1998) Enabling the imple-mentation of evidence based practice: a conceptual framework.Quality in Health care 7:149–158

Lave J (1991) Wenger E, Cambridge university press: Situatedlearning: Legitimate peripheral participation

Lee J, Suh E-h, Hong J (2010) A maturity model based copevaluation framework: A case study of strategic cops in a Koreancompany. Expert Systems with Applications 37:2670–2681

Lesser EL, Storck J (2001) Communities of practice and organi-zational performance. IBM systems journal 40:831–841

Levac D, Colquhoun H, O’Brien KK (2010) Scoping studies:advancing the methodology. Implementation Science 5:1–9

Li LC, Grimshaw JM, Nielsen C, Judd M, Coyte PC, Graham ID(2009) Evolution of Wenger’s concept of community of prac-tice. Implementation Science 4:11

Li LC, Grimshaw JM, Nielsen C, Judd M, Coyte PC, Graham ID(2009) Use of communities of practice in business and healthcare sectors: A systematic review. Implementation Science 4:16

Loyarte E, Rivera O (2007) Communities of practice: A model fortheir cultivation. Journal of Knowledge Management 11:67–77

McDermott R (1999) Nurturing three-dimensional communitiesof practice. Knowledge Management Review 11:26–29

McDermott R (2002) Measuring the impact of communities whathappens when you need to show proof that the communities areworking and are worth the investment? This article explainshow to make sense of a chain of community activities, outcomesand value in the language of the business Knowledge Manage-ment Review 5:26–29

Meessen B, Bertone MP (2012) Assessing performance of com-munities of practice in health policy.

Parkes MW (2011) Diversity, emergence, resilience: Guides for anew generation of ecohealth research and practice. EcoHealth8:137–139

Parkes MW, Charron DF, Sanchez A (2012) Better together: field-building networks at the frontiers of ecohealth research. In: Eco-health Research in Practice: Innovative Applications of an EcosystemApproach to Health, Charron DF (editor), Insight and Innovationin Development, New York, NY: Springer and Ottawa, Canada:International Development Research Centre, pp 231–253.

Patton MQ (2011) Developmental evaluation: Applying complexityconcepts to enhance innovation and use, New York: The GuilfordPress

Perrin B (2002) How to—and how not to—evaluate innovation.Evaluation 8:13–28

Ranmuthugala G, Cunningham FC, Plumb JJ, Long J, Georgiou A,Westbrook JI, Braithwaite J (2011) A realist evaluation of therole of communities of practice in changing healthcare practice.Implementation Science 6:49

Ranmuthugala G, Plumb JJ, Cunningham FC, Georgiou A, West-brook JI, Braithwaite J (2011) How and why are communities ofpractice established in the healthcare sector? A systematic reviewof the literature BMC Health Services Research 11:273

Rogers PJ (2008) Using programme theory to evaluate compli-cated and complex aspects of interventions. Evaluation 14:29–48

Scarso E, Bolisani E (2008) Communities of practice as structuresfor managing knowledge in networked corporations. Journal ofManufacturing Technology Management 19:374–390

Scarso E, Bolisani E, Salvador L (2009) A systematic frameworkfor analysing the critical success factors of communities ofpractice. Journal of Knowledge Management 13:431–447

Serrat O (2010) Knowledge solutions: tools, methods and ap-proaches to drive development forward and enhance its effects.Asian Development Bank.

Spiegel J, Breilh J, Beltran E, Parra J, Solis F, Yassi A, et al (2011)Establishing a community of practice of researchers, practitio-ners, policy-makers and communities to sustainably manageenvironmental health risks in Ecuador. BMC internationalhealth and human rights 11:S5

Stein J, Stren R, Fitzgibbon J, MacLean M (2001) Networks ofKnowledge: Collaborative Innovation in International Learning,Toronto: University of Toronto Press

Stolee P, Conklin J (n.d.) Assessment of impact of SHRTNactivities. http://seniorshealthknowledgenetwork.ca/sites/seniorshealthknowledgenetwork.ca/files/Model_to_guide_the_evaluation_of_SHRTN_2_.pdf. Accessed 7 Oct 2013.

Van Eerd D, Cole D, Keown K, Irvin E, Kramer D, BrennemanGibson J, Kohn M, Mahood Q, Slack T, Amick B III, Phipps D,Garcia J, Morassaei S (2011) Report on knowledge transfer andexchange practices: a systematic review of the quality and types ofinstruments used to assess kte implementation and impact, Tor-onto: Institute for Work & Health

Verburg RM, Andriessen J (2006) The assessment of communitiesof practice. Knowledge and Process Management 13:13–25

Webb JC, Mergler D, Parkes MW, Saint-Charles J, Spiegel J, Walt-ner-Toews D, Yassi A, Woollard RF (2010) Tools for thoughtfulaction: the role of ecosystem approaches to health in enhancingpublic health. Canadian Journal of Public Health 101:439–441

Wenger E (1998) Communities of Practice: Learning, Meaning, andIdentity, Boston: Cambridge University Press

Wenger E (2004) Knowledge management as a doughnut: shapingyour knowledge strategy through communities of practice. IveyBusiness Journal 68:1–8

Wenger E, McDermott R, Snyder WM (2002) Cultivating Com-munities of Practice: A Guide to Managing Knowledge, Boston:Harvard Business School Press

Wenger E, Trayner B, de Laat M (2011) Promoting and assessingvalue creation in communities and networks: a conceptualframework. http://wenger-trayner.com/wp-content/uploads/2011/12/11-04-Wenger_Trayner_DeLaat_Value_creation.pdf.Accessed 7 Oct 2013.

Yap R, Robben J (2012) Measuring the value of online commu-nities and networks of practice for business. In: Proceedings ofthe 8th International Conference on Networked Learning.

Zhang J, Luna-Reyes LF, Nakashima M, Gil-Garcia JR, Sayogo DS,Mellouli S (2011) Building and sustaining a transnational andinterdisciplinary research group: lessons learned from a NorthAmerican experience. In: System Sciences (HICSS), 2011 44thHawaii International Conference on System Sciences.


http://seniorshealthknowledgenetwork.ca/sites/seniorshealthknowledgenetwork.ca/files/Model_to_guide_the_evaluation_of_SHRTN_2_.pdf



http://wenger-trayner.com/wp-content/uploads/2011/12/11-04-Wenger_Trayner_DeLaat_Value_creation.pdf

http://wenger-trayner.com/wp-content/uploads/2011/12/11-04-Wenger_Trayner_DeLaat_Value_creation.pdf

Documents

Evaluating Communities of Practice and Knowledge Networks ... · Scopus, Web of Science, Medline, EMBASE, CINAHL, ERIC, EconLit, and Business Review—captured frame-works from a