Perceptions of traumatic brain injury network participants about network performance

Brain Injury, June 2010; 24(6): 812–822

Perceptions of traumatic brain injury network participants aboutnetwork performance

MARIE-EVE LAMONTAGNE1–4, BONNIE R. SWAINE2,3, ANDRE LAVOIE1,FRANCOIS CHAMPAGNE2,4, & ANNE-CLAIRE MARCOTTE5

1Unite de traumatologie - urgence - soins intensifs, Centre de recherche FRSQ du CHA universitaire de Quebec,

Quebec, Canada, 2Universite de Montreal, Montreal, Canada, 3Centre interdisciplinaire de recherche en readaptation

du Montreal Metropolitain (CRIR), Montreal, Canada, 4Programme d’Analyse et Evaluation des interventions en

sante (AnEIS), Montreal, Canada, and 5Agence d’evaluation des technologies et des modes d’intervention en sante

(AETMIS), Montreal, Canada

(Received 5 October 2009; revised 11 March 2010; accepted 18 March 2010)

AbstractBackground: Networks have been implemented within trauma systems to overcome problems of fragmentation and lackof coordination. Such networks regroup many types of organizations that could have different perceptions of networkperformance. No study has explored the perceptions of traumatic brain injury (TBI) network participants regarding networkperformance.Objective: To document the perceptions of TBI network participants concerning the importance of different dimensionsof performance and to explore whether these perceptions vary according to organization types.Methodology: Participants of network organizations were surveyed using a questionnaire based on a conceptual frameworkof performance (the EGIPSS framework).Results: Network organizations reported dimensions related to goal attainment to be more important than dimensionsrelated to process. Differences existed between the perceptions of various types of network organizations for some but notall domains and dimensions of performance.Conclusion: Network performance appears different from the performance of an individual organization and theconsideration of the various organizations’ perceptions in clarifying this concept should improve its comprehensivenessand its acceptability by all stakeholders.

Keywords: Brain injury, integrated healthcare delivery, network, evaluation, performance

Introduction

Given their increasing complexity and specialization,many healthcare systems, including trauma systems,experience fragmentation and lack of integration[1–3]. To overcome these problems and to improveservice delivery, researchers and policy-makershave advocated for the implementation of networkswithin these systems [4, 5]. A network is defined

as a set of three or more organizations that areformally or informally connected together by variousties [6, 7] such as communication, collaboration,coordination or referral strategies. The concept of anetwork focuses on the ties or relations betweenorganizations [7] rather than on the individualorganizations. Networks have been implemented inmany countries to improve services for individuals

Correspondence: Marie-Eve Lamontagne, MSc, Unite de recherche en traumatologie - urgence - soins intensifs, Centre de recherche du CHA, Hopital del’Enfant-Jesus, Local H-041, 1401, 18e Rue, Quebec (Quebec) G1J 1Z4, Canada. Tel: (418) 649-0252 ext. 6066. Fax: (418) 649-5733. E-mail: [email protected]

ISSN 0269–9052 print/ISSN 1362–301X online � 2010 Informa Healthcare Ltd.DOI: 10.3109/02699051003789252

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

Nor

th T

exas

on

11/1

0/14

For

pers

onal

use

onl

y.

with various health conditions, including mentalhealth problems [8], dementia [4], frailty [9] andtraumatic brain injury (TBI) [5]. Despite theirincreasing popularity, little research attention hasbeen devoted to evaluating network performance,which remains a vague construct. This lack of clarityis likely to induce difficulties in network performanceevaluation and consequently limits one’s ability touse evaluation results to improve networkperformance.

Performance is not an easy concept to appraise; it isdescribed as one of the most elusive notions oforganizational theory and it has been understooddifferently over time and among organizations [10].Early frameworks of performance were one-dimensional and focused, for example, only ontechnical processes or goal attainment (e.g. specificoutcomes) [10]. For instance, a network could havebeen labelled as performing if it reached its goals or ifits internal processes ran smoothly. These earlynotions were criticized because they offered only afragmented understanding of performance.Thereafter, researchers proposed a variety of multi-dimensional frameworks that encompass multipledimensions of performance to foster a more compre-hensive conceptualization of the concept. Theseframeworks proposed examining simultaneously avariety of dimensions such as effectiveness, quality,internal climate, or satisfaction, to better appraise theperformance of an organization. The number andthe nature of the dimensions varied according to theframework examined, creating a variety of waysto conceptualize performance. Indeed, a recentliterature review by Klassen et al. [11] identified111 different multidimensional performance frame-works, 54 of which addressed health systemperformance. However, not one of these frameworkswas specific to network performance and the scientificliterature does not report the use of multidimensionalperformance frameworks for network evaluation.Because of this dearth of evidence, one cannot basethe choice of dimensions to include in a networkperformance framework upon existing frameworks orupon previous evaluation experiences.

Ideally, the choice of dimensions to include in anetwork performance framework should be based onscientific evidence gleaned from experimental stud-ies [12]. However, there is no experimental way tojudge whether one dimension is ‘better’ than anotherin assessing the performance of a network [13]. Theabsence of empirical evidence thus creates a situa-tion where it has been deemed appropriate anduseful to consider experts’ opinions [12] to developperformance frameworks. Network participants, whostrive for integration and coordination with theirpartners on a daily basis, could be considered asbeing network experts and their opinion may be a

useful source of information in defining what con-stitutes network performance.

Moreover, the consideration of the opinions ofvarious types of network participants shouldpromote a vision of network performance thatreflects the various stakeholders’ interest and prior-ities [14–16]. Indeed, networks within trauma sys-tems may link many types of member organizations,such as acute care facilities, rehabilitation facilities,health authorities and accreditation bodies, that havedistinctive cultures, mandates and clinical realities[17, 18]. Network participants are thus likely to havediffering perceptions regarding what dimensionsshould be included in an evaluation of networkperformance. Performance-related research hasexplored differences in stakeholders’ perceptionsusing a multiple-constituency approach [19]. Thisapproach stipulates that a multiple constituency (orstakeholders) of an organization could hold differentperspectives about what constitutes it’s performance.For example, in a survey of stakeholders’ preferencesin Ontario (Canada), Tregunno et al. [16] foundthat hospital (physicians, nurses and managers) andnon-hospital (home care workers and paramedics)stakeholders had different opinions with regard tothe performance dimensions of emergency depart-ments. In Belgium, Guisset et al. [20] arrived at asimilar conclusion based on their finding that nurses,physicians, pharmacists and administrative hospitalstaff paid importance to different performance indi-cators. However, a study by Minvielle et al. [15] ofhospital stakeholders’ views on hospital performancein Belgium found no significant variation in theopinions of different groups (physicians, other ser-vice providers, administrative staff). Rather, thegroups valued many performance dimensionsequally, especially those related to professional andpersonal achievement values.

Because the multi-constituency approach stipu-lates that it is unjustified or arbitrary to choose onestakeholder perspective (e.g. managerial perspective)over another, it suggests incorporating the variousperspectives to create a vision of an organization’sperformance reflecting the evaluative criteriadeemed important by all stakeholders. If a perfor-mance framework it is deemed responsive andrelevant to their concerns, the stakeholders aremore likely to accept it. The adoption of a perfor-mance framework is one of the first stages in theperformance evaluation process [21, 22] that pre-cedes the determination, measurement and analysisof performance indicators. The acceptance of acommon network performance framework increasesthe likelihood that the evaluation results will be usedby many stakeholders to improve network perfor-mance [23, 24] and result in service deliveryimprovement.

TBI network performance evaluation 813

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

Nor

th T

exas

on

11/1

0/14

For

pers

onal

use

onl

y.

In Quebec (Canada), regional networks across theprovince are responsible for the implementation andco-ordination of a continuum of care for personswith TBI. Every 4 years the TBI networks undergoan accreditation process led by a government-appointed advisory board. Due to the increasingemphasis on performance within the accreditationprocess in the healthcare sector [25, 26], theadvisory board is shifting the focus of its evaluationso that in future the accreditation status of TBInetworks will be based on their performance.Knowing the perceptions of TBI network membersregarding network performance will facilitate thedevelopment of an appropriate, balanced and usefulnetwork performance framework. In this context, itwas felt important to document the perceptions ofTBI network members concerning the importance ofdifferent dimensions of performance and to explorewhether these perceptions vary across organizationmember types. It was hypothesized that (1) differ-ences would exist with respect to the importanceattributed to domains and dimensions of perfor-mance and (2) perceptions would vary according toorganization type.

Methods

From November 2006 to November 2008, themember organizations of TBI networks in Quebecwere surveyed. The networks typically link onetrauma centre designated to provided specializedtrauma care with facilities providing in- and out-patient rehabilitation services and with regionalhealth authorities, responsible for the organizationof health services and the implementation of thegovernment’s health and social service priorities intheir sociodemographic region. Most of the networksinclude three or four organizations, but one networkincludes six organizations while another links13 different organizations. Forty-three individualorganizations make up the networks: 10 acute carefacilities, 18 rehabilitation facilities and 15 regionalhealth authorities. Each network is further linked tothe provincial health and social services departmentand to the advisory board responsible for theaccreditation process.

Procedure

With the help of the advisory board president, acontact person was identified in each of the 43organizations. For acute care and rehabilitationfacilities, clinical/administrative program co-ordina-tors were contacted by phone to explain the studyand ask their organization to participate. If theyagreed, they were asked to recruit a team of fivepersons (a physician, a rehabilitation professional, a

clinical co-ordinator, a mid- and a high-level man-ager) involved in the management of TBI patients attheir facility and having an interest in networkperformance measurement, to represent their orga-nization. Teams rather than individuals were sur-veyed to reduce the burden on individual cliniciansand managers and to provide the group with anopportunity to discuss performance issues in astructured manner. For advisory boards, regionalhealth authorities and the health and social servicesdepartment, representatives were contacted byphone and asked to complete the form eitherindividually or in teams of two, due to their smallnumbers within each organization. This provided atotal of 52 potential participants.

The contact person received the questionnaire,along with a cover letter, detailed written instruc-tions, sociodemographic data collection forms andconsent forms. The teams recruited by each contactperson were asked to meet and answer a question-naire developed specifically by the research team forthis study. The questionnaire was based upon theEGIPSS (Evaluation Globale et Integree de la

Performance des Systemes de Sante) performanceframework [10]. The EGIPSS was chosen as theconceptual base of the questionnaire because it is ageneric performance framework developed to reflectthe performance of a wide range of organizations,from a single facility to an entire healthcare system.EGIPSS is considered comprehensive, having fourlarge domains regrouping 16 dimensions that encom-pass the many visions of performance that exist inthe literature. In addition, the framework incorpo-rates supplementary notions of dynamism and bal-ance associated with the multidimensional characterof performance that are absent in other models of theconcept. Indeed, it specifies that relations existbetween the domains and dimensions and that anychange in a domain or dimension would be reflectedin others (i.e. changes in process lead to changes inresults, changes in internal climate can result inprocess changes, etc.).

Essentially, the questionnaire consisted of a list ofthe 16 dimensions and their definitions (Appendix).For each dimension, examples pertaining to net-works were provided to aid understanding.Respondents were asked to rate, as a group, theperceived importance of each dimension using ascale (0%: Not at all important to 100%: Extremelyimportant). They were also invited to providequalitative comments explaining their response andthey could suggest additional dimensions notincluded in the EGIPSS framework that theythought should be considered when evaluating anetwork’s performance. The questionnaire was pre-tested with three individuals working within thenetworks but not participating in the study to ensure

814 M.-E. Lamontagne et al.

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

Nor

th T

exas

on

11/1

0/14

For

pers

onal

use

onl

y.

that the wording was clear and that the exampleswere pertinent. Only a few minor changes in wordingwere required. Participants were asked to return thequestionnaire within 4 weeks. Each organization’sresearch ethics board approved the study.

Analysis

Importance scores relating to each dimension werecompiled based on the responses provided in thecompleted questionnaires and presented as medianpercentages with their corresponding inter-quartileranges. The median importance scores for each ofthe framework’s four domains were calculated usingthe importance scores for all dimensions pertainingto the particular domain. However, to examine thedifferences between organization types, importancescores were not compared but rather their rankorder. For each network participant (organization),the dimensions were ranked based on their impor-tance score, rank 1 being attributed to the mostimportant dimension and rank 16 to the leastimportant. When participants attributed the sameimportance score to more than one dimension, arank corresponding to the mean of possible rankvalues was attributed. For instance, if three dimen-sions received a score of 100%, they each received arank of 2, corresponding to the mean of 1þ 2þ 3.Because of the small number of organizationssurveyed and the limited distribution of the scores,Kruskall-Wallis analysis of variance was used toexamine whether differences existed among themedian importance scores attributed to differentdomains and dimensions and among the medianimportance score attributed by different organizationtypes. Post-hoc comparisons were performed usingMann-Whitney U-tests, with an � corrected formultiple comparisons (p¼ 0.05/number of tests per-formed [27]). Kendall rank correlations were alsoused to examine the relations between the rankingsattributed by the different organization types.Statistical analyses were conducted with SPSS 16.0software and content analysis of qualitative data [28]was conducted with Nvivo 7.0 software.

Results

Forty-six of the 52 questionnaires were completedand returned. Because two rehabilitation facilityteams, accustomed to working closely together,decided to meet and respond jointly to the ques-tionnaire, a response rate of 90.4% (47/52) wasobtained. Two regional health authorities and tworehabilitation facilities did not return the calls andone rehabilitation facility could not participate dueto the timing of the study. Overall, 142 personsparticipated in the survey. Twenty-five per cent of

the participants worked in acute care facilities, 63%in rehabilitation facilities and 10% in regional healthauthorities while 2% worked in a governmentdepartment or as part of an advisory board (thelatter two being referred to as ‘Other’ organizationsfrom here on). Participants had a mean professionalexperience of 17.6 years (SD¼ 9.5) and reportedworking in their current position for an average of7.8 years (SD¼6.1). Thirty-nine per cent wereclinicians while 54% held administrative positionswithin the TBI programme/facility and the remain-der held positions that combined clinical andadministrative activities. Participants’ characteristicswere similar across settings, except that individualsworking in acute care were older (Z¼�2.89,p¼ 0.004) and had more professional experience(Z¼�2.36, p¼ 0.018) than those working in reha-bilitation facilities.

Importance of domains and dimensions of the EGIPSS

framework

With regard to the importance paid to the domainsof the EGIPSS framework, the median importancescores varied from 90% for the Goal attainment

domain to 85% for the Adaptation and the Production

domains, to 80% for the Value maintenance domain.Significant differences existed between the impor-tance scores of each domain (�2

¼ 13.04, p¼0.005).Post-hoc comparisons showed that the medianimportance score for the Production domain wassignificantly lower than that for the Goal attainment

domain (Z¼�3.453, p¼ 0.001).Table I presents the median importance scores

and inter-quartiles ranges for each of the 16 dimen-sions of the EGIPSS framework as well as p-valuesresulting from post-hoc comparisons of the medianperceived importance scores.

The median importance scores attributed to thedimensions fluctuated from 100% for the Capacity

to attract the clientele and Continuity dimensions to57.5% for the Quantity of care and services dimension.Significant differences were found between theimportance scores of the 16 dimensions(�2¼ 218.9, p< 0.001). Based on post-hoc compar-

isons, the Quantity of care and services dimension wasthe most different dimension, being significantly lessimportant than all the others except the Productivity

and Ability to adapt to requirements and tendencies

dimensions. The perceived importance of Capacity to

attract the clientele was higher than the perceivedimportance of the Effectiveness dimension and fromthe nine dimensions that received lower scores. TheContinuity dimension was more important than sixother dimensions. The Ability to adapt to requirements

and tendencies was less important that the sevendimensions perceived as the most important and the


Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

Nor

th T

exas

on

11/1

0/14

For

pers

onal

use

onl

y.

Tab

leI.

Med

ian

imp

ort

ance

score

san

din

terq

uar

tile

ran

ges

attr

ibu

ted

by

par

tici

pan

tsto

the

16

dim

ensi

on

sof

per

form

ance

and

p-v

alu

esre

sult

ing

from

post

-hoc

com

par

ison

s.

EG

IPS

Sd

imen

sion

s(n¼

16)

Med

ian

score

s(%

)

Inte

r-q

uar

tile

ran

ges

Capacitytoattracttheclientele

Continuity

Abilitytomeettheclient’sneeds

Effectivness

Efficiency

Equity

Quality

Capacitytoacquireressources

Collaborationclimate

Abilitytomobilizecommunitysupport

Abilitytoinnovateandtransform

Concensuswithfundamentalvalues

Satisfactionofclientsandpartners

Productivity

Abilitytoadapttorequirementsandtendencies

Quantityofcaresandservices

Cap

acit

yto

attr

act

the

clie

nte

le100.0

[90–1

00]

–

Con

tin

uit

y100.0

[85–1

00]

0.1

68

–A

bilit

yto

adap

tan

dm

eet

the

clie

nt’

sn

eed

s

95.0

[80–1

00]

0.0

32

0.3

94

–

Eff

ecti

ven

ess

90.0

[80–9

0]

0.0

00

0.0

05

0.0

79

–E

ffic

ien

cy90.0

[80–9

5]

0.0

01

0.0

50

0.3

38

0.3

14

–E

qu

ity

90.0

[80–1

00]

0.0

20

0.3

25

0.8

62

0.0

80

0.3

45

–Q

ual

ity

90.0

[80–1

00]

0.0

19

0.3

06

0.8

50

0.0

98

0.3

86

0.9

55

–C

apac

ity

toac

qu

ire

ress

ou

rces

85.0

[75–9

5]

0.0

00

0.0

06

0.0

42

0.6

86

0.2

37

0.0

05

0.0

76

–

Collab

ora

tive

clim

ate

82.5

[75–9

0]

0.0

00

0.0

02

0.0

21

0.5

25

0.1

39

0.0

27

0.0

35

0.8

84

–A

bilit

yto

mob

iliz

eco

mm

un

ity

sup

port

80.0

[60–9

0]

0.0

00

0.0

00

0.0

00

0.0

18

0.0

02

0.0

00

0.0

01

0.0

88

0.0

97

–

Ab

ilit

yto

inn

ova

tean

dtr

ansf

orm

80.0

[80–9

0]

0.0

00

0.0

00

0.0

02

0.1

01

0.0

12

0.0

03

0.0

03

0.3

80

0.4

19

0.3

51

–

Con

cen

sus

wit

hfu

nd

amen

tal

valu

es80.0

[70–9

0]

0.0

00

0.0

01

0.0

08

0.2

53

0.0

63

0.0

13

0.0

19

0.5

20

0.5

86

0.2

90

0.9

27

–

Sat

isfa

ctio

nof

clie

nts

and

par

tner

s

80.0

[80–9

0]

0.0

00

0.0

00

0.0

06

0.1

91

0.0

19

0.0

04

0.0

06

0.4

86

0.5

66

0.2

43

0.6

50

0.9

55

–

Pro

du

ctiv

ity

80.0

[80–8

5]

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

05

0.0

06

0.2

00

0.0

36

0.0

32

0.0

15

–A

bilit

yto

adap

tto

req

uir

emen

tsan

dte

nd

enci

es

75.0

[60–8

0]

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

02

0.0

02

0.1

25

0.0

05

0.0

13

0.0

04

0.7

60

–

Qu

anti

tyof

care

and

serv

ices

57.5

[25–7

5]

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

00

0.0

15

0.0

13

–

Sig

nif

ican

td

iffe

ren

ces

(p<

0.0

00

42)

ind

icat

edin

ital

ic.


Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

Nor

th T

exas

on

11/1

0/14

For

pers

onal

use

onl

y.

Productivity dimension was less important than sixother dimensions.

Ranking of dimensions based on type of network

organization

Table II presents, for each type of organization,mean and absolute ranks for the EGIPSS’s domainsand dimensions. It also shows the �2 and p valuesresulting from comparisons between the mean ranksof the different types of organizations. According tothe type of organization, mean ranks ranged from1.1–3.8 for the domains and from 3.3–15.4 for thedimensions. With regard to domains of performance,Goal attainment received the highest mean rank forthree types of organization while Adaptation receivedthe third highest rank in terms of importance fromall organizations. However, Value maintenance wasthe only domain for which the mean rank variedsignificantly across organization type.

With regard to dimensions of performance,Capacity to attract the clientele received the highestmean rank (3.3) from both rehabilitation facilitiesand Others. The lowest mean rank (15.4) wasobserved for Quantity of cares and services. OnlyConsensus with fundamental values, Productivity andQuantity of care and services mean ranks werestatistically different across organization types.Globally, the ranking of dimensions for each of the

four types of organizations all correlated significantlywith each other, with the exception of the rankings ofacute care facilities that were not linked to thoseof the Others. Indeed, there was a low but significantcorrelation between rehabilitation facilities’ rankingsand Others’ rankings (�¼0.43, p¼0.02) andbetween regional health authorities’ and acute carefacilities’ rankings (�¼ 0.49, p¼ 0.01). A moderatecorrelation was found between regional healthauthorities’ rankings and that of the Others(�¼ 0.58, p¼0.002). The highest correlations wereobserved between rehabilitation facilities’ rankingsand regional health authorities’ (�¼ 0.68, p< 0.001)and acute care facilities’ (�¼ 0.68, p<0.001)rankings.

Comments and explanations

Many respondents provided comments explainingtheir scores and took the opportunity to share theirgeneral views about the TBI networks. Threedimensions, namely Capacity to attract the clientele,Continuity and Ability to adapt and meet the client’s

needs, received high importance scores and werealso described by network participants as being themain goals of a network. Indeed, to explain the highscore attributed to Capacity to attract the clientele

dimension, respondents explained that Quebec’sinter-organizational networks were initially created

Table II. Mean ranks and absolute ranks for domains and dimensions of the EGIPSS framework, by type of organization.

Acute carefacilities

Rehabilitationfacilities

Regional healthauthorities Others

MR AR MR AR MR AR MR AR �2 p

EGIPSS domains (n¼ 4)Adaptation 2.6 3 2.7 3 2.4 3 2.5 3 0.59 0.90Value maintenance 2.0 1 2.8 4 2.2 2 3.8 4 9.21 0.03

Production 3.1 4 2.5 2 3.2 4 2.6 2 5.48 0.14Goal attainment 2.2 2 1.9 1 2.1 1 1.1 1 5.25 0.15

EGIPSS dimensions (n¼16)Capacity to attract the clientele 6.1 4 4.3 1 3.3 1 3.3 1 5.25 0.15Continuity 5.3 3 6.1 4 4.9 2 6.1 5 1.22 0.75Ability to adapt and meet the client’s needs 4.8 1 8.1 6 5.7 4 8.6 10 7.21 0.07Effectiveness 8.6 10 8.2 7 7.8 6 3.6 2 6.70 0.08Efficiency 6.9 5 5.7 2 8.1 8 4.8 4 5.78 0.12Equity 7.3 6 6.0 3 5.2 3 7.3 6 1.86 0.60Quality 5.2 2 6.6 5 6.3 5 4.5 3 2.24 0.52Capacity to acquire resources 8.4 9 8.5 8 7.9 7 12.6 14 4.6 0.20Collaborative climate 8.3 8 10.1 12 8.2 9 12.3 12 6.40 0.09Ability to mobilize community support 11.7 14 10.2 13 10.0 12 12.3 13 2.57 0.46Ability to innovate and transform 8.8 11 9.9 11 9.8 11 7.5 7 2.25 0.52Consensus with fundamental values 7.7 7 10.2 14 8.6 10 13.5 15 10.14 0.02

Satisfaction of clients and partners 8.9 12 8.7 9 10.3 13 8.0 9 2.43 0.49Productivity 12.0 15 9.5 10 12.4 14 7.5 8 9.46 0.02

Ability to adapt to requirements and tendencies 11.6 13 12.0 16 12.8 15 8.9 11 2.52 0.47Quantity of cares and services 12.5 16 11.5 15 14.5 16 15.4 16 10.82 0.01

MR¼mean ranks; AR¼ absolute ranks; Significant differences indicated in italic (p< 0.05).


Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

Nor

th T

exas

on

11/1

0/14

For

pers

onal

use

onl

y.

to optimize accessibility of care and to provideservices to all individuals with TBI. They alsoreported that networks were supposed to fosterseamless service organization. Some organizationscommented that they paid high importance to thedimension Ability to adapt and meet the client’s needs

because they felt that the complexity and specificityof TBI requires services to be customized based onthe needs of each individual. The implementation ofnetworks was thus considered a reflection of atrauma system’s Ability to adapt and meet the client’s

needs.Participants explained that they gave a high

importance score to Equity because it is a funda-mental value of care for the entire Canadian publichealthcare system. Some participants commented onthe potential for inequity within networks across theprovince, particularly in the presence of a third partypayer (such as Quebec’s automobile insuranceboard). In the past, depending on the injury mech-anism (i.e. automobile or work accident as opposedto a fall or sports-related injury), some persons mayhave received more services or received them in amore timely fashion compared to others. Althoughthis situation has been corrected, respondents appearto remain sensitive about equity issues relating to theage and area of residence (rural vs urban) of theirTBI clients.

Respondents described the Efficiency dimensionas an important dimension reflecting the added valueof network service delivery: ‘It seems very importantto know the value added to the services offered byour network [when compared to a less integratedorganization of service]’ (comment from an acutecare facility). Some participants mentioned theyscored Quality, Effectiveness and Satisfaction of clients

and partners as slightly less important becausethey perceived these dimensions as being difficultto measure and as being influenced by factorsbeyond the control of network members(e.g. patient comorbidities, motivation, expectations,resource availability, external constraints). Capacity

to acquire resources was also perceived as beyond anetwork’s control, particularly because of currentbudget constraints and the lack of human andfinancial resources in the healthcare system. Acutecare organizations added ‘their administrative rulesweren’t flexible enough to facilitate resource acqui-sition or sharing of resources with other networkmembers’. Indeed, health professionals cannot cur-rently be moved from one facility to work in anotherwithin the network to address human resource needsand personnel shortages.

Respondents perceived the Collaborative cli-

mate dimension as a facilitator important but notessential to the various integration activitiesacross the organizations (e.g. communication,

sharing and collaboration). Consensus with fundamen-

tal values was scored as moderately important andperhaps difficult to attain: ‘It’s hard to have commonvalues while having different mandates’ (commentfrom a rehabilitation facility team).

Ability to innovate and transform was considered bysome as a somewhat important component ofcontinuous quality improvement, resulting fromclinicians’ and a teams’ ability to question theirclinical practices and from research and programmeevaluation projects. The slightly lower score attrib-uted to Ability to adapt to requirement and tendencies

could be explained by a comment from an acute carefacility, that mentioned ‘All trends aren’t synony-mous with improvement: efforts to change must becarefully analysed to insure they bring an addedvalue to patients’.

Comments with regards to the dimension Ability to

mobilize community support underscore that mobili-zation of partners in the community is required tocomplete the continuum of care essential to individ-uals with TBI. However, members of an acute carefacility mentioned that ‘Mobilization of the commu-nity is not a priority for an acute care setting’.Finally, most of the comments pertaining to Quantity

of care and services and its low score underscored thatthis dimension does not by itself provide interestinginformation and that it needs to be examined inrelation to others to gain significance in the contextof network performance evaluation.

Eight organizations (three acute care facilities,three rehabilitation facilities, one regional healthauthority and one Other) suggested 10 additionaldimensions to the EGIPSS framework, the mostfrequently suggested dimensions pertaining to thegovernance of the network. Three organizationsstressed the importance of evaluating the manage-ment and the leadership abilities of the personsresponsible for the network and one mentioned theimportance of evaluating the leadership of thenetwork in the community. Two organizationsstrongly suggested that the ‘social participation’ ofTBI individuals as an outcome could be used toreflect network performance. Other dimensionssuggested by one organization included: self-assess-ment of the network, evaluation of the appropriate-ness of patients’ trajectory within the continuum ofcare, support and recognition of participants fortheir implication in the network, inter-professionalconfidence and maintaining clinical expertise.

Discussion

This study addressed gaps in the knowledge aboutnetwork participants’ perceptions of network perfor-mance. It demonstrated that all performance


Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

Nor

th T

exas

on

11/1

0/14

For

pers

onal

use

onl

y.

domains and dimensions included in the EGIPSSframework were not perceived of equal importanceby the different network organization membersand that this perception may vary according toorganization type.

The first hypothesis was that differences wouldexist in the perceived importance of the domains anddimensions of performance for the evaluation of TBInetworks. One important result of this study is thatwith regard to the domains investigated, Goal

attainment was deemed significantly more importantthan Production. In others words, network organiza-tions seem to feel that more importance should bepaid to the evaluation of outcomes than to theevaluation of the processes involved. This opinion ismost likely because network participants work inhealthcare settings that promote a patient/client-centred philosophy, where patient outcomes andsatisfaction related to the Goal attainment domainwould be deemed more important than the more‘technical’ aspects of services. This result is interest-ing because the use of common outcome measureswithin the Goal attainment domain could be advan-tageous for a network. Indeed, outcome measuresare described as promoting a shared vision betweenservice providers by keeping the focus on patientsand thus encouraging collaboration between clini-cians [29, 30]. The use of such measures couldtherefore contribute to strengthening the links andthe integration between network organizationmembers.

With regard to performance dimensions, partici-pants perceived Capacity to attract the clientele,

Continuity and Ability to adapt and meet the client’s

needs as more important than Ability to adapt to

requirements and tendencies, Productivity andQuantity of care and services. The three dimensionsperceived as most important were also viewed asfundamental to networks, thus supporting the liter-ature describing these as being the ultimate goals ofnetworks [8]. Participants also viewed the dimensionCapacity to attract the clientele as closely linked toaccessibility concerns. One important goal of theTBI networks is to act as a comprehensive regional‘net’ catching individuals with TBI and preventingthem from falling through the system’s cracks. Thesystems of care for individuals with TBI havepreviously been described as having accessibilityissues [31], particularly for patients who have com-pleted in-patient rehabilitation yet still require ser-vices [32, 33]. Similarly, Continuity and Efficiency

are considered major goals of many integrationinitiatives [4, 34]. The higher importance attributedby respondents to dimensions that are key features ofnetworks indicate that the participants’ conceptionof network performance is linked to networktheory. Many authors state that performance is a

contingent concept in that performance is closelyassociated with how organizations are conceptual-ized [10, 19]. The high importance attributed toCapacity to attract the clientele, Continuity, Ability

to adapt and meet the client’s needs, Equity andEfficiency contrasts with the findings from studiesconducted within single organizations (i.e. hospital)that stress the importance of evaluating dimensionspertaining to human relations [15] or critical pro-cesses of care [16]. This important finding highlightsthe idea that network performance is different fromthe performance of an individual organization.Consequently, network performance should be eval-uated using criteria that best reflect the added valueof organizations working together for a commongoal. Indeed, Mandell and Keast [33] insist thatnetwork performance measures should include thedegree to which linkages that exist among membersare tight or loose, the degree to which all relevantparties are included in the network and the degree towhich the network is supported by key actors bothinside and outside the network. These elementscorrespond to the Continuity and Ability to mobilize

community support dimensions of the EGIPSSframework.

Because many dimensions were deemed impor-tant by network participants, the results support thenotion that network performance is a multidimen-sional concept that should be appraised using manydimensions simultaneously [35]. In the authors’view, the comprehensive nature of the performanceconcept and the dynamic relationships between thedifferent domains and dimensions of the EGIPSSframework offer an alternative solution to the long-standing discussion of process vs outcomes measures[29, 30], by enabling evaluators to consider bothprocesses and outcomes rather than one or the other.Indeed, while it is important to document theoutcome(s) of an organization, including those of anetwork, understanding the network processesunderlying these outcomes allows one to ‘open theblack box’ of a network and ultimately modifyprocesses to improve service delivery. Based on thestudy results and in accordance with Mandell andKeast [33], the importance of some dimensionsshould be emphasized more than others whenevaluating network performance. For example, theadvisory board responsible for the accreditationprocess may want to consider attributing more‘weight’ to dimensions perceived as most important.These dimensions could be monitored on a contin-uous basis, while others deemed less importantcould be evaluated less frequently.

The second hypothesis was partially confirmed inthat perceptions regarding the importance of somedomains and dimensions do vary across organizationtypes. In spite of differences in organizational


Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

Nor

th T

exas

on

11/1

0/14

For

pers

onal

use

onl

y.

mandates, cultures and values, there is a certaindegree of association between the rankings attrib-uted by the different types of organizations. Thevarious network organizations appear to generallyagree on what is important to measure to reflectnetwork performance. This coherence, also observedin an another context [15], is likely due to the highdegree of collaboration already existing among TBInetwork organizations across Quebec. In the lastdecade, these organizations have had to work closelytogether to meet the evaluation standards of theadvisory board. This collaboration may have buff-ered differences in the organizations’ visions ofnetwork performance. Working together for nearlya decade, the organizations may have reached acertain level of maturity and confidence allowing fora more shared vision about their network perfor-mance. These results may thus not be generalizableto ‘younger’ trauma networks or to those outsideQuebec.

The differences observed across organization typespartly support the idea that network participantshave various conceptions of the specific componentsof network performance. Consequently, evaluatingnetworks from only one perspective (e.g. based onlyon the perspective of one type of facility or on thatheld by health authorities) could result in an unbal-anced and limited evaluation and subsequently affectthe adoption and use of the evaluation results by allstakeholders. It is expected that the adoption of acomprehensive network performance framework,encompassing the domains and dimensions deemedimportant by all stakeholders, will make the imple-mentation of the new performance evaluation pro-cess in TBI networks much easier for all concerned.

The study’s strengths and limitations should benoted. The high participation rate of all types ofnetwork participants suggests that the perspectives ofmost stakeholders are represented. Indeed, thechoice of the EGIPSS framework as the conceptualbasis of the study conditioned the observed results.However, the fact that participants suggested fewadditional dimensions supports the choice of frame-work in that it provided participants with a startingpoint for their thoughts about network performance.The questionnaire was developed for this study;although pre-tested, some definitions of the domainsand dimensions were not clear to some participants.The fact that teams of individuals completed thequestionnaire may have allowed some individualswith stronger opinions to exercise more control overthe group’s responses when arriving at a consensus.Some individuals may not have been able to expresstheir own perceptions. Nonetheless, the exerciselikely promoted discussions among TBI team mem-bers working within a same organization. In

addition, social desirability bias may have beenresponsible for the relatively high importance scores.

The methodological approach used in this studyallowed individual organizations to identify thedimensions deemed important in the evaluation ofnetwork performance and the identification of vary-ing perceptions of different organization types.Future research could explore the results of discus-sions about network performance between organiza-tions linked within the same TBI network.

Conclusion

This article describes the perceptions of individualorganizations within the TBI networks in Quebecabout a topic of growing interest, the evaluation ofTBI network performance. The results underscorethe specificity of network performance, as well as theadded value of considering many perspectives inclarifying this concept. These results will supportthe development of an evaluation tool to measure theperformance of TBI networks. Ultimately, improvedevaluations should help trauma systems andnetworks provide high quality service to individualswith TBI.

Acknowledgements

Preparation of this article was supported by schol-arships awarded to the first author by the CanadianInstitutes for Health Research, the ANEIS program,and the Programme de recherche en readaptation eten Integration sociale (PRISST).

Declaration of interest: The authors reportno conflicts of interest. The authors alone areresponsible for the content and writing of the paper.

References

1. Fleury MJ. Integrated service networks: The Quebec case.Health Service Management Research 2006;19:153–165.

2. Hoyt DB, Coimbra R. Trauma systems. Surgical Clinics ofNorth America 2007;87:21–35, v–vi.

3. Cope DN, Mayer N, Cervelli L. Development of systems ofcare for persons with traumatic brain injury. Journal of HeadTrauma Rehabilitation 2005;20:128–142.

4. Lemieux-Charles L, Chambers LW. Network evolution andeffectiveness. The Dementia care networks’ study. Toronto:Dementia care networks’ study; 2002.

5. Vander Laan R, Brandys C, Sullivan I, Lemsky C. Integrationthrough a city-wide brain injury network and best practicesproject. Neurorehabilitation 2001;16:17–26.

6. Borgatti S. The network paradigm in organizational research:A review and typology. Journal of Management 2003;29:991–1013.

7. Brass DJ, Galaskiewicz J, Greve HR, Tsai W. Taking stock ofnetwork and organizations: A multilevel perspective. Academyof Management Journal 2004;47:795–817.


Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

Nor

th T

exas

on

11/1

0/14

For

pers

onal

use

onl

y.

8. Fleury M-J, Mercier C. Integrated local networks as a Modelfor Organizing Mental Health services. Administration andPolicy in Mental Health 2002;30:55–73.

9. Hebert R, Durand PJ, Dubuc N, Tourigny A. Frail elderlypatients. New model for integrated service delivery. CanadianFamily Physician 2003;49:992–997.

10. Sicotte C, Champagne F, Contandriopoulos AP, Barnsley J,Beland F, Leggat SG, et al. A conceptual framework for theanalysis of health care organizations’ performance. HealthService Management Research 1998;11:24–41; discussion -8.

11. Klassen A, Miller A, Anderson N, Shen J, Schiariti V,O’Donnell M. Performance measurement and improvementframeworks in health, education and social services systems:A systematic review. International Journal of Quality HealthCare 2009;22:44–69.

12. Campbell SM, Braspenning J, Hutchinson A, Marshall MN.Research methods used in developing and applying qualityindicators in primary care. British Medical Journal 2003;326:816–819.

13. Kenis P, Provan KG. Towards an exogenous theory of publicnetwork performance. Public Administration 2009;87:440–56.

14. Gagliardi A, Lemieux-Charles L, Brown A, Sullivan T,Goel V. Stakeholder preferences for cancer care performanceindicators. International Journal of Health Care QualityAssurance 2008;21:175–189.

15. Minvielle E, Sicotte C, Champagne F, Contandriopoulos AP,Jeantet M, Preaubert N, Bourdil A, Richard C. Hospitalperformance: Competing or shared values? Health Policy2008;87:8–19.

16. Tregunno D, Ross Baker G, Barnsley J, Murray M.Competing values of emergency department performance:Balancing multiple stakeholder perspectives. Health ServiceResearch 2004;39:771–791.

17. Kodner DL. Spreeuwenberg C. Integrated care: Meaning,logic, applications, and implication—a discussion paper.International Journal of Integrated Care 2002;2:1–6.

18. Leutz WN. Five laws for integrating medical and socialservices: Lessons from the United States and the UnitedKingdom. Milbank Quarterly 1999;77:77–110, iv–v.

19. Connolly T, Conlon EJ, Deutch SJ. Organizational effective-ness: A multiple-constituency approach. Academy ofManagement Review 1980;5:211–217.

20. Guisset A-L, Sicotte C, Leclercq P, D’Hoore W. Definitionde la performance hospitaliere: Une enquiete aupres desdivers acteurs strategiques au sein des hopitaux. SciencesSociales et Sante 2002;20:65–102.

21. Adair CE, Simpson E, Casebeer AL, Birdsell JM,Hayden KA, Lewis S. Performance measurement in health-care: Part II. State of the science findings by stage of

performance measurement process. Healthcare Policy2006;2:56–78.

22. Gibberd RW. Performance measurement: Is it now morescientific? International Journal of Quality Health Care2005;17:185–186.

23. Finucane PM, Barron SR, Davies HA, Hadfield-Jones RS,Kaigas TM. Towards an acceptance of performance assess-ment. Medical Education 2002;36:959–964.

24. Goddard M, Davies HT, Dawson D, Mannion R,McInnes F. Clinical performance measurement: Part2—avoiding the pitfalls. Journal of the Royal Society ofMedicine 2002;95:549–51.

25. Adair CE, Simpson E, Casebeer AL, Birdsell JM,Hayden KA, Lewis S. Performance measurement in health-care: Part I. Concepts and trends from a state of the sciencereview. Healthcare Policy 2006;1:85–104.

26. Smits PA, Champagne F, Contandriopoulos D, Sicotte C,Preval J. Conceptualizing performance in accreditation.International Journal of Quality Health Care 2008;20:47–52.

27. Pett MA. Nonparametric statistics for health care research:Statistics for small samples and unusual distributions.Thousand Oaks, CA: Sage Publications; 1997.

28. Huberman AM, Miles MB. Data management and analysismethods. In: Denzin N, Lincoln Y, editors. Handbooks ofqualitative research. CA: Sage; 1994. pp 428–444.

29. Goddard M, Davies HT, Dawson D, Mannion R,McInnes F. Clinical performance measurement: Part 1—getting the best out of it. Journal of the Royal Society ofMedicine 2002;95:508–510.

30. Mannion R, Davies HTO. Reporting health care perfor-mance: Learning from the past, prospects for the future.Journal of Evaluative Clinical Practice 2002;8:215–228.

31. Cope DN, Raynolds WE. Systems of care. In: Silver JM,McAllister TW, Yudofsky SC, editors. Textbook of traumaticbrain injury. Washington DC: American Psychiatric Pub;2005, pp. 559–569.

32. Lefebvre H, Pelchat D, Swaine B, Gelinas I, Levert M-J. Theexperiences of individuals with a traumatic brain injury,families, physicians and health professionals regarding careprovided through the continuum. Brain Injury 2005;19:585–597.

33. Mandell M, Keast R. Evaluating network arrangement.Toward revised performance measure. Public Performanceand Management Review 2007;30:574–597.

34. Fleury MJ. Quebec mental health services networks: Modelsand implementation. International Journal of Integrated Care2005;5:e07.

35. Leggat S, Leatt P. Cadre d’evaluation du rendement dessystemes de prestatin de services de sante integree.Healthcare Management Forum 1997;10:19–26.


Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

Nor

th T

exas

on

11/1

0/14

For

pers

onal

use

onl

y.

Appendix: Domains and dimensions of the EGIPSS framework with definitions adapted to the

network context

Domain (n¼ 4) Dimension (n¼ 16) Definition

Adaptation Ability to adapt and meet theclient’s needs

Ability of a network to meet the needs of the individuals with TBI and theirfamily or to adapt the services to suit the evolution of these needs.

Ability to adapt to requirementsand tendencies

Ability of a network to modify the services to meet external requirementsand tendencies generated by the evolving environment.

Ability to mobilize communitysupport

Range and intensity of the support obtained by a network within acontinuum of services, in its community, from a political and a societalstandpoint.

Ability to innovate and transform Ability of a network to initiate changes and to promote the continuousimprovements of its services.

Capacity to acquire resources Ability of a network to obtain and maintain financial, human, material andinformational resources required to exercising its mandate.

Capacity to attract the clientele Ability of a network to draw in and transfer persons with TBI requiringservices provided within the network.

Values maintenance Collaborative climate Characteristics of the climate in the network with regards to its influence onthe way members are able to collaborate.

Consensus with fundamentalvalues

Consensus about the attitudes and the values deemed important to thenetwork.

Production Productivity Optimization of the production of services according to the availability ofresources.

Quantity of care and service Number of hours of services provided by the network.Continuity Set of formal procedures allowing for a logical arrangement of the different

constituents of a system to reach the system’s goal.Quality Set of process attributes that induce the best possible result, as defined by

knowledge, technologies, expectations and social norms.Goal attainment Equity Similar and accessible services provided to all persons with TBI with regards

to the needs of the individuals, the regions, the groups, etc.Efficiency Relationship between the results obtained by the network and the human,

material and financial resources used to reach these results.Effectiveness Degree of attainment of the objectives determined by network members.Satisfaction of clients and partners Degree of appreciation of the services provided within the network by

persons with TBI, their families and by network partners


Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsity

of

Nor

th T

exas

on

11/1

0/14

For

pers

onal

use

onl

y.

Documents

Perceptions of traumatic brain injury network participants about network performance