Upload
shana-goodman
View
218
Download
2
Tags:
Embed Size (px)
Citation preview
What About the Independence of Evaluations in Switzerland?
A Comparative Perspective
Prof. Dr. Fritz SagerCenter of Competence for Public Management University of Bern
SEVAL Annual Conference 2015Geneva, 4 September 2015
Palmer Hilton, Chicago11 April 2013, 7 a.m.
2
Independence in Evaluation Standards
> “The trustworthiness of evaluators decisively influences the feasibility and effectiveness of an evaluation. In order to be considered trustworthy by the various stakeholders, the following features are crucial: integrity, independence as well as social and communicative competences” (SEVAL)
> “evaluators aspire to construct and provide the best possible information that might bear on the value of whatever is being evaluated” (American Evaluation Association)
> “unbiased conduct and reporting” (German Evaluation Society DeGEval)
3
Some questions
> Is it all normal that a commissioning agency asks me to change my results?
> Am I the only one to experience pressure?> How do other evaluators cope with this situation?> How does Switzerland look in international comparison?
4
Comparing results from the USA, UK, Germany and Switzerland
> USA by Morris & Clark (2013); UK by the LSE GV314 Group (2013); Germany by Stockmann et al. (2011); Switzerland by Pleger & Sager (2015)
> Research design: studies were conducted by using an online questionnaire surveying members of their respective evaluation societies
> Sample size: 940 (USA); 204 (UK), 132 (Germany), 157 (Switzerland)
> Evaluators in the USA, Germany and Switzerland are relatively highly educated
> Primary employment setting of the respondents when conducting evaluations are heterogeneous
5
Experience with pressure and influence
6
Survey USA(N = 905)
Survey Germany (N = 132)
Survey Switzerland(N = 123)
Survey UK(N = 173)
0
10
20
30
40
50
60
70
80
90
51
22
48
17
42
79
50
83
7
02
0
Changing Requirements of Stakeholder
7
> Present findings more positively or negatively> Omit or downplay negative findings> Draw different conclusions> Use invalid/old/different data> Distortion of content> Results were determined in advance by the stakeholder
Dealing with misrepresentation pressure
> 28% of AEA members and 12% of the SEVAL members claimed that no changes were made
> 16% of AEA members and 9% of SEVAL members admitted that changes were made which constituted misrepresentation
> 90% of DeGEval members and 90% of SEVAL members conceded that they had reformulated at least one sentence as a consequence of stakeholder pressure
> 57% of the respondents from DeGEval and 48% of SEVAL members stated that they had presented findings more positively on at least one occasion
8
Potential preventive action
9
Other
Improve data collection/documentation
Have independent third party intervene
Establish formal evaluation protocol
Emphasize evaluator’s responsibility to stand by the data
Discuss the possibility of negative findings at beginning
Develop fuller understanding of goals/purposes/roles
Collaborate more with stakeholders
Better educate stakeholders in methodology/statistics
0 10 20 30 40 50 60
56
14
14
12
33
51
51
16
22
13
6
12
10
8
8
20
11
13
Study USA(N = 340)
Study Switzerland(N = 51)
Findings
> Pressure on evaluators to misrepresent findings is a common phenomenon in all four discussed countries
> There is a large range of pressure> Individuals who commission evaluations are identified as
the main influencing actor> While German commissioning agencies are more prone
to put pressure on evaluators, German evaluators do not show stronger tendencies to surrender to pressure than the other countries’ respondents
10
Conclusion: Evaluators between a rock and a hard place
> Independence of evaluations is not a self-evident fact> Preventive measures are needed in order to reduce pressure
by commissioners> Evaluators can be torn between commissioners’ demands
and compliance with professional standards, i.e. Evaluation Standards
> Independence must take a prominent place and higher significance in Evaluation Standards
11
Changing Requirements of Stakeholder
Content of Misrepresentation Request Studies
Survey USA
(N = 875)Survey Germany
(N = 121-123)Survey Switzerland
(N = 61-63)
Open-ended question Closed questions
Present findings more positively 38%(n = 130).
78%(n =123 )
76%(n = 48)
Present findings more negatively Unlevied. 30%(n = 123)
27%(n = 17)
Omit or downplay negative findings 25%(n = 857) Unlevied. Unlevied.
Change language—neither positive nor negative
13%(n = 43)
80%(n = 122)
87%(n = 54)
Use inappropriate/different methodology or statistical procedures
6%(n = 21)
72%(n = 121)
63%(n = 39)
Draw different conclusions 6%(n = 20)
53%(n = 123)
56%(n = 35)
Show inappropriate concern for implications of results
4%(n = 12) Unlevied. Unlevied.
Use invalid/old/different data 3%(n = 11)
57%(n = 121)
45%(n = 28)
Distortion of content Unlevied 55%(n = 123)
55%(n = 34)
Results were determined in advance by the stakeholder Unlevied Unlevied 41%
(n = 25)
Other 5%(n = 18) Unlevied 30%
(n = 16)
12