Embed Size (px)
Citation preview
THE MATURITY OF
MONITORING AND EVALUATION SYSTEMS OF
THE BRAZILIAN PUBLIC ADMINISTRATION
TEAM
Selma Maria Hayakawa Cunha Serpa
Hiroyuki Miki
SUPERVISION
Glória Maria Merola da Costa Bastos
Maridel Piloto de Noronha
ADVISOR
Paulo Carlos Du Pin CalmonUniversity of Brasília
Bangkok, October 2015
FEDERAL COURT OF ACCOUNTS - BRAZIL
4th International Conference on National Evaluation Capacities
Support
Objectives
Evaluate Brazilian Ministries` (Federal Departments) capacity to demand, do and use evaluation
Describe their evaluation systems
Conceptual Framework
Analysis Framework
Evaluative
Demands
Focus I
Focus I - Do governmental decision-makers
have well defined program objectives, targets, goals, budget and other resources?
know which, when and for what purpose evaluative information is needed ?
know who needs the evaluative information to be produced ?
Production of
evaluative knowledge –
supply
Focus II
Focus II – Do Brazilian Ministries have (to produce evaluative information)
Institutionalized organizational process?
Evaluative procedures and practices?
Skilled resources to develop or to ask for evaluation?
Evaluative rules and organizational support (resources to do evaluation) ?
Organizational
capacity learning
Focus III
Evaluation
use
Focus IV
Evaluation Systems Maturity LevelsMaturity Level Description Score
Not structured
A system or focus is considered not structured when the mechanisms and instruments needed to characterize the evaluation systems are partially present and they are not enough and do not regularly meet the decision-makers` needs of evaluative knowledge
> 50
Incipient
A system or focus is considered incipient when the mechanisms and instruments needed to characterize the evaluation systems are present, but they are partially sufficient and do not regularly meet the decision-makers` needs of evaluative knowledge
≥ 50 e ≤ 60
Intermediate
A system or focus is considered intermediate when the mechanisms and instruments needed to characterize the evaluation systems are present, partially sufficient and regularlymeet the decision-makers` needs of evaluative knowledge
> 60 e ≤ 70
Improved
A system or focus is considered Improved when the mechanisms and instruments needed to characterize the evaluation systems are present, sufficient and satisfactorily meet the decision-makers` needs of evaluative knowledge
> 70 e ≤ 80
Advanced
A system or focus is considered advanced when the mechanisms and instruments needed to characterize the evaluation systems are present and entirely adequate to meet the decision-makers` needs of evaluative knowledge
> 80
Methodology
Web-survey (as of 26, August to October 4th, 2013)
Target group - 2,062 governmental decision-makers at 28 ministries (Secretariats
(DAS 6), Directors (DAS 5) and Coordinators (DAS 4 and DAS 3))
750 respondents (36.4%)
Descriptive statistics and content analysis
Multi-Criteria Decision Analysis (constructivist approach) – MCDA-C (index – ISA-
Gov)
Minister
Director B1
Secretariat A Secretariat B Secretariat C
Director B2
Coordinator General Coordinator Project Manager
Executive Secretariat
73.1%
18.3%
7.1%
1.1%
Survey Target Group
0.4%
0.5%
0.8%
1.3%
1.5%
5.9%
10.3%
15.7%
0.8%
0.3%
0.5%
5.6%
11.1%
22.6%
22.7%
0.0% 10.0% 20.0% 30.0% 40.0% 50.0%
Branco (sem resposta) → 0,4%
Secretário(a) Executivo(a) → 1,1%
Diretor(a) Geral → 1,4%
Coordenador(a) → 1,8%
Secretário(a) → 7,1%
Diretor(a) → 16,9%
Outros → 32,9%
Coordenador(a) Geral → 38,4%
Percentage - per job position
% Respondentes % Ñ Respondentes
Others
Directors
Secretariats
Coordinators
General Directors
Executive Secretariats
Non identified
Respondents Non Respondents
General Coordinators
17.65%
6.60%
19.14%
33.56%
23.05%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Mais de 6 anos
5 a 6 anos
3 a 4 anos
1 a 2 anos
Menos de um ano
Period of time at the current job position
< 1 year
1 to 2 years
3 to 4 years
5 to 6 years
> 6 years
Respondents Profile
57 %
ResultsMaturity Level per Focus
Capacity to do evaluation
0
10
20
30
40
50
60
70
80
90
Pontuação noFoco 1
(Demanda)
Pontuação noFoco 2 (Oferta)
Pontuação noFoco 3
(Aprendizado)
Pontuação noFoco 4
(Utilização)
GERAL
59.17
49.57
82.50 80.00
63.96
Focus I score(Demand)
Focus II score(Supply)
Focus III score(Organizational
Capacity Learning)
Focus IV score(Evaluation use)
General score
ResultsBreakdown of Foci - Dimensions of Analysis
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
60.48
74.00
32.46
72.46
41.3346.00
50.00
59.11
82.50 80.00 80.00 80.00 80.00
63.96
Main evaluation practices
ResultsComparative analysis between evaluation demand x supply – Focus I x Focus II
Focus I
I -
Supply
Focus I - Demand
Focus Radar
Ministry
Governmental Evaluation Systems Purposes
Organizational Learning
Performance Monitoring
Evidence-Based Policy
Accountability
6 X X
8 X X X X
12 X X
14 X X
17 X X
21 X X X X
26 X X X
27 X X X X
28 X X X
Purposes of Evaluation Systems
Maturity Index of Governmental Evaluation Systems - iSA-Gov
Advanced
Improved
Intermediate
Incipient
Not Structured
Thank you!
Selma Maria Hayakawa Cunha Serpa
Federal Court of Accounts - Brazil
