Embed Size (px)
Citation preview
1
EVALUATION OF US DISPUTE RESOLUTION PROGRAMS FOR
ENVIRONMENTAL AND COMPLEX PUBLIC POLICY
DISPUTES
Andy RoweGHK International
2
DISCLAIMER
This presentation has been prepared by the author and represents only my views. The agencies participating in or contracting the work on which this presentation is based do not necessarily subscribe to any or all of what is contained in the presentation. Andy Rowe
3
CHEAPER FASTER BETTER?
There appears to be more belief than evidence underlying this claim
This is really a statement about program effectiveness:“Collaborative processes are more effective than a reasonable or likely alternative process”
The evaluation system being used by important U.S. federal and state agencies is starting to provide evidence for this claim of effectiveness
4
BACKGROUND
The evaluation system specifically addresses environmental and complex public policy disputes
Features of the system:• Takes a results-based accountability / performance management
approach• Takes an improvement orientation• Addresses expected internal and external performance reporting
requirements (state and federal)• Self administered• Has a high level of face validity for both ADR and evaluation
professions• Is proving to have a high level of validity and reliability
Focus on utilization. Results will:• Be easy to interpret• Be useful for improving performance• Be easily comprehended by ADR professionals
5
APPLICATIONS
U.S. Federal agencies• USEPA Conflict Prevention and Resolution Center• US Institute for Environmental Conflict Resolution• Other federal environmental agencies through contact to
USIECR
U.S. State agencies• Oregon Dispute Resolution Commission• Florida Conflict Resolution Consortium
Other applications• Base for Maryland Alternative Dispute Resolution• Applied in pilot project of 9th Circuit
6
A PERFORMANCE MANAGEMENT APPROACH
Primary goal is to use evaluation as a mechanism to focus organizations on results and achieve success• While satisfying evaluation goals and standards• Address possible Government Performance Results Act
(GPRA) requirements
John A. Koskinen’s three GPRA questions are what the evaluation attempts to address (John McLaughlin version):• What is your program / agency supposed to achieve?• How will you know when you have achieved this?• How are you doing now?
8
IMPLEMENTATION
Questionnaires to parties, practitioners and project manager at conclusion of the process and a follow-up to parties at a later date• Mail, web and email administration options
Response rates are high (will exceed 70% in Oregon pilot) and 100% of practitioners (included their contracts)
Information was systematically tested and has proved valid and reliable
Meets performance standards for the design• Managers are “can answer” to “fluent” for first two questions• Resource requirements = 10 hours per case for admin, analysis &
reporting• Culture of the organizations shifts towards results focus• Internal stakeholders very satisfied, external not yet known
9
OUTCOME CHARTS ARE THE KEY TO THE DESIGN
Similar to logic models / logframes but much more focused on results
Outcome charts are best developed collaboratively with the program managers / staff
Illustrations that follow are from the Public Policy Program of the Oregon Dispute Resolution Commission• This is an ADR collaborative process – similar to
applications in other agencies
10
Increased efficiency of agencies through use of
ADR
Decreased cost of resolving
disputes
Increase agency and public satisfaction with process and
results of ADR
OREGON PUBLIC POLICY DISPUTE RESOLUTION PROGRAM OUTCOMES
ADR is successful (time and resources spent in
disagreement and conflict are now redirected to
more constructive purposes)
ADR in State Government
Oregon state
agencies have
practices and
systems that
ensure ADR is
available
Practice of ADR
supports agency
mission, and
receives positive
feedback Good Process
Good agreements
Case Management
Dispute Systems Designs
Mediator Roster
Dispute resolution systems are improved
Roster provides expedited access to
qualified and appropriate providers
11
EXTRACTED OUTCOMES FOR OREGON CASE MANAGEMENT
ADR successful (resources spent in disagreement are now used more constructively)
Good agreements Good outcomes from process
ADR more effective (better benefits for resources expended) than other
options for this dispute
All parties involved in an ADR process are
satisfied that the process was fair and open.
Durable and implementable
agreement reached using
ADR
Use of ADR narrows disagreements
Agreements reached with
ADR are complete - there
are no hard issues left or
deferred.
Right parties continue to be engaged, new parties
added as required.
Use of ADR helps adversarial parties
collaborate
Party in
teractions con
structive
Appropriate mediator leads ADR
AD
R u
sed
app
ropriately
Design of the ADR process is appropriate for the dispute
and needs of parties
Right parties engaged in process
ADR is used where it is the best approach for
this case
Stakeholder capacity to use ADR is improved through experience with
this case
Government decision making is improved through use of the process
Non-ADR processes used where the best approach
13
Simplifying Complex Practices
Outcome charts seek to display the responses to the first of the GPRA questions:• What is my agency/unit trying to achieve?
The complexity of any one element of ADR practice is difficult to articulate in a single outcome but this complexity is required to respond to the second GPRA question:• How will we recognize this?• The mechanism I am currently using to simplify this is nested outcomes
– mini outcome charts under individual outcomes Even the nested outcomes can be at too high a level of
generalization to be able to respond to the third GPRA questions• How are we doing now?• This is either addressed through further nesting or directly by the
questions we ask
14
Appropriate Neutral Guides
the Process
Appropriate neutral – parties satisfied with
neutral
Appropriate neutral – neutral understands
issues
Appropriate neutral – manages process well
Appropriate neutral – available for process
Appropriate neutral – parties understand
process
At not time did one of the parties dominate to the detriment of the process or others
The neutral helped us manage our time well.
The neutral made sure that the concerns of all parties were addressed.
The neutral made sure that the concerns of all parties were heard.
The parties followed the direction of the neutral.
The neutral ensured that all parties were fully engaged in the process.
When things got tense the neutral was always able to find ways to move ahead constructively
Nested OutcomesExpressed as questions
Nested Outcomes Charted Outcome
ILLUSTRATION OF CHARTED AND NESTED OUTCOMES
15
FCRC Case "A" Nested Outcome Results for Right Neutral
6.0 7.0 8.0 9.0 10.0
Appropriate neutral guides theprocess
Appropriate neutral - partiesunderstand process
Appropriate neutral - neutralunderstands issues
Appropriate neutral - neutralmanages process well
Appropriate neutral - neutralavailable
Score 0 (lowest) to 10 (highest)
16
FCRC Case "A" Outcome Results
6.0 7.0 8.0 9.0 10.0
Right parties are effectively engagedAppropriate neutral guides the processFCRC designs an appropriate process
Disagreement narrowsBest information used by parties
Parties communicate and collaborateParties satisfied with the process
Collaborative process is more effectiveParties reach complete agreements
All parties are satisfed with what they haveAgreements are implemented
Agreements endure Intergovernmental coordination is improved
Government decision making is improvedAgency and party capacity is improved
Score 0 (lowest) to 10 (highest)
18
OTHER ELEMENTS IN THE EVALUATION SYSTEM
Collaborative Processes• Program Intake• ECR/ADR/Collaborative Processes (longer and shorter
forms• Conflict Assessment / Collaborative Assessment /
Situation Assessment• Meetings: information gathering / public input• Dispute System Design
Other ADR Services• Training• Roster program• Contract for neutral services
19
THE EVALUATION DESIGN IS SUCCESSFUL
Have established a program theory for ADR in environmental and complex public policy disputes
Approach is being implemented by a growing number of agencies
Positive peer external reviews (practice and evaluation)
Information is helping program improve
Information is addressing internal and external accountability requirements
21
MAIN REMAINING CHALLENGES
Incremental Benefits• Inherently difficult because of nature of many ADR cases• In some settings this can be addressed without difficulty
Effectiveness• Inherently difficult because of nature of most or perhaps all ADR cases• Lack of useful applied research in this area
Both incremental benefits and effectiveness are highly charged concepts in the field• Many strong believers have vigorously argued the merits of ADR based
on intuitive appeal, inappropriate straw dogs, limited conceptual rigor and poor evidence
• These are also linked to an even more charged debate between proponents of process versus proponents of agreements
There is a compelling logical and some empirical evidence that ADR will prove to deliver incremental benefit and be an effective approach when used appropriately
22
CHALLENGE 1: INCREMENTAL BENEFITS
Disputes tend to be heterogeneous• Arises from some or all of: characteristics of the dispute, parties,
practitioner, general setting in time and place• Very challenging to find a counterfactual• Counterfactuals will always need to be unique to the case• Disputes often engage a variety of resolution options over their
life Litigation is not the only alternative
• When litigation is the likely alternative we can get some reasonable information
• For many disputes the likely alternatives are: do nothing, legislation or administrative rule making
Addressing the challenge• The strength of the program theory, fact that claims made by the
program theory are not exceptional “likely to” should suffice as an evaluation judgment of incremental benefits
23
CHALLENGE 2: EFFECTIVENESS
Costs and benefits occur in three phases:• Prior to the dispute resolution process. Many costs have
already occurred such as fact finding, establishing the technical, legal and economic claims, identifying rights
• During the dispute resolution process. Main costs are time of parties, attorneys, practitioner, travel. Ability to pay can vary dramatically (e.g. dispute involving a major utility, environmental groups and individual small farmers)
• After the dispute resolution process. The hypothesis is that a good agreement will significantly reduce costs
Shelf life of this information is relatively short
The classes and characteristics of costs and benefits have not been enumerated, nor weights or likely ranges
24
ADDRESSING THE CHALLENGES
Incremental Benefits• Use litigation as an alternative where feasible• Undertake GOOD case studies to create illustrative
comparators that can be referred to• Use “likely to” approach in the interim
Effectiveness• Case studies to establish the categories, ranges and
coefficients• Include limited number of cost questions in the meantime• Manage the dance of “doing no harm” by not collecting
bad cost data in the interim
25
MY CLAIM
1. Getting the organization to focus on results generates immediate and significant benefitsa) Outcome chart program intent is clear = improved
evaluabilityb) 3 GPRA questions requirement for monitoring outcomes =
performance information
2. Worst case Improved evaluationa) Utilization enhancedb) Can report on outcomesc) Credible information for program improvement
3. Better case Organizations adopt an evaluative and performance culture
26
CONTACT INFORMATION
Andy RoweGHK International2707 Stratford Rd.
Columbia SC 29204 (803) 782-2182 voice (803) 782-3618 fax
