The Challenges of Measuring Success in Collaborative ... challenges white paper.pdfChallenges of Measuring Success in Ecosystem Management Ecosystem Management Initiative Œ Draft

The Challenges of Measuring Success in Collaborative Ecosystem Management

-Draft-

Elizabeth McCance, Sarah Kopplin,

Sheila Schueller, and Steven Yaffee

May 2002

2002 Ecosystem Management Initiative, School of Natural Resources and Environment, University of Michigan, Ann Arbor, MI 48109-1115;

[email protected], 734-615-6512

Draft � Do NOT cite or quote.

Table of Contents INTRODUCTION .......................................................................................................................................................1

THE CASES .................................................................................................................................................................1 Chequamegon-Nicolet National Forest .................................................................................................................1 Chicago Wilderness ...............................................................................................................................................2 Fish Creek .............................................................................................................................................................3 Kankakee River Basin............................................................................................................................................4 Oak Openings ........................................................................................................................................................5

LESSONS.....................................................................................................................................................................5

CHALLENGES AND SOLUTIONS..........................................................................................................................1

1. NO UNIVERSAL DEFINITION OF ECOSYSTEM MANAGEMENT ............................................................2

CHALLENGES..............................................................................................................................................................2 Integration of Conflicting Goals ........................................................................................................................2 Difficulties with Scale .......................................................................................................................................3 Role of Collaboration ........................................................................................................................................3 Understanding Systems Thinking......................................................................................................................4

POTENTIAL SOLUTIONS ..............................................................................................................................................5

2. VAGUE, ILL-DEFINES, OR IDEALISTIC GOALS ..........................................................................................7

CHALLENGES..............................................................................................................................................................7 Unachievable Goals ...........................................................................................................................................7 Implementation Versus Progress .......................................................................................................................8 Running to Stay in Place....................................................................................................................................8

POSSIBLE SOLUTIONS .................................................................................................................................................9

3. MULTIPLE STAGES OF ECOSYSTEM MANAGEMENT............................................................................10

CHALLENGES............................................................................................................................................................10 No Universal Indicator of Success...................................................................................................................10 Successful Implementation Versus Successful Strategy..................................................................................11 Strategies at a Variety of Stages ......................................................................................................................12

POSSIBLE SOLUTIONS ...............................................................................................................................................12

4. MULTIPLE TIME SCALES................................................................................................................................13

CHALLENGES............................................................................................................................................................13 Time Lags ........................................................................................................................................................13 Lack of Monitoring Programs..........................................................................................................................14 Interconnected Strategies.................................................................................................................................15


5. MULTIPLE SPATIAL SCALES .........................................................................................................................16

CHALLENGES............................................................................................................................................................16 Scaling Up .......................................................................................................................................................17 Importance of Larger Scales ............................................................................................................................17 Uncertain Causal Linkages ..............................................................................................................................17


6. COMPLEX ECOLOGICAL SYSTEMS.............................................................................................................19

CHALLENGES............................................................................................................................................................19 Natural Variability ...........................................................................................................................................19 Confounding Variables....................................................................................................................................20 Representative Indicators.................................................................................................................................20

Coarse-scale Measures.....................................................................................................................................21 POSSIBLE SOLUTIONS ...............................................................................................................................................21

7. LACK OF ARTICULATED SOCIAL AND PROCESS GOALS.....................................................................22

CHALLENGES............................................................................................................................................................22 Lack of Social and Process Goals....................................................................................................................23 Fuzzy Concepts................................................................................................................................................23 Benefits of Well-Defined Process....................................................................................................................24


8. EXTRANEOUS AND CONFLICTING GOALS ...............................................................................................25

CHALLENGES............................................................................................................................................................26 Small Effort in a Big Pond...............................................................................................................................26 Complications with Indicator Selection...........................................................................................................26 Small Token Projects.......................................................................................................................................26 Incompatible Goals..........................................................................................................................................27


9. SINGLE STRATEGY VERSUS HOLISTIC GOAL .........................................................................................28

CHALLENGES............................................................................................................................................................28 Strategy Not Sufficient ....................................................................................................................................28 Short-term Versus Long-term Success ............................................................................................................29 Understanding Incentives ................................................................................................................................30 Unintended Effects ..........................................................................................................................................31 Multiple Measures ...........................................................................................................................................32


10. CAUSAL LINKAGES.........................................................................................................................................33

CHALLENGES............................................................................................................................................................33 Simultaneous Use of Strategies .......................................................................................................................34 Poorly Designed Monitoring Programs ...........................................................................................................34


11. UNREALIZED EVALUATION PLANS...........................................................................................................36

CHALLENGES............................................................................................................................................................36 Other Priorities ................................................................................................................................................37 Overwhelmed by Task.....................................................................................................................................38 Data Collection Versus Analysis .....................................................................................................................39

POTENTIAL SOLUTIONS ............................................................................................................................................40

12. SCALE OF EVALUATION ...............................................................................................................................42

CHALLENGES............................................................................................................................................................42 Fine Scale Indicators........................................................................................................................................42 Coarse Scale Indicators....................................................................................................................................43

POTENTIAL SOLUTIONS ............................................................................................................................................44

Introduction

Challenges of Measuring Success in Ecosystem Management Ecosystem Management Initiative � Draft (do not cite or quote)

1

Introduction

Ecosystem management is becoming an increasingly popular strategy for addressing complex

conservation issues. While more and more organizations are using ecosystem management

approaches to address environmental problems, there is not yet an accepted framework to

evaluate ecosystem management programs. There is both an internal and external interest in

learning how to evaluate ecosystem management. For those organizations involved in an

ecosystem management program, evaluation is needed to measure progress. How is the program

working? Are the goals being met? From an outside perspective, many stakeholders, including

funders, are interested in determining how successful ecosystem management is in addressing

complex environmental and social issues. Is ecosystem management a good approach? Are

there certain factors that make a program more or less successful that should be replicated in

other programs?

As a first step to develop a framework to evaluate ecosystem management, we undertook a

qualitative investigation of five programs. Within each ecosystem management case, we

interviewed three to seven people from different participating organizations. The purpose of the

interviews was to determine how various participants viewed success, their own progress, and

how they might be dealing with evaluation questions. The five cases included in this study are:

Chequamegon-Nicolet Forest, Chicago Wilderness, Fish Creek, Kankakee River Basin, and Oak

Openings. While information about each case is laced throughout the document, basic

information is described below.

The Cases

Chequamegon-Nicolet National Forest The ecoregion in question is a 1.5 million acre national forest and an overlapping 700,000-acre

watershed that empties into Lake Superior on the eastern side of Chequamegon Bay in northern

Wisconsin. The Chequamegon Bay Watershed harbors many rare and endangered plants and

animals, and the Chequamegon-Nicolet National Forest contains some of Wisconsin's largest and

least disturbed stands of sugar maple-hemlock forest, and globally imperiled pine barrens and

dry forest communities.

Introduction


2

Change in the forest composition and cover, runoff from agricultural fields, development, and,

potentially, mining, could have or are having an impact on the long-term health and biodiversity

of the Sloughs and the Chequamegon Bay Watershed. Ownership of the watershed is divided

among federal and state agencies, private timber companies, and the Bad River Reservation. No

all-encompassing formal partnership exists in this ecoregion, but there has been collaboration

and informal or temporary partnerships (and/or plans for stronger ties) among DNR, TNC,

USFS, local colleges and universities, and the Bad River Indian Reservation.

TNC worked with several partners to develop their conservation plan in 1997, including US

Forest Service, Wisconsin DNR, tribal members, Northland College, county resource staff,

county Land Conservation Department, and other local scientists. TNC is now focusing on

setting up an advisory group that would consist of local long-term residents, and people from

some agencies and those involved in local town government. Within the forest service, there is

an internal team, which is working to revise the Chequamegon and Nicolet Forest Plans into one

joint plan.

Overall, the goals of all agencies are to restore much of the forested area to something more

similar to historical forests and to protect the health of the watershed. The Forest Service has

additional multiple-use goals beyond conservation, such as recreation and timber use. TNC�s

goals focus on improving the health of the watershed. Their approach is to influence private and

public management practices through the forest plan revision process and through public

education and easements. The DNR is working to collect the kind of information � state-wide

biological inventory � to allow for priority sites to be identified for conservation or restoration

and to provide for more informed development of management plans.

Chicago Wilderness Chicago Wilderness extends across northeastern Illinois, southeastern Wisconsin and

northwestern Indiana, covering nine counties of the metropolitan Chicago area. Within these

nine counties are approximately 200,000 acres of publicly owned natural areas. These areas

along with other natural remnants in the region are the focus of Chicago Wilderness. The natural

remnants include many natural communities including, woodlands, savannas, prairies, and

wetlands. While the sprawling city contributes to the major threats to the natural communities,

the region retains some of the best examples of remaining prairie and savanna, as well as some

Introduction


3

threatened and endangered species. Almost all of the remaining natural areas are already in

public ownership or are being maintained by the private owners. The two main priorities for

Chicago Wilderness are how to manage these remnants and how to connect and enlarge them

through strategic acquisitions to best conserve the region�s biodiversity.

A group of concerned organizations realized that these habitat patches would best be managed in

cooperation with one another, that a regional perspective was needed if the region�s biodiversity

was to be saved. Therefore a group of organizations banned together to form Chicago

Wilderness. Membership in the consortium has always been inclusive, and new organizations

are continually joining. Currently there are 130 members. The mission of Chicago Wilderness

is �the protection, restoration, and stewardship of the natural communities of the Chicago region

through fostering their compatibility with the human communities whose lives they enrich.�

Fish Creek Fish Creek is a tributary of the St. Joseph River, and its watershed lies primarily in the

northeastern corner of Indiana, but also in parts of Ohio. Fish Creek became a conservation

concern when several Federally Endangered mussel species were found.

In the early 1990s, a partnership was formed between The Nature Conservancy, FWS, and

Indiana and Ohio Departments of Natural Resources to address threats to these mussel species.

As soil erosion and sedimentation into the creek is a major threat, the first task was to look for

problem areas of soil erosion within the watershed. Land use in the area is primarily agricultural,

and therefore strategies to minimize soil erosion tend to target farming practices. Great emphasis

is placed on converting farmers to conservation tillage practices. In addition, trees are being

replanted along the creek. The Nature Conservancy feels that they are implementing the plan

agreed to by the partnership, but that the planning meetings are no longer needed. The

collaboration has evolved into a more informal relationship, as they all feel comfortable with one

another�s work.

Meanwhile, a contaminants spill in the area gave the Indiana and Ohio Departments of Natural

Resources money from a settlement to be used to restore the same watershed. This spill

refocused the attention of the Departments of Natural Resources onto another partnership

agreement. They formalized a new partnership through an MOU, known as the Fish Creek

Introduction


4

Council, and drafted a restoration plan, which includes many of the same issues and strategies

that the original partnership was and continues to work on. Both efforts are going on

simultaneously to achieve the same goals, although the partnership resulting from the oil spill is

more limited in area and scope and operates under a more formalized structure. The Nature

Conservancy is not a formal member of the second partnership, but often collaborates on specific

projects. The work of both partnerships is very similar and often overlapping, but in this paper

the Fish Creek Partnership refers to the original partnership.

Kankakee River Basin At the turn of the last century, the Kankakee River Basin was dominated by the Grand Kankakee

Marsh. It was the largest inland marsh in North America � 100 miles long and between one and

fifteen miles wide � a 500,000 acre expanse teeming with plant life and wildlife native to

northern Illinois and Indiana. In Indiana, the marsh and river were drained and dredged between

the late 1800�s and 1920, and the Kankakee River, once a 240-mile long, meandering stream,

was straightened and is now only 90 miles long. With the Grand Marsh drained, agriculture

claimed the wetlands, and the price paid was sedimentation of the stream and flooding of the

developing farm and urban landscape.

Today, it is as though there are two different Kankakee Rivers; in Indiana it is an agricultural

ditch, and in Illinois it is a sand-plagued, yet more natural stream. Despite this difference, the

remnants of the Grand Kankakee Marsh are still home to over 220 rare species, including the

federally endangered Indiana bat and the state-listed American bittern, northern leopard frog and

salamander mussel. Nearly the entire eastern population of sandhill cranes depends on the

Kankakee watershed during migration.

As early as the 1930s, efforts were being made to restore the Grand Kankakee Marsh. Although

most of those efforts were unsuccessful, today there are several programs that are now working

towards restoration of this marsh. Many agencies are involved � federal, state, local, nonprofit,

etc. � and there are three main projects underway: 1) the US Fish and Wildlife Service Grand

Kankakee Marsh National Wildlife Refuge, 2) the US Army Corps of Engineers Flood Control

Study, and 3) the Indiana Grand Kankakee Marsh Restoration Project (also known as the North

American Project). Even though these efforts are related, there is not an overall vision, plan, or

program.

Introduction


5

Oak Openings Oak Openings, a region once covered by Oak Savanna with some prairie and wetlands mixed in,

lies to the west of Toledo and is threatened by the expanding metropolis. Several organizations

and agencies, including Toledo Metroparks, The Nature Conservancy, and Ohio Department of

Natural Resources own reserves within the region. These agencies among others have come

together to try to conserve and restore remaining habitat, although the first Oak Openings

working group has been dissolved. It included several development interests and it seemed to

the main facilitator that it was more of an information session with little real progress.

A new group has since formed, composed of just conservation minded organizations that has

developed a Green Space Plan, which outlines parcels ideal for acquisition and restoration.

Currently the members are trying to build public and political support for the plan. In the

meantime, each agency is working on their own holdings to restore certain parts to native habitat.

Other education and advocacy efforts are also underway by a variety of organizations.

Lessons

The rest of this paper describes the various lessons we learned from the interviews. Why is

evaluation of ecosystem management difficult? In addition to articulating the various challenges

to evaluation and the types of behavior that they spawn, we also make a first attempt at offering

some potential solutions. These potential solutions include both activities currently being tried

by one or more of the cases and ideas that we think might be useful.

The lessons learned fall into two general categories. Many challenges to evaluating ecosystem

management stem from the complex nature of ecosystem management itself. Other challenges

are more logistical difficulties or deal with problems of implementing evaluation plans. Table 1

identifies the main challenges found as well as some of the solutions to deal with these issues.

Each challenge is discussed at length below.

Challenges and Solutions


1

Challenges and Solutions

Lessons Learned

1. Because there is no universal definition of ecosystem management, there is no �ideal� or �standard� against which to measure success of cases.

2. Knowing when success is achieved is difficult when goals are vague, ill defined, and/or idealistic.

3. Evaluations, particularly cross-case comparisons, must recognize that there are several different phases of ecosystem management, which should be evaluated separately.

4. Because systems, goals, and strategies operate on multiple time scales, measuring progress must account for time lags and out-of-synch strategies.

5. Because systems, goals, and strategies operate on multiple spatial scales, it is difficult to infer regional level success from site level measures of success.

6. Measuring success is difficult because ecological systems are inherently complex and variable, complicating interpretation of data, identification of useful indicators, and disentangling of confounding variables.

7. Indicators of social and process goals are not as recognized by practitioners as are ecological goals, and therefore are often not articulated or measured, making it difficult to evaluate overall progress.

8. When member organizations of ecosystem management have diverse agendas, evaluation is complicated by goals that are extraneous, conflicting, and/or competing with ecosystem management goals.

9. Because successful implementation of strategies may not equate with overall ecosystem management success, this cannot be used as reliable measure of progress.

10. The lack of detectable causal linkages, such as in the case of time lags or simultaneous use of many strategies, complicates evaluation of strategy success.

11. Evaluation is difficult, because, although its importance is recognized, evaluation steps are often not put into practice, leaving little information on which to base progress.

12. Because both fine and coarse scale indicators have limited usefulness, it is difficult to determine the correct scale at which to conduct an evaluation.

1. No Universal Definition of Ecosystem Management


2


Because there is no universal definition of ecosystem management, there is no �ideal� or

�standard� against which to measure success of cases.

Challenges

There are many different definitions of ecosystem management, although many authors agree on

a few core principles. These include: attention to varied and broad spatial and time scales,

systems thinking, consideration of humans as part of the ecosystem, social goals, goals related to

biodiversity and ecosystem functions, integration of social and ecological goals, adaptive

management, and collaboration. Although some of the core principals of ecosystem

management are not highly disputed, there is a lack of a concise definition of ecosystem

management, making it difficult both to have an �ideal� or �standard� measure of success against

which all cases can be compared and to compare cases to one another. Difficulties in evaluation

arise in the many different components in a definition of ecosystem management, including

goals, scale, collaboration, and systems thinking.

Integration of Conflicting Goals The ultimate challenge is how to measure successful balance or integration between conflicting

goals, since that is an integral part of ecosystem management. It is an approach that is supposed

to apply an understanding of how human and natural systems interrelate and incorporate

ecological and social concerns into the same end point. On the ground, this is more difficult than

in theory. For example, Toledo Metroparks must balance a variety of organizational goals to

meet the desires of its many constituents. The Metroparks� board �is very much interested in

preserving natural areas, but they are also interested in providing more recreation opportunities

to the public�. We have to somehow find a balance between public use of areas, because our

money comes from tax levies�� What is a successful balance between soccer fields and restored

prairies? How can such a balance be measured without coming up with a meaningless index?

Or it may be that it is impossible to achieve the stated goals, and therefore no measure of

successful balance.



3

Difficulties with Scale Ecosystem management definitions do not clarify the scale of an ecosystem. Part of most

definitions of ecosystem management is that it occurs on the ecoregion scale. However, what

entails an ecosystem remains unclear, especially from an organizational perspective. Is a

watershed big enough, or is it too big logistically? The cases examined here ranged considerably

in scale. Oak Openings covers a relatively small region defined by the former lake plain area,

whereas Fish Creek is defined by the creekshed, but The Nature Conservancy is considering the

need to tackle the whole watershed of the St. Joseph River. The Kankakee River Basin

represented a large area with several different ecosystems, and Chicago Wilderness defines its

area of coverage by a bioregion shaped by the glaciers, although on a practical basis uses county

lines and encompasses a nine county area. At what scale should restoration take place? Can

efforts on such different scales be meaningfully compared? Challenges specific to spatial scales

are further discussed in issue #5.

Role of Collaboration Collaboration is usually part of the definition of ecosystem management, but is it a necessary

component, and if so, at what level? Due to all the challenges of ecosystem management �

working on a large scale across administrative and disciplinary boundaries, thinking across

multiple dimensions, etc. � collaboration is considered essential to achieve ecosystem

management. But if goals are accomplished in the absence of collaboration, is that unsuccessful

ecosystem management? Does there need to be a central organization to call it an ecosystem

management case, or does a collection of individual efforts working towards similar goals count?

Do the efforts need to be truly collaborative or just cooperative? Often perceptions of good

collaboration are taken as evidence of success, but success should not be assumed. Instead, the

resulting products of the collaboration should also be examined.

It is easy to get a sense of whether agency members feel like they are collaborating with other

agencies more or less than they would like too (e.g. �we do keep an eye on each other and

communicate frequently. Maybe not as frequently as I�d like to see us do,� or �I think that if we

had a more formal group that met 2-3 times a year that would help things become more

streamlined�). However, what is not clear is the extent to which that feeling of desire or lack

thereof for increased collaboration is a good measure of whether the case is successful or not in



4

terms of collaboration. For example, TNC Fish Creek is quite content with their current informal

collaboration relationship with DNR: �Other agencies trust what we�re doing out here. There�s

not really a need to get together and discuss. I mean � they know what we�re doing, they believe

in it � just go do it.�

Certainly there are cases where collaboration is not fruitful. A DNR practitioner comments that

the word �team� is �one of those four-letter words that I�m not particularly fond of.� He finds that

when teams became trendy they were over-used and that �there is a lot of work that is much

better accomplished and you are more likely to get it done if you know what your charge is and

just do it. Teams can certainly slow things down.� The level at which teamwork is �needed� to

succeed at ecosystem management is unclear, and therefore levels of collaboration do not

necessarily respond to progress in ecosystem management. Commenting on the different

activities of different organizations, one Kankakee Basin worker comments, �because we have

similar goals and objectives, and no one really cares who does it as long as it gets done.� In the

case of Federal Agencies, some cooperation is mandated by law (sometimes even with specific

legal agreements not to �negatively impact one another�). Is the ecosystem management

requirement of collaboration then fulfilled? When collaboration is required it can lead to

practitioners �not really doing any more than they have to� and thereby not putting together the

best project they could. But what is the best project? Is there an optimal level or form of

collaboration?

Understanding Systems Thinking Ideally, systems thinking should be an integral part of both implementation and evaluation of

ecosystem management. Systems thinking is a method used to see the interconnections between

different activities and outcomes. To achieve ecosystem management goals, there needs to be

synergistic effects between a variety of strategies. An effective evaluation should analyze how

various strategies may be working together to fulfill overall goals. As a Chicago Wilderness

representative commented, when all the different goals including �more than 300

recommendations� are viewed together, �the whole smorgasbord of things� can become quite

overwhelming. In all the cases, many projects that contribute to overall goals are being

undertaken by single agencies in site specific areas. The result is that there may be progress on

many little projects, but this may not be achieving overall success or a good balance between



5

goals, or that certain components will be left by the wayside while energy is put into progress in

other components. When they are not organized by categories or into a hierarchy, prioritization

and overall assessment are challenging.

Potential Solutions

An oversight committee to help formulate a common goal between organizations would be

useful in that there would be something definitive to measure progress towards. Recognizing the

need for this, a Kankakee participant noted, �the next logical step would be a bi-state oversight

committee, which is not in place now. It would need to be all the federal agencies and the state

agencies. One of these big, unholy, ungodly stakeholder parties � that�s the only way something

that big is going to happen.� A case cannot be assessed as an ecosystem management case unless

it has a central viewpoint � either an existing organization, or an outside evaluator taking the

ecoregion-wide perspective.

At least part of the solution to measuring success of different components of ecosystem

management is to organize or categorize them in some way. A USFS practitioner expects that

success would have to be measured separately for different kinds of �goal areas that you could

measure,� such as �how did you do in biological diversity? How did you do in meeting

recreational needs? How did you do in terms of meeting a sense of place, if that�s one of the

goals�?� The challenge, then, is figuring out what those categories, components, or sets of goals

are. It may be different for each case, rather than based on a universal definition of ecosystem

management since there is none. A Chicago Wilderness practitioner sees the development of

three categories of indicators: �1) the state of biodiversity, 2) the state of the habitat, 3) the state

of what is driving those habitats.� He feels that if focus on these three areas remains consistent

during evaluation, then the other components (such as specific strategies and social goals) will be

evaluated in the process.

For the question of collaboration, an optimal level or form of collaboration is likely to be

context-dependent and not determined by any definition of ecosystem management.

Alternatively, what needs to be measured is not the amount or degree of collaboration itself

(number of joint meetings, decisions made together, number of years in a partnership), but the

quantity or quality of products of that collaboration or lack thereof (e.g. adding new alternatives

to the forest plan revision, setting up a large scale public education program). In that way you



6

would not need to rely on measures of successful collaboration such as �a sense of whether folks

that are outside of the agency feel collaborated with.� Instead you could look at what is the

ultimate goal of working with group x, and are they achieving that goal (despite or because of

�partnerships�). That is the way practitioners in the Chequamegon-Nicolet region currently

measures success of collaborative efforts: ��we got together a group of conservation folks�and

we all came with the same message [to the forest service] that we need a broader range of [forest

plan] alternatives�In the last few forest service meetings that we�ve had they said they would

create a natural range of variation alternative and they�ve also talked about taking the maps that

we�ve put together and using that as alternative too. For me that�s a huge step forward.�

Developing a common work plan across agencies would increase an overall or systems thinking

perspective of ecosystem management success. For an outside evaluator, if such an over-arching

plan did not exist (which it does not currently for any of the five cases, although Chicago

Wilderness is working on one) all the different agencies� strategies would need to be viewed as a

whole to see where each contributes to different ecosystem management components. A regional

perspective can foster integration, prioritization, and balancing of different components.

Chicago Wilderness hopes that �by putting all of [the member�s] individual goals and objectives

together in the work plan, the goal is to have them integrate and to talk with each other.� They

also hope to discern their main priorities and link them strategically to the budget. An overall

perspective at Kankakee reveals ways to alleviate conflicts between competing components: �I

think that all these programs can dovetail together to make a real nice ecosystem in the Kankakee

basin, but there�s a fine dance that has to be made through this. The farmers have to understand

that they�re not going to drain every square inch, and they�re not going to dump it into a river

channel that�s going to whish it all away across the state line into IL�And the environmental

people have to understand that there�s a farm economy and there�s a lot of people living in the

basin that depend on that river for drainage.�

As an overall solution to this problem of measuring against a non-existent ideal, one can take the

approach suggested by Yaffee (1999). That is, that heterogeneity of ecosystem management

definitions is good, and that different cases will be along a continuum of resource management

philosophies. This can be used to our advantage if we can see how they differ in their progress

2. Vague, Ill-defined, or Idealistic Goals


7

towards their own goals, and evaluate separately how close their goals might come to different

definitions of ecosystem management.


Knowing when success is achieved is difficult when goals are vague, ill defined, and/or

idealistic.

Challenges

In many cases, the ultimate goal of an ecosystem management project is to conserve or restore an

ecosystem. Other goals are similarly idealistic: �to preserve, restore and enhance all animals and

plants that are endangered or threatened with becoming endangered; restore and preserve a

natural diversity and abundance of flora and fauna; perpetuate the migratory bird resource; and

provide the public with additional high quality wildlife-dependent, public use, and environmental

education opportunities� (as said by the US Fish and Wildlife Service for the purpose of the

Kankakee National Wildlife Refuge). A different agency in the Kankakee River Basin hopes to

restore the river back to its free-flowing state. Vague and idealistic goals abound in ecosystem

management projects, and this is problematic for measuring success for many overlapping

reasons. 1) It is difficult to measure progress towards ill-defined goals. 2) There is a tendency to

monitor strategy implementation rather than overall progress. And 3) measuring forward

progress is hard because many projects are doing all that they can to merely prevent further

destruction. In other words, the goals are too idealistic.

Unachievable Goals In many cases, the way the goals are framed makes the task of measuring progress to attain them

difficult, because the goals are too vague, too broad, or too idealistic to achieve. For example, a

common goal of ecosystem management is to sustain viable populations. As a Fish Creek

participant said about the freshwater mussels: �We often don�t know what a �viable� population

is.� He goes on to say that their strategies are good for the mussels, but the population density

might be so low that it would be impossible to re-establish a viable population. As a result of not

understanding what is meant by the stated goal, practitioners often rely instead on coarse-scale

assessments, which may be laden with assumptions. As seen in the Fish Creek Case, �we�re

measuring success basically by acres in conservation tillage, acres brought out of cropland,�



8

even though their stated goal is in terms of mussel populations. Although Fish Creek

practitioners use biological monitoring data collected by contractors, their coarse measures of

farming practices are a more accessible evaluation metric. Similarly, a project leader in the

Chequamegon Bay Watershed case felt that ultimate success was �probably hard to measure, but

we could just assume that good forest management on these areas is going to lead to good water

quality downstream too.�

Implementation Versus Progress There is a tendency to monitor implementation of strategies rather than progress towards overall

goals. Success of ecosystem management can be amorphous ecological concepts or changes in

people�s attitudes or ethics (e.g., sustainable mussel populations, support of the public, harmony

among stakeholders), making their measurement difficult. However, measures of progress in

implementing strategies seem to be concrete and measurable. For example, interviewees

mentioned number of acres in conservation tillage, increase in vertical structure of the forest, and

number of people visiting a demonstration plot as indicators of progress. As summarized by a

Fish Creek participant, �[measuring success] is sort of hard because the strategic plan has goals

in it like �control soil erosion to tolerable levels for the ecosystem.� Nobody knows what that is.

We know that controlling soil erosion is good for the watershed, but nobody has yet figured out

what tolerance levels are for the animals that live in the stream.� This makes evaluation of

progress towards goals difficult, especially since strategy success may not equate with overall

goal success. (See issue #9)

Running to Stay in Place Sometimes ecosystem management projects are doing all that they can to prevent further

destruction of the landscape, making it very difficult to measure forward progress. Facing

increasing development in the Oak Openings region from the growing city of Toledo, an Oak

Openings leader expresses this frustration: �Sometimes we feel like the dike is leaking in 50,000

holes and we�re sticking our fingers in all over the place�it�s just a reality that there�s so much

impact on the landscape, it�s just enormous.� Partners in Oak Openings are discussing various

strategies to save natural areas in the region from being paved over. This is progress towards

mitigating threats, but not necessarily reflective of the goal to restore oak savannas.



9

Possible Solutions

In general, ecosystem management projects need to clarify and articulate specific goals or even

objectives within goals. Without a clearer understanding of the goals themselves, success will be

impossible to attain. In developing a conservation design scheme, which articulates objectives,

indicators, and thresholds for selected conservation targets, Chicago Wilderness is well on their

way. �We have been using the Recovery Plan for the vision, the overall vision of increasing and

improving grassland bird habitat. And then the next part of the design is to say, ok - what is our

more specific goal?� Chicago Wilderness is commencing with three conservation targets,

grassland birds, savanna herps, and woodlands but eventually hope to expand the process to

include all of the targets.

Projects need to use all available information to make vague goals clearer and thereby measures

of attaining those goals clearer. For example, a US Forest Service leader for the Chequamegon-

Nicolet National Forest Case explains that one of their goals is to bring the land base closer to a

�range of natural variability.� That is, they want to get closer to a historical state of the forest.

Since it is unclear what this state is exactly, they are using all available ecological information �

from paleoecological to climate data � to transform this vague concept into real ecological

objectives. To get at what that goal is exactly, he says �We�ve got some estimates of range of

natural variability that came out of research work from universities in the area�interpreting the

general land office notes, survey notes� We�re using the data that we have available from the

1830s along with some pollen data, archeological information and other things we have for

climatic conditions and knowledge of how the glaciers moved � all of this comes together to

make an estimation of what we think was the range of natural variability across a long period of

time�We�re trying to get a sense of that to get closer to the systems that the species that are here

now evolved under � how close can we estimate that. We�re still working on that.�

Ultimately, success is better measured incrementally, as a process of moving in the right

direction (change over time) versus reaching some particular point. Chicago Wilderness uses a

process called �conservation design� to break goals into specific objectives with thresholds.

�The way I conceptualize it (conservation design) is that it kind of gives teeth or creates a

blueprint, gives specifics for how to do the management and how to tell if you are successful.�

3. Multiple Stages of Ecosystem Management Projects


10

In essence, conservation design is a way to start measuring progress towards goals, by defining

specific objectives and thresholds within each goal.

The problem of working hard just to slow deterioration is a common one, and it needs to be

addressed in evaluation efforts. Lofty goals can serve to motivate projects. However, it is often

useful to couple these with modest steps to show progress along the way. In some cases the best

sign of progress may be a measure of threat mitigation rather than forward progress towards a

idealized goal.


Evaluations, particularly cross-case comparisons, must recognize that there are several different

phases of ecosystem management, which should be evaluated separately.

Challenges

Some ecosystem management projects have been in existence for several decades, others have

just recently come together, and still others are merely ideas on paper. There are milestones to

be noticed at each of these stages, whether the project is just in the planning stages, whether it is

working on implementing strategies, or whether it is trying to maintain some level of ecosystem

management. Furthermore, ecosystem management efforts often entail numerous smaller

projects, each of which has distinct stages that can be evaluated separately. Clearly measuring

success is dependent upon the stage of the project, and evaluation will need to be an on-going

process. Conservation of ecosystems is not a simple intervention that can then be evaluated; it is

an on-going effort. It is particularly important to acknowledge the different stages when

comparing successes across cases. Challenges to evaluation associated with different stages

include: 1) what is important to the success of a project may depend on the stage of the project;

2) measuring success often leads to the evaluation of implementation strategies rather than the

results of the strategies; and 3) different aspects of a project can be at different stages.

No Universal Indicator of Success Different things may be important to the success of a project depending on the stage of the

project. As exemplified in the Fish Creek Case, a partnership that may have been important to



11

the success of planning may not be as important for the actual implementation. �It was really a

useful partnership to get it started�and then once the project was established, the credibility and

trust was there - you know, there was not a big need for project partners to meet; it [grew into]

more of an informal thing over the phone.� So a snapshot assessment should not include that the

lack of collaboration is a failure, because it was used when needed and they still partner when the

opportunity arises, although it may bring into question whether or not the approach is ecosystem

management. Similarly, collaboration should not necessarily be taken as a sign of success. Oak

Openings members are involved in several formal and informal groups, but the initial working

group has been dissolved. Does this mean that the first working group was not successful in its

mission? Or is this a natural succession of stages? Furthermore, because Oak Openings and

Fish Creek are at such different stages, comparisons between the two should only be made with

caution.

Measuring success is difficult because it is hard to know what is needed at each stage of the

project. Moreover stages may be iterative, so successful completion of one stage should not be

taken as an indicator of progress in the next. For example, there is often a disconnect between

the creation of a management plan and the actual implementation of the plan. Measuring the

success of a management plan is complicated because while the mere existence of a management

plan or a restoration plan may be a sign of success, it cannot be considered a complete success.

This is seen in the Chicago Wilderness Case: �The purpose [of the meeting] last November was

to try to figure out how to implement the Recovery Plan� Ok, now we�ve got this lovely

Recovery Plan - what do we do with it?�

Successful Implementation Versus Successful Strategy A consequence of having several stages is that measuring success often leads to the evaluation of

implementation of strategies rather than the results of the strategies. For example, an initial step

in most ecosystem management projects is to set up an oversight group comprised of

stakeholders. The intention of such a group is to get people working together to accomplish

agreed-upon goals. In many instances, when questioned about measuring success, people will

say that one measure is that the group actually got together � not what the group eventually

accomplished. The Oak Openings Project is a case in point � success is getting people to work

together (not necessarily what they eventually achieve together). �In fact, that�s one of the goals



12

of this campaign - to provide more of a formal relationship among these groups where we can

agree on some common goals - Just the whole process of participating in this together.�

However, implementation of a strategy does not necessarily mean that the strategy was

successful in achieving the desired goals.

Strategies at a Variety of Stages Evaluation efforts should acknowledge that there are various stages that exist within the different

projects. In the Kankakee River Basin Case, different organizations are at different stages in the

overall process to restore the watershed. For example, the Grand Kankakee Marsh Restoration

Project has been working on implementing their goals since 1993, the Friends of the Kankakee

have been in the planning stage for just about a year, and the US Fish and Wildlife Service is in

the very early planning stages for a national wildlife refuge, i.e., they are working to getting the

refuge off paper and onto the ground. Each of these efforts will lead to improvement of the

Kankakee River Basin, but evaluation needs to take into consideration the different phases that

each subproject is in. Again, there is a need for systems thinking to link all the activities

together.

Possible Solutions

Practitioners and outside evaluators need to recognize separate phases within projects as well as

the different meaning of success at each. This is apparent to a Nature Conservancy

representative in the Fish Creek Case: �I would hope that someday we are at a point where we

can begin to step away from the actual implementation of things� I would hope that 15 years

after that, it begins to turn more into a monitoring type of thing� I guess the ultimate success

would be seeing native species repopulating the larger St. Joseph River. That�s probably too

ambitious, [but] that�s ultimate success.� As a result, evaluation processes should also have

multiple stages.

Creation of different success measures for each stage of a project is a useful solution. For

example, participants in the Chequamegon-Nicolet case developed criteria to evaluate the

planning phase, including the following. 1) Is there a contribution to unique and significant

features? 2) Is there a contribution to social vitality? 3) Is there a contribution to economic

vitality? 4) Are we aiming for ecosystem sustainability? 5) What is the administrative capability

� are you looking at things you can possibly accomplish or are you getting beyond your ability to

4. Multiple Time Scales


13

manage it? 6) How is the timing of things � how big and how fast of a change is suggested, what

is your social responsibility to the community? 7) Is there evaluation and adaptability � does the

plan include an evaluation system that would let you know whether strategies are working? Is the

plan put together in a way that you can adapt it and make course corrections? Furthermore,

successful implementation of that plan would be measured by checking progress towards goals,

and successful maintenance would mean you are keeping the status quo � i.e., not letting it slip,

meeting the demands of new stresses, and revising where and when needed. Such criteria can

help a project measure success no matter what stage they are in.

Along the same lines, one could break up the measures in terms of proximate and ultimate

success. Using the Fish Creek Case as an example, a proximate measure of progress would be

the number of acres that have been converted to conservation tillage, whereas an ultimate

measure of reaching a goal would be whether or not water quality improved or the diversity of

native fauna increased. The Fish Creek Council focuses on water quality measures, and The

Nature Conservancy considers both types of measures.


Because systems, goals, and strategies operate on multiple time scales, measuring progress must

account for time lags and out-of-synch strategies.

Challenges

Goals are often visions for the future state of things; strategies are often short-term attempts at

reaching long-term goals. The systems in place to implement the strategies to reach those goals

are often delayed by bureaucratic or logistic realities. The fact that just about everything works

on different time scales leads to many problems when trying to measure success. 1) It can be

difficult to evaluate the effectiveness of a strategy within an evaluation�s duration or even a

project�s lifetime because of time lags in the system. 2) Projects tend to move forward before

monitoring programs are in place. And 3) sometimes one strategy needs to finish in order to see

the success in another strategy.

Time Lags Time lags create difficulties in evaluation because the more time that elapses between

implementation of a strategy and its effect, the harder it is to find causal relationships. And with



14

most strategies in ecosystem management, a lot of time is needed to see results. For example,

ecological results often require several years, or even decades, to become apparent. In the Fish

Creek Case, this problem is seen by the connection between conservation tillage and water

quality. When asked whether they can see changes in the water quality that correlate with the

change in tillage, the response was: �I wouldn�t say that�We hope they are [related]. I think it�s

such a long-term change though.� Forest regeneration is another example of a strategy that may

take years before anyone can determine if the regeneration strategy met biodiversity goals. As

put so aptly by a Chequamegon Forest worker, �There�s really very little that you change in a 10-

year period. Take 150 and I could do something.� Given these ecological constraints, ineffective

or insufficient strategies may continue to be used for long time periods and never really achieve

attended goals. This means that strategies are followed, even though they have not necessarily

been proven effective. And reciprocally, it may take years to demonstrate that a given strategy

was successful.

Not all time lag issues are ecological. Most of the work in ecosystem management takes time,

or as one person said, �It takes a long time to move a mountain in the direction you want to move

it.� This is exemplified by bureaucratic operations in the Kankakee River Basin Case. Federal

interests have been involved in the project since the early 1990s � the idea for the national

wildlife refuge has been on the table since 1996 � yet, nothing has happened. As a Fish and

Wildlife Service leader stated, �Well, we had a political hoop to jump through.� Forward

progress on the refuge plan has been forestalled for a number of reasons.

Lack of Monitoring Programs When projects do move forward, it is often before monitoring programs are in place. Without

monitoring from the beginning, it is nearly impossible to know the baseline status of the resource

or to be able to measure the success of any early strategies. Knowing the baseline from where a

project is starting is crucial to the eventual measures of success. For example, Chicago

Wilderness estimates that it will take three years to develop a regional database. The regional

database will provide an invaluable set of monitoring data, but how does one account for the

many things that are going to occur on the ground before the actual database can be configured?

Measuring success is difficult in this example because it will be tricky to measure the early

progress of regional strategies.



15

Interconnected Strategies Sometimes one strategy needs to be finished before success in another strategy can be seen. In

many cases, public trust is essential for progress to occur. In the Kankakee River Basin Case, the

US Fish and Wildlife Service knows that they cannot progress without first garnering the support

of the public. �You can sit in a room with thirty people and explain until you are blue in the

face, but until they have personal contact with you, and they realize that you are an OK person,

[it does not feel like progress]. So, I look at progress as [making] little lists of who I�ve got to

contact, and every time I contact a person and get good information out to them, I feel like that�s

progress. Will it achieve success? Who knows what their attitude really is?�

Similarly, sometimes strategies are modified before causal links can be determined.

Organizations often change strategies for a variety of reasons. For example, a variety of

restoration activities are used to restore savanna and prairie habitat in Oak Openings. The land

managers discuss perceptions of effects of these strategies informally and proceed to make

modifications on these informal discussions. �Oh we discuss it constantly, so we are constantly

changing our management practices.� Constant change of strategies challenges evaluation

efforts, because often not enough time has elapsed to see the effects of any one strategy.

Generally the modifications are not based on any evaluation data.

Possible Solutions

An essential aspect of measuring success is knowing where the project started and whether or not

it has made a forward progression toward accomplishing its goals. Without monitoring systems

in place from the outset, success cannot be measured. One way to achieve this is to create a

game plan for gauging success. This is exemplified by the Army Corps of Engineers in the

Kankakee River Basin Case: �The way that we gauge that we are on target is that at the outset of

the study, we created a task oriented network analysis... we created a fairly detailed game plan.

We look at where we are with that game plan � where we are behind and where we are ahead and

where we are right on, and the progress that we needed to make by now to finish by August of

next year, and as long as we�re following that original plan, getting the answers that we need

along the way, then we are going to presume that we are not going to have any real shockers in

the future and we�ll stay on that plan.�

5. Multiple Spatial Scales


16

Measures of the same strategy will mean different things depending on the context. When asked

about evaluating success over the past 10 years and over the next 10 years, a Kankakee leader

said: �I think it�s easy to do. If we were focused on the Kankakee River in IL, it would be

different because in IL it�s a natural river; it�s a valuable resource. There�s already an active

recreational industry around it; the local economies get a lot of money from the Kankakee River.

In Indiana, where the river has been dredged and the marsh dried, it�s exclusively a liability. So,

we are starting from ground zero, and anything at all is progress. So, even if everything stops

right now, the Kankakee Marsh County Park in Lake County is 2,000 acres of wetlands, and 5

years ago, it was only 500 acres. So, even small definite progress shows big results.�

In addition, short-term and long-term goals need to be measured on different time scales and

with different emphases. For example, in the Fish Creek Case, measuring the success of whether

or not more farmers are signed up in the conservation tillage program will be very different from

whether or not mussels are able to repopulate sub-watersheds.

Another solution is to look at progress rather then success. �A lot of that change will take place

over many decades, and the primary thing you can do then is find measures or indicators of

change in the right direction.� This is the case with Chequamegon-Nicolet National Forest as

with several other cases. Small changes in the right direction should be accounted for when

measuring the success of goals that may take many years to realize.


Because systems, goals, and strategies operate on multiple spatial scales, it is difficult to infer

regional level success from site level measures of success.

Challenges

Dealing with various spatial scales is an ongoing problem for ecosystem management projects.

Defining the scale of the ecosystem is difficult in and of itself, as projects range from the scale of

a single watershed to a national forest to an ecoregion. Because most strategies are implemented

and organizations operate on smaller scales than that of the ecosystem management project,

evaluation at a regional scale is difficult for three reasons. 1) It is difficult to scale up monitoring

data and information from the site to the regional level within cases. 2) The importance of

regional level evaluation is not always recognized, and 3) spatial scale issues make it difficult to



17

define causal linkages. Additionally, cases operating on vastly different scales may not be

comparable in cross-case analyses.

Scaling Up It is hard both to conceptualize what needs to be evaluated at the larger scale and to logistically

amalgamate the site level information to represent the regional scale. In general, goals are often

at the regional level but strategies are intended for the site level. For example, Chicago

Wilderness contains numerous smaller natural areas, housing a variety of community types, and

much of the work to restore these communities is on the ground restoration, although the

coalition views and makes recommendations about all these areas as a collective. One Chicago

Wilderness practitioner commented, �We don�t have a really unified view of the ecosystem that

we can report on conveniently.� In addition, it is hard to evaluate the system at both the site and

regional level: �Again we are struggling with that� because the site level is the level at which

people will be doing the monitoring, and they have to be able to say something about that site for

the people managing that site, but then we also want to be able to extrapolate up to the region.�

The Oak Openings Case is dealing with similar issues in that each of the many townships in the

region makes their own local decisions, making it difficult both to implement region wide

strategies and to evaluate the status of the region. As a result, measuring success in such cases is

hard to comprehend. In expressing frustration about success measures after a planning process,

one participant commented, �one of the things I wanted to get us to look at was measures of

success of the whole project versus measures of success of what we do specifically here. And

that is really hard.�

Importance of Larger Scales The importance of regional level evaluation is not always recognized, even when site level is.

Evaluation on a site level may have direct benefit to a manager, but seeing the benefits of

working for a region-wide evaluation are less obvious, given the costs involved, because it does

not directly inform a manager about their work. Chicago Wilderness is making an effort to

recognize the importance of and design regional level evaluation.

Uncertain Causal Linkages Spatial scale issues can also make it difficult to determine causal linkages. When strategies are

applied on a site basis, but evaluation is occurring across the region, it is difficult to determine if



18

a regional goal has not been met because the strategy is not working or because it has not been

fully implemented at every site, or it is working differently at different sites, etc. As a member

of Chicago Wilderness noted, �It is a lot easier to conceptualize at the site level, because you

can say well we did not burn enough last year, we need to burn more this year. But at a region

you have to figure out how to know where the problem is - we did not reach it because county x

or county y, and we have not really figured that out yet.� Other difficulties in determining causal

linkages are discussed in section 10.

Possible Solutions

One way to deal with spatial scale issues is to keep monitoring at the site scale and to create a

database that can be used at the regional level. In addition, regional level measures are needed.

Chicago Wilderness is trying to tackle this by creating a central database into which everyone

can enter their data: �If each county puts in their own data, suddenly we have regional

information.� �The database would be comprehensive in terms of monitoring data and research

data, and all kinds of information so that everyone can go into one stop shopping.� And the

result is that �maybe we end up knowing more than we think we know because everybody is

sharing.�

It is also important to realize that measuring success at a regional scale may include several

successes at smaller scales. Additionally, being able to accomplish one piece of the bigger

puzzle may be one means to measuring success of the overall picture. In the Kankakee River

Basin Case, the Corps has one issue to deal with � flood reduction for the entire basin. If this is

accomplished, several other aspects of the ecosystem will benefit. It is one piece of the larger

restoration of the watershed, but the success of that piece is essential. For the Corps, measuring

success �is real easy. It doesn�t have anything to do with what the FWS is doing with the refuge,

or what any other particular special interest group may be looking for, whether they be

sportsters, fishermen, or people who are out to restore or protect the environment, farmers who

are looking to maximize croplands� while I have to deal with all of those folks, my assignment

is not so broad�I am trying to accomplish [one] particular thing with the basin as a whole �

flood reduction.�

6. Complex Ecological Systems


19


Measuring success is difficult because ecological systems are inherently complex and variable,

complicating interpretation of data, identification of useful indicators, and disentangling of

confounding variables.

Challenges

Measuring success of the ecological components of ecosystem management is a challenge,

because ecological processes themselves are so complex and difficult to track. Ecological

complexity and variability 1) call into question the reliability and interpretation of ecological

data, 2) confound measures of the direct effects of certain strategies and 3) make it difficult to be

certain which variables best represent a given ecological process or phenomenon. A common

but somewhat problematic response to these challenges is to abandon attempts to measure

ecological phenomenon accurately, and to rely instead on coarse-scale assessments of success,

which may not be sufficiently informative for adaptive management.

Natural Variability Natural variability over time and space may make practitioners feel that ecological data that they

or others have collected is unreliable or too difficult to interpret. For example, variations in rain

events preclude practitioners at Fish Creek from making conclusions on water quality at present.

��it just seems like the variations you get from year to year � different things are going to affect

the quality of the system, so you just can�t take 1 or 2 years worth of data and draw any

conclusion.� Natural variation is, however, a part of this and any system, and thus a

comprehensive assessment of success would have to take the effects of this kind of variation into

account.

Ecological variants that do not fit the expected pattern may reveal important information, but

require an additional (and often expensive) study to understand. At Fish Creek, The Nature

Conservancy is doing riparian zone reforestation to presumably help the mussel populations, yet

they found that �one of the densest populations of mussels happens to be in a spot where there

are few trees, in narrow riparian area. So, could just be the cold water above, who knows? I

don�t know, it�s a tough thing to judge.� The tendency may be then to ignore ecological variants



20

and assume that strategies are meeting the needs of most cases, rather than looking at the effect

of other ecological variables, such as local land-use, that might explain their observation.

Furthermore, because ecological variability is so complex, even the scientists collecting data for

land managers may argue on its interpretation. This only serves to further frustrate practitioners

with ecological variability since what they really need are best guesses. For example, with

respect to invertebrate and fish communities in Fish Creek �our scientists are arguing over

whether there was some problem with data collection or data analysis because some scientists are

saying no it didn�t decline and other scientists are saying yeah it did.� The result is that land

managers do not rely on ecological data and instead just keep doing what they feel is right.

Confounding Variables It is difficult to know the direct effects of a strategy when other variables are not controlled for.

Because of the large number of variables that contribute to any ecological phenomenon, a

controlled experiment with replication is needed to focus on the effects of single variables.

However, most practitioners, because of lack of organization, funding, or experience may not use

the scientific method to test, for example, the effects of prescribed burns and herbicide on prairie

restoration. As described by a land manager working to restore savanna in Oak Openings, �It

probably should/could be scientific method� but it is a lot of what I affectionately call trial and

error, but with land management techniques.� Lack of scientific methods will likely lead to

ecological puzzles: �and sometimes we have some puzzles like we do some burning and

sometimes we alter from spring to fall burning to favor forbs over grasses and sometimes we are

not really sure what is happening in some of the things. We just give it time.� Even though lack

of time is sometimes given as a reason for not setting up controlled experiments, the time

investment required to reach a conclusion with controlled experiments may be less than using

trial and error, especially since different ecological regions may not respond to treatments in the

same ways.

Representative Indicators Because ecological processes are complex, it is difficult to determine which variables or

indicators best represent a process. A lack of appropriate indicators in turn makes measuring

progress towards ecological goals problematic. At Chicago Wilderness they are attempting to

establish �indices that people believe are representing what you are trying to represent.� For



21

example, they are currently working on developing indices of grassland bird populations, which

is involving looking at currently used measures and asking practitioners to refine them by

evaluating what would be useful and what would not be. There is often no accepted measure of

important ecological attributes, because the direct cause of those attributes is unknown. For

example, vertical structure is an indicator of forest health used by the USFS in the Chequamegon

Forest. Whether this variable has ecological significance or captures the necessary aspects of the

forest health goal is unclear. Similar problems exist for measures of �sustainable� and �viable�

populations.

Coarse-scale Measures Ecological complexity may lead practitioners to abandon efforts to measure it accurately. If

practitioners find ecological data uninformative or too daunting to make sense of, it may lead

them to abandon efforts to monitor ecological processes accurately or comprehensively, and to

rely instead on very coarse-scale assessments of ecological success, such as forest is better than

crops, or more species better than fewer (as discussed in lesson #12). These assessments may

not provide adequate information to judge whether strategies are in fact improving ecosystem

health and thus would fail to alert practitioners of the need to change strategies.

Possible Solutions

The difficulties associated with ecological variability over space and/or time may be reduced by

looking at overall trends. To deal with variation over time in ecological data, TNC Fish Creek

measures progress by looking at trends over several years as opposed to any point in time. �So

you just can�t take 1 or 2 years worth of data and draw any conclusion. There�s a trend we hope

to see vs. one year�s data�right now we�re just trying to build that trend. Taking the annual data

and plotting it out.� Oak Openings also looks for trends, �Sometimes we are not really sure what

is happening in some of the things. We just give it time. I would say it would not even be

unusual to give something like 5 years.�

Similarly, variation over space could be dealt with by looking at trends over several sampling

points. Data collected by different agencies could foster this, if collaboration has led to those

data, such as inventory of specific species, being collected in a similar fashion, as in the case of

DNR and USFS in Wisconsin. For example, �occurrences of rare plants and animals, outstanding

examples of natural communities and aquatic features throughout WI� are collected.

7. Lack of articulated social and process goals


22

Application of the scientific method or even meta-analysis of existing data may clarify causal

relationships. More applications of the scientific method to evaluate strategy effects would be

helpful, although even in controlled experiments there is some level of statistical uncertainty to

deal with. Adaptive management would need to incorporate some decision-making criteria (a

threshold level of effect or number of times a result was obtained) for whether they would

change strategy based on results. Where agencies themselves were not able to engage in

controlled experiments, meta-analysis of existing data could prove very useful (see value of

evaluation, lesson # 11).

Tapping into existing scientific knowledge would increase understanding of complex ecological

processes and phenomenon. This would help both with understanding treatment effects and

determining appropriate indices of ecological phenomenon. On the other hand, waiting for all the

ecological knowledge to accrue is obviously not a solution. Chicago Wilderness is trying to build

on indices already in use by specific groups or counties, and where none exists to start with

educated guesses. It needs to be �an evolving iterative process that will be refined over time as

people are able to fill in those blanks.� Working with universities provides some help with

knowledge, but has definite limitations, namely different agendas and short-lived graduate

student projects.


Indicators of social and process goals are not as recognized by practitioners as are ecological

goals, and therefore are often not articulated or measured, making it difficult to evaluate overall

progress.

Challenges

When asked about measures of success and goals, almost all of those interviewed immediately

jumped to ecological goals � numbers of acres protected, restored hydrology, preserved

biodiversity, etc. Rarely were social and or process goals even articulated in the interviews,

which is telling of the situation in the field. Without the articulation of social and process goals,

evaluation efforts are challenging, because 1) it is difficult to determine what to evaluate with

respect to these areas; 2) when goals are mentioned, they tend to be fuzzy concepts, and 3) the

lack of such goals illustrates a lack of systems thinking, making it difficult to conduct a system-



23

wide evaluation. Lack of articulated social and process goals leads to two resulting behaviors: 1)

a reliance on gut instincts instead of measured results, and 2) processes that may lead to

undesirable results because they were not evaluated. Furthermore, successful implementation of

social and process goals are often needed to achieve ecological goals, and such incremental steps

could provide a way of measuring progress.

Lack of Social and Process Goals Most organizations within ecosystem management cases are more ecologically oriented, and

therefore ecological objectives are frequently given priority over social and process objectives in

collective work. It is difficult to evaluate progress towards something that is not considered a

goal. However, even though many ecosystem management projects have goals such as

ecological integrity or health, the ecosystem can not be separated from its human components.

Therefore social well-being issues are often important to ultimately achieve the long-term

ecological goals and should be worked on and evaluated.

Fuzzy Concepts When social or process goals are mentioned, they tend to be fuzzy concepts that are not easy to

measure. For example, Fish Creek workers know that establishing trust with the farmers is an

important component. As stated by one member, �Being in such an agricultural setting, the

farmers � it takes a long time to get their trust, and to finally get their trust is quite an

accomplishment.� But how should something like trust be measured? Currently in Fish Creek,

it is being measured by instinct � how well the project managers feel they know the farmers.

Collaboration is also often measured through instinct. According to a Chequamegon participant,

�One [measure] is just my sense of whether folks that are outside of the agency feel collaborated

with.� Even though he feels he has a good sense of the public, the same respondent admitted,

�But if I was able to know that�we had a good sense of harmony amongst the pieces, that

would be useful.�

Many social constructs are difficult to articulate, but this does not mean that appropriate

measures and indicators should not be developed. Instead many practitioners rely on their

instincts to measure progress. A Chequamegon worker noted, �my gut reaction is that the more

people get educated about forest management, the better management is going to go on in the

watershed.� While many ecosystem management practitioners are trained in the skills of



24

ecological methodology and testing, fewer have skills in social science experimentation.

Linkages between, for example education and watershed management could be demonstrated if

the appropriate mind-set and skills to carry out the work existed. This lack of commonly thought

of ways to measure social/process success means that they get short shrifted relative to

ecological monitoring. While many ecological variables are measured for the sake of monitoring

with no a priori connection to evaluation questions, measures of social and process indicators are

not even collected in monitoring schemes. Such a mindset is often not present. However, there

is some evidence of improvement. Oak Openings implemented a public awareness campaign,

and then hired a consultant to conduct an evaluation of its affects.

Benefits of Well-Defined Process A lack of process goals can lead to haphazard implementation of strategies, because there is no

overall plan for how the process should unfold. The lack of planning procedures can make

measuring progress difficult, because incremental steps are not identified. For example, the

project coordinator for Chicago Wilderness recognizes the need to integrate various efforts

together if regional goals are going to be achieved. While it may seem obvious that

communication is the key for linking people and projects together there is not yet an articulated

process for improving communication. �There definitely has been communication among

people. Sometimes it is because they happen to sit on more than one working group, and it is

sometimes because I am forcing them to have conversations or whatever it might be.� This

haphazard approach makes it difficult to determine what strategies to meet

communication/integration goals might be effective.

In the atmosphere of urgency surrounding most conservation efforts, the predominate attitude is -

just get things done, doesn�t matter how. While this attitude may be a benefit in some situations,

the lack of well-thought out processes can lead collaborations down undesirable paths. In order

to help its member agencies implement needed work, Chicago Wilderness started infusing

money into various efforts. This strategy had the unintended effect of causing members to lose

sight of the overall goals and instead look to Chicago Wilderness for project money. Leaders of

the collaboration are now working to change these perceptions. As noted by one, �I am trying to

almost change a mindset of how a lot of the members think � oh we will just go to Chicago

Wilderness for money � and they don�t really get what we are trying to do here.�

8. Extraneous and Conflicting Goals


25

Possible Solutions

Suggestions to deal with this complication cover a broad range. At one end of the spectrum is

heightening awareness of the importance of social and process goals and how measuring their

progress can be a predictor of overall success. When asked what she would see as a good social

measure of success, a member of Oak Openings replied, �My measure of success from the social

level would be that when there is a public hearing that people would attend and start speaking up

for quality of life issues which would include natural areas.� Getting to the point where people

recognize that ecological issues and social ones are interconnected is an important goal for

ecosystem management.

At the other end of the spectrum is an understanding of the value of qualitative evaluations.

While there might be a tendency to develop some numerical indicator to parallel some of the

ecological ones developed, such an indicator may not be very useful. A member of the

Chequamegon-Nicolet project noted that �to boil that down to a number, I don�t know what that

would mean to me.� It is unlikely that a social or processes index would be useful, except maybe

on a broad comparative scale (e.g. all the national forests) where narratives would be impractical.

Within a case qualitative information may be more useful for improving upon strategies. As is

true with all indicators, attention must be paid to interpretation. For example a change in

awareness does not necessarily indicate a change in behavior.

A first start to aiding evaluation is the inclusion of social and process goals within ecosystem

management plans. Identification of goals makes tracking of progress towards those goals

easier. Providing a list of sample easy-to-measure goals may help agencies think about these

issues.


When member organizations of ecosystem management have diverse agendas, evaluation is

complicated by goals that are extraneous, conflicting, and/or competing with ecosystem

management goals.



26

Challenges

Evaluation of ecosystem management cases where member organizations have goals beyond the

scope of the ecosystem management project is difficult for three reasons. 1) The sub-

organization working towards ecosystem management goals may be overwhelmed by larger

organizational goals. 2) Indicator selection may be complicated because not all organizational

efforts contribute towards the ecosystem management goals. 3) Organizations become spread so

thin that various project may have insignificant contributions to overall goals. 4) Organizations

may have other goals which are incompatible with the ecosystem management goals.

Small Effort in a Big Pond A branch of a larger organization may be working very diligently, and apparently effectively

towards ecosystem management goals, only to be thwarted later by the larger organization�s

overriding decisions, budget processes, etc. Maumee State Forest has a hard time carrying out

many of its initiatives because its parent organization, Ohio DNR, has many other initiatives

underway in the state. Because land near Toledo is expensive compared to the rest of the state,

acquisition funds are often spent where land is much cheaper. The result is that the State Forest

has difficulties securing necessary funds for these innovative projects, because Oak Openings is

only one of many DNR priorities. Success to them is just initiating a project.

Complications with Indicator Selection Indicator selection can be complicated by organization goals extending beyond the ecosystem

management project. For example, a prime strategy for Oak Openings is to acquire land, and

Toledo Metroparks is efficient at securing dollars to do so. However, dollars dedicated to land

acquisition would be a misleading indicator for Oak Openings, because Metroparks uses the

same fund to also purchase land outside of the Oak Openings region. Many of the larger

organizations and agencies � The Nature Conservancy, Fish and Wildlife Service, EPA, that are

involved in these ecosystem management projects are also involved in similar activities outside

of the region, therefore, care must be taken in interpreting measures taken at the whole

organization level.

Small Token Projects Organizations can have their resources stretched very thinly when they get involved in too many

projects, making it is hard to do many things well. For example, the Maumee State Forest wants



27

to contribute to Oak Openings goals. Thus, they have initiated a prairie restoration project, but

the scant resources available to put into this project in light of all the other work that needs to get

done within the means of their mission means that this is little more than a token effort. In these

cases, project initiation should be interpreted with caution, as it may never be completed or it

may be on such a small scale that it can not contribute significantly to the regional effort.

Incompatible Goals Sometimes organizations pursue multiple goals that may not be compatible, which ultimately

impinges their ability to fully reach one or more of these goals. The Toledo Metroparks is very

involved in the preservation of the Oak Openings Region, but at the same time it must serve as a

recreation facility for the public. When asked if there is a balance between recreation and

preservation, the answer was, �No. They are incompatible. They really, are, ultimately they are

incompatible.� The issue of goal compatibility relates back to the definition of ecosystem

management (issue #1). Does ecosystem management need to include both social and ecological

goals? Do they need to be compatible? Some practitioners feel that a balance between social

and ecological goals cannot be achieved. Also how would one compare a case emphasizing

social goals versus one emphasizing ecological goals? Is one doing better than the other?

Possible Solutions

There are several different ways to approach the complications imposed by participant

organizations having multiple agendas. For large organizations, in which only one branch is

involved in an ecosystem management project, giving that branch significant decision-making

authority can make a big difference not only in getting the work done, but also in simplifying

evaluation efforts. As an Oak Openings representative notes, � �I think that�s one of the really

good things about TNC is that we are given a lot of autonomy in our project offices to make

these decisions.�

It also simply needs to be realized that many participant organizations cannot devote one

hundred percent of their time and resources to the ecosystem management goals. This in itself is

not a problem, but evaluation efforts should acknowledge it, as it can lead to other evaluation

complications such as spatial and temporal scale problems. Indicators need to reflect the work

that a given organization is doing towards the ecosystem management goals, and not overall

organizational work.

9. Single Strategy versus Holistic Goal


28

The main solution to this difficulty is to develop and implement a work plan. A work plan

demonstrates how the work of various organizations helps to achieve the whole. For example,

Chicago Wilderness is asking groups to show how projects will further overall goals �as opposed

to just a hopscotch approach of this is a good project and this one sounds interesting.� When it

can be articulated how various smaller projects fit into an overall scheme, evaluation of the entire

plan becomes easier.


Because successful implementation of strategies may not equate with overall ecosystem

management success, this cannot be used as reliable measure of progress.

Challenges

In the cases, there is a strong tendency to measure success by looking at whether or not certain

strategies are implemented. In some instances the next step is taken, the step to see if the

strategy is working as intended. However, the success of a strategy may sometimes be

misleading and should not be equated with ecosystem management success for several reasons.

1) Strategies may not be sufficient to reach ecosystem management goals. 2) Short-term success

may not mean long-term success. 3) Strategies may not really be implemented to achieve

ecosystem management goals (at least not directly). 4) Some strategies may have unintended

consequences. And 5) differential progress on different strategies makes a measure of overall

success difficult.

Strategy Not Sufficient While it may be thought that a given strategy is working well, it may not be sufficient to achieve

the goal. Ecological complexity and lack of causal linkages may make evaluation difficult. For

example, Fish Creek has been working diligently for many years to convert farming practices to

conservation tillage in order to reduce sediment loads in the river. Reduction of sediment will

certainly help the endangered mussels, which is one of the ultimate goals of the project. Whether

or not this primary strategy will be sufficient to recover the mussels remains unknown. �If we

get up to 70% [conservation tillage] and the indexes that we are monitoring are not at least

trending up or maintaining, I think then we are going to have to say to ourselves, �Did we miss

the target?�� In trying to convince the public of the worth of a National Wildlife Refuge, a



29

Kankakee participant recognizes that progress does not necessarily lead to success. �every time I

contact a person and get good information out to them, I feel like that�s progress. Will it achieve

success? Who knows what their attitude really is?� FWS is making progress is talking with

people about the reserve, whether or not this will translate into the creation of a reserve remains

unknown. The danger here is that a successful evaluation of a strategy can give false impression

of overall success.

Sometimes strategies are not sufficient to achieve goals, because difficult to address or

overwhelming threats are avoided. For example, hydrology issues are a big threat to the

preservation of the Oak Openings. �We�ve always said that water, ground water withdrawal, is a

big problem, but, at the same time, a lot of us have always said, what can we do? Is there

anything that we can do realistically about it? I�m not sure that there is.� So even if all

strategies implemented are successful, goals may still not be reached, because some threats were

never addressed.

Short-term Versus Long-term Success A strategy may have short-term success but then break down in the long-term. We saw a number

of examples where this might be the case. In Fish Creek, monetary incentives as well as

whatever other creative means that present themselves, are used to convert farmers to adopt

conservation tillage. This strategy appears to be effective, however, what if a developer were to

come along and offer a lot more money? Instilling a conservation mind-set in the community

has not necessarily been a part of the strategy, so any gains made today may not hold in the long-

term. Members of Oak Openings are devising a land acquisition plan, yet it is being kept quiet.

�[The plan] has not been publicized yet, because as you know, land acquisition can create some

real polarity among the community and we just had a real severe backlash.� The implementation

of such a plan may have long-term consequences that prevent achievement of their goals.

Another example, common to all cases, is the fact that most politicians only work on projects

that will help them get elected. As one Oak Opening member said, �These politicians are only

going for short term.� Some strategies needing political support may be geared towards short-

time frames, but then be disregarded if there is a change in political leadership.



30

Understanding Incentives Caution should be taken at every stage of ecosystem management in terms of interpreting the

significance of various actions, because not all strategies are implemented purely for the purpose

of achieving the ecosystem management goal. Sometimes organizational members are required

by legal mandates to implement certain strategies. The strategies may be conflicting or not

completely compatible with ecosystem management goals, or they just might not be the most

efficient means of achieving goals. This situation is a problem for government agencies. As put

by a USFS employee working in Chequamegon, �I�m a public servant and I�m trying to figure

out how to apply what the public has said they want on their land.� Public involvement is

generally desired in ecosystem management, but if consensus or compatible goals are not

reached, the implementation of a variety of strategies may counter one another. In Oak

Openings, �the biggest challenge is finding the balance between public access and preservation.�

In the Kankakee, The Army Corps of Engineers is legally mandated to reduce flooding, which

aids restoration but may impede work of others also trying to restore the area. As said by a Corps

member, �For me, it�s easy because I have an assignment from Congress and it�s spelled out in

the law�It�s intended to deal with one issue and that issue is flood reduction.� Complying with

the law can be interpreted as success; the mandate can focus people onto one aspect, but this

diminished the whole picture view.

In addition to legal reasons, strategies can be implemented just because a good opportunity

arises. Funding and good PR are two reasons why strategies that may not be the most direct path

to achieving ecosystem management goals are implemented. As one person involved in Oak

Openings said, �Honestly, I think the Port Authority was doing it [participating in the working

group] for good PR. So they could say we are working on things, we are trying to preserve the

Oak Openings.� Or in the case of Fish Creek, Monsanto donated $275,000 to promote the

conservation tillage strategy. While reducing sediment has positive ecosystem effects,

conservation tillage can mean increased herbicide use. Although people argue both sides of the

issues, The Nature Conservancy conducted a study on the side effects of this strategy and they

are comfortable that it is a good one to go with. Generally the same herbicides are used with

both conventional farming techniques and no tillage. {Should these statements be removed?

TNC does not see it as a perverse incentive; this more reflects our views of Monsanto}

Following opportunistic strategies is not necessarily a bad way to get necessary work done,



31

however an evaluation of such a case must not fall into the trap of just looking at the strategies

being used. A more holistic picture is needed to evaluate how the overall effort is progressing

towards its goals.

Because multiple reasons may lie behind the use of a strategy, caution is needed when evaluating

them. For example, participation in planning meetings should not be interpreted as commitment

to or interest in group goals. Members of Oak Openings attended working group meetings for a

variety of reasons. � �My director told me yeah you can go to the meeting, but don�t make any

commitments.� Attendance of organizational members at meetings should be interpreted

carefully, because often they have little to no decision making authority. This is particularly a

problem for organizations that are headquartered outside of the ecoregion such as the case with

Ohio DNR and Maumee State Forest. Furthermore, planning is not a good indicator of action.

Many organizations do well at the planning stages but then fall short during implementation,

which means that evaluators must also look at what is actually implemented, not just agreed to.

While organizations can agree that certain strategies would be worthwhile, someone must be

willing to implement the strategy. Chicago Wilderness is at a point where, �We�ve got the

priorities. What we really need to do now is go out and find who is interested in doing this work.

And what is it that excites the members and what do they have energy to do? Because if they

don�t have the energy to do it, it�s not going to get done.� And if nothing gets done, then goals

will never be reached.

Unintended Effects Well-meaning strategies can have unintended effects. For example, Chicago Wilderness started

a granting program to bring more resources to the work of its members, but this granting process

ended up influencing how some members think about the coalition. A project manager laments,

�We don�t want to be known as Chicago Wilderness the granting foundation where our members

just look to us as a source of money.� Instead of unifying them toward working collaboratively,

in some instances the granting program created a competitive atmosphere. In this case, an

indicator measuring project success may mislead evaluation of overall success, because while

more funds may be going to projects, the process may be undermining other goals, such as the

building of collaborative relationships.



32

Multiple Measures Differential progress on components or conflicts among components makes an overall index of

success meaningless. Progress towards different goals can take place independently such that

work on one goal may be going well while work on a different goal is not. This is especially

complicated to deal with when success of different goals is mutually exclusive. As a USFS

practitioner expressed it �ecosystem management success is looking at how you are balancing

competing goals, and that can not be measured broadly.� If all different goals are �scored� and

then pooled, �pluses� and �minuses� might cancel each other out and the overall broad score

would be meaningless. �For example,� he says, � as you are trying to accomplish a healthier

vertical structure, what else did you do for competing needs? Did you meet the needs in one area

at a bigger cost in another area? Are those measurable? Yes, they are. But they are measurable

primarily in specifics�to try and make an overall grade of the total package�how does that

meet that nice philosophical goal? I don�t have an idea of how you�d score it that way.�

Possible Solutions

When strategy success does not equate with overall ecosystem management success, evaluation

must determine not only how strategies further ecosystem management goals, but also where the

gaps in progress are. Chicago Wilderness, in general is trying to take a more strategic approach.

�We have money for year one, but the point of identifying work for years two and three is to

figure out where we want to fundraise, so we are not just limited to those federal funds and also

again to think more strategically and not just do a great project here and a great project there but

have them build on each other and capitalize on what we have invested.�

Another tactic taken by Chicago Wilderness is the development of criteria for what makes a

good ecosystem management project. This not only helps their implementation keep on track

with goals, but it aids evaluation in that it charts how the progress of different strategies connects

to larger goals. Chicago Wilderness is asking groups to show how projects will further goals �as

opposed to just a hopscotch approach of this is a good project and this one sounds interesting.�

Chicago Wilderness�s State of the Region program is furthermore designed to help illuminate

where the gaps are, threats that strategies are currently not addressing.

Never does anything go exactly according to plan. Strategies will be undertaken for a variety of

reasons. Various opportunities arise. Just because strategies are not implemented according to a

10. Causal Linkages


33

schedule does not mean that they do not contribute to progress. However, during an evaluation,

noting reasoning behind implementing a particular strategy may help to uncover how its progress

is leading to overall goals and show where certain strategies are primarily being undertaken for

other reasons.

Not only does adaptive management help to determine cause and effects, it is also an effective

means of determining when strategies have unintended consequences. �So you have to have an

adaptive management process with good monitoring so that you can see whether or not the

system turns in a direction that you intend it to.�

Blending the conflicting needs of people may be done subjectively or impartially. For example,

the USFS member in Chequamegon explains a scenario of doing a good job on improving

biological conservation, but as that is done �it reduces access or changes from early successional

to late successional habitat and the grouse hunters and people who love deer are very unhappy.

So now I have a negative on one and I have a plus on the other. Did I succeed or fail?

Interviewer: What do you think? Is there an answer to that? Respondent: Yes, but it�s a

subjective answer.� Being impartial, on the other hand, may mean equal lack of success across

components. As a Kankakee practitioner says, �I will tell you that there�s the full spectrum of

folks and how they see things� we will probably finish the study having no one happy. Maybe

the best answer is that we mistreat every party somewhat equally.�

10. Causal Linkages

The lack of detectable causal linkages, such as in the case of time lags or simultaneous use of

many strategies, complicates evaluation of strategy success.

Challenges

Evaluation of strategies rests in part on being able to determine the effects of the various

strategies implemented. If such causal linkages are not determined, it is impossible to tell if the

observed effects are the result of the strategy in question, a different strategy, some combination

thereof, or some outside factor. Despite the importance of causal linkages to evaluation, such

linkages are difficult to find for a number of reasons. 1) There are often long time delays

between cause and effect. 2) Multiple spatial scales make it difficult to determine causal

linkages. 3) Often many strategies are used simultaneously. 4) Monitoring is often not designed

10. Causal Linkages


34

to answer causal questions. The problems to measuring successes posed by multiple temporal

and spatial scales are addressed in different sections. (See lessons # 4 and 5).

Simultaneous Use of Strategies Causal linkages are difficult to find when many strategies are used simultaneously, which is

often the case. For example, multiple strategies can be used to reach the same goal or multiple

organizations can implement similar strategies. In Oak Openings, an effort to educate the public

about the region followed several routes. Two organizations put together a travelling exhibit and

a third organization mounted both a radio ad campaign and posted billboards along the

highways. A private consulting firm was hired to conduct a survey after the radio ad campaign,

and in that they asked participants where they had heard of Oak Openings. So it is possible to

separate out the effects of different strategies through evaluation techniques. However, because

the travelling exhibit strategy was not part of the evaluation nor was there a pre-test component,

it is hard to say how much the exhibit might have made the public more receptive to the other

two strategies. Use of multiple strategies can be effective and efficient, but the evaluation needs

to be designed to distinguish between the different effects if any lessons are to be learned. As a

member of Chicago Wilderness summed up the problem of simultaneous strategies, �There are

so many moving pieces that it can be bewildering even for the people in the thick of it.�

Knowing the effects of each strategy makes it easier to assess success.

Poorly Designed Monitoring Programs Most of the time monitoring is not designed to answer causal questions. Either there is

monitoring for monitoring sake or little to no monitoring. All five cases had some sort of

monitoring going on. In most cases, monitoring efforts were at the site level and orchestrated

only within a single organization. In addition, in most cases, monitoring efforts were relatively

long term and set up to collect data believed to be important or useful. However, as one person

noted, �We have got so much information that we almost have no information.� This overload

of monitoring data occurs when evaluation questions are not established a priori to guide

monitoring efforts. Without specific evaluation questions, there is a tendency to monitor

everything (that is easy to monitor). Not only is this tendency an inefficient use of resources, it

tends to cloud evaluation prospects, because of the wealth of information. While monitoring can

answer questions such as is the population increasing or decreasing, it often can not say anything

10. Causal Linkages


35

about why such a trend is being observed. If the goal is reached, perhaps knowledge of causal

linkages is not needed, however, for the strategies to be implemented elsewhere or if the goals

are not met, it would be nice to know which strategies had which effects.

Often such full scale monitoring does not occur because there are simply many barriers and costs

to implementing complete evaluation schemes. Monitoring is time and labor intensive. Often

adequate resources do not exist to carry it out. Evaluation schemes should work to overcome

this challenge by developing simple, easy to measure indicators. Lack of easy to implement

monitoring programs often leads to relaxed attitudes about monitoring. In Oak Openings, for

example, �We really kind of informally document that we are doing that. It would probably be

better if we did a more formal documentation of that but we just don�t have time to set it up. It

probably should� could be scientific method involved and so forth but it is a lot of what I

affectionately call trial and error.�

The lack of concrete causal linkages or ways to measure them, often lead practitioners to rely on

anecdotal evidence. For example, one member of Oak Openings said, �How do I know that

[there is increases public awareness]? People say I heard this on the radio, or people would ask

about native plants.� Anecdotal evidence can be used as a measure of progress, but it does not

build a very strong case if funding or other resources are on the line.

Possible Solutions

Because causal linkages are difficult to find, one of the best solutions is to set up the situation to

improve chances of finding causal linkages. Noting that �trends are tough in a two year time

frame,� a Chicago Wilderness member suggests that periodic conferences to assess status might

work because �you are trying to set yourself up so that you get better trend information in the

long term.� In other words, it will take time to develop causal linkages, but as long as this is a

goal then evaluation work can build towards this aim.

Staying focused on the evaluation goals will also help to illuminate progress and trends over

time. As noted by a Chicago Wilderness member, �State of the Region can report on how the

birds and bunnies are doing, but if you keep the discipline of looking at the state of the living

system and habitats then you will naturally enough look at programs and stressors. We said that

some big things had to happen to address those stressors, how are we doing?� Consistency in

11. Unrealized Evaluation Plans


36

evaluation is important to track progress, particularly where it is not occurring as one might

hope.

Because development of indicators can be labor intensive and difficult, a member of Chicago

Wilderness suggests starting subjectively and then building up to indicators. In other words, start

with expert opinion to determine if the status of the resource is �poor,� �fair,� or �good.� This

was the approach take by State of the lakes Ecosystem Conference (SOLEC) in the beginning.

Subjective measures will help to keep participants focused on the evaluative components and

therefore searching for the correlations and linkages.

Probably the most called for method for establishing causal linkages in the literature is through

the adoption of an adaptive management approach. Certainly if evaluation plans are integrated

into strategy implementation plans, the likelihood of finding causal linkages and the ability to

draw conclusions about a strategies� effectiveness greatly increases. It is much harder to

evaluate a strategy post facto than it is to set up the evaluation plan along with the

implementation. One way to work towards adaptive management is to change the general way in

which monitoring is conducted. Monitoring programs should be designed to answer specific

evaluation questions about the use of strategies. Rather than placing time and effort into

collecting data that probably will never be used, more effort should go into anticipating

evaluation needs. What information does the program need to know about its progress? This

linkage between strategies and monitoring is often overlooked.


Evaluation is difficult, because, although its importance is recognized, evaluation steps are often

not put into practice, leaving little information on which to base progress.

Challenges

One of the largest challenges to measuring success of ecosystem management cases is that in

most cases there is a lack of an existing and functioning evaluation plan. Meaningful evaluation

requires that the agency recognizes the value of evaluation, develops an evaluation plan that

corresponds to their goals, and then actually carries out this plan. There are mental and

organizational barriers to achieving each of these requirements, which makes evaluating

ecosystem management success difficult both from a within-agency or case perspective and for



37

an outsider evaluating one or many cases. Furthermore, lack of evaluation can lead to the

problem of non-adaptive management, since the success of individual strategies is not routinely

questioned. For example, agencies may pursue the same strategy for many years without

considering alternative strategies.

Many of the barriers to evaluation are the challenges discussed in this paper, but specifically the

mind-set and organizational barriers to evaluation are what will be discussed here. These are

important to consider, because even if an ideal evaluation system were developed, its

implementation may fail within an agency due to these types of barriers. Evaluation plans are

often not implemented because 1) there are other competing priorities, 2) it becomes an

overwhelming task, or 3) it is incomplete because the results are not perceived as useful.

Other Priorities Although the value of evaluation may be recognized, it is rarely made a priority task. There is a

general consensus that at least at some level (see lesson # 12), it is important to have measures of

success. For example, a TNC worker in the Chequamegon Bay Watershed said �we should be

asking these questions before we implement a strategy, because if we can�t measure our

effectiveness how do we know whether we�re spending our money in the right spots.� Similarly,

a Chicago Wilderness participant said �it is certainly something we should think about, and I

think every step of the way we will look back over our shoulders and say did this make sense and

did this make good use of our time and energy and so on.� Chicago Wilderness practitioners

have also recognized the value of evaluation in terms of being able to show potential funders a

clear need for their continued support and to prove accountability to the public.

In a few situations the need for evaluation is not recognized. For example, the value of regional-

level evaluation is not acknowledged in cases where local sites each focus on the need for their

own monitoring plans. Individual members in Oak Openings are working on their own internal

monitoring plans, but as of yet there is no region wide monitoring effort. Chicago Wilderness is

currently experiencing the balancing act between the needs of site level evaluation for each

organization and of region-wide evaluation for the whole effort. As noted by a coordinator of

the evaluative processes of conservation design, �there is a little bit of skepticism from some of

the land managers who think things like � we are already doing this; we don�t need this kind of



38

thing.� In these cases, important uses of evaluation, including how regional information can

bring support to local efforts, are not fully understood.

Despite the value of evaluation generally being recognized, it is most often not a priority.

Evaluation plans are generally not as formalized as lists of objectives and strategies, and most

time spent on the job is devoted to doing, not evaluating. In the mix of so many pressures on

time, money, and resources, implementing evaluation plans often gets lowest priority. This stems

from actual challenges to measuring success [see other lessons], or the daunting nature of

evaluation. But it seems that it also is due in large part to a lack of an evaluative or analytical

mind-set. This is indicated, among other things, by a) initial responses to the question of how to

measure success, b) statements that reflect a focus on doing, and c) a lack of use of collected

data.

Overwhelmed by Task The result of a lack of an evaluation mindset or of making evaluation an organizational priority is

that many forms of evaluation are lacking, not systematic or not documented. Subjective (gut

feelings) or anecdotal (bird sightings, comments from people) evaluation presents the problem of

how to move from the individual evaluation level to an institutional one. To recognize the

importance of evaluation is one thing, to have some ideas about how that might be done is

another. Even some of those practitioners who had strong feelings about the need for evaluation,

had not thought deeply about what measures they might or should be using. For example, in

response to how he might measure success of a currently implemented forest-management

education program in the Chequamegon-Nicolet, a project director replied, �I don�t know.. that�s

a good point. We obviously want to see change in certain forest management out there. I�m not

sure how you actually measure it� I guess we haven�t thought that far in advance�I have a

hard time, other than just saying that my gut reaction is that the more people get educated about

forest management, the better management is going to go on in the watershed.� This initial

response was not atypical. That is, several practitioners had difficulty listing potential indicators

of success and discussed gut feelings or anecdotal information instead. Exceptions to this include

a practitioner from the USFS, which is required by law to have documented evaluation systems,

and Chicago Wilderness, which is actively engaged in developing its evaluation systems.



39

In addition to the use of intuitive evaluations, another sign of feeling overwhelmed by the task of

evaluation is the documentation of strategies used. There is a tendency to equate evaluation of

strategy effectiveness with documentation of what is being done. For example, in response to the

question, �Are you [determining effective use of herbicide] systematically?� an Oak Openings

project director replied, �More of it�s random. It�s really obvious � we don�t really need a plot.

[Our site manager] is good at documenting what we do where.� Similarly, the emphasis on doing

from another practitioner within Oak Openings: �Of course evaluation is important, but if we

don�t do any work, then what is there to evaluate?� Another Oak Openings participant

commented, ��I think we are learning as we go along. We are trying to keep pretty good

records of what we do in terms of restoration work.� Annual reports for funders or work or

progress reports also tend to document what has been done, and thus are not particularly useful

for evaluative purposes.

Data Collection Versus Analysis In situations where there are some data, the focus tends to be on data collection rather than

analysis. Many agencies recognize the need to collect data and may have quite extensive

monitoring programs to collect substantial amounts of data. However, documented data that are

collected without a clear plan of how that data will be used or analyzed for evaluative purposes

can be uninformative. How useful these data are, or how the data will be used are not usually

recognized. This is clear from looking at the amount of data collected versus analyzed or applied

to decision criteria by agencies. Many cases use monitoring data just to establish trends and

potentially an early warning system, but not really to apply it to any strategy or decision-making

process at the moment. The result is that they then operate on a best-management practice

principal, or a �time will tell� approach. For example, much of the data collected on Fish Creek

is to date not used but is filed or �on a web site, and we can look at it any time. I think it�s pretty

much just data collection right now. I don�t think anyone�s really analyzed it.� Similarly

monitoring data on rare plant species in the Oak Openings region is sent in to the natural heritage

database, but the information is not analyzed to answer evaluation questions. Available data

could be used more effectively to evaluate success, i.e. certain informative analyses have not

been done to link strategies with anticipated goals.



40

Potential Solutions

Potential solutions must be looked at from two angles: 1) ways agencies themselves can improve

internal evaluation, and 2) ways that an outside evaluator can make the most of existing

information. Stimulating thought on how ecosystem management could be measured is needed

for both internal and external evaluation.

There are several things agencies could do to move from valuing evaluation to practicing it.

These include: a) integration of evaluation into the site plan, b) increasing accountability, c)

dedicating time and personnel to evaluation, and d) enhancing capacity through the use of

volunteers. To really evaluate the effectiveness of a strategy, how it is going to be evaluated

should be determined a priori. That is, there need to be evaluative measures for each goal and

strategy before they are implemented. As evidenced by the different level of evaluation done by

USFS, increased accountability requires evaluation, or at least some look at progress. Chicago

Wilderness is planning on using public conferences in this way: �Holding a conference is a

moment of truth and people who said they would do things are held accountable to ok now you

are going to stand up in front of 500 people and if you just screwed off and didn�t do your paper,

it is going to hurt.� Finally, some practitioners realize the need for a job position or a leader

devoted to evaluation, developing indicators, compiling and analyzing data sets: �Someone

needs to agree to carry out work necessary for evaluation.� Such positions do exist (e.g. referred

to as �a DNR guru on performance measures,� or the �leader of DNR�s ecosystem management

planning team�). The question is whether there are enough of them working on local scales often

enough to be able to assess success continuously.

Some organizations have had great success from the use of volunteers. �Citizen-scientists� can

be trained as monitors, and because there can be large numbers of trained volunteers, they can

collect much more data than is often possible by staff personnel. For volunteers to be used

effectively, however, the type of data that they will collect and the how this data will be analyzed

should be well planned out in advance.

Another solution is to make the most of all existing information. An outside evaluator could

combine data from all sources and analyze it in ways that have not been done. There are many

data sets already available that could be used for evaluation, or additional information could be

obtained from agencies not directly participating in the ecosystem management project. For



41

example, county data on the numbers of acres in conservation tillage acres, which is apparently

available, could be amassed for the �control� study sites and then water quality and land use

across sites could be correlated. An additional source of knowledge may come from long-term

employees of agencies. Practitioners that have been there for a while get a feeling for progress

from their long personal history, so even if evaluation is not systematic or integrated into plan, it

may be easier to evaluate success of cases where the personnel have low turnover.

Simply thinking more about evaluation, objectively and subjectively, leads to the development

of ways to measure ecosystem management success. A trend for all interviews was that in the

beginning of the conversations on measures of ecosystem management success (before specific

questioning) the interviewees made comments to the effect that they don�t have any (yet) and

don�t know what they would use. But upon further questioning, several ideas and opinions did

emerge. Figuring out what the important questions are � what is that they want to know � leads

to designing ways to measuring it. Practitioners realize that as they are forced to think about

evaluation more, it becomes a more integrated part of what they do. According to The Nature

Conservancy Office in the Oak Openings, �Our measures of success were more general in terms

of acreage and awareness, and we just went through this latest round of site conservation

planning that was a bit more formal and we have to think about these things. And measures of

success have become a much bigger aspect of our work.�

Starting modestly and subjectively is Chicago Wilderness�s approach. �It does not have to leap

forward with full armor, but I think the important thing is to get Chicago Wilderness committed

to an event where it says, this is what we think we know about where we are today and this is

what we are going to do to measure progress.� It seems that with SOLEC, thinking about

evaluation even subjectively served as an important guide to developing actual indicators. �It has

gone from the very subjective to heading in a much more quantified direction, which is very

much needed and creates institutional accountability and lots of good things, and it enhances the

ability to communicate to the public and the media and decision-makers.� Starting subjectively

has advantages, in that it is less intimidating than �bar graphs and pie charts and chart progress �

well that is overwhelming.�

12. Scale of Evaluation


42


Because both fine and coarse scale indicators have limited usefulness, it is difficult to determine

the correct scale at which to conduct an evaluation.

Challenges

There are an infinite number of indicators to choose from in measuring progress towards goals,

and these vary from very coarse to very fine scale variables. Both coarse and fine scale indicators

have their pros and cons, which are discussed below. The challenge is finding the right number

and level of indicators, and that is likely to depend on the component of ecosystem management

success to be measured, the available resources for monitoring, and the opinion of the agencies

involved.

Fine Scale Indicators Fine scale indicators are clearly measurable items, but with limited time and money, only a few

can be measured. Examples of fine-scale indictors seen in the cases looked at are sedimentation

level, acres in conservation tillage, lupine density, and number of people attending a field trip.

They are generally easy to measure and thus are the most often type of data collected. The

number of fine scale items that could be measured is infinite. However, given limited time,

resources, and money only a few are actually done. As a forest service representative in

Chequamegon-Nicolet put it �you know that the ecosystems are more than just the species

elements. There are also the interactions and...biochemical, energy exchange.. a whole variety of

things to know whether a system works well. And to actually think that we�re going to get all the

measures to say � did we do well? We�re just not going to go there. We�re not going to be able to

afford it, as a society.� A TNC project director in the Chequamegon Bay Watershed feels the

same limitation on what they can measure: �I would say there�s a lot of money out there to do

restoration projects, but not a lot of money out there to do long-term monitoring whether its was

a good use of your money or not.�

A limited number of fine-scale measures is insufficient for measuring ecosystem management

success. That only a few fine-scale items can be measured by any one agency poses a problem to

measuring success, because fine-scale measures are not likely to be informative unless a) there

are many (since we do not know which of the few are most informative) and b) they are a part of



43

a synthesizing analysis (which is rarely done, see value of evaluation, lesson # 11). This may be

especially true for social or process issues where easily measured fine-scale numbers reveal little

about progress towards goals.

Coarse Scale Indicators The result is that most practitioners turn instead to very coarse-scale indicators, which are also

not sufficient for measuring success. For example, in Fish Creek, where TNC has a goal of

restoring riparian habitat in an agricultural setting, �the way we rate it, is if there�s anything that

isn�t corn and soybeans out there, we�re doing something better to reduce threats to the

ecosystem.� How much better, relative to other strategies or other cases, is not clear from this

kind of assessment. Similarly, Oak Openings practitioners measures success of prairie restoration

project as seeing new native species appear from year to year, which may be recorded or just

anecdotal evidence. Course scale indicators have the benefit of being easy and cheap, because

they are not actually measuring anything in a systematic way, but may be too coarse to reveal

possibly important deviations from success. Often there are indicators which are easy to

measure, but do not really get at the crux of issue. For example, a TNC Chequamegon Bay

Watershed worker noted, �The easy ones would be we�ve reached x amount of people in x

amount of years�but how do you know that you�ve changed anybody�s attitude� I really don�t

know.�

Because of the difficulty of balancing the use of fine and coarse scale indicators, there is no

standard resolution at which success can be compared. This is true both among the different

agencies within a case and across different cases. Within Chequamegon-Nicolet, for example,

the Forest Service, may have more resources available to it than other members do and have

monitoring programs required by law and so focus on many finer-scale items to measure. The

Fish Creek TNC, on the other hand, with the local office having less than 3 personnel, may have

much more limited resources and only do the monitoring for which they are equipped. That is,

they will use primarily a few medium scale-measures � number of acres in certain land-use types

� to gauge success. This makes it difficult to measure the success of this case, since different

components of ecosystem management will be measured at different resolutions and success in

one component may be an artifact of the resolution at which it was measured. Similarly,

measuring success at different resolutions make it difficult to compare across cases. It is



44

unreasonable to compare a case that relies on anecdotal evidence with one that uses more

scrutinizing fine-scale measures of success.

Potential Solutions

There are several approaches that cases are taking or plan to take to develop an idea of the right

level of coarseness at which to measure progress: a) a �filter approach�, b) the �no-brainer

category�, c) feedback approach, d) subjective start, and e) the �two-prong approach.�

Filter approach. A USFS project manager in the Chequamegon-Nicolet area finds that a

relatively coarse filter approach to measuring progress would avoid the problems associated with

fine-scale measures (too many and too expensive) and may be a viable concept to use for

monitoring and managing by. He finds that �no matter how fine we think we are, we are still

coarse�but through a combination of science and pragmatism you arrive at some level of

coarseness that you feel you�re probably measuring it well enough.� This level of coarseness

would tell you if �something�s going awry�whether the system turns in a direction that you

didn�t intend it to.� That is, it filters out what you don�t need to know, but catches the important

information. The pragmatic side of finding the right level of coarseness: go through a hierarchy

of indicators from coarse to fine scale until you find the first measurable item. The science side:

use all available scientific information. His forest planning team is still developing such

indicators, but an example given was vertical structure as one indicator of forests moving in the

right direction.

No-brainer category. A solution to the scale-problem may be to identify components of

ecosystem management or certain goals for which a very coarse scale is sufficient, i.e. that do

not require fine-scale assessment. Several cases alluded to this in some way, such as noted by a

Chequamegon Bay Watershed participant, : �sometimes you really need to go on a gut reaction.

We could spend billions of dollars to figure out whether we�re making a difference, but do we

really need to do that, or do we know that harvesting aspen over and over vs. trying to convert

that stand to what it once was is a good thing to do. We kind of can know that without spending

the money on monitoring.� This might be especially relevant in cases where information is

needed immediately. For example, at Kanankakee they were �considering doing a survey for the

presence of the bat in the areas where we might do work�The FWS said, �well, we could do

that, but it�s going to take a lot of time and money to do a survey� we believe that we have



45

enough evidence to tell you that the IN bat is present�� and that is sufficient knowledge for what

they need to do right now.

Feedback approach. As a way to prioritize which of the many possible fine scale measures are

relevant and how they might be used, Chicago Wilderness starts with educated guesses for

indices for various conservation targets, and then gets reactions from stakeholder. They �send

them out to people and say � ok what do you think of this? And people are saying oh it is too

high or it is too low. Here is what is wrong with it, and then keep adjusting it until everybody

feels comfortable with it.�

Subjective start. For the items for which there are no data, Chicago Wilderness is taking a

subjective approach to measuring success. They rely on expert opinion and consensus to

determine where they are on a spectrum from bad to excellent in terms of the status of a

resource. They use these �gas gauges,� or subjective measures of status, to work towards the

development of more objective indicators. The advantage of doing this is that they then do not

only focus on the few items they are already measuring: �[going] subjective gives you a chance

to look around and say let�s not get caught by the streetlight affect, which is don�t get caught

reporting just what is under the streetlight. Let�s look around and see what needs reporting on,

and let�s do subjective reporting on that and turn on more streetlights.�

Two-prong approach. Chicago Wilderness is taking a two-prong approach � fine scale

(conservation design) and coarse scale (state of the region). There is a need for this, because

�conservation design is developing at a very fine scale and detailed and it is going to take a very

long time. Somehow we need to jumpstart an objective approach.�

Another potential solution to the problem of fine-scale items being too many for any one agency

to measure is for agencies to pool or compare their data collecting resources. Some are certainly

doing this (e.g.WI DNR and USFS share data) or have plans to do this (e.g. Chequamegon TNC

plans to work with monitoring program of Bad River Indian Reservation). However, it seems

more often agencies are doing parallel data collection rather than dividing the labor of

monitoring and then combining it to evaluate success � e.g. one agency doing public awareness

and the other plant monitoring. However, there are limitations to data sharing. For example, even

though EPA, IN DNR, and OH DNR are doing similar types of monitoring, TNC may be



46

�spending about $30,000 a year to the Midwest Biological Institute to do our sampling�. TNC

does this because they feel a time lag before others� data is in report form for public

consumption. Universities could also be tapped into for the fine-scale data that they may have

collected for a different purpose, but would still be of use to cases to evaluate success of their

strategies, or changes from baseline. At the same time, agencies could compare data collected

from similar areas and discover that fine-scale monitoring does not need to be applied to all areas

because they behave similarly. A strategy can be tested in one area and assume to work in other

areas (although at some level these assumptions should be checked). In any case, an outside

evaluator would be able to pool and compare the many sources of information to make use of

scattered fine-scale indicators.

Cha

lleng

es o

f Mea

surin

g Su

cces

s in

Eco

syst

em M

anag

emen

t Ec

osys

tem

Man

agem

ent I

nitia

tive

� D

raft

(do

not c

ite o

r quo

te)

0

Tabl

e 1:

Com

paris

on o

f cha

lleng

es o

f mea

surin

g su

cces

s in

eco

syst

em m

anag

emen

t to

pote

ntia

l sol

utio

ns

Cha

lleng

es to

Mea

surin

g Su

cces

s Po

tent

ial S

olut

ions

1.

The

re is

no

univ

ersa

l def

initi

on o

f eco

syst

em m

anag

emen

t. •

Diff

icul

t to

mea

sure

succ

ess b

alan

ce a

mon

g m

ultip

le c

ompo

nent

s. •

Diff

icul

t to

know

on

wha

t sca

le to

mea

sure

succ

ess

• H

ow sh

ould

succ

essf

ul c

olla

bora

tion

be m

easu

red?

•

How

to m

easu

re su

cces

sful

syst

ems t

hink

ing?

• Es

tabl

ish

over

sigh

t com

mitt

ee to

form

ulat

e co

mm

on g

oal

• D

efin

e di

stin

ct g

oal a

reas

that

cou

ld b

e m

easu

red

• M

easu

re su

cces

s sep

arat

ely

for d

iffer

ent c

ompo

nent

s •

Eval

uate

pro

duct

s of c

olla

bora

tives

rath

er th

an st

ruct

ure

• D

evel

op a

syst

ems p

lan

to se

e ho

w e

ach

org/

stra

tegy

may

be

cont

ribut

ing

to o

vera

ll go

als.

• Ev

alua

te c

ases

aga

inst

a c

ontin

uum

of E

M p

hilo

soph

ies.

2. G

oals

are

som

etim

es v

ague

, ill

defin

ed, a

nd/o

r ide

alis

tic.

• If

goa

ls c

an n

ot b

e ac

hiev

ed, h

ow c

an su

cces

s be

mea

sure

d?

• Le

ads t

o a

tend

ency

to m

onito

r im

plem

enta

tion

of st

rate

gy ra

ther

than

pr

ogre

ss to

war

ds su

cces

s. •

Goa

l of r

esto

ratio

n m

ay b

e ul

timat

e, b

ut c

urre

nt su

cces

s may

be

prev

entin

g fu

rther

des

truct

ion.

• Fr

om v

isio

ns, s

peci

fy c

oncr

ete

goal

s, an

d fr

om g

oals

, spe

cify

obj

ectiv

es

• U

se a

ll av

aila

ble

info

to tr

ansf

orm

a v

ague

con

cept

into

a m

easu

rabl

e ob

ject

ive.

•

Dev

elop

incr

emen

tal m

easu

res o

f pro

gres

s

3. T

here

are

diff

eren

t sta

ges o

f eco

syst

em m

anag

emen

t. •

Diff

eren

t thi

ngs m

ay b

e im

porta

nt to

succ

ess d

epen

ding

on

stag

e of

pr

ojec

t. •

Aga

in le

ads t

o a

tend

ency

to m

easu

re st

rate

gy im

plem

enta

tion

• D

iffer

ent a

spec

ts o

f a p

roje

ct c

an b

e at

diff

eren

t sta

ges

• C

riter

ia su

cces

s sho

uld

diff

er fo

r var

ious

stag

es.

• D

evel

op m

easu

res o

f bot

h pr

oxim

ate

and

ultim

ate

succ

ess.

• Pr

ojec

ts in

diff

eren

t sta

ges s

houl

d no

t be

dire

ctly

com

pare

d.

4. S

yste

ms,

goal

s, an

d st

rate

gies

ope

rate

on

mul

tiple

tim

e sc

ales

. •

Tim

e la

gs m

ake

it di

ffic

ult t

o m

easu

re th

e su

cces

s of a

stra

tegy

. •

Proj

ects

tend

to m

ove

forw

ard

befo

re m

onito

ring

prog

ram

s are

in p

lace

•

Som

etim

es o

ne st

rate

gy n

eeds

to fi

nish

in o

rder

to se

e su

cces

s in

anot

her

stra

tegy

.

• Li

nk st

rate

gies

to a

n ov

eral

l gam

e pl

an, i

n or

der t

o m

easu

re su

cces

s.

• R

ecog

nize

var

ying

deg

rees

of c

hang

e an

d re

lativ

e di

ffic

ulty

in a

chie

ving

th

at c

hang

e.

• M

easu

re sh

ort-t

erm

and

long

-term

goa

ls se

para

tely

. •

Look

for t

rend

s in

the

dire

ctio

n of

des

ired

chan

ge.

Cha

lleng

es o

f Mea

surin

g Su

cces

s in

Eco

syst

em M

anag

emen

t Ec

osys

tem

Man

agem

ent I

nitia

tive

� D

raft

(do

not c

ite o

r quo

te)

1Cha

lleng

es to

Mea

surin

g Su

cces

s Po

tent

ial S

olut

ions

5.

Sys

tem

s, go

als a

nd st

rate

gies

ope

rate

on

mul

tiple

spat

ial s

cale

s. •

Cha

lleng

es in

scal

ing

up m

onito

ring

data

from

site

to re

gion

leve

l. •

Impo

rtanc

e of

regi

onal

leve

l eva

luat

ion

not a

lway

s rec

ogni

zed.

•

Spat

ial s

cale

issu

es m

ake

it di

ffic

ult t

o se

e ca

usal

link

ages

. •

Cas

es o

pera

ting

on v

astly

diff

eren

t sca

les m

ay n

ot b

e co

mpa

rabl

e.

• Sc

ale

up si

te le

vel d

ata

into

a re

gion

al re

pres

enta

tion.

•

Dev

elop

bot

h si

te le

vel a

nd re

gion

al in

dica

tors

.

6. E

colo

gica

l pro

cess

es a

re in

here

ntly

com

plex

and

var

iabl

e.

• Q

uest

ions

aro

und

relia

bilit

y an

d in

terp

reta

tion

of e

colo

gica

l dat

a.

• C

onfo

undi

ng m

easu

res o

f dire

ct e

ffec

ts o

f cer

tain

stra

tegi

es.

• H

ard

to k

now

whi

ch v

aria

ble

best

repr

esen

t an

ecol

ogic

al p

roce

ss o

f ph

enom

ena.

• Lo

ok a

t ove

rall

trend

s. •

Use

exi

stin

g kn

owle

dge

abou

t eco

syst

ems t

o he

lp in

terp

ret r

esul

ts.

7. In

dica

tors

of s

ocia

l and

pro

cess

goa

ls a

re n

ot a

s rec

ogni

zed

by p

ract

ition

ers

as a

re e

colo

gica

l goa

ls a

re th

eref

ore

are

not a

rticu

late

d or

mea

sure

d.

• Ev

alua

tion

of so

cial

and

pro

cess

goa

ls is

diff

icul

t whe

n th

ese

goal

s are

not

sp

ecifi

ed.

• So

cial

and

pro

cess

goa

ls te

nd to

be

fuzz

y co

ncep

t tha

t are

not

eas

ily

mea

sure

d.

• H

ard

to m

easu

re p

rogr

ess w

hen

incr

emen

tal s

teps

are

not

iden

tifie

d.

• C

ases

can

go

dow

n un

desi

rabl

e pa

ths

• A

rticu

late

soci

al a

nd p

roce

ss g

oals

and

obj

ectiv

es.

• Q

ualit

ativ

e m

easu

res m

ay b

e m

ore

usef

ul th

at q

uant

itativ

e on

es.

Cha

lleng

es o

f Mea

surin

g Su

cces

s in

Eco

syst

em M

anag

emen

t Ec

osys

tem

Man

agem

ent I

nitia

tive

� D

raft

(do

not c

ite o

r quo

te)

2

Cha

lleng

es to

Mea

surin

g Su

cces

s Po

tent

ial S

olut

ions

8.

Man

y ag

enci

es h

ave

dive

rse

agen

das.

• Su

b-or

gani

zatio

nal g

oals

may

be

over

whe

lmed

by

larg

er o

nes

• In

dica

tor s

elec

tion

can

be c

ompl

icat

ed w

hen

not a

ll or

g ef

forts

con

tribu

te

to E

M g

oals

•

Som

e pr

ojec

ts m

ake

insi

gnifi

cant

con

tribu

tions

to o

vera

ll go

als

• So

me

org

goal

s may

be

inco

mpa

tible

with

EM

goa

ls

• Ev

alua

tion

of E

M sh

ould

look

at w

hole

pic

ture

, eve

n if

org

is o

nly

wor

king

on

a sm

all p

iece

.

• Fo

cus j

ust o

n or

gani

zatio

nal r

esou

rces

des

igna

ted

for E

M g

oals

. •

Dev

elop

ove

rall

wor

kpla

n of

how

smal

l pro

ject

s fit

into

larg

er sc

ope.

9. S

trate

gy su

cces

s may

not

equ

ate

with

ove

rall

ecos

yste

m m

anag

emen

t su

cces

s.

• St

rate

gies

may

not

be

suff

icie

nt to

reac

h EM

goa

ls.

• Sh

ort-t

erm

succ

ess m

ay n

ot m

ean

long

-term

succ

ess

• So

me

stra

tegi

es h

ave

unin

tend

ed c

onse

quen

ces.

• D

iffer

entia

l pro

gres

s on

diff

eren

t stra

tegi

es c

ompl

icat

es o

vera

ll ev

alua

tion.

• U

se e

valu

atio

n to

unc

over

gap

in st

rate

gies

nee

ded

to a

chie

ve g

oals

. •

Dev

elop

crit

eria

for w

hat c

onst

itute

s a g

ood

EM p

roje

ct.

• N

ote

reas

onin

g be

hind

impl

emen

tatio

n of

stra

tegi

es.

• Ev

alua

te st

rate

gy a

ffec

ts to

mak

e su

re it

cau

sed

desi

rabl

e re

sults

.

10. C

ausa

l lin

kage

s are

diff

icul

t to

find.

•

Long

tim

e de

lays

bet

wee

n ca

use

and

effe

ct.

• H

ard

to se

e ac

ross

mul

tiple

spat

ial s

cale

s •

Ofte

n m

any

stra

tegi

es a

re u

sed

sim

ulta

neou

sly

• M

onito

ring

is o

ften

not d

esig

ned

to a

nsw

er c

ausa

l que

stio

ns

• U

se e

xper

imen

ts to

unc

over

cau

sal l

inka

ges.

•

Con

sist

ency

in e

valu

atio

n ef

forts

will

hel

p to

trac

k pr

ogre

ss

• St

art w

ith su

bjec

tive

asse

ssm

ent,

and

build

up

to u

se o

f obj

ectiv

e in

dica

tors

. •

Beg

in e

valu

atio

n at

beg

inni

ng o

f stra

tegy

impl

emen

tatio

n.

11. T

he v

alue

of e

valu

atio

n is

reco

gniz

ed b

ut n

ot p

ut in

to p

ract

ice

• R

esou

rce

barr

iers

to im

plem

enta

tion

• La

ck o

f ide

as o

n ho

w to

eva

luat

e •

Tend

ency

to e

quat

e ev

alua

tion

with

wha

t is b

eing

don

e •

Tend

ency

to fo

cus o

n da

ta c

olle

ctio

n an

d no

t ana

lysi

s •

Lack

of i

n-pl

ace

eval

uatio

n sy

stem

to u

se

• U

se a

vaila

ble

info

rmat

ion

to p

iece

toge

ther

bes

t pos

sibl

e ev

alua

tion

• G

et p

eopl

e fo

cuse

d on

eva

luat

ion

so it

get

s inc

orpo

rate

d in

to p

lans

.

Cha

lleng

es o

f Mea

surin

g Su

cces

s in

Eco

syst

em M

anag

emen

t Ec

osys

tem

Man

agem

ent I

nitia

tive

� D

raft

(do

not c

ite o

r quo

te)

3Cha

lleng

es to

Mea

surin

g Su

cces

s Po

tent

ial S

olut

ions

12

. Ver

y fin

e an

d ve

ry c

oars

e sc

ale

indi

cato

rs e

ach

have

lim

ited

usef

ulne

ss.

• O

nly

a fe

w fi

ne sc

ale

indi

cato

rs c

an b

e m

easu

red

with

lim

ited

reso

urce

s •

A li

mite

d nu

mbe

r of f

ine

scal

e in

dica

tors

is in

suff

icie

nt fo

r mea

surin

g ov

eral

l suc

cess

. •

Coa

rse

scal

e m

easu

re o

ften

don�

t ref

lect

real

succ

ess

• H

ard

to c

ompa

re c

ases

with

out a

stan

dard

reso

lutio

n fo

r eva

luat

ion.

• U

se c

oars

e fil

ters

to u

ncov

er w

anin

g si

gns o

f pro

blem

s •

Rel

y on

indi

cato

rs fo

r whi

ch c

oars

e sc

ale

is su

ffic

ient

/mea

ning

ful

• U

se fe

edba

ck to

dev

elop

indi

cato

rs a

ccep

tabl

e by

all

stak

ehol

ders

•

Star

t with

subj

ectiv

e m

easu

res,

and

grad

ually

dev

elop

indi

cato

rs.

• U

se c

oars

e sc

ale

and

fine

scal

e ap

proa

ches

sim

ulta

neou

sly.

Documents

The Challenges of Measuring Success in Collaborative ... challenges white paper.pdfChallenges of Measuring Success in Ecosystem Management Ecosystem Management Initiative Œ Draft