Assessing the Evaluation Capacity of Large Nonprofit Organizations Altarum Institute 1 A Detailed Account of the Methods, Findings, and Lessons Learned from the YMCA Evaluation Capacity Assessment
Table of Contents
Introduction .......................................................................................................................................... 3
YMCA Evaluation Capacity Assessment Purpose and Goals ...................................................... 3
Overview of Evaluation Capacity Assessment Process ................................................................ 4
Purpose and Organization of This Report .................................................................................... 5
Step 1: Becoming Acquainted with Y Culture, Structure, and Evaluation Needs .......................... 6
Organizational Structure ............................................................................................................... 7
Organizational Culture ................................................................................................................. 9
Evaluation Capacity .................................................................................................................... 10
Evaluation Capacity Assessment Process ................................................................................... 12
Lessons Learned ......................................................................................................................... 17
Step 2: Evaluation Capacity Assessment Survey ............................................................................. 18
Methods ...................................................................................................................................... 18
Results ........................................................................................................................................ 20
Lessons Learned ......................................................................................................................... 25
Step 3: Virtual Focus Group .............................................................................................................. 27
Methods ...................................................................................................................................... 27
Results ........................................................................................................................................ 34
Lessons Learned ......................................................................................................................... 35
Step 4: Key Informant Interviews ..................................................................................................... 37
Methods ...................................................................................................................................... 37
Results ........................................................................................................................................ 40
Lessons Learned ......................................................................................................................... 43
Step 5: Site Visits ................................................................................................................................. 45
Methods ...................................................................................................................................... 45
Results ........................................................................................................................................ 49
Lessons Learned ......................................................................................................................... 52
Summary of Lessons Learned ........................................................................................................... 54
Conclusions from the Evaluation Capacity Assessment Process ................................................... 57
Y Specific Conclusions .............................................................................................................. 57
Assessing the Evaluation Capacity of Large Nonprofit Organizations Altarum Institute 3 A Detailed Account of the Methods, Findings, and Lessons Learned from the YMCA Evaluation Capacity Assessment
Introduction Nonprofit organizations are increasingly challenged to provide a growing number of community services,
influence public policy, and be innovation leaders in their fields, while at the same time facing resource
constraints and increased pressure from funders to be accountable for achieving tangible results. Many
nonprofits are turning to organizational capacity building as a strategy to run more effective and efficient
operations, enhance sustainability, and produce better results from their programs (Connolly and York, 2002).
Specifically, some organizations are undertaking evaluation capacity building. By building evaluation
capacity, these organizations seek to improve their accountability, competitiveness for funding, and
effectiveness by using results from high-quality evaluations to enhance program development and operations.
The trend in building evaluation capacity is increasingly being supported by social sector funders to improve
nonprofit performance and maximize the impact of philanthropic investments (Welsh and Morariu, 2011).
As activity in this area expands, a challenge for nonprofit organizations and funders alike is the lack
of clarity and comprehension around what it means for an organization to posses evaluation capacity
and what the best approaches are for assessing such capacity. There is a dearth of research in the
emerging field of evaluation capacity, and what research exists uncovers no singular or widespread
definition of evaluation capacity or best practices for its assessment (Nielson, Lemire, and Skov,
2011). For example, more simplistic definitions such as the ability to perform high-quality
evaluations and use evaluation findings are available, but other definitions suggest that evaluation
capacity is significantly more complex, intrinsically linked to an organizations structures, processes,
culture, human capital, technology, and other attributes that traditionally comprise an organizations
capacity more broadly. Additional research into how organizations define evaluation capacity and
approach evaluation capacity assessment can contribute valuable knowledge to the field and expand
understanding of how nonprofits can approach this work.
YMCA Evaluation Capacity Assessment Purpose and Goals
The purpose of this report is to describe the methods, findings, and lessons learned through an
evaluation capacity assessment process completed with the YMCA (the Y) in 2012. The National
Council of the Young Mens Christian Association of the United States of America (YMCA of the
USA or Y-USA) is one of the top ten largest nonprofit organizations in the United States, and the
nations leading nonprofit committed to strengthening communities through youth development,
healthy living, and social responsibility. For more than 165 years, the Y has worked to strengthen
communities through programs and services that build healthy spirit, mind, and body for all. There
are approximately 2,700 local Ys across the country (collectively, the Y movement) serving more
than 10,000 communities and 21 million individuals, including 12 million adults and 9 million youth
under the age of 18. As a mission-driven organization, the Y must use evaluation and measurement to
assess whether they are reaching their goals and advancing their cause. The Y must also be
accountable for producing results that demonstrate the impact the Y has on the lives of individuals
and within communities. T
