Managing Evaluation Managing Evaluation ProjectsProjects

CRJS 4466EA

Evaluation Project Management

“a process of considerations, decisions, and activities engaged in by the leadership of the evaluation project to facilitate its conduct” (Bell, 1994)

“the evaluation mandate is the direction on the analytic purpose and intended use of evaluation findings given to evaluation management by the project sponsor” (Bell, 1994)

Project management is more art than science No prescription exists This lecture focuses on five areas that are

essential to effective project management– Clarifying the evaluation mandate– Staffing and organizing for success– Making assignments productive– Monitoring interim progress– Ensuring product quality and usefulness

Clarifying the evaluation mandate

Gaining initial agreement– Should be formalized orally or in writing– Evaluation management need to clarify the evaluation user, purpose,

scope, design and method, resources, schedule, and other technical requirements from the sponsor

– Both parties should check for congruence of mandate interpretation– Develop framework– When multiple purposes exist, evaluation management should discuss with

the sponsor the relative importance of the different purposes Checking the mandate during the evaluation

– A new legislative proposal or executive initiative may cause sponsors to change their views about preferred measurements

– Project managers need to involve sponsors in discussion about mandate throughout the evaluation

– Sometimes shifts cause a complete mid-stream negotiation of mandate

Staffing and organizing for results

“the project staff should embody the qualifications needed to conduct the planned evaluation activities on schedule in a high quality and effective manner…they should be available and motivated….they should be organized into teams with well-defined roles” (Bell, 1994)

Selecting appropriate staff

First step – staffing matrix– Substantive and methodological qualifications needed

to conduct the project are arrayed in rows in the matrix and the identities of candidate staff members are listed in the columns

– Cells in the matrix are marked for individuals whose qualifications demonstrated in past similar or related evaluations match those needed to conduct the tasks and activities required for this evaluation (Bell, 1994)

Substantive qualifications

Knowledge, skills and experience that demonstrate familiarity with the program and subject area

The ideal substantive qualifications encompass knowledge of all pertinent aspects of the subject area within and around the scope of the evaluation (Bell, 1994)

Administration and management, financing, client intake, case management, and management information systems are functions in many social programs. Knowledge in these functional areas is transferable across programs (Bell, 1994)

Methodological qualifications

Should reflect the requirements of each stage of a project – evaluation design and instrument development, data collection, data analysis and interpretation and report-writing

Distinguishing candidates who have capabilities in quantitative and qualitative methods and techniques is also beneficial (Bell, 1994)

Interpersonal relations/communications capabilities

Should be appropriate to member roles on the team

Avoid staff who are doctrinaire or zealous Seek neutral and objective staff

Level of training

Need mix of junior and senior staff appropriate to the problem-solving challenge of the project (Bell, 1994)

Organizing for results

Teams should be flat matrix work groups Members can be on more than one team Advisable to form a synthesis group to shepherd the efforts of the

technical teams– Involved in most aspects of planning evaluation assignments,

monitoring technical progress, and interpreting and integrating products and results from completed evaluation activities

– In the synthesis group, the evaluation part are brought together and shaped through internal peer review into the information products that will present the evaluation results (Bell, 1994)

– Some evaluation projects also benefit from an external advisor panel comprised of experts

Making assignments productive

Evaluation management must ensure that each staff member has at all times a clear and well-defined assignment governing his or her efforts on the evaluation

“Effort spent on gaining well-conceived and agreed upon assignments should not strain the evaluation resources and schedule. A project must maintain momentum by integrating the effort required for planning and initiating assignments into daily, weekly, and monthly activities. Only a very small share of total project labor and a moderate share of total evaluation management effort should be devoted solely to assignment making” (Bell, 1994)

Shaping individual assignments

Management determines set of interim projects needed for the next evaluation period

Management matches each assignment within the actual capabilities demonstrated on other evaluations by the individuals

In making assignments, management should encourage open discussion with project staff

Ground rules for reporting unanticipated problems should be established (Bell, 1994)

Formalizing assignments

Management and project staff should enter into an agreement specifying the expected product, the major milestones in product development, the resources set aside for the assignment, and the expected completion date (Bell, 1994)

Monitoring interim progress

In addition to the technical progress on evaluation products, the monitoring reports should inform evaluation management about the calendar time and labor spent and remaining for each product (Bell, 1994)

Well specified milestones

Progress milestones should exist for each evaluation product

Well-timed monitoring

Progress monitoring is appropriate when the forward momentum of the project is enhanced or maintained through monitoring

Monitoring milestones for each product should be defined by considering the steps in product development

A monitoring approach based solely on elapsed time should be avoided (Bell, 1994)

Ineffective use of monitoring

Managers who conduct the monitoring must be able to interpret monitoring information and respond appropriately

They need to know how to revise assignments accordingly

If concerns exist re. middle manager monitoring efforts, senior managers need to step up their presence

Milestones may be well defined but their indicators may be flawed (Bell, 1994)

Source

Bell, J. (1994). Managing Evaluation Projects. Handbook of Practical Program Evaluation (Chapter 21). San Francisco, California: Jossey-Bass Inc.