View
213
Download
0
Category
Preview:
Citation preview
Managing Evaluation Managing Evaluation ProjectsProjects
CRJS 4466EA
Evaluation Project Management
“a process of considerations, decisions, and activities engaged in by the leadership of the evaluation project to facilitate its conduct” (Bell, 1994)
“the evaluation mandate is the direction on the analytic purpose and intended use of evaluation findings given to evaluation management by the project sponsor” (Bell, 1994)
Project management is more art than science No prescription exists This lecture focuses on five areas that are
essential to effective project management– Clarifying the evaluation mandate– Staffing and organizing for success– Making assignments productive– Monitoring interim progress– Ensuring product quality and usefulness
Clarifying the evaluation mandate
Gaining initial agreement– Should be formalized orally or in writing– Evaluation management need to clarify the evaluation user, purpose,
scope, design and method, resources, schedule, and other technical requirements from the sponsor
– Both parties should check for congruence of mandate interpretation– Develop framework– When multiple purposes exist, evaluation management should discuss with
the sponsor the relative importance of the different purposes Checking the mandate during the evaluation
– A new legislative proposal or executive initiative may cause sponsors to change their views about preferred measurements
– Project managers need to involve sponsors in discussion about mandate throughout the evaluation
– Sometimes shifts cause a complete mid-stream negotiation of mandate
Staffing and organizing for results
“the project staff should embody the qualifications needed to conduct the planned evaluation activities on schedule in a high quality and effective manner…they should be available and motivated….they should be organized into teams with well-defined roles” (Bell, 1994)
Selecting appropriate staff
First step – staffing matrix– Substantive and methodological qualifications needed
to conduct the project are arrayed in rows in the matrix and the identities of candidate staff members are listed in the columns
– Cells in the matrix are marked for individuals whose qualifications demonstrated in past similar or related evaluations match those needed to conduct the tasks and activities required for this evaluation (Bell, 1994)
Substantive qualifications
Knowledge, skills and experience that demonstrate familiarity with the program and subject area
The ideal substantive qualifications encompass knowledge of all pertinent aspects of the subject area within and around the scope of the evaluation (Bell, 1994)
Administration and management, financing, client intake, case management, and management information systems are functions in many social programs. Knowledge in these functional areas is transferable across programs (Bell, 1994)
Methodological qualifications
Should reflect the requirements of each stage of a project – evaluation design and instrument development, data collection, data analysis and interpretation and report-writing
Distinguishing candidates who have capabilities in quantitative and qualitative methods and techniques is also beneficial (Bell, 1994)
Interpersonal relations/communications capabilities
Should be appropriate to member roles on the team
Avoid staff who are doctrinaire or zealous Seek neutral and objective staff
Level of training
Need mix of junior and senior staff appropriate to the problem-solving challenge of the project (Bell, 1994)
Organizing for results
Teams should be flat matrix work groups Members can be on more than one team Advisable to form a synthesis group to shepherd the efforts of the
technical teams– Involved in most aspects of planning evaluation assignments,
monitoring technical progress, and interpreting and integrating products and results from completed evaluation activities
– In the synthesis group, the evaluation part are brought together and shaped through internal peer review into the information products that will present the evaluation results (Bell, 1994)
– Some evaluation projects also benefit from an external advisor panel comprised of experts
Making assignments productive
Evaluation management must ensure that each staff member has at all times a clear and well-defined assignment governing his or her efforts on the evaluation
“Effort spent on gaining well-conceived and agreed upon assignments should not strain the evaluation resources and schedule. A project must maintain momentum by integrating the effort required for planning and initiating assignments into daily, weekly, and monthly activities. Only a very small share of total project labor and a moderate share of total evaluation management effort should be devoted solely to assignment making” (Bell, 1994)
Shaping individual assignments
Management determines set of interim projects needed for the next evaluation period
Management matches each assignment within the actual capabilities demonstrated on other evaluations by the individuals
In making assignments, management should encourage open discussion with project staff
Ground rules for reporting unanticipated problems should be established (Bell, 1994)
Formalizing assignments
Management and project staff should enter into an agreement specifying the expected product, the major milestones in product development, the resources set aside for the assignment, and the expected completion date (Bell, 1994)
Monitoring interim progress
In addition to the technical progress on evaluation products, the monitoring reports should inform evaluation management about the calendar time and labor spent and remaining for each product (Bell, 1994)
Well specified milestones
Progress milestones should exist for each evaluation product
Well-timed monitoring
Progress monitoring is appropriate when the forward momentum of the project is enhanced or maintained through monitoring
Monitoring milestones for each product should be defined by considering the steps in product development
A monitoring approach based solely on elapsed time should be avoided (Bell, 1994)
Ineffective use of monitoring
Managers who conduct the monitoring must be able to interpret monitoring information and respond appropriately
They need to know how to revise assignments accordingly
If concerns exist re. middle manager monitoring efforts, senior managers need to step up their presence
Milestones may be well defined but their indicators may be flawed (Bell, 1994)
Source
Bell, J. (1994). Managing Evaluation Projects. Handbook of Practical Program Evaluation (Chapter 21). San Francisco, California: Jossey-Bass Inc.
Recommended