NM 6068 NM 6068 2009/102009/10

Tracy Culkin Tracy Culkin

Learning OutcomesLearning Outcomes

• Aim – revision of evaluation and its techniques

• Objectives –by the end of the lesson you will be able to:

• Give a definition of evaluation

• Describe the different evaluation methods

• Consider which evaluation models and tools would be appropriate for your dissertation

Chapter 3 Chapter 3 This chapter should clearly articulate a systematic evaluation This chapter should clearly articulate a systematic evaluation

plan of how the change will be measured, monitored and plan of how the change will be measured, monitored and evaluatedevaluated..

• It should demonstrate a planned systematic evaluation of how the change will be measured, monitored and audited.

• The methods of evaluation should be critically analysed with reference to relevant evaluation theory.

• The proposed analysis of the evaluation data should include a discussion of anticipated factors which may hinder or help the process.

• A critical analysis of key factors identified in relation to self-evaluation of the desired skills of a change agent.

DefinitionDefinition

• ‘Evaluation is primarily concerned with determining the merit, worth or value of an established policy or a planned intervention’

Clarke (1999, p 3-4)

History & Emergence of different kinds History & Emergence of different kinds of evaluationof evaluation

• Increase in funding from the government from the 1960s to remedy social problems – led to a huge increase in evaluation methods being applied to education, and health & social welfare programmes.

• There emerged three major approaches to evaluation:

• Evaluation for Accountability• Evaluation for Development• Evaluation for Knowledge

Evaluation for AccountabilityEvaluation for Accountability

• Is about whether there is clear evidence that the program or policy has ‘caused’ any discernible effects or changes in outcomes – accountability is to those usually external to the organisation, i.e. government sponsors

(Hall & Hall 2004, p32)

Evaluation for DevelopmentEvaluation for Development

• Development evaluation is committed to establishing a close relationship between the evaluator and the programme participants -its about human interactions, and that the delivery of the program is the focus of the evaluation

(Hall & Hall 2004, p33)

Evaluation for KnowledgeEvaluation for Knowledge

• Is about generating understanding and explanation – to unravel complex interactions, explore the issues underlying social problems and to examine the appropriateness of program provision

(Hall & Hall 2004, p 34)

Evaluation Models used in EducationEvaluation Models used in Education

• Formative Evaluation

• Summative Evaluation

Why Evaluate?Why Evaluate?

• To determine in this instance if the planned implemented change has been effective.

• ‘Evaluation has long been recognised as fundamental to good practice’ (Green & South , 2006, p 4).

• ‘Evaluation also has a major role in protecting the public from inappropriate or harmful practices’ (Green & South , 2006, p 4).

What am I evaluating?What am I evaluating?

• The process of implementing a planned change

• The outcomes of the implemented planned change -to determine if the aims & objectives of the intervention were met

• Your skills as a change agent

Evaluation & ChangeEvaluation & Change

• Evaluation is integrally linked to the change process.

• Benefits of Evaluation -

• ‘…..identify important contributions that the inclusion of a planned process of evaluation can make to the successful management of change’ (Skinner, 2004, p6).

• ‘a planned, systematic and rigorous evaluation process as a key part of successful change management’ (Skinner, 2004, p7).

Evaluation & ChangeEvaluation & Change

• Benefits of Evaluation Allows for adjustments to the change process - by giving the

opportunity to reflect and analyse. Can contribute to learning

Resistance to Evaluation & ChangeResistance to Evaluation & Change

• In some organisations there is resistance to systematic planned evaluation (Doyle et al, 2000).

The Phases of EvaluationThe Phases of Evaluation

• Planning and design

• Gathering and analysing data

• Identifying findings

• Conclusions

• Recommendations

• Disseminating results

The Processes of EvaluationThe Processes of Evaluation

• Monitoring through observation and measurement

• Comparing data with criteria for achievement in implementation

Evaluation Models/ToolsEvaluation Models/Tools

• Discuss utilisation of model used to Evaluate Change Process, e.g. Carney (2002)

• A Reflective model could be utilised to discuss Evaluation of Self as a Change Agent

• Evaluation Tools - Audit could be used

AuditAudit

Audit/Evaluation ToolsAudit/Evaluation Tools

• For more information on the above do access the online learning section for this module NM 6068 –ELibrary –scanned chapters section, where you will find 6 scanned chapters on this subject.

• The session ‘Evaluation Strategies for Clinical Change’ outlines specific strategies on how to evaluate. This session is led by Mary Fisher-Morris, who is Head of the Research and Clinical Audit Department, at The Countess of Chester NHS Hospital Foundation Trust.

References/ResourcesReferences/Resources

• Clarke A (1999) Evaluation Research. London: Sage• Doyle M, Claydon T & Buchanan D (2000) ‘Mixed results, lousy process: the

management experience of organizational change’. British Journal of Management, 11 (3): S59-S80

• Green J & South J (2007) Evaluation. Maidenhead: Open University Press• I Hall & D Hall (2004) Evaluation and Social Research . Basingstoke: Palgrave

Macmillan• Weiss CH (1998) Evaluation: Methods for Studying Programs and Policies. 2nd edn.

Upper Saddle River, NJ: Prentice Hall• The World Wide Evaluation Gateway http://www.policy-evaluation.org/• Resources for Methods in Evaluation and Social Research

http://gsociology.icaap.org/methods/• Skinner, D (2004) Evaluation and change management: rhetoric and reality. Human

Resource Management Journal, 14(3): 5-19• The CAQDAS Networking Project http://caqdas.soc.surrey.ac.uk/