Embed Size (px)
Citation preview
An Action Research Study to Select an Effective
Model to Evaluate Consultation within Two
Educational Psychology Services
A thesis submitted to The University of Manchester for the degree
of Doctor of Educational and Child Psychology in the Faculty of
Humanities
2016
Adrienne Eddleston
School of Education
2
Table of Contents
List of Figures…………………………………………………………... 9
List of Tables…………………………………………………………… 10
Abstract…………………………………………………………….. …. 12
Declaration……………………………………………………………... 13
Copyright Statement…………………………………………………… 14
Acknowledgements……………………………………………………. 15
List of Acronyms………………………………………………………. 16
Chapter One: Introduction
1.1 Outline of Chapter……………………………………………… 18
1.2 Rationale……………………………………………………….. 18
1.2.1 National Context……………………………………....... 18
1.2.2 Local Context………………………………………........ 19
1.3 The evaluation of EP practice…………………………………. 20
1.4 Summary of Research…………………………………………. 21
Chapter Two: Literature Review Stage One
2.1 Chapter Introduction………………………………………….. 22
2.1.1 Literature Review Stage One……………………………… 22
2.1.2 Stage One Search Strategy……………………………....... 22
2.2 Setting the Context………………………………………........ 23
2.2.1 Historical context……………………………………........ 23
2.2.2 Role definition within the EP profession……………........ 27
2.3 Role Confusion………………………………………………. 30
2.4 Reviews of the role of the EP………………………………... 31
2.5 Welsh context……………………………………………….. 34
2.6 Role of scientist practitioner………………………………… 35
2.7 Consultation…………………………………………………. 36
2.7.1 Background to Consultation…………………………...... 37
2.7.2 Wagner’s Consultation Model…………………………... 39
2.8 Accountability within the current developments in
Educational Psychology Services…………………………… 40
3
2.8.1 Political and economic context………………………….. 40
2.9 Evaluation of Educational Psychology Services……………... 42
2.9.1 Hampshire Educational Psychology Service……………. 43
2.10 Measuring EP impact through consultation………………… 44
Chapter Three: Literature Review Stage Two
3.1 Chapter Outline…………………………………………........ 47
3.2 Service context……………………………………………… 47
3.3 Professional practice framework search strategy…………… 48
3.3.1 Evaluation framework selection process……………….. 48
3.3.2 The Interactive Factors Framework…………………….. 49
3.3.3 A framework for psychological assessment
and intervention………………………………………… 50
3.3.4 The Constructionist Model of Informed Reasoned
Action……………………………………………........... 51
3.3.5 Research and Development in Organisations…………… 51
3.3.6 Activity Theory…………………………………………. 52
3.3.7 Target Monitoring and Evaluation…………………....... 52
3.3.8 Appreciative Inquiry……………………………………. 53
3.3.9 Friedman’s Results Based Accountability Model……… 53
3.3.10 Measuring the impact of casework………………........ 54
3.4 Summary……………………………………………………. 54
3.5 Research Questions……………………………………........ 55
Chapter Four: Methodology
4.1 Chapter Introduction……………………………………….. 56
4.2 Context of the study………………………………………... 56
4.3 Chapter outline……………………………………………... 57
4.4 Philosophical considerations……………………………….. 59
4.4.1 Ontology………………………………………………. 59
4.4.2 Positivism…………………………………………........ 61
4.4.3 Relativism……………………………………………… 61
4.4.4 Critical realism………………………………………… 62
4.4.5 Epistemology………………………………………….. 63
4
4.4.6 Axiological position…………………………………… 64
4.5 Research design……………………………………………. 64
4.6 Action Research design……………………………………. 66
4.7 Consideration of other Research designs……………. ……. 67
4.8 Models of Action Research………………………….. ……. 68
4.9 Research and Development in Organisations……………… 69
4.10 Research phases………………………………………….. 70
4.11 Pre-research Phase and Phase One……………………… 70
4.11.1 Stage 1- awareness of need……………………. …… 71
4.11.2 Stage 2- invitation to act………………………. …… 71
4.11.3 Stage 3 and 4- clarifying organisational and
cultural issues and identifying stakeholders………… 71
4.12 Research Phase Two…………………………………….. 72
4.12.1 Stage 5- agreeing the focus of concern……………… 73
4.12.2 Stage 6- negotiating the framework for information
Gathering……………………………………………. 73
4.12.3 Stage 7- gathering information………………............ 73
4.12.4 Stage 8- processing information with
Stakeholders…………………………………………. 74
4.13 Research Phase Three……………………………………. 74
4.13.1 Stage 9- agreeing areas for future action and
Stage 10- action planning…………………............... 75
4.14 Data gathering methods………………………………… 76
4.15 Data collection…………………………………………… 77
4.16 Self-report questionnaire………………………................ 78
4.16.1 Pre-pilot questionnaires……………………………... 78
4.16.2 Post-pilot questionnaires……………………………. 79
4.17 Initial focus groups………………………………........... 80
4.18 Research diary…………………………………………... 82
4.19 Phase Two data analysis………………………………… 82
4.20 Questionnaires………………………………….............. 83
4.21 Content Analysis………………………………............... 84
4.22 Phase Three data gathering methods and data
Analysis………………………………………………… 84
5
4.22.1 Semi-structured interviews………………………... 85
4.22.2 Final focus groups………………………………… 85
4.22.3 Thematic Analysis…………………………………. 86
4.22.4 Thematic Analysis process………………………… 86
4.23 Critique of methods……………………………............. 88
4.23.1 Action Research framework………………………. 88
4.23.2 Questionnaires…………………………………….. 89
4.23.3 Focus groups……………………………………… 90
4.23.4 Thematic Analysis………………………………… 90
4.23.5 Content Analysis…………………………………. 91
4.23.6 Multi-methods design validity/reliability………… 91
4.24 Ethical considerations………………………………… 92
Chapter Five: Results
5.1 Chapter outline………………………………………… 94
5.2 Pre-research Phase and Phase One……………………. 95
5.3 RADIO Stages 1-4……………………………………. 95
5.4 Research Phase Two………………………………….. 95
5.5 Stage 5 and Stage 6- agreeing the focus of concern
and negotiating the framework for information
gathering…………………………………………......... 96
5.5.1 Systematic analysis……………………………….. 96
5.5.2 Step One: Framework for selection
process……………………………………………… 96
5.5.3 Step two: Systematic analysis……………………... 97
5.5.4 Step Three: Exclusion of frameworks based on
Paradigms……………………………….................. 99
5.5.5 Results Based Accountability Model……………… 99
5.5.6 Step Four: Further exclusions of
Frameworks………………………………………… 100
5.5.7 Target Monitoring and Evaluation…………………. 100
5.5.8 Research and development in Organisations………. 100
5.5.9 Measuring the impact of casework………………… 101
5.5.10 Step 5- Inclusion of frameworks……………......... 101
6
5.6 RADIO Stage 7: Gathering information…………………….. .. 102
5.6.1 Audit of previous knowledge questionnaire……………… 102
5.6.2 Data analysis outcome…………………………………… 102
5.6.3 Initial focus groups………………………………………. 108
5.6.4 Data analysis outcome……………………………………. 108
5.6.4.1 COMOIRA………………………………..…………. 109
5.6.4.2 IFF…………………………………………………… 113
5.6.4.3 AI……………………………………………………. 118
5.6.4.4 DECP……………………………………………….. 122
5.6.4.5 AT………………………………………….………… 124
5.7 Preference questionnaire…………………………………….. 126
5.8 Data analysis feedback meeting…………………………….. 129
5.9 Piloting of professional practice frameworks………………. 130
5.9.1 COMOIRA: Quantitative data analysis outcome………. 130
5.9.2 COMOIRA: Qualitative data analysis outcome………… 133
5.9.3 AI: Quantitative data analysis outcome………………… 137
5.9.4 AI: Qualitative data analysis outcome………………….. 140
5.10 RADIO Stage 8: processing information
with stakeholders…………………………………………… 144
5.11 Phase Three…………………………………………………. 144
5.12 RADIO Stage 9: agreeing areas for future action…………… 144
5.12.1 Final focus groups………………………………………. 144
5.12.2 Data analysis…………………………………………….. 145
5.12.3 Data analysis outcome: COMOIRA…………………….. 145
5.12.3.1 Change………………………………………………. 146
5.12.3.2 Accountability………………………………………. 149
5.12.3.3 Applying Psychology……………………………….. 151
5.12.3.4 Future Implications…………………………………. 152
5.13 Data analysis outcome: AI………………………………….. 154
5.13.1 Evaluation………………………………………………. 155
5.13.2 Strengths……………………………………………….. 157
5.13.3 Limitations…………………………………………….. 159
5.14 RADIO Stage 10: action planning…………………………. 160
7
5.15 RADIO Stage 11 and Stage 12: implementing action
and evaluating action………………………………………. 160
Chapter Six: Discussion
6.1 Chapter Introduction………………………………………… 161
6.2 RQ 1a and 1b………………………………………………… 162
6.3 RQ1a: What are the EPs’ views on the most
effective methods of evaluating consultation in
relation to assessing consultation as a process………………. 164
6.3.1 COMOIRA: Applying Psychology……………………… 165
6.3.2 AI: Strengths……………………………………………. 166
6.3.3 Application of psychology within the
assessment of consultation………………………………. 167
6.3.4 Change…………………………………………………… 168
6.4 RQ1b: What are the EPs’ views on the most
effective methods of evaluating consultation in
relation to providing outcomes which demonstrate
accountability to employers and service users……………… 172
6.4.1 COMOIRA: Accountability…………………………… 173
6.4.2Appreciative Inquiry: Evaluation………………………. 175
6.4.3 Demonstrating accountability…………………………. 175
6.4.4 Evaluation……………………………………………… 176
6.5 Future implications…………………………………………. 177
6.6 Personal reflections…………………………………………. 180
6.6.1 Action Research Model………………………………… 180
6.6.2 Future implementation………………………………….. 181
6.7 Theoretical contribution to knowledge……………………… 182
6.8 Limitations of the research…………………………………... 184
6.8.1 Model selection…………………………………………. 184
6.8.2 Transferability of findings……………………………… 184
6.8.3 Time implications……………………………………….. 184
6.8.4 Methodological considerations………………………….. 185
6.9 Implications for practice…………………………………… 185
6.9.1 Data driven……………………………………………… 185
8
6.9.2 Applying psychology…………………………………… 186
6.9.3 Stakeholder and service user feedback………………… 187
6.9.4 Future implications……………………………………… 187
Chapter Seven: Conclusion
7.1 Chapter Introduction………………………………………… 189
7.2 Future Research Directions…………………………………… 190
Word Count: 48, 650
9
List of Figures
Figure 4.1 Service Structure 2014-2015……………………………….. 57
Figure 4.2 The Simple Relationship between Epistemology, Methodology
and Method………………………………………………… 63
Figure 4.3 Action – reflection cycle……………………………………. 67
Figure 5.1 Thematic Map for COMOIRA…………………………….. 146
Figure 5.2 Change……………………………………………………… 146
Figure 5.3 Accountability……………………………………………… 149
Figure 5.4 Applying Psychology………………………………………. 151
Figure 5.5 Future Implications………………………………………… 152
Figure 5.6 Thematic Map for AI……………………………………….. 154
Figure 5.7 Evaluation…………………………………………………. 155
Figure 5.8 Strengths……………………………………………………. 157
Figure 5.9 Limitations………………………………………………….. 159
10
List of Tables
Table 1.1 Overview of the thesis……………………………………… 21
Table 3.1 EPS Evaluation Framework………………………………… 53
Table 4.1 Phases of the Research Process…………………………….. 58
Table 4.2 Four Philosophical Research Paradigms…………………… 60
Table 4.3 RADIO model……………………………………………… 69
Table 4.4 Pre-Research Phase and Phase One………………………… 70
Table 4.5 Research Phase Two……………………………………….. 72
Table 4.6 Research Phase Three……………………………………… 74
Table 4.7 Data Gathering Methods…………………………………… 77
Table 4.8 Phase Two Data Gathering………………………………… 78
Table 4.9 Phase Two Data Analysis………………………………….. 83
Table 4.10 Phase Three Data Gathering and Analysis…………………. 85
Table 4.11 Phase of thematic analysis…………………………………. 87
Table 5.1 Phases of the Research ……………………………………. 94
Table 5.2 Frameworks for the Selection Process…………………….. 97
Table 5.3 Framework proforma……………………………………… 98
Table 5.4 Audit of previous EP knowledge Q1……………………… 103
Table 5.5 Audit of previous EP knowledge Q2……………………… 105
Table 5.6 EP responses from the Audit of previous Knowledge `
Questionnaire……………………………………………… 107
Table 5.7 Content analysis of data for the COMOIRA model………. 110
Table 5.8 Content analysis of data for the IFF model……………….. 113
Table 5.9 Content analysis of data for the AI model………………… 118
Table 5.10 Content analysis of data for the DECP framework……….. 122
Table 5.11 Content analysis of data for the AT model……………….. 124
Table 5.12 Preference Questionnaire Rank responses………………… 127
Table 5.13 Preference Questionnaire responses ……………………… 128
Table 5.14 Post-pilot feedback questionnaire: COMOIRA…………. 131
Table 5.15 Open-ended responses from the post-pilot COMOIRA
questionnaire……………………………………………… 134
Table 5.16 Post-pilot feedback questionnaire: AI…………………… 138
11
List of Tables (continued)
Table 5.17 Open-ended responses from the post-pilot AI
questionnaire……………………………………………. 141
Table 6.1 Final Focus Group Themes……………………………... 163
Table 6.2 RQ1a – Links to the dataset……………………………. 165
Table 6.3 RQ1b – Links to the dataset……………………………. 173
Table 6.4 Future Implications……………………………………… 178
12
Abstract Background: This research was conducted within two Welsh bordering Local Authorities (LAs) across two Educational Psychology Services (EPSs) that changed their service delivery model to consultation following joint training. The Principal Educational Psychologist (PEP) sought to evaluate the EPSs and the services they deliver, but previous methods, including sending questionnaires to head teachers and parents, were not considered robust enough or to have sufficient depth. The PEP therefore commissioned this piece of research to source a clear and useful measure to evaluate consultation. Participants: Across the two LAs, the PEP, ten generic Educational Psychologists (EPs) and one Senior Educational Psychologist (SEP) participated in the study. Methods: This study used the Research and Development in Organisations (RADIO) model of Action Research (AR). A literature review was initially conducted to identify potential evaluation models. Thereafter, within the AR phases, different data were collected and analysed with stakeholders to ascertain EPs’ use of evaluation models and their preferences and perspectives as practitioners. Data collection incorporated both focus groups and questionnaires, which generated quantitative and qualitative data which were analysed through a variety of methods, including content analysis, thematic analysis and questionnaire analysis software. Findings: Within the AR design, data gathered during the earlier phases of the research were used to guide next steps in the research process. This led to the identification of two preferred models: The Constructionist Model of Informed Reasoned Action (COMOIRA) and Appreciative Inquiry (AI). These were then piloted across the EPSs and each EP completed a post-pilot questionnaire. In the final phase of the research two focus groups were held to consider the usefulness of the COMOIRA and AI models as service evaluation tools. The data generated a number of themes relating to accountability, applying psychology, change, evaluation and strengths; and raised issues relating to future implications. Conclusion: The AR design of the research facilitated a collaborative approach for shared decision making around the development of an EPS evaluation framework. The study identified positive and valuable aspects with regards to both the COMOIRA and AI models when these were implemented to evaluate consultation. Both models have the potential to assess aspects of the consultation process and could conceivably contribute towards providing outcomes which demonstrate accountability to employers and service users. Additionally, professional practice models could be used to evaluate wider aspects of EP practice and be of broader benefit to EPSs.
13
Declaration
No portion of the work referred to in the thesis has been submitted in support of an
application for another degree or qualification of this or any other university or other
institute of learning.
14
Copyright Statement
i. The author of this thesis (including any appendices and/or schedules to this
thesis) owns certain copyright or related rights in it (the “Copyright”) and she has given
The University of Manchester certain rights to use such Copyright, including for
administrative purposes.
ii. Copies of this thesis, either in full or in extracts and whether in hard or
electronic copy, may be made only in accordance with the Copyright, Designs and
Patents Act 1988 (as amended) and regulations issues under it or, where appropriate, in
accordance with licensing agreements which the University has from time to time. This
page must form part of such copies made.
iii. The ownership of certain Copyright, patents, designs, trademarks and other
intellectual property (the “Intellectual Property”) and any reproductions of copyright
works in the thesis, for example graphs and tables (“Reproductions”), which may be
described in the thesis, may not be owned by the author and may be owned by third
parties. Such Intellectual Property and Reproductions cannot and must not be made
available for use without the prior written permission of the owner (s) of the relevant
Intellectual Property and/or Reproductions.
iv. Further information on the conditions under which disclosure, publication and
commercialisation of this thesis, the Copyright and any Intellectual Property and/or
Reproductions described in it may take place is available in the University IP Policy
(see http://documents.manchester.ac.uk/DocuInfo.aspx?DocID=487), in any relevant
Thesis restriction declarations deposited in the University Library, The university
Library’s regulations (see http://www.manchester.ac.uk/library/aboutus/regultations)
and in The University’s policy on Presentation of Theses.
15
Acknowledgements
I would firstly like to thank my husband Craig. You have supported me every step of
the way over the past 6 years. Your constant words of encouragement and “you can do
it” attitude have kept me going. Thank you to my lovely boys, Henry, now 15 years and
Tom, now 10 years. You have offered me lots of well needed distractions along the way
and kept me laughing.
Thank you to my wonderful parents. For everything you have done and continue to do
for me. I am so lucky to have you.
Thanks to my lovely friend Lou for your patience and guidance in the final stages of
this process.
Also, I would like to thank the EPs who have taken part in this study. Without your
involvement I couldn’t have completed this thesis.
I would especially like to thank Cathy Atkinson who has supervised me throughout this
thesis and has provided me with lots of support and guidance for which I am very
grateful.
16
List of Acronyms
Acronym Meaning
AEP Association of Educational Psychologists
AI Appreciative Inquiry
AR Action Research
AT Activity Theory
BPS British Psychological Society
CA Content Analysis
COMOIRA Constructionist Model of Informed Reasoned Action
CPD Continued Professional Development
CYP Children and Young People
DfE Department for Education
DfES Department for Education and Skills
DES Department of Education and Science
ECM Every Child Matters
EPs Educational Psychologists
EPiTs Educational Psychologists in Training
EPS Educational Psychology Service
GAS Goal Attainment Scaling
HEPs Hampshire Educational Psychology Service
IFF Interactive Factors framework
LA Local Authority
LEAs Local Education Authorities
NAPEP National Association of Principal Educational Psychologists
PAF Problem Analysis Framework
PEP Principal Educational Psychologist
RADIO Research and Development in Organisations
RBA Results Based Accountability
PCP Person Centred Planning
SEF School Effectiveness Framework
SEN Special Educational Needs
SENCOs Special educational Needs Coordinators
17
List of Acronyms (continued)
TA Thematic Analysis
SEP Senior Educational Psychologist
TME Target Monitoring and Evaluation
UCL University College of London
WAG Welsh Assembly Government
18
Chapter One: Introduction
1.1 Outline of Chapter
The changing political climate has resulted in a number of significant socio-legislative
changes (Fallon, Woods & Rooney, 2010; Gersch, 2009). Furthermore, these changes
have occurred alongside the Government’s agenda to improve the quality of public
services. A key element to this agenda was that services should be delivered on the
basis of evidence-based practice (Clegg, 2005; MacKay, 2002). In addition, there has
been increasing pressure on Educational Psychology Services (EPSs) to be accountable
for the services they provide and to demonstrate their unique contribution.
The focus of this piece of research was to establish an evaluation tool which would
collect relevant information on the impact of service delivery within two Welsh Local
Authorities (LAs). It focused on finding a method that would facilitate a group of
Educational Psychologists (EPs), to successfully and appropriately evaluate consultation
and gather relevant data about the effectiveness of the EPSs for the LA.
This chapter considers the rationale for the study and outlines both the national and
local context for the research. Following on from the rationale, further information
outlining the structure of the study and the remaining chapters will be provided.
The author will refer to herself as ‘the researcher’ throughout the study. This
terminology reflects the researcher’s axiological position (see Section 4.4.6) and
acknowledges her participation in the research process. Despite her involvement in the
AR process the researcher tried to maintain the stance of ‘researcher’ within this
context. This involved maintaining neutrality and reflexivity in relation to her
contribution and how this could potentially impact upon the other participants and the
wider research process.
1.2 Rationale
1.2.1 National context
The changing educational climate driven by the previous Coalition Government had a
significant impact on some EPSs. The size of their services was reduced and posts
disappeared due to the reduction in budgets. Discussions took place about the future of
19
EPSs on a national and local level following the publication of the ‘SEN Green Paper’
(Department for Education (DfE), 2011) and the ‘School Funding Reform: next steps
towards a fairer system’ (DfE, 2011). These led to decisions being made about the
trading of EPSs and the streamlining of their work (Association of Educational
Psychologists (AEP, 2011).
The Welsh Assembly Government (WAG, 2008) launched the ‘School Effectiveness
Framework’ (SEF), a key policy for educational reform. WAG planned to implement
this through improving the quality of teaching and leadership. The SEF (WAG, 2008)
was founded on a set of principles and expected activities. A key element to the SEF
framework was the focus on improving the quality of public services with an emphasis
on the effective use and analysis of data. Hence, data should be collected in a
systematic manner with a clear focus upon outcomes which can then be used to inform
judgements about the effectiveness of services. These national reforms led to LAs
exploring potential ways to collect information to validate and evaluate their services
(Dunsmuir, Brown, Iyadurai & Monsen, 2009).
1.2.2 Local context
This research was conducted within two Welsh bordering LAs. The two EPSs involved
in the research had recently gone through the process of merging and have one PEP who
manages both services. Across the two LAs there are eleven generic EPs and two SEPs.
In September 2014 the two EPSs changed their service delivery model following joint
training from Patsy Wagner, an EP with a longstanding specialism in consultation
(Wagner, 1995; 2000). All the EPs’ now offer consultation as a service delivery model.
Each EP has a patch of schools which are allocated individual sessions on an annual
basis and group consultation is also delivered to all primary schools.
The PEP has, in recent years, been searching for effective ways to evaluate the EPSs
and the services they deliver. Evaluation methods which have already been trialled have
included sending questionnaires to head teachers which used rating scales to measure
the level of satisfaction. Additionally, a questionnaire survey was carried out in 2010 to
gather parental views on EP practice. The PEP was dissatisfied with these methods of
information gathering as they were not considered to be robust enough or in sufficient
depth. Furthermore, the evaluation methods were not developed systematically; they did
20
not focus on consultation nor gather feedback from the EPs working within the services.
In order to determine a clear and useful measure to evaluate consultation meaningfully,
the PEP prioritised the development of a theoretically sound, systemically developed
and evidence-based evaluation tool. It was subsequently agreed that the author, having
the dual role of being an EP within one of the EPSs and a doctoral student completing a
research degree, was best placed to research and investigate this area.
This study has particular pertinence when considering a piece of research commissioned
by the DfES which suggested that the quality of most Special Educational Needs (SEN)
Services evaluation was poor (Gray, 2001). In addition, the National Association of
Principal Educational Psychologists (NAPEP) (Hampshire Educational Psychology
Service, 2010) demonstrated that some EPSs were struggling to find an adequate
evaluation instrument to measure the impact of what they do. This research highlighted
the need for an effective evaluation tool within EPSs and this study hopes to contribute
to this area of research.
1.3 The evaluation of EP practice
The issue of how to measure impact for a client group with whom EPs do not work
directly has been greatly debated (Cherry, 1998; Dowling & Leibowitz, 1994; Dunsmuir
et al., 2009; Sharp, Frederickson & Laws, 2000; Turner, Randall & Mohammed, 2010).
There have been many problems associated with what is measurable, whether what is
measured correlates with the actual outcomes and the real value added by the EPs’
involvement (Cherry, 1998; Turner et al., 2010). In the past there has been an over
reliance on process indicators and a focusing on identifying outcomes that are easily
measurable (Currie, 2002). Turner et al. (2010) discussed how the data resulting from
such measures were not robust enough and, as such, the information generated did not
shed enough light on the input and effectiveness of EPs. In the real world, it is
fundamental that evaluation encapsulates the complexity of the EP role. Therefore, for it
to be meaningful, the evaluation needs to be able to measure the direct work of an EP,
as well as evaluating the outcomes of EP advice and consultations, where the EP works
through others to bring about change.
21
1.4 Summary of Research
This research aimed to establish a tool that could be implemented across two EPSs and
that would offer an effective method of evaluating consultation. It was conducted
through an Action Research (AR) framework. This method was selected as it was
deemed to be compatible with the aims of the research. Unlike traditional research,
where the researcher makes claims about what is certain and true, the AR process is also
the methodology (Cohen, Manion & Morrison, 2000). It provided the researcher with a
collaborative and consistent process where stakeholders were able to interact with the
research (Cohen et al., 2000; McNiff & Whitehead, 2011; Reason & Bradbury, 2001).
The Research and Development in Organisations (RADIO) model was selected to guide
the AR process (Timmins, Sheperd & Kelly, 2003). RADIO is a collaborative AR
framework which was developed to help Educational Psychologists in Training (EPiTs)
to manage their school improvement work (Timmins et al., 2003). RADIO provided a
robust structure which enabled the researcher to maintain a partnership with
stakeholders and colleagues (Timmins et al., 2003). Furthermore, as reported in
previous research, RADIO also provided a useful process and planning tool (Ashton,
2009; Lightfoot, 2013; Timmins et al., 2003; Timmins, Mohammed, McFadyen &
Ward, 2006). An overview of the thesis is presented below (see Table 1.1).
Table 1.1: Overview of the thesis
Chapter One Introduction Sections 1.1- 1.4
Chapter Two Literature Review Stage One Sections 2.1- 2.10
Chapter Three Literature Review Stage Two Sections 3.1- 3.5
Chapter Four Methodology Sections 4.1- 4.24
Chapter Five Results Sections 5.1- 5.15
Chapter Six Discussion Sections 6.1- 6.9.4
Chapter Seven Conclusion Sections 7.1- 7.2
22
Chapter Two: Literature Review Stage One
2.1 Chapter Introduction
The following chapter sets out the literature review for the study which was conducted
in two separate stages. The first part of the literature review focuses upon the changing
role of the EP and the need for EPs to define their distinctive contribution within an
increasingly accountable environment. It considers how professional practice
frameworks have been implemented to address these challenges and to secure the
centrality of psychology within EP practice. The second part of the literature review
outlines the search strategy that sourced the professional practice frameworks for the
research. It then considers the professional practice frameworks in further detail.
2.1.1 Literature Review Stage One
The first part of the literature review sets the context for the development of the
evaluation tool. This section of the literature firstly considers the role of EPs within the
rapidly changing context in which they work. It then moves on to focus on the effect of
these changes on EPSs and how socio-legislative changes furthered the need to justify
the ‘distinctive contribution’ that the EP brings. It considers the role of the EP as
scientist-practitioner and the need for professional practice frameworks. Because the
format for service delivery in the EPSs within this study is consultation, this is also
briefly explored within the literature review, before it moves on to consider the drive for
accountability within the current political and economic context. Finally, the evaluation
of EPSs is discussed, prior to the second stage of the literature review.
2.1.2 Stage One Search Strategy
The literature for stage one was gathered using a number of methods. Initially, the
search was broad and general, using key words such as: ‘role of the EP’;
‘accountability’; ‘reviews of the EP role’; ‘consultation’; ‘evaluation of EPSs’; ‘EP
practice frameworks’. This generated literature from websites such as Google Scholar
and sites associated with Government publications. The Welsh agenda and local agenda
were also examined. A number of Government office websites generated information
around the Welsh context and the local context (e.g., Assemblywales.org). The
researcher acknowledges that these sources were not peer reviewed but they provided a
broad range of literature which was considered alongside the studies located within the
23
peer review journals. Ovid was used to carry out advanced searches of the electronic
databases: PsychINFO, Sage full text Index, ERIC, Australian/British Education Index
and BEI. Some limitations were applied to the search which looked for full articles in
the English Language. A variety of terms, for example, ‘educational psychologists’,
‘evaluation’, ‘self-evaluation’, ‘accountability’, ‘public sector’, ‘distinctive contribution
of the EP’, ‘professional practice frameworks’ were used to produce a wide range of
references. Combinations of these were trialled (see Appendix A). The professional
networks that support the profession: EPNET (an online professional forum) and the
National Association of Principal Educational Psychologists (NAPEP) were both
consulted. An abstract review was conducted of the sourced literature and papers were
included if they were considered to be relevant, were available in full text and were
written in English.
Back copies of individual journals were hand searched; this included Educational
Psychology in Practice, the main UK practitioner journal. The references within
selected papers were reviewed which led to the consideration of a number of related
papers which were also pertinent to the literature.
2.2 Setting the context
This section of the review begins by providing an overview of the historical context of
the EP profession. This was identified as an important aspect of the literature as it
provided an insight into how the many educational reforms placed upon the EP
profession changed the role of the EP and the context in which they work. The effect of
these reforms was significant for the EP profession and impacted upon the relationship
between EP practice and its theoretical foundations. The confusion that subsequently
built up around the role of the EP raised many questions about the unique contribution
the EP offered.
2.2.1 Historical context
In the early part of the 20th century, views around educational provision for children and
young people (CYP) with SEN were positioned within a medical model of ‘deficits’.
The associated mental testing movement meant children were given medically
diagnosed categories. This provided the initial impetus for the development of the
profession of educational psychology (Wagner, 2000). Psychometric testing by early
24
psychologists can be traced back to Sir Cyril Burt, the first EP. He was employed as the
London County Council’s psychologist from 1913 to advise on providing individual
educational treatment. Aspects of his role included research, psychometrics and work
with schools (Lindsay, 1985).
The later growth of the child guidance movement, which derived from the child
guidance support systems imported from the US, led to the location of EPs in a
psychiatric clinic setting. The first child guidance clinic opened in London in 1927.
Work in the clinics was carried out by a team which included an EP, a child psychiatrist
and a social worker. This contributed to the further constriction of the EP role to tester.
The prevalent psychological model was one of individual deficit, leading to the need for
clinical diagnosis (Lindsay, 1985; Wagner, 2000). Another small strand of educational
psychology that emerged around this time was through the teaching profession with
teachers in schools training to become EPs. This influence brought with it a strong
educational component, working with teachers on identified problems (Lindsay, 1985).
Under the 1944 Education Act, significant reforms to the education system were
brought into effect. The Act was mainly directed at mainstream education but it
addressed certain aspects of education for children with SEN. However, it still focused
on a medical model of disability. Children with SEN were placed into eleven categories
of disability and special schools were still seen as the most appropriate places to educate
pupils with SEN. However, at this time it was the role of the medical officer, not the EP,
to examine the child and issue a certificate to show whether a child was “suffering from
any such disability” (34(5)).
By the 1960s educational psychology was a small profession encompassing these
different strands of educational psychology. Work was divided between that in a child
guidance clinic and that in a school psychology service. In the 1960s and 70s, the
profession moved towards an approach favoured by behavioural psychologists and tried
to move away from the medical child deficit model (Kelly, Woolfson & Boyle, 2008).
Behaviourists had rejected the medical model and were advocating an approach that
dealt with only what they could observe. This work stressed the possibility of modifying
the problems of children with SEN and placed the responsibility with the teacher
(Lindsay, 1985) The Summerfield Report (1968) provided the driver for this change in
25
perspective (HMSO, 1968). The report identified future priorities as having greater
focus on preventative work and there was a call for EPs to be involved in early
identification and intervention (Kelly et al., 2008). The report recommended a major
expansion of EP training and posts and in response to this the profession saw a rapid
increase in the number of EPs in service in the 1970s and 80s. Alongside the increase in
numbers there were also several other factors that were shaping EPSs at that time. EPs
in England and Wales in the 1970s developed many exciting new ways of working to
supplement and replace earlier patterns of working. Gillham (1978) was particularly
influential at the time and called for the application of psychology within the text
“Reconstructing Educational Psychology”. The child guidance model of delivery was
developing. The philosophy of ‘giving psychology away’ (Miller, 1969) which had
become prevalent in the 1970s had stimulated EPs to work primarily with parents and
teachers in a training capacity. The growing numbers of EPs began to alter the balance
in SEN procedures (Lindsay, 1985). Rather than medical officers carrying out
assessments of children with SEN, by 1975 this role was carried out by EPs and there
was a move from a medical towards a psychoeducational influence (Lindsay, 1985).
There was a growing disenchantment with normative assessment techniques, especially
tests of intelligence and challenges were made towards the way difficulties were
assessed. Methods of intervention with individual children broadened to encompass a
wide range of humanistic and behavioural therapies. EPs had also developed a role as
consultant to schools on matters of policy and as providers of further training for
teachers which shifted the balance away from simply working with referred children
(Kelly et al., 2008).
The introduction of the Warnock Report (1978) had a dramatic effect on the work of the
EP. The report formed the basis of the 1981 Education Act’s policies on SEN. The 1981
Education Act radically changed the conceptualisation of SEN. It introduced the idea of
statements for SEN and an integrative and inclusive approach, based on common
educational goals for all children regardless of their disabilities. It advocated a
continuum of SEN, rather than discrete categories. It suggested that only two per cent of
CYP required separate educational provision but another eighteen per cent required
special provision within mainstream schools. The Act required Local Education
Authorities (LEAs) to take advice from an LEA employed EP which meant that LEAs
had to have a psychological service (Lindsay, 1985). Alongside this legislation came a
26
huge increase in bureaucracy. There was an increase in the proportion of EP time spent
on individual assessments and this decreased the time available for preventative and
systems work. Fears raised by Lindsay (1985) were founded and there was a reverse in
the development of the EP profession and EPs were forced into the role of assessors and
gatekeepers.
Over the next decade EPs became a valuable resource to schools to the extent that they
supported them to get additional resources. Faupel and Norgate (1993) argued that this
was “perhaps the single greatest disaster” (p. 132) for EPSs as EPs were seen to be the
providers of additional resources to schools and, as such, became administration officers
at the expense of being applied psychologists. For some EPs this became a privileged
and powerful role and accorded them a great deal of status as they became major
decision makers within the LA structure (Fallon et al., 2010; Faupel & Norgate, 1993).
It also secured a position for EPs who wanted to be seen as ‘helpful’ to schools or the
LA and, at the time, ensured that EP staffing levels would be maintained despite the
threat of delegation of budgets (Wagner, 2000).
The 1994 Education Act replaced by the 1996 Education Act brought about the Special
Educational Needs Code of Practice, (DFES), (2001). This legislation was seen as
according status to EPs through the legislation of LA statutory assessment processes
and it continued to constrain the range and development of EPs’ functions (DfES, 2001;
Fallon et al., 2010; Farrell, Squires, Woods, Lewis, Rooney & O’Connor, 2006,
Norwich, 2000). EPs were placed at the centre of local authority. The EP role became
restricted to that of ‘gatekeepers’ for special educational provision (Frederickson &
Reason, 1995; Miller & Frederickson, 2006; Woods, 1994). Following on from this, the
pressure for statements led to EPs spending more time carrying out statutory assessment
work at the expense of providing early intervention when the child’s needs were first
identified (DfEE, 1997; DfEE, 2000; Pennick & Lagunowitsch, 2010).
This gatekeeping role brought significant costs for the EP profession which became
swept back into a medical model of delivery (Faupel & Norgate, 1993). The role and
responsibilities of the EP became confused which undermined the profession’s
credibility (MacKay, 2002). It was this distorted view about the role of EPs that
underpinned some of the negative evaluations of EPs’ contributions, and potential
27
contributions, to children’s development, by some educationalists (Farrell et al., 2006).
This bureaucratic role also created conflict with the theoretical roots of educational
psychology relating to the application of psychology (Kelly & Gray, 2000). Reflecting
on the call from Gillham (1978) for the application of psychology, a special edition of
the Association of Educational Psychologists’ (AEP) journal Educational Psychology
in Practice (1999), celebrated 21 years since Reconstructing Educational Psychology,
the authors considered how far the ideals written in the book had crossed over into EP
practice (Leyden,1999). Farrell et al. (2006) reported that most of the contributors felt
that the ideas had still not been implemented into mainstream practice and that the
aspects of EP practice considered outdated and ill-judged in 1978, such as intelligence
quotient (IQ) tests and the consequent ‘gatekeeper’ role, were still being practised.
2.2.2 Role definition within the EP profession
So far the literature has presented a historical context highlighting the constantly
shifting role of the EP. This constantly changing working context created confusion
around the role of the EP and potentially undermined the profession. The next section
begins to explore how more recent legislative changes further contributed to the
difficulties the EP profession experienced when attempting to define its role and how
this added to both public and professional confusion (Gersch, 2009; Kinderman, 2005)
around the role of the EP (Farrell et al., 2006).
This consistent theme of “reconstruction,” “reformulation” and “refocusing” of the
profession (Fallon et al., 2010, p.2) has continued over the past decade and the
profession is currently undergoing another period of rapid and immense change in light
of the new SEND Code of Practice (DfE, 2014). Government reforms have included:
‘Every Child Matters’ (DfES, 2004); the ‘Common Assessment Framework’ (2004); and
changes to EP training from a one year Masters to a three year Doctorate to bring it in
line with clinical psychology training. All of these reforms have resulted in a further
period of considerable adjustment for EPs. They have also had a significant impact
upon the delivery systems of EPSs and created a much wider spectrum of possible
work. One of the consequences of such role expansion has been further decrease in role
clarity (Stobie, 2002). As such, the EP profession was placed in a position that rendered
it rather ambiguous and therefore subject to further role conflict and confusion.
28
Consequently, many EPs in the twenty first century continued to find it difficult to
describe their role (Ashton & Roberts, 2006; Stobie, 2002).
The drive for integration into children’s services resulting from these legislative reforms
created a multi-agency focus with many EPs working in more diverse roles, with a
greater range of role partners (AEP, 2008; Farrell et al., 2006). On the one hand, the
literature considers how multi-agency working increases the feelings of professional
identity for EPs (Gaskell & Leadbetter, 2009; Willdridge, 2013). However, on the other,
for some professionals the blurring of boundaries that can occur within multi-agency
working can prove stressful (Moran, Bunn & Bifulco, 2007) with the overlapping of the
EP role with other professional groups leading to the erosion of professional identity for
some EPs. The diversity of the EP role meant that EP service delivery differed greatly at
both a service wide level and between individual EPs within services. This brought with
it a number of complex challenges and created confusion about what it is that EPs do
(Boyle & Lauchlan, 2009; Fallon et al., 2010). The multi-agency working context led to
questions being asked around whether other agencies do jobs which are similar to parts
of the EP role. This is currently evident within the EPSs in which this study is taking
place with EPs being asked to define their distinctive role and offer clarity about the
unique contribution they make in comparison to other services (Kelly & Gray, 2000;
Stobie, 2002). Consequently, this created a challenging context and, as highlighted
within the literature, within such working contexts EPs can begin to lose sight of the
beliefs, hopes and aspirations with which they entered the profession and become
anxious and uncertain about their futures (Cameron, 2006). Cameron and Monsen
(2005) suggest that the uncertainty around the professional role of the EP is one of the
main reasons to explain why many EPs appear to be experiencing an ‘identity crisis’.
The ‘identity crisis’ and ‘insecurity complex’ within the EP profession has been widely
acknowledged within the literature (Boyle & Lauchlan, 2009; Farrell et al., 2006;
Gaskell & Leadbetter, 2009; Love, 2009; Norwich, 2005; Willdridge, 2013) and are
ongoing issues within the EPSs contributing to this study.
In addition, the literature highlights that alongside the ambiguity and confusion around
the nature of the EP role there is also evidence indicating that the ideas EPs have about
their role are not necessarily congruent with those to whom they deliver service
(Mackay, 2002). Research indicates disparities between what schools want, what EPSs
29
want to deliver and a mismatch between what EPSs think they should be doing and
what users perceive as their role (Ashton and Roberts, 2006; Kelly & Gray, 2000).
Findings from a national survey of schools (Kelly & Gray, 2000) suggested that there
were conflicts between what schools are looking for and what EPs want to offer.
Boyle and MacKay (2007) discussed how these disparities and conflicts were
consistently reported in studies on the views of schools from the 1970s to the early
1990s. In the 1990s MacKay and Boyle conducted a secondary school study (Boyle &
MacKay, 1990) and a combined study in primary and secondary schools looking at
what schools expect the role of the EP to be (Boyle & Lauchlan, 2009). MacKay and
Boyle (1994) interviewed head teachers in 115 schools across Scotland. They noted that
after more than a decade of reconstruction of psychological services, in which a major
theme has been the reduced emphasis on direct work with individual children, the
traditional role of individual assessment was one that continued to be strongly endorsed
by teachers. This research also highlighted discrepancies that existed between EPs and
teachers in their perceptions regarding the EP role. Boyle and MacKay (2007) later
carried out follow up research finding a continued mismatch between what teachers
wanted and believed they were getting and what EPs felt that teachers needed and
believed they were receiving. The traditional Scottish EP role was based on individual
casework and the new roles were related to interventions based on consultation,
training, research and involvement in policy and organisational change. However, while
EPs tended to believe the changes in service delivery were a valuable enhancement of
practice, teachers reported valuing the traditional EP roles and preferred models of
service delivery that offered a continuing commitment to these.
Research by Ashton and Roberts (2006) also highlighted a gap between the perceptions
of a group of Special Educational Needs Coordinators (SENCos) and EPs. This small
scale study was prompted through discussions with newly qualified EPs from the same
LA who felt anxious because they were unclear about the unique contribution they
could make to schools. The study aimed to explore which aspects of the EP role were
valued by SENCos and also EPs themselves. It compared the views of the SENCos with
those of the EPs in the study. This research found that EPs had their own ideas about
what their services should look like which were not necessarily the same as those of
their clients. Most of the SENCos valued the ‘traditional’ EP role of individual
30
assessment, which often led to statutory assessment and giving their expert advice to
staff. The EPs in the study valued working through consultation, the relationship they
shared with the school, the different perspectives they brought to understanding
different problems and the fact that they included the views of the pupils. Within this
piece of research there was also a suggestion of role conflict within the EP profession
itself. The findings showed that the EPs themselves were struggling to agree on what
they should be offering to schools. However, this piece of research captured the view of
only a small group of eight EPs within one EPS and it is questionable to what extent the
findings can be generalised.
The findings of Ashton and Roberts (2006) reflect the current context within the EPSs
which this study is taking place. With recent job losses and a reduced service capacity
there have been frequent changes within the service delivery of one of the EPSs. This
EPS is currently delivering group consultation but the EPs within the team are not in
general agreement about whether they should be offering this or whether schools
actually want this aspect of service delivery. This demonstrates that decisions about
what EPSs should look like may not be congruent with what the service users want.
Furthermore, this supports the literature which demonstrates that EPs do not necessarily
agree with each other about what they want to offer (Ashton & Roberts, 2006). It is not
surprising; therefore, that confusion has built up around the role of the EP.
2.3 Role confusion
The literature has considered how, alongside the historical context, more recent
legislative changes have contributed further to the confusion around the role of the EP
and that this role confusion existed amongst EPs and service users. Unsurprisingly, this
has created significant challenges for the EP profession over the years in relation to the
application of psychology. Indeed, as highlighted by Norwich (2000), the profession
spends a great deal of its time contemplating how much psychology is used in everyday
work. It has been suggested that due to the complex historical background and changing
socio-legislative context the gulf between educational psychology and its academic
roots could be greater than in any other area of applied psychology (Kelly et al., 2008).
As the EP profession looked to address the challenges faced with regard to defining its
role through the application of psychology, there was a realisation that social
31
constructionism could provide a way of working that could bridge the gap between
theory and practice and move EPs away from the ‘within child’ focus towards a
preventative way of working (Kelly & Gray, 2000). However, the application of social
constructionist theory to EP practice brought with it significant challenges and
“generated its own minefield of conflicts about role focus and about the relationship of
educational psychology to education” (Kelly et al., 2008, p. 21).
As argued by Kelly et al. (2008), these challenges caused a delay in academic
developments reflecting the theory of social constructionism which could have been
applied to EP practice. This lack of cohesion between theory and practice and the
subsequent scarcity of frameworks which might have enabled theoretical models to be
embedded in professional practice created a lack of clarity and consistency within EPS
delivery. Subsequently this led to the implementation of ‘loose approaches’ and the
failure to adopt a collective approach across the EP profession. This challenge of
adapting theory to practice is a further reason put forward for the lack of clarity around
the role of the EP (Norwich, 2000) and the need for EPs to constantly reflect upon and
question their role (Boyle & Lauchlan, 2009).
As is evident within the literature, the EPs involved in this research have found it very
difficult to describe and define their role (Corban, 2011; Love, 2009). Until very
recently the EPs have maintained a ‘gatekeeper’ role within the LAs and the delegation
of budgets to schools has been a very recent development. Subsequently, services are
under increased scrutiny and the EPs are being asked to define their role. Furthermore, it
has become apparent that a great deal of confusion exists between LA officers around
the role of the EP.
2.4 Reviews of the role of the EP
The literature has explored how the changing legislative and working context for EPs
has created confusion around the role. In addition, the challenges faced around the
application of psychological approaches led to the implementation of ‘loose approaches’
(Kelly et al., 2008). Continued debates around what EPs do and whether an EP is the
most appropriate professional to involve in a particular piece of work highlighted to the
profession that it needed to provide clarity around its purpose to ensure that LAs and
service users could be confident about the unique role they offer (Cameron, 2006;
32
Farrell et al., 2006). In response to this context a number of reviews on the role of the
EP were undertaken and these will now be detailed within this section. The debate
around the distinctive contribution the EP brings is particularly pertinent to the EPSs
involved in this study as they are under increasing pressure to provide evidence to
demonstrate this (Cameron, 2006). This adds weight to the rationale for this piece of
research which is looking to find an effective service evaluation model of consultation.
As outlined, questions around the distinctive contribution of the EP profession have
been one of the driving forces behind reviews of the EP profession (Ashton & Roberts,
2006). There have been numerous and frequent reviews of the role of the EP at an
academic, practitioner, governmental and professional level (Fallon et al., 2010). Some
of these have been broad ranging, such as ‘The Currie Report’ (Scottish Executive,
2002) and the ‘Review of the Functions and Contribution of Educational Psychologists
in England and Wales in light of “Every Child Matters: Change for Children” (Farrell
et al., 2006). Within the parameters of this review it is only possible to briefly consider
the findings by the DfEE (2000), the Scottish Executive (2002) and the review by
Farrell et al. (2006). The Welsh context will also be summarised as it is pertinent to the
context of this research.
Within the Green Paper ‘Excellence for all Children: Meeting Special Educational
Needs’ (DfEE, 1997), the government made a commitment to look at ways to shift the
balance of EP work towards earlier intervention and support. The follow up report
‘Meeting Special Educational Needs: A Programme of Action’ (DfEE, 1998) led to a
reappraisal of the future role and training of EPs (Kelly & Gray, 2000; Kelly et al.,
2008). A working group were tasked to determine future priority areas for EPSs. The
subsequent report: ‘Educational Psychology Services (England): Current Role, Good
Practice and Future Directions’ (DfEE, 2000) defined several future priorities for EPs.
These included:
• To have a continuing role in working with children with SEN;
• Consultation, problem solving and solution focused approaches;
• “To apply psychology in an educational context” (p. 71)
33
The “Review of the Provision of Educational Psychology Services in Scotland” (Currie
Report) (Scottish Executive, 2002) published performance indicators and the core
functions of EP work were identified as: assessment, intervention, consultation, training
and research. The report considered that these five functions should be delivered at three
levels: the individual level (child), at the group level (class/group/family) and at the
organisational level (school/LA) (Boyle & Lauchlan, 2009; Fallon et al., 2010). These
five functions were reflected within the British Psychological Society (BPS) (2006)
National Occupational Standards for Applied EPs (Fallon et al., 2010).
Following these reviews there were considerable changes to EP roles and
responsibilities and a change in the training route for EPs. This context presented
challenges and opportunities for the evolving role of educational psychology and
formed the backdrop for the DfES funded review of the role of the EP (Farrell et al.,
2006). This was one of the most comprehensive reports of EP service delivery in recent
years (Kelly et al., 2008). It considered the views of a range of stakeholders as to the
distinctive contribution that EPs can make in light of the five Every Child Matters
(ECM) outcomes and reforms (DfES, 2004).
The Farrell et al. (2006) review generated recommendations for the future role and
function of the EP. As with the review by the Scottish Executive (2002), Farrell et al.
(2006) also recognised that the core functions of EP work should be done at the level of
the individual child, the group and the organisation. It is not feasible to outline all the
recommendations from the review; instead, the most pertinent ones will be outlined.
Despite these recommendations being delivered before the onset of traded services in
England, it is important to note that the picture in Wales is very different to England.
Farrell et al.’s (2006) review remains relevant to the context for this study as these
recommendations are only just beginning to be implemented:
• The extent to which the role and function of EPs is distinctive: Farrell et al.
(2006) stated that documentation about the range of work offered by an EP
service should be explicit about the psychological nature of their contribution
and when responding to a particular request for EP involvement, EPs should
clarify the specific nature of the work required and the psychological
contribution that they can offer and, where appropriate, clarify whether an
34
alternative provider is available who might be able to carry out the work with
the same impact.
• The impact of a reduction in EPs’ role in statutory work: Farrell et al. (2006)
considered how EPs should continue to have a key role in the statutory
assessment of children with the most complex needs but that they should take
advantage of the trend in the reduction of statutory work to expand and develop
their activities in different areas where their skills and knowledge can be used to
greater effect e.g. group and individual therapy, staff training and systems work.
2.5 Welsh context
In addition to the English context, it is important to consider the Welsh context as this is
also relevant to the EPSs within this study. The Welsh Assembly Government (WAG)
set out their priorities for the future through a document entitled ‘Children and Young
people: Rights to Action’ (WAG, 2011) this presented a vision of a fairer, more
prosperous, healthier and better-educated country. WAG adopted the United Nations
Convention on the Rights of the Child (UN, 1989) as the basis of all work for CYP in
Wales. This was translated the rights into seven core aims for CYP. These were that
CYP:
• Will have a flying start in life;
• Will have a comprehensive range of education and learning opportunities;
• Will enjoy the best possible health and are free from abuse, victimisation and
exploitation;
• Will have access to play, leisure, sporting and cultural activities;
• Are listened to, treated with respect, and have their race and cultural identity
recognised;
• Have a safe home and community which supports physical and emotional
wellbeing;
• Are not disadvantaged by poverty (WAG 2011, p.1)
The national approach includes developing policy, improving services for CYP, and
evaluating the difference made to their lives so that achievements can be demonstrated.
This political agenda, informs work at the authority level, in the form of the Children
35
and Young People’s Partnership Plan which aims to translate the seven core aims into
actions. From this LAs need to assess and demonstrate how services support children
and young people and the impact that services have for young people.
2.6 Role of scientist practitioner
The outcomes of the many extensive reviews on the role of EPs highlighted to the
profession that the application of psychology is a fundamental part of the EP role. Each
supported a movement away from the ‘historical’ EP role which has been outlined as
one of over-involvement in administrative gate-keeping tasks towards the provision of a
more sound use of psychological knowledge across the variety of contexts in which EPs
work (Farrell et al., 2006) (see Section 2.4). As such there has been an explicit move for
EPs towards a reconciliation of the pragmatic and the scientific through the
“conceptualisation” of the role as that of a “scientist-practitioner” (Fallon et al., 2010;
Lane & Corrie, 2006). This envisages EPs implementing scientific principles and
methods within their practice (Fallon et al., 2010; Lane & Corrie, 2006; Miller &
Frederickson, 2006). Fallon et al. (2010) discuss how alongside developing
understanding around the core functions of EPs there has also been a move towards
integrating psychological frameworks into EP working practice as a tool for drawing
out “psychological knowledge, skills and understanding” (p. 4).
Kelly et al. (2008) argued that practice frameworks are an essential resource for the EP
profession and that they provide a means of addressing the challenge that EPs face with
regard to clarifying the complex relationship between theory and practice and enabling
the integration and cohesion of complex theoretical perspectives with equally complex
practice methodology. Effective practice frameworks can be seen to secure the
centrality of psychology and the distinctiveness of the EP’s contribution in their
professional interactions. Such frameworks also enable the integration of functions
across different pieces of work (Fallon et al., 2010; Frederickson & Miller, 2008;
Gameson, Rhydderch, Ellis & Carrol, 2005; Kelly et al., 2008; Leadbetter, 2005;
Timmins et al., 2003; Woolfson, Whaling, Stewart, & Monsen, 2003).
The need to more clearly demonstrate the scientist-practitioner aspect of the EP role
provided the impetus for the EPSs within this study to select consultation as a model of
service delivery to schools. As highlighted within the literature, consultation has
36
emerged as a way of addressing the shift in the nature and balance of EP work and
linking theory to practice. Consultation approaches have increasingly become seen as an
important aspect of EP work (Athun, 2000; Dunsmuir et al., 2009; Kelly & Gray, 2000).
Indeed, the Scottish Executive (2002) and Farrell et al. (2006) highlighted consultation
as a core function of the EP role (Boyle & MacKay, 2007; Dennis, 2004). The
significance of consultation is further highlighted by its inclusion in the UK core
curriculum for initial training and within later professional development in EPSs.
(Dickinson, 2000; Kennedy, Cameron & Monsen, 2009). Before consultation is
considered in more detail within section 2.7, it is important to distinguish between the
professional practice model of consultation and the professional practice frameworks
that have been developed to be used by EPs within their practice. Wicks (2013)
provides a clear explanation of the difference between professional practice models,
such as consultation, (Wagner, 2000) which support the application of theoretical
models, and, ‘executive’ professional practice frameworks which can be applied across
all areas of educational psychology practice, at any level and do not stipulate the
psychology that should be used by the EP (Wicks, 2013). It is important to clarify for
the reader that within this research, consultation is employed by the EPs as the model of
service delivery. Several of the professional practice frameworks that are summarised
within Sections 3.3.2 – 3.3.10, have been referred to within the literature as ‘executive
frameworks’ (Kelly et al., 2008; Wicks, 2013) and have been used by EPiTs and EPs to
scaffold their practice (Woolfson, 2008). Unlike previous research, this study considers
these professional practice frameworks as potential tools for the evaluation of
consultation.
2.7 Consultation
The following section will consider consultation in more detail as it was selected by the
PEP of the EPSs within this study as the initial focus for service evaluation.
As previously outlined, the EPSs within this study selected a consultation model of
service delivery as a way of integrating the application of psychology into their working
practices and demonstrating the distinctive contribution they can offer. This move
towards more consultative ways of working is highlighted within the literature and
Larney (2003) discusses how, in recent years, this has been one of the most significant
changes within educational psychology. Many EPSs within the UK deliver service
37
through a consultation model but this approach varies widely (Wagner, 2000). For
some, traditional methods of working are often maintained because of LA requirements
and Wagner (2000) noted that EPSs are at different places within this model. The EPSs
in the two Welsh LAs taking part in this research have adopted Patsy Wagner’s
consultation model (Wagner, 1995; 2000; 2008) having been trained by Wagner in
2013. This is the predominant method of service delivery across the two EPSs who have
been relatively autonomous in their decision making, until recent economic and political
pressures took effect. Before the Wagner consultation model (Wagner, 1995; 2000;
2008) is outlined a brief background to consultation will be provided and the four most
prevalent consultation models within the literature will then be detailed.
2.7.1 Background to Consultation
Consultation originated in the USA where school psychologists started to use and
implement the approach within their school work (Conoley & Conoley, 1982; Coneley,
Conely, Ivey & Scheel, 1991; Gutkin & Curtis, 1990).
As far back as 1981, research into the medical model of service delivery suggested its
effectiveness was questionable. School psychology services were predominantly
focused on the assessing, diagnosing and treating children’s diagnoses and far less focus
was given to the environments in which children function (Gutkin & Curtis, 1990).
School based consultation was seen as a feasible solution and an efficient use of time
and resources. School based consultation emerged as one of the preferred models of
school psychology practice over the next 10-20 years (Dunsmuir et al., 2009; Gutkin &
Curtis, 1990).
There are a number of different consultation models available to EP practitioners. It is
not possible within the scope of this literature review to discuss each model so the
following four models will be briefly outlined: Mental Health Consultation;
Behavioural Consultation; Process Consultation and Organisational/Systems
Consultation (Larney, 2003). These models have been selected as they have been
“particularly influential within the field of school-based consultation” (Sheridan &
Cowan, 2004, p. 599). There are key differences which relate to the psychological
theories that underpin the various models and these influence the process that is adopted
within the consultation process (Kennedy et al., 2009). However, Miller and
38
Frederickson (2006) argue that all the models have some shared features. These include
the participation of a consultee when identifying and exploring a problem and working
with them to bring about change (Kennedy et al., 2009; Wagner, 1995).
Mental Health Consultation was developed by Gerard Caplan in the 1960s (Caplan,
1970; Larney, 2003). As the longest standing approach it was born within mental health
rather than educational settings (Miller, 1996). Larney (2003) considered how this
consultation model might be amenable to adaptation to the school contexts; however, it
has not been widely applied by EPs for several reasons. It is perceived as being too
psychodynamically-oriented (Larney, 2003; Watkins, 2000). Additionally its lack of
supporting empirical evidence has contributed to its lack of popularity within
psychological practice (Gutkin & Curtis, 1999; Larney, 2003; Wilkinson, 2006).
Process and organisational/systems share many features in common (Larney, 2003).
They are rooted in the psychology of groups and organisations and, typically, the client
is a group within an organisation or an entire organisational system itself (Gutkin &
Curtis, 1999). Process consultation is often associated most prominently with the work
of Schein (Miller, 1996; Schein, 1998). It aims to make people more aware of the events
or processes in their environments and the ways in which these affect their work
(Schein, 1998). In organisational/systems consultation, the aim is to work towards
effecting change at the organisational /systems level. The literature indicates that both
the process and the organisational/systems models are potentially useful approaches to
consultation in school contexts but have not enjoyed huge popularity. One of the main
reasons for this has been that these models are more alien to teachers than the
behavioural approach described earlier and both models require detailed training input
for teachers if they are to succeed (Gutkin & Curtis, 1999). This has been a barrier to
school psychologists adopting them on a large-scale basis.
The literature highlights how behavioural consultation is the model which is most
widely used and researched within school psychology both in the US and the UK
(Gutkin & Curtis, 1999; Larney, 2003) and is drawn upon most widely within Wagner’s
consultation model (Wagner, 1995; 2000; 2008). The model has its roots in social
learning theory and it aims to work with consultees to identify and manipulate relevant
person-environment variables to improve, eliminate and/or prevent identified problems
39
(Larney, 2003) and is viewed as a more straightforward approach (Miller, 1996). The
role of the consultant is to lead the consultee through a structured problem solving
process as a means of producing change (Larney, 2003; Wilkinson, 2006). One of the
major criticisms of this approach relates to its lack of focus on the nature of the
consultant-consultee relationship and how this can determine whether a favourable
outcome will be achieved. Gutkin and Curtis (1999) discuss how an increasing amount
of research has begun to address this issue and the behavioural consultation approach
continues to be the model chosen by most practitioners.
2.7.2 Wagner’s Consultation model
As highlighted in the above literature, a number of EPSs nationally have selected
consultation as the medium through which they deliver psychology services to schools.
A wide range of these services, along with the EPS in this study, adopted the Wagner
model of delivery (Larney, 2003). Therefore there will now be a brief synopsis of the
literature around the Wagner model.
The Wagner model of consultation has been in development across the UK within EPSs
for the past 26 years and is still in the process of ongoing development (Wagner, 2008).
This model is discussed in three key works on consultation (Wagner, 1995; 2000; 2008)
and is generally described as a collaborative approach focusing on a joint investigation,
problem-solving, planning and intervention and evaluation and review process
(Woolfson et al., 2003). It aims to promote change at the level of the individual child,
the group/class or the organisation/whole school level (Gutkin & Curtis, 1999; Wagner,
1995). Wagner’s approach draws upon the behavioural consultation model but it is also
derives features from a wide range of psychological theories: narrative therapy, social
constructionism, systems thinking, solution-focused theory, and personal construct
psychology. These psychological models are able to reflect the complex context in
which EPs work (Wagner, 1995; 2000).
The approach places great emphasis on the equal role between EP and consultee as the
‘perceived expert’ role of the EP can be ‘deskilling’ due to the fact that the EP is
potentially place in a position of power (Wagner, 2000). It encourages the EP to work
with the professionals involved and not necessarily directly with the child thus enabling
EPs to empower the team members around the child that are most appropriately placed
40
to support (Leadbetter, 2006; Pennick & Lagunowitsch, 2010). This, it is argued,
improves its efficiency as the clients are best supported by those who work more
directly with the child (Cording, 2011; Gutkin & Curtis, 1999). It involves shared
assessment and intervention and recognises the professional skills of teachers (Idol,
1990; Larney, 2003).
2.8 Accountability within the current developments in Educational Psychology
Services
So far the literature review has considered how the constantly changing role of the EP
has contributed to confusion around the role. In turn, this has contributed to negative
assessments of the profession (Fallon et al., 2010). For some EPs this led to an ‘identity
crisis’ and loss of professional identity (Boyle & Lauchlan, 2009; Fallon et al., 2010;
Farrell et al., 2006 ; Gaskell & Leadbetter, 2009; Love, 2009; Norwich 2000;
Willdridge, 2013). In response to this numerous reviews of the EP profession were
undertaken to explore the distinctive contribution of the EP and these identified that EPs
should be applying psychological principles and methods to their practice. Consultation
was identified as one of the EPs core functions within reviews of the EP role (Scottish
Executive, 2002) and has subsequently been adopted by a number of EPSs nationally as
the medium through which they deliver psychology services to school. As with many
EPSs, the EPSs within this study selected a consultation model of service delivery as a
way of integrating the application of psychology into their working practices and in
order to demonstrate their distinctive contribution. This is particularly relevant within
the increasingly accountable environment in which EPs are finding themselves situated.
2.8.1 Political and economic context
The following section of the literature review considers the issue of accountability
within the current political and economic context of EPSs. It highlights how it is not
enough just to implement a psychological model of service delivery but how thorough
evaluation is also required in order to assess the effectiveness of such models,
considering that EPs are under intense scrutiny to demonstrate accountability evidenced
through their performance. The need to demonstrate that what EPs do during
consultation makes a difference, adds weight to the rationale for this piece of research
which look to find an effective service evaluation model of consultation.
41
Political reforms, arising from the 2008 global economic crisis and the introduction of
the Coalition Government in 2010, forced massive reductions in public spending, cuts
in LA funding and changes to the delivery of public services including EPSs. The
publication of the ‘School Funding Reform-Proposals for a fairer system’ (DfE, 2011)
placed the funding of EPs under the responsibility of the LA enabling local decisions to
be made about funding for EPs and whether this should be delegated to schools. In
Wales there was a continued drive for the Education department, through the School
Funding Regulations (2010), to delegate funding to schools with a commitment to
increasing the delegation rate to 85% within four years. Schools can then make their
own decisions about how to spend this money (www.assembly.wales)
In response to this changing social, political and educational climate the AEP published
the circular: ‘Principles for the delivery of Educational Psychology Services’ (2010)
which highlighted the impact on EPSs. It outlined how the imposition of major
reductions in public spending led to significant cuts in the funding for LAs.
Subsequently, LAs began reducing many teams within their Children’s Services
departments, including EPSs. Between Autumn 2010 and Autumn 2011 approximately
200 substantive EP posts disappeared from LAs (AEP, 2011). The impact across EPSs
has been very varied with some services experiencing minor changes, and others facing
complete re -structuring. The AEP (2011) define the nature of the current services as
ranging from:
1) The ‘traditional’ LA model (fully core funded)
2) The ‘LA plus’ model (semi-traded)
3) Other models (fully traded/commissioned)
Running alongside these significant changes was the Government’s continuing agenda
to improve the quality of public services. Willdridge (2013) discussed how the drive
towards improving the effectiveness, efficiency and quality of public services had been
on the Government agenda for many years. Both the White Paper: Modernising
Government (Cabinet Office, 1999) and the National Audit Office (2001) called for an
improvement in the evaluation, monitoring and the reporting of outcomes to
stakeholders. In 2008 the WAG launched the SEF a key policy for educational reform
founded on a set of principles and expected activities which include annual self-
42
evaluation updates using Estyn’s (Her Majesty’s Inspectorate for Education and
Training in Wales) Common Inspection Framework to identify areas for improvement.
These LA self-evaluations would form part of the basis of Estyn’s inspections. The data
produced by services to demonstrate their impact were then considered by LAs and used
to inform decisions about the most efficient and effective deployment of public
resources.
This political and economic situation has subjected the role of the EP to increased
scrutiny (Fallon et al., 2010).The drive towards improved service delivery with
comprehensive impact data means that EPS s need to be more accountable than ever for
their performance and ensure increased consistency in the standards of their service
delivery (Farrell et al., 2006; Timmins et al., 2006; Willdridge, 2013). EPSs are no
longer sheltered from market pressures and protected by their statutory function (Pugh,
2010). This reflects the working context of the EPSs within this study and managers are
currently being asked to present to LA officers what a 30% reduction in EP posts would
mean to the LA. In addition, they are also being asked to consider a traded model of
service delivery. This demonstrates how the work of the EP needs to be evidenced
through their performance and the results they produce (Farrell et al., 2006). This clear
focus on accountable practice has created, now more than ever, the need for EPs to
provide evidence based practice. Gersch (2009) argues that for educational psychology
to be a valued and adequately funded profession with a secure future, EPs need to be
seen as relevant, valuable, useful and capable of providing real solutions for people.
2.9 Evaluation of Educational Psychology Services
The drive for increased accountability has highlighted to the EP profession that it needs
to be regularly updating the information it produces in order to monitor services, to
determine whether they are meeting their goals (Boyne, Gould-Williams, Law &
Walker, 2004) and to answer questions related to: ‘what works ‘and ‘what added value
do they bring’? (Gersch, 2009). Such evidence can help to safeguard effective services
and can assist with the drive for austerity and efficiency by helping to highlight the most
cost effective ways to deliver positive changes (Lloyd & Harrington, 2012). In order to
provide such evidence it is critical that EPSs evaluate the services they deliver and this
provided the impetus for this piece of research. One of the specific areas this study
wanted to address was how self-evaluation data could be used to communicate to the
43
public and LA officers about the effectiveness of the EPSs (Kaslow, 2004). The
findings will also be used to inform future decisions about the evaluation of service
delivery. The importance of this piece of research was highlighted by the Lamb Inquiry:
Special Educational Needs and Parental Confidence (DCSF, 2009), which emphasised
the importance of the evaluation of different EPS service models and their impact on
outcomes for CYP. Turner et al. (2010) concurred and discussed how EPs need to
provide evidence of the positive impact they bring.
A particularly relevant question to address, having reviewed the literature about current
evaluation within the field of EPSs, is with regard to how capable public organisations
are of self –evaluation (Boyne et al, 2004). A piece of research commissioned by the
DfES suggested that the quality of most SEN Services evaluations was poor (Gray,
2001). An evaluation carried out to look at how EPSs evaluate themselves (Hampshire
Educational Psychology Services (HEPS), 2010) found that this was an area that needed
to be addressed. This piece of research is especially pertinent to this study so will be
considered in more detail within the Section 2.9.1.
2.9.1 Hampshire Educational Psychology Service evaluation
In 2010 HEPs: Research Evaluation Unit conducted an evaluation on behalf of the
National Association of Principal Educational Psychologists (NAPEP) which
considered how EPSs currently evaluate themselves. Due to the poor response rate (24
EPSs) only limited information was collected but the collated responses highlighted that
EPSs were struggling to find an adequate evaluation instrument to measure the impact
of what they do.
One of the questions asked was: How do you evaluate the impact of the service? Many
services indicated that they had either adopted or been looking to apply recognised
models of evaluation. The most common amongst these was the Friedman Results
Based Accountability model (Friedman, 1997) but other models included RADIO
(Timmins et al., 2003). NAPEP (2010) received reports of nine services using a scaling
system of some sort. Three were recognisable as Target Monitoring and Evaluation
(Dunsmuir et al., 2009) and one had piloted Goal Attainment Scaling (Dunsmuir et al.,
2009). Three services described assessing impact against a standardised measure of
some sort, for example, Strengths and Difficulties Questionnaire.
44
NAPEP (2010) outlined how the comments indicated that EPSs recognised that
evaluating impact was “the big question” they needed to address. Many were grappling
with how best to achieve this but did not feel as though they had satisfactory
mechanisms in place. The difficulties inherent in the task were highlighted with the
diversity of the EP role making it difficult to measure the less tangible aspects of the
difference EPs really make. The responses demonstrated that it was not always clear
when a meaningful post measure should be taken or how to define the real indices of
success. There are inherent difficulties in separating the impact of the EP’s contribution
from the others who actually put the advice into practice. NAPEP (2010) considered
that whilst many of these issues were valid they were not impossible to resolve.
2.10 Measuring EP impact through consultation
The literature highlights that one of the fundamental reasons behind the difficulties
associated with measuring the difference EPs make is the fact that it is such a
complicated process (NAPEP, 2010). Indeed, when considering the difficulties
associated with evaluating consultation, it can be possible, to understand why this might
be the case. One factor is that within the consultation model EPs are working with a
wide range of people across a variety of different settings (Farrell et al., 2006). Another
added complication is that it is often the consultee rather than the EP who implements
the recommendations. This leaves EPSs with the challenge of defining outcomes that
are measurable and can show a positive impact within a practice which is particularly
complex (Dunsmuir et al., 2009).
The issue of how to measure impact for a client group with whom EPs do not work
directly with has been greatly debated (Cherry, 1998; Dowling & Leibowitz, 1994;
Dunsmuir et al., 2009; Sharp et al., 2000; Turner et al., 2010). There are many problems
associated with what is measurable and whether what is measured correlates with the
actual outcomes and the real value added from an EP’s involvement (Cherry, 1998;
Turner et al., 2010). Sharp et al. (2000) make the distinction between ‘output measures’
(what is done) and ‘outcome measures’ (what is achieved). Through their research they
found that output measures did not provide adequate information for service
improvement purposes. The focus upon the need for outcome rather than output
measures was also emphasised in a paper by Dunsmuir et al. (2009). This research
45
identifies that the challenge is to “define outcomes that are measurable” (p. 54).
However, further literature indicates that there has been an over reliance on process
indicators and only focusing on identifying outcomes that are easily measurable (Currie,
2002). The data resulting from measurable outcomes are not considered to be robust
enough. Turner et al. (2010) discuss how such information does not shed enough light
on the input and effectiveness of an EP. This relates in part to the complexity of the EP
role and there are also inherent difficulties attached when attempting to apply a
scientific, reductionist paradigm to this area of ‘soft’ systems. Many attempts to
evaluate qualitative methods appear to have resulted in quantitative measurements.
Turner et al. (2010) suggested that process measures should be used alongside
quantitative measures thus making evaluation more comprehensive. However, how to
link output data to the input from the EP still proves problematic (Turner et al., 2010).
In 2006 Scottish EPSs drafted quality indicators. These suggested that evidence of
impact for CYP should be generated through both quantitative and qualitative data and
should incorporate both process and outcome data (Turner et al., 2010). Also, the nature
of the relationship, and where the EP is located within the “chain of impact” should
always be considered (Turner et al., 2010, p.316). However, the challenge of
highlighting the specific input and impact of an EP still remains. For evaluation to be
meaningful this challenge still has to be addressed (Turner et al., 2010). Therefore,
evaluation needs to capture the complexity of the EP role. Furthermore, it needs to find
a way to measure the impact of the EP during consultation where they are working with
others to bring about change.
This review of the literature has highlighted that the EP profession would benefit from
finding a suitable self- evaluation tool and the reasons for this, but demonstrates that
there are challenges associated with this task. It is essential that such an evaluation tool
is effective in order for it to be valuable to EPSs (Gray, 2001). Dunsmuir et al. (2009)
argued that for a self-evaluation tool to be effective it should collect data in a systematic
manner in order to inform reliable judgements about the effectiveness of EPSs. The
literature contends that the EP profession would benefit from a formalised transparent
framework of evaluation which can be embedded in the day-to-day world of an EP
(Currie, 2002; Turner et al., 2010). Turner et al. (2010) suggested that this is a role for
applied psychologists and they should use their knowledge and skills to refine the tools
46
and processes required for evaluating their work. With this in mind, this study looked to
establish a tool which would facilitate the EPSs to evaluate consultation.
The next stage of the literature review: Literature Review Stage Two, details the
sourcing of the professional practice frameworks that were selected for the research
process.
47
Chapter Three: Literature Review Stage Two
3.1 Chapter Outline
The second part of the literature review begins by briefly providing an overview of the
service context in which this research was undertaken before outlining the search
strategy that sourced the professional practice frameworks for the research. It then
considers the different professional practice frameworks in further detail and finally
presents the research questions.
3.2 Service context
As outlined within Literature Review Stage One, Wagner’s (1995, 2000) model of
consultation is the method of service delivery adopted across the two services in which
this piece of research took place. Prior to March 2013 a range of practice and
perceptions existed around consultation. This led to the organisation of further joint
consultation training from Patsy Wagner in March 2013 which acted as a refresher
course prior to a re-launch of consultation in September 2014.
The two Welsh EPSs participating in this research have, in recent years, been searching
for effective ways to evaluate the services they deliver (refer to Section 1.2.2).
This piece of research was prompted by the drive from the EP management team to
develop an evaluative framework tool which could be utilised in an efficient manner
and embedded within a consultation model of service delivery. The focus of this
research was at the individual level and concentrated on how individual consultations
could be monitored and recorded through the integration of professional practice
frameworks. As argued by Kelly et al. (2008), the review and evaluation of consultation
is a key aspect of individual consultations and should take place at each distinct
consultation meeting with the aim of reviewing and evaluating the meeting. Therefore,
the overall aim of this study was to establish an evaluation tool that could collect
relevant information on the impact of the EPSs service delivery. It focused on finding a
method that could facilitate EPs, to successfully and appropriately evaluate individual
teacher consultations and gather relevant data about the effectiveness of the EPSs for
the LA.
48
As outlined in Section 2.6 practice frameworks have arisen within the professional
practice of EPs as a means of addressing the challenge that EPs face with regard to
clarifying the complex relationship between theory and practice and to answer questions
around the distinctive contribution they bring (Fallon et al., 2010; Frederickson &
Miller, 2008; Gameson et al., 2005; Kelly et al., 2008). Wicks (2013) discussed the
potential of ‘executive’ professional practice frameworks and how they can encourage
effective evaluation which in turn should increase efficiency of EP working (refer to
Section 2.6). However, the lack of research within this area means that it is difficult to
know the extent of their use and, furthermore, their effectiveness as evaluative tools.
Wicks (2013) called for more extensive use of practice frameworks and for further
research to look at the relevance of such frameworks within EP practice models, such as
consultation. This research considered whether the implementation of professional
practice frameworks could provide a way for the EPSs within this study to formalise
high quality evaluation within their consultation model of service delivery (Wimbush &
Watson, 2000).
3.3 Professional practice framework search strategy
The following section outlines the literature search strategy that took place during this
second stage of the literature review. This informed the selection of the most relevant
evaluation frameworks that were then taken forward and included in the systematic
selection process (refer to Section 5.5.1). Initially the search concentrated on finding EP
professional practice models and frameworks. However, following these literature
investigations and from discussions at thesis panel it became clear that further models
outside the field of educational psychology needed further consideration. Therefore, the
literature search for the second stage of the literature review was multifaceted in its
approach. Firstly it looked to identify research around evaluation models/frameworks
within EPSs and then it went on to explore evaluation tools from other professional
disciplines.
3.3.1 Evaluation framework selection process
Relevant literature was identified using a number of different search strategies and
sources of information (as outlined in Section 2.1.2). The research conducted by HEPs
(2010) was a useful source and provided information from EPSs about the evaluation
tools they were currently using. The search strategy was then extended to include other
49
types of service delivery and evaluation models, outside the field of Educational
Psychology. Overall, this yielded some useful and potentially appropriate models:
• A framework for psychological assessment and intervention
• Appreciative Inquiry
• Activity Theory
• The Constructionist Model of Informed Reasoned Action
• Friedman’s Results Based Accountability Model
• Measuring the impact of Casework
• Research and Development in Organisations
• Target Monitoring and Evaluation
• The Interactive Factors Framework
Following this review a systematic selection process was implemented through which
certain practice models were either included in the research or excluded from the
research if they were not deemed to be relevant. It is not appropriate to outline this
selection process within this section of the review as a detailed explanation is provided
within Section 5.5.1 as part of the AR process. A brief summary of the frameworks,
selected for the initial stages of this piece of AR will now be provided.
3.3.2 The Interactive Factors Framework (IFF)
The IFF was developed and integrated into the problem-analysis process by
Frederickson and Cline (2002).
The Problem Analysis Framework (PAF) was devised as arguably the first bespoke
framework to inform and structure EP practice (Monsen, Graham, Frederickson &
Cameron, 1998). It was described as a process whereby information is structured and
then analysed in a way that facilitates understanding. The PAF was developed at
University College London (UCL) and used to guide EPiTs practice and help them to
structure case information (Monsen et al., 1998). As defined by Woolfson et al. (2003)
the PAF had an emphasis on the generation of and testing of hypotheses; problem
solving and the evaluation of evidence and psychological knowledge. However, this
nine step model was perceived as very detailed and lengthy for daily use in EP practice
(Woolfson et al., 2003). PAF was considered to be effective in a small scale study with
ten EPs (Kelly et al., 2008). However, it is important to note that this study only
50
considered this one particular model and as highlighted by Wicks (2013) no
comparisons were made at the time with other professional practice frameworks.
Following a period of research relating to the PAF framework, a number of areas for
development were identified (Monsen et al., 1998). It was seen to require clearer
guidance for the identification of “initial guiding hypotheses” and also for “resultant
problem dimensions” (Frederickson & Cline, 2002, p. 286). This research led to the
development of the IFF, an abridged version of the PAF that could be used by practising
EPs (Frederickson & Cline, 2002). The IFF emphasised the importance of using a
holistic and systemic approach to problems (Wicks, 2013).
The IFF offered a five phase approach, with action points for the EP (Woolfson et al.,
2003). It required EPiTS to actively consider different levels of analysis and their
interactions when formulating initial guiding hypotheses. It aimed to retain a systematic
structure to guide EPs’ thinking, actions and reporting. Key principles within the IFF
were the need for transparency of EP practice, collaborative working between the EPs
and the various problem stakeholders, recognising EPs, parents and carers, teachers and
children all bring their own perspective (Frederickson & Cline, 2002).
3.3.3 A framework for psychological assessment and intervention (DECP)
The Framework for Assessment and Intervention (DECP, 1999) framework was
published by the Division of Educational and Child Psychology (DECP). This intended
to guide assessment by EPs and presented a change in perceptions of children’s
difficulties recognising the interplay of the systems around the child (Wicks, 2013). The
framework recognised that, although psychological assessment was a highly
individualised, complex and creative process, there were some fundamental under-
pinning principles relevant to all EPs (DECP, 2002). It acknowledged that the
assessment of CYP had moved away from the positivist frameworks which dominated
the EP profession for many years and, consequently, that models of assessment should
reflect a more constructionist paradigm. The framework recognised how psychological
assessment involved the use of a variety of tools, techniques and approaches drawing
upon relevant theory and evidence based research. It should also generate an
understanding of what is happening, who is concerned, why there is a problem and what
can be done to make a difference to the situation. It sought to provide information on
51
the processes of learning, the young person’s cognition, social and emotional
development and the impact of the context on those areas (DECP, 2002).
3.3.4 The Constructionist Model of Informed Reasoned Action (COMOIRA)
Gameson and colleagues at the University of Cardiff developed COMOIRA (Gameson
et al., 2003; Gameson & Rhydderch, 2008), as a flexible professional practice model.
COMOIRA stressed the importance of practice being informed by theory and was used
to guide the practice of trainee EPs (TEPs) during their fieldwork placements (Gameson
et al., 2003; Gameson & Rhydderch, 2008).
COMOIRA encourages and summarises the idea of rebuilding. It focuses on processes
which consider and promote change at the organisational, systems, group or individual
level. It can be used at these different levels in a variety of ways. COMOIRA comprises
a ‘core set ‘of psychological principles and has eight key decision points, each with a
set of functions which are supported by a series of reflective and reflexive questions
(Gameson et al., 2003) (see Appendix B). The emphasis on the reflective and reflexive
practice that EPs use to facilitate change is one of the key features of COMOIRA
(Gameson & Rhydderch, 2010). COMOIRA emphasises the importance for
practitioners to ‘adapt’ and ‘personalise’ the model. This will ensure that it is suited to
the service users and fits the EPs’ personal style and paradigm base. However, it is
crucial to maintain the key features and central concepts of social constructionism and
systematic thinking.
3.3.5 Research and Development in Organisations (RADIO)
The RADIO approach was developed by Knight and Timmins (1995) to guide the
school improvement work of the EPiTs on The University of Birmingham training
course (Timmins et al., 2003). RADIO was informed by the work of Schein (1989)
which related to organisational culture and relationships between the researcher and
stakeholders (Timmins et al., 2003). RADIO (Timmins et al., 2006) is a collaborative
AR framework which supports the process of research from planning to delivery.
RADIO has been used by EPiTs to evaluate their consultations. There are different
research phases which include clarifying concerns, researching the concern and creating
change. The AR method allows for many types of approaches, for example, interviews,
questionnaires, surveys, case study.
52
3.3.6 Activity Theory (AT)
AT was first developed by Vygotsky in the late 1920s and it was further refined by
Leontiev, Vygotsky’s student (Greenhouse, 2013). Subsequently, a second and third
generation model was developed by Engestrom (1999) for understanding the
perspectives and networks that occur across interconnected activity systems
(Greenhouse, 2013). Engestrom’s model can be used to understand forms of multi-
agency working. AT provides a framework for understanding human activity within the
sociocultural context it occurs and it is capable of mapping the complex systems and
practices that present within multi-agency teams. (Greenhouse, 2013). Leadbetter
(2008) details how second-generation AT can be used when considering an individual
child’s learning and development. EPs can use the framework to conceptualise their
thinking and the planning of a piece of work and this can be then shared with others e.g.
teachers. This model has been used by trainee psychologists for planning purposes and
as a tool for considering possible ways of working with staff in schools (Leadbetter,
2008). AT can be used as a descriptive tool and to examine and make sense of systems
and situations.
3.3.7 Target Monitoring and Evaluation (TME)
TME was designed to be used within a consultative model of service delivery. It derived
from Goal Attainment Scaling (GAS) which was originally devised to evaluate the
outcomes of mental health interventions (Dunsmuir et al., 2009). TME was developed
to overcome the difficulties identified with GAS but retained many of the key features
e.g. it is constructed for individuals. TME has a five point code to measure the level of
progress. An action plan is drawn up with the consultee. Three goals can be set that
focus and organise the delivery of the plan, ensuring coherence and continuity across
professions and settings. It can be completed with school staff, families and with the
child (Dunsmuir et al., 2009).
TME was trialled as a pilot study by eight assistant EPs and thirteen EPs based in two
LEAs in the South of England. Dunsmuir et al. (2009) considered the value and
usefulness of the TME system as an evaluation tool for EP practice. The findings
showed that TME provided a robust way to gather target oriented feedback on
interventions and valuable data for performance management measures, quality control,
53
and accountability. The individualised nature of the measurement can be used for
casework review and to supplement standardised and qualitative outcomes indicators.
3.3.8 Appreciative Inquiry (AI)
AI was developed in the mid-1980s by David Cooperrider and colleagues for their work
in organisations (Cooperrider,Whitney & Stavros, 2008). It is based on the belief that
groups within organisations often share a set of assumptions which cause them to act
and think in certain ways. Often these assumptions operate at an unconscious level. A
key aspect of AI is the belief that it is important to verify these assumptions every so
often to check they are still current (Hammond, 1998). A four process cycle is used to
evaluate/investigate these assumptions (see Appendix C):-
1) Discover – what works well, searching for the best experiences, successes
2) Dream- what would work well in future/ develop a vision
3) Design – working together to plan/prioritise processes around how to achieve
the vision
4) Destiny/deliver – making the vision real/, implementation of proposed design
3.3.9 Friedman’s Results Based Accountability Model (RBA)
One of the North Wales EPS already uses a model base on Friedman’s RBA framework
(Friedman, 1997). This is based on a four-quadrant framework which aims to answer
questions about the service delivery (see Table 3.1 below).
Table 3.1: EPSs Evaluation Framework
Quantity Quality
Input/Effort How much Service did we deliver?
How well did we deliver the Service?
Output/Effect
How much effect/change did we produce?
What quality of change/effect did we produce?
54
3.3.10 Measuring the impact of Casework (MtiC)
Turner et al. (2010) developed a method of evaluating casework that could be
administered by EPs with a view to capturing the impact that an EP had on outcomes
for pupils. An evaluation tool was developed that combined “actual outcomes, evidence
of impact and EPs’ reflections on their casework” (Turner et al., 2010, p. 327). The
casework evaluation was carried out individually by the EPs and the method used was
still in an evolutionary state during the piece of research (Turner et al., 2010). This
approach was embedded in a constructionist relativist paradigm.
3.4 Summary
The findings from the literature review demonstrated that many EPSs in the UK were
struggling to find a tool to evaluate their impact and this was ‘the big question’ that still
needed to be addressed (HEPS, 2010). The senior managers within the two Welsh EPSs
have been looking for ways to answer key questions about EPS delivery: ‘Is it
effective?’ When is it effective?’ and ‘For whom is it effective?’ and in response to this,
commissioned this piece of research. Furthermore, the literature highlighted that in the
experience of many EPs it can become difficult to incorporate evaluation as an aspect of
routine practice (Dunsmuir et al., 2009). This piece of research aimed to discover a
professional practice framework that could be utilised in an efficient manner and used to
evaluate what EPs do though individual consultations with teachers.
To the author’s knowledge there is no current existing research comparing a range of
professional practice frameworks and considering which are most effective at evaluating
EP consultations with teachers. Therefore, this is the first research of its kind. It is
envisaged that this work will lead to a better understanding around how to evaluate
consultation within EPSs. Hopefully, this may encourage further research through the
wider application of the selected evaluation method/methods. Indeed, neighbouring
EPSs have expressed an interest in this piece of research and wish to be involved in
future communications.
55
3.5 Research Questions
This literature review generated the following research questions:
RQ1: What are the EPs’ views on the most effective models of evaluating
consultation in relation to:
1a) Assessing consultation as a process?
1b) Providing outcomes which demonstrate accountability to employers and
service users?
56
Chapter Four: Methodology
4.1 Chapter Introduction
Within Chapters Two and Three the rationale behind the study was outlined and
research questions were constructed to facilitate this piece of research. The research
questions were designed to enable the researcher to explore frameworks that assisted a
group of EPs within two EPSs, to successfully and appropriately evaluate consultation
and gather relevant data about the effectiveness of the EPSs for the LA. The research
questions were as follows:
RQ1: What are the EPs’ views on the most effective models of evaluating
consultation in relation to:
1a) Assessing consultation as a process?
1b) Providing outcomes which demonstrate accountability to employers and
service users?
4.2 Context of the study
This study took place in two separate EPSs working across two bordering Welsh LAs.
At the time of the study there was one PEP, two SEPs and eleven EPs. The structure of
the EPS is illustrated in Figure 4.1 below. The PEP manages both EPSs and each
service has an allocated SEP. However, at the time of the study one of the SEPs had
been seconded to a different post within the LA for a fixed period of time. The PEP and
SEPs meet regularly with LA officers at a strategic level. The two EPSs have half-
termly meetings at one of the office locations within the LAs. Out of the eleven EPs,
one was unavailable during the time period of this study.
57
Service A Service B
Figure 4.1: Service structure 2014-2015
4.3 Chapter outline
This chapter firstly explores the ontological, epistemological and axiological position
adopted by the researcher. An outline of the research design is then provided, followed
by an exploration of the AR methodology and a discussion around the selected RADIO
model. Following on from this the different phases of this piece of AR are presented
according to the RADIO framework which guided the research process (Timmins et al.,
2003). Consideration is then given to the chosen research data gathering methods and
the rationale for their selection. Information is then provided about the data analysis
methods used at the various stages of the research. A critique of the methodology will
then follow and the chapter ends by exploring the ethical considerations that guided the
researcher throughout the study. An overview of the different phases of the research
process is presented within Table 4.1 below.
PEP
SEP SEP
6 EPs
5 EPs
58
Table 4.1: Phases of the Research Process
Phase of research
RADIO Stages/Activities
Purpose Outcome Research Question
Pre-research
1-3 Exploration and clarification of focus of research with PEP
To develop rationale for research
Literature review
Exploration and development of research questions
Phase 1 Clarifying concerns
3-4 Meeting with PEP and SEP
Refine rationale for research
Identified stakeholders and the process for collaboration, feedback/ discussion.
Exploration and development of research questions
Phase 2 Research methods
5-8 Meeting with PEP and SEP.
Exploration of literature review
Identified need for refining frameworks
Gathering Information to inform Phase 3
Designed proforma for systematic analysis
Information gathering to inform choice of evaluation model/framework
Refined frameworks to take to next phase of research
Meeting with PEP and SEP
Negotiated methods for data gathering
Action research using multi- methods design
Research proposal shared with EPs
Sharing of information in collaborative manner
Participant consent forms designed
Initial focus groups and questionnaires
Information gathering
Quantitative and qualitative Data
59
Data analysis
Selection of model/framework for piloting stage
Collate frameworks for piloting
Piloting of COMOIRA and Appreciative Inquiry followed by feedback questionnaire
Information gathering
Quantitative and qualitative Data
Phase 3 Organisational change mode
9-10 Final focus groups
Information gathering
Qualitative data
1a 1b
Data analysis
Selection of evaluation model/framework to be implemented by EP services
Meeting with PEP and SEP
4.4 Philosophical considerations
Before consideration is given to the AR methodology it is important to firstly explore
the ontological, epistemological and axiological views shaping this piece of research.
These will now be outlined in order to assist the reader in understanding the researcher’s
stance.
4.4.1 Ontology
Ontology is the philosophy of being (McNiff & Whitehead, 2011). It is concerned with
how we view the nature of reality (Marrow, 2007) and considers how we see and
experience the world (Allison & Pomeroy, 2000). For this piece of research several
paradigms were considered by the researcher. These were outlined by Guba and Lincoln
(1994) and include: positivism, post positivism, critical theory and social
constructivism. These paradigms are presented in Table 4.2 below:
60
Table 4.2: Four Philosophical Research Paradigms
Positivism Post-Positivism Critical Theory
Constructivism
Epistemology (Nature of knowledge)
Objective view of the world. Verified hypotheses are considered as facts. Findings are true.
Modified objectivist. Pure objectivity is impossible.
Subjective view of the world. Knowledge is reflected through the perspective of the researcher.
Subjectivist, researcher and participants construct knowledge together.
Ontology (Nature of Reality)
There is a reality. Aims to uncover truth/facts using scientific methods.
There is a reality which is predictable only in terms of probability.
Virtual reality shaped by social, political, cultural, economic, ethnic and gender factors.
Relativism-reality is multiple and is constructed differently by people. Dependant on individual.
Methodology Experimental, verifying hypotheses, quantitative methods.
Modified experimental. May include quantitative and qualitative methods.
Involves dialogue with participants as source of information.
Qualitative, research through dialogue.
Research methods
Questionnaires, Pre-test, post-test intervention and control group.
Pre-test, post-test. Questionnaires, Interviews, Triangulation.
Ethnographic, interviews, Biographies.
Focus groups, group interviews.
*Adapted from the work of Allison and Pomeroy (2000); Gephart (1999); Guba and
Lincoln (1994)
The researcher adopted a ‘critical realist’ stance which sits within the post-positivism
paradigm (see Section 4.4.4) but before this is explored the positions of positivism and
relativism are considered.
61
4.4.2 Positivism
Traditional research assumed a positivist view of the world. Positivism asserts that there
is a reality out there to be discovered (Allison & Pomeroy, 2000). Positivism takes an
objective view of the world and seeks to establish this through an empiricist; hypothesis
testing model where knowledge can be elicited through direct observation (Gephart,
1999; Robson, 2002). Science is seen as a way of discovering the truth by gathering
data through quantitative methods; a means by which we can begin to understand the
world so that we are able to predict and control it. The positivist view assumes that only
what is observable can provide us with the clearest knowledge and that which cannot be
observed is of less worth (Cohen, Manion & Morrison, 2000).
Positivism was a dominant force within social research until there was a shift away
from it in the middle part of the 20th century. It became criticised for its underlying
philosophy and application (Robson, 2002). Anti-positivists rejected the belief that there
is only one truth (Cohen et al., 2000) and concerns were raised about the limits of the
methods associated with positivism. Critics argued that the use of quantifiable measures
created a significant challenge when applying them to the study of social contexts. It led
to the omission of important data, stripping contexts from meanings and excluding the
perspectives of the participants (Cohen et al., 2000; Gephart, 1999; Sarantakos, 1998).
4.4.3 Relativism
Relativists emphasise that the social world can only be understood if we consider the
interpretations of individuals who are part of it and claim that it is only possible to
determine what is probable (Allison & Pomeroy, 2000; Cohen et al., 2000). Those who
hold a relativist position would reject the positivistic belief that it is possible to govern
human behaviour by universal laws. Relativism values qualitative methods (Robson,
2002). It assumes that knowledge is subjective and there is no objective truth.
Relativists assert that reality is multi-faceted and situations are complex, constantly
changing and are affected by their context. There are in existence multiple
interpretations and perspectives based around the same events (Cohen et al., 2000).
Robson (2002) discusses how, in its most extreme form, relativism asserts there is no
external reality that can exist outside that of human consciousness. Relativism has been
criticised for its inability to substantiate the information it obtains through the
application of approaches that cannot be verified (Robson, 2002).
62
4.4.4 Critical realism
As outlined in Sections 4.4.2 and 4.4.3 positivist and relativist paradigms use two
different lenses to understand phenomena. Both paradigms have received criticisms for
presenting incomplete accounts of social behaviour (Cohen et al., 2000). For this piece
of real world action research it was decided that neither a purely positivist or relativist
paradigm would provide an appropriate ontological or epistemological position for the
researcher to place themselves. Instead the researcher adopted the post-positivist
paradigm of critical realism. A critical realist stance provides a third way between
positivism and relativism which allows for a more balanced approach (Robson, 2002).
Critical realism was one of the most prominent of the post–positivist paradigms and it
was largely established by the writings of Bhasker in the 1970s (Zachariadis, Scott &
Barrett, 2010). The critical realist stance suggests that no scientific facts are beyond
dispute and that knowledge is a social product (Jefferies, 2011). Within this paradigm,
the researcher’s and the participant’s views are valued and it is assumed that the
researcher and the researched cannot be separated (Braun & Clarke, 2006). The
foundations of critical realism lie within a realist ontology (Bhasker, 1978; Corson,
1991). The theory being that a reality exists that is multi-layered, independent of mind
and cannot be reduced simply to how we experience it (Corson, 1991; Easton, 2010;
Patomaki & Wight, 2000). Critical realists assert that we construe rather than construct
the real world (Easton, 2010). They would argue that within the:
Real world there are entities, such as organisations, which have powers to act
and are liable to be acted upon by others. These entities can also have their own
internal structures, such as departments and individuals which in turn, have
their own powers (Easton 2010, p.128).
As such, critical realism recognises the realities of everyday practice (Clegg, 2005). It
acknowledges that subjective data exist, considers the ways individuals make meaning
of their experiences and how the broader social context impacts on those meanings.
Importantly, critical realism enables us to accept the fallibility of our knowledge and
recognise the possibility of getting things wrong (Braun & Clarke, 2006; Zachariadis et
al., 2010). The researcher believed that a critical realist paradigm supported research
63
within the real world context of this study. This paradigm choice guided the selection of
an AR methodology.
4.4.5. Epistemology
Whilst ontology considers the nature of reality, epistemology addresses how that reality
is known (Morrow, 2007). Epistemology is the philosophy of knowledge or of how we
come to know. It questions how knowledge is created, acquired and communicated to
other human beings (Allison & Pomeroy, 2000; Cohen et al., 2000; McNiff &
Whitehead, 2011). Carter and Little (2007) consider how epistemology can be seen to
be a way of justifying knowledge (see Figure 4.2 below). The diagram demonstrates the
influence that epistemology exercises upon the research methodology. The
epistemological stance is central to the research process and can be seen to affect the
data gathering methods which provide the basis of the knowledge produced.
Figure 4.2: The Simple Relationship between Epistemology, Methodology, and
Method (Carter and Little, 2007)
As critical realism asserts that the two approaches of positivism and relativism can co-
exist this allows the researcher to carry out scientifically grounded research whilst
incorporating the views of the participants and giving consideration to the social context
(Lightfoot, 2013; Robson, 2002).
64
4.4.6 Axiological position
Axiology has to do with the role of values and the influence these have upon the
research process (Marrow, 2007; McNiff & Whitehead, 2006). It is important for the
researcher to acknowledge their own values and how these may influence the way in
which the research is conducted, interpreted and subsequently reported.
AR is often considered as value laden as it assumes that the researcher, in relation to
everything else within the research, influences, and is influenced by others. AR
researchers bring with them their own values. They undertake enquiries with others and
often view themselves in relation to others, in terms of their practices, ideas and their
environment. AR involves incorporating methods that nurture respectful relationships. It
demonstrates a commitment towards ‘I/we’ forms of enquiry (McNiff & Whitehead,
2011).
The researcher values the principles of both collaboration and participation and wanted
to transfer these beliefs to the research process. This intention guided the selection of an
AR methodology. One of the central aspects of the research design was its participatory
nature which placed the collaboration of staff members at the centre of the process. It
was important to recognise the researcher’s interactive part in this study and consider
the implications of an active researcher role. The researcher needed to be reflective
about the decisions and actions taken during the research process and maintain an
awareness of how her values could influence this study. The researcher did believe that
this piece of AR research would create an opportunity to bring about some change and
improve practice within one of the systems within the EPSs. Due to extensive reading
and research around the EP practice frameworks the researcher had undertaken in the
early stages of the AR process it was important to recognise the potential for bias. The
researcher had thoughts around her preferred frameworks and the models that would be
best fit for the EPS to evaluate consultation and it was important to reflect upon this as
the research progressed.
4.5 Research design
Critical realism is unique in that it does not commit to a single type of research design
but rather endorses an extensive variety of research methods which are chosen
according to the type of the project and the aims of the study (Zachariadis et al., 2010).
In keeping with the critical realist stance of the researcher, this piece of AR incorporates
65
a multi- methods approach. It focuses on collecting, analysing and mixing both
quantitative and qualitative data with the understanding that combining both approaches
facilitates a clearer understanding of the research. This piece of research uses both focus
groups and questionnaires to collect data. Through the implementation of a multi-
methods approach the researcher aimed to bring greater rigour to the research process
(Creswell & Clark, 2006). The triangulation of methods can be seen to provide
contextually richer data and, thus, enables the researcher to be more confident about the
findings of the research process (Robson, 2002).
The aim of this piece of research was to find a professional practice framework that
offered an effective tool for evaluating individual EP consultation within the
commissioning LAs. With this in mind, a predominantly qualitative research paradigm
was selected as it was considered the most applicable to the aims of the study. As
outlined by Robson (2002) the purpose of evaluation is to assess the effects and
effectiveness of something. This study sought not only to evaluate the effectiveness and
relevance of the evaluation frameworks that were piloted but it also looked to explore
ways in which these models could be adapted. This was a summative piece of research
but aimed to have a formative effect on future service developments.
This study involved the participation of other EPs. Therefore, a fundamental aspect of
the research design was that it could be carried out within a collaborative relationship
with those staff members central to the process. An AR methodology was selected as it
facilitated the participatory requirement and evaluatory nature of the study. Kidd and
Kral (2005) discussed how participatory action research is a dynamic process that
develops from the unique needs, challenges and learning experiences specific to a given
group. The implementation of this approach has also been shown to promote change at
an organisational and individual level (Robson, 2002; Simm & Ingram, 2008). It has
been recognised as a model that promotes professional development, can contribute to
evidence based practice and is effective in changing practice and embedding change
(Levin & Rock., 2003; Simm & Ingram, 2008; Torrance, 2004). As such, this
methodology realised the aims of the PEP and the researcher in finding a framework for
the evaluation of consultation that could be used across two EPSs. It enabled the
combination of local knowledge and research knowledge. The PEP wanted to involve
the EPs in this piece of research and to use the data generated to inform future planning
66
(Brydon-Miller, Greenwood & Maguire, 2003). It was hoped that through this piece of
AR the EPs would be empowered to understand and manage their own questions and
recognise that they could go on to change issues effectively with a sense of autonomy
(Gameson et al., 2003).
4.6 Action Research design
AR originated in the work of John Collier in the 1930s (McNiff & Whitehead, 2003), it
was subsequently developed by the work of Kurt Lewin in the 1940s (McNiff &
Whitehead, 2003) and became popular in education in the 1950s (McNiff & Whitehead,
2006). AR was developed out of critical theory and introduced an action element.
Critical theory aimed to understand a particular situation before considering how to
change it. AR considered the action that was required in order to introduce change. AR
is a flexible, contextually responsive methodology which can be integrated with other
methods. This approach can bring rigour and a voice to the research process (Cohen et
al., 2000). The purpose of AR is to influence or change some aspect of whatever is the
focus of the research. Robson claims that improvement is central, “first, the
improvement of a practice of some kind; second, the improvement of the understanding
of a practice by its practitioners; and third, the improvement of the situation in which
the practice takes place” (Robson, 2002, p. 215).
Unlike traditional research where the researcher makes claims about what certain and
true, the AR process is also the methodology. This methodology provides the researcher
with an interactive inquiry process where the researcher is able to interact with the
research process. It provides the researcher with the opportunity to generate a living
theory of practice and seeks to bridge the gap between theory and practice (Cohen,
Manion & Morrison, 2007; McNiff & Whitehead, 2011; Reason & Bradbury, 2001).
AR researchers generally assume that there is no one answer, that knowledge is created
and any answer is tentative. AR researchers do not look for generalizable outcomes but
seek to produce personal theories and invite others to learn with them. (McNiff &
Whitehead, 2011).
AR is a cyclical process which involves planning a change, acting, observing what
happens following the change, reflecting and then planning further action and repeating
the cycle (Robson, 2002). McNiff and Whitehead (2011) discuss how one cycle of AR
67
leads on to another. Figure 4.3 below demonstrates an action-reflection cycle which
involves the process of observing, reflecting, acting, evaluating, modifying and then
moving in a new direction.
Figure 4.3: Action-reflection cycle (adapted from McNiff & Whitehead, 2011. p. 9)
4.7 Consideration of other research designs
Prior to the selection of the AR design other data analysis methods were considered by
the researcher. Analytic methods such as discourse analysis (Potter & Wetherell, 1987)
and grounded theory (Glaser, 1992) were studied and discounted for the reasons
outlined below.
Discourse analysis was considered as this researcher wanted to use focus groups within
the research process. Discourse analysis lends itself to the analysis of focus group data
and is a popular method used by psychologists when analysing discourse (Willig, 2003).
Different versions of discourse analysis exist, for example, discursive psychology and
Foucauldian discourse analysis but these methods share a common concern around the
role of language in the construction of social reality (Willig, 2003). Discourse analysis
differs from other qualitative methods in that it attempts to understand the way in which
reality is produced. It is considered to be a methodology which has a social
68
constructivist epistemology and believes that social reality is something we create
through social interactions (Gerrig, 2004). This method was discounted as it and its
epistemological positon does not fit with the critical realist stance of this piece of
research.
A further method considered was that of grounded theory. Grounded theory can be
viewed as a methodology and positions itself within a realist epistemology. It is an
inductive approach that looks to generate a theory which is grounded in the data. There
is a belief within Grounded Theory that knowledge is increased by generating new
theories rather than analysing data within existing ones (Biggerstaff, 2012; Cresswell,
1998). There are different versions of grounded theory. Glaser’s version and Strauss’s
version incorporate different processes of theory generation through their coding
systems (Heath & Cowley, 2004). This approach was discounted by the researcher as
the constant comparison analysis that it requires was considered too time consuming,
given the multiple phases of the research. Furthermore, it is recommended that when
using Grounded Theory, the literature review is conducted once the research has
commenced (Ng & Hase, 2008). This created difficulties for the researcher as the main
body of the review of literature was done in the early stages of the RADIO cycle in
order to inform the next part of the process. This method was discounted as the
researcher was not looking for one single theory, instead recognising the need to
describe the different interpretations of the participants.
4.8 Models of Action Research
The researcher chose the RADIO model to structure the design of this study. Before
exploring the reasons behind the selection of this approach the researcher will briefly
consider some of the other action research models that exist.
Illuminative evaluation, pioneered by Stephen Parlett (Parlett & Deardon, 1977), takes
into account the wider contexts within which educational programmes function and it is
primarily concerned with the idea of description and interpretation rather than with
measurement and prediction (Parlett & Hamilton, 1972). It aims to discover what it is
like to participate and looks to address a complex range of questions. Two main
concepts underpin this approach, the ‘instructional system’ and the ‘learning milieu’.
69
There is no set format provided for illuminative evaluation which the researcher was
looking for in order to conduct this piece of research (Kelly et al., 2008)
Living theory is an approach which aims to generate theories and understanding in AR
and is distinguished by the asking of questions by the individual focused on producing
explanations for their educational influences. The researcher is seen to be a creator of
knowledge via the process of pursuing ways to improve practice (McNiff & Whitehead,
2006; Whitehead, 2009).
4.9 Research and Development in Organisations
Although living theory and illuminative evaluation were considered, RADIO was used
by the researcher in order to provide a greater structure for the work. The RADIO model
(Timmins et al., 2003) was selected as it has been used in previous EP studies (e.g.
Ashton 2009; Lightfoot, 2013) and has been found to offer a useful framework for
structuring AR within the context of EP practice. RADIO is a collaborative AR
framework which was developed to help EPiTs manage their school improvement work
(Timmins et al., 2003). RADIO is a 12 step model and it takes into account the
complexity of factors in the workplace. It allows flexibility in the methods of data
collection and analysis (Timmins et al., 2003).An overview of the RADIO model is
presented in Table 4.3 below.
Table 4.3: RADIO model
RADIO PHASES RADIO STAGES Clarifying concerns (Phase One) Research methods mode (Phase Two) Organisational change mode (Phase Three)
1. Awareness of need 2. Invitation to act 3. Clarifying organisational and cultural issues 4. Identifying stakeholders 5. Agreeing the focus of concern 6. Negotiating the framework for data gathering 7. Gathering information 8. Processing information with stakeholders 9. Agreeing areas for future action 10. Action planning 11. Implementing /action
70
12. Evaluating action
The researcher considered RADIO to be contextually appropriate. It provided a robust
structure which enabled the researcher to maintain a partnership with stakeholders and it
facilitated the focus of the research. A further reason for its selection was that RADIO
has been shown to provide a useful process and planning tool which encourages
collaborative research between the researcher and participants in order to create positive
change (Ashton, 2009; Lightfoot, 2013; Timmins et al., 2003; Timmins et al., 2006).
4.10 Research Phases
Sections 4.11 – 4.23 will be presented according to the RADIO design that provided the
framework for this piece of research. The researcher has chosen to present it in this way
in order to guide the reader through the research process as it proceeded. Each phase of
the research will discussed chronologically and during each phase the research methods,
data collection tools and data analysis methods that were involved in that phase of the
research will be explored.
This piece of research incorporated the first 10 stages of the RADIO model, although
the whole RADIO structure was considered. The RADIO framework used to guide the
research as presented in Table 4.1. This provides an overview outlining how all of the
phases in this piece of research fit into the stages of the RADIO model.
4.11 Pre- research Phase and research Phase One
The Pre -Research Phase and Phase One of the study incorporated Stages 1-4 of the
RADIO process. The stages of RADIO covered across these phases of the research are
summarised In Table 4.4 below. The elements of each stage of the process are then
discussed in turn.
Table 4.4: Pre-research Phase and Phase One
RADIO STAGE RADIO ACTIVITY
Stage 1: Awareness of Need
Researcher’s discussions with PEP identified the need for research into the evaluation of EP consultation
Section 4.11.1
Stage 2: Invitation to Act Section 4.11.2
71
PEP requests research
Stage 3: Clarifying organisational and cultural issues
Initial literature review confirming the need for further research
Section 4.11.3.
Stage 4: Identifying stakeholders
Stakeholders identified
Section 4.11.3.
4.11.1 Stage 1-awareness of need:
The need to evaluate consultations was brought to the researcher’s attention through
discussions with the PEP. The PEP wanted to establish an evaluation tool which would
collect relevant information on the impact of our service delivery. As a doctoral student,
the researcher was requested to find a method that enabled the EPs within the two
services, to successfully and appropriately evaluate consultation and gather relevant
data about the effectiveness of the EPSs for the LA. This prompted Stage 2 of the
RADIO process.
4.11.2 Stage 2 –invitation to act:
The PEP, at the request of the commissioning LA, requested this piece of work,
focussed on the evaluation of consultation within the two services, from the researcher.
The research was central to her continuing professional doctorate programmes, funded
by the LA.
4.11.3 Stage 3 and 4- clarifying organisational and cultural issues and identifying
stakeholders:
In order to better understand and to gain an insight into existing frameworks a ‘rich
picture’ was formulated by the researcher prior to the research being undertaken
(Checkland & Scholes, 1990). A review of the literature was undertaken which
confirmed the need for further research in this area. The PEP, SEP, EPs and
commissioning LA were identified as the primary stakeholders. The PEP and one SEP
were identified as the key stakeholders with whom the research information would be
processed as they were motivated to initiate change within the EPSs. As a member of
one of the EP teams, the researcher had an understanding of the context in which the
research was conducted. The processes for collaboration with the stakeholders were
identified. It was agreed that regular meetings would be held with the PEP and SEP.
72
The joint team meetings, held half-termly between the two services, would have an
allocated time set aside to focus on the particular stage of research in progress at the
time.
Following the contracting of the research, Phase Two of the RADIO process
commenced. This was the primary data collection phase of the research and will be
described in the sections below.
4.12 Research Phase Two
Phase Two of the research incorporated Stages 5-8 of the RADIO process. The stages
are summarised within Table 4.5 below. These will be discussed in the order that they
were conducted within the research.
Table 4.5: Research Phase Two
RADIO STAGE RADIO ACTIVITY
Stage 5: Agreeing focus of concern
The PEP and researcher agreed the focus of the research and the main aims of the research.
Section 4.12.1
Stage 6: Negotiating framework for information gathering
Systematic Literature Review An appropriate methodology and research design was selected: Action Research Design RADIO
Section 5.5.1 Section 4.12.2 Section 4.6 Section 4.9
Stage 7: Gathering information using agreed methods
Information was gathered using the agreed methods Initial focus groups Questionnaires Research Diary Content Analysis
Section 4.12.3 Section 4.17 Section 4.16, 4.20 Section 4.18 Section 4.21
Research findings were shared with the Section 4.12.4
73
Stage 8: Processing information with stakeholders
stakeholders following data analysis
4.12.1 Stage 5- agreeing the focus of concern:
The PEP and researcher agreed the focus of the research and designed the research
questions (refer to Section 4.1). The main aims of the research were to find a way to
evaluate consultation that would enable EPs to assess consultation as a process and
provide outcomes which would demonstrate accountability to employers and service
users.
4.12.2 Stage 6-negotiating framework for information gathering:
An appropriate methodology and research design was selected to address the research
aims (Atkinson, 2011). The process for this is outlined in Sections 4.5 and 4.6. The PEP
and one SEP met with the researcher and a decision was made to use an AR
methodology which collected both quantitative and qualitative data. This proposal was
then presented to the participant EPs during a joint meeting. The rationale for selecting
AR and RADIO as appropriate for this participatory research is previously described in
Sections 4.6 and 4.9.
In relation to the systematic literature review, through discussions with the PEP and
SEP it was agreed that a pro forma would be designed by the researcher in order to
carry out a systematic analysis of relevant professional practice models. The purpose
behind this was to select models which could then go on to be piloted by the EPs across
the two EPSs. It was agreed that this would then bring a specific focus to the research
and guide the methodology and research design process in Stage 6. The framework for
the selection process is discussed in detail in Chapter 5 as part of the cycle of research
(see Section 5.5.1). This process led to the selection of five professional practice models
which were taken forward in to the next stage of the research process.
4.12.3 Stage 7-gathering information:
Information was gathered using the data collection methods agreed with the
stakeholders. This involved conducting one initial focus groups with each EPS, each
individual EP completing an audit of previous knowledge questionnaire and a
preference questionnaire. These data were then analysed to inform the selection of the
74
frameworks to be piloted. The piloting of the two selected frameworks COMOIRA and
AI was undertaken. A planning meeting was held with all participants prior to the
trialling of each individual model. This facilitated discussions around the
implementation of the model and additional questions that could be added to the
frameworks to support the evaluation process (refer to Appendix I).These discussions
were recorded within the researcher diary and prompted the researcher to share some
additional resources with the EPs (refer to Appendix O). Each model was then piloted,
in turn, for a six week period. The EPs were asked to use the models to evaluate three
individual teacher consultations. The EPs conducted the individual consultation and
then used the frameworks as an evaluative tool to collate their personal views.
Following the piloting of each model feedback questionnaires were completed. These
data were then analysed to inform Stage 8 of RADIO.
4.12.4 Stage 8-processing information with stakeholders:
As the research progressed the PEP and SEP had regular meetings with the researcher in
order to share the research findings following the data analysis process. Members of the
EPS were consulted regularly during joint team meetings and on an impromptu basis to
ensure that the model was developed collaboratively and reflected the needs of the
Service (Timmins et al., 2003). Different phases of the model were re-visited during the
research to encourage a flexible way of working.
4.13 Research Phase Three
Phase Three of the research incorporated Stages 9 and 10 of the RADIO process (as
outlined in Table 4.6 below).
Table 4.6: Research Phase Three
RADIO STAGE RADIO ACTIVITY
Stage 9: Agreeing areas for future action
Final focus groups were conducted and data analysed and feedback to stakeholders Final focus groups Thematic Analysis
Section 4.13.1 Section 4.22.2 Section 4.22.4
75
Section 10: Action planning
Decisions were due to be made around the professional framework that would be implemented in the future to evaluate EP Consultations
Section 4.13.1
4.13.1 Stage 9-agreeing areas for future action and Stage 10- action planning:
During Phase Three of the research a decision was made not to conduct the semi-
structured interviews that had been originally planned. Within the initial research design
all the EPs were going to take part in a semi-structured interview which would follow
the professional framework piloting process. However, during a pre-planned meeting in
Phase 3 of the RADIO model, the EPs had a collaborative discussion around the next
stage of the research process. The general feedback from the EPs indicated that, rather
than participating in an interview, it was much easier for the EPs just to attend one final
focus group. From a logistical point of view the two teams felt that this would be easier
to convene. Several of the EPs shared that since the piloting of the first framework
(September-October 2014) several weeks had elapsed and, as such, it would be difficult
to respond in detail to a set of interview questions about that specific framework. They
believed that having a group discussion alongside fellow colleagues would aid the final
selection of an evaluative framework. The EPs considered the focus groups to be more
advantageous than the individual semi-structured interviews as they would enable them
to come together as a group, discuss the models and recall important information. They
believed that the focus groups alone would generate sufficiently rich data and that the
interviews were, therefore, unnecessary and unfeasible within their heavy workload.
In collaboration with the PEP and SEP, the EPs agreed that it would be sufficient to
collate the post-pilot feedback questionnaire responses as these had documented their
views adequately and it would be possible to qualify this data further within the final
two focus groups. As this was a piece of AR and involved a collaborative process a
decision was made, following this direct feedback, not to conduct the interviews. The
researcher recognised that one to one interviews would have provided an opportunity to
explore the individual participants’ experiences. These would have collected rich data as
it would be likely that more detailed responses would be gathered.
76
Following the data analysis of the post-pilot feedback questionnaires the two final focus
groups were conducted with the two EPSs. Due to the time constraints of this research,
Stages 10, 11 and 12 of the RADIO model were not completed by the point at which
this thesis was submitted. During Stage 9 the findings of the focus groups were shared
with the stakeholders. The next stages of the RADIO model are due to be implemented
following on from this piece of research but will not be formally documented by the
researcher.
Having provided an overview of all phases of the RADIO process, the following
sections will look specifically at the processes of data collection and data analysis
central to the research, which occurred at Phase 2 and Phase 3 of the research.
4.14 Data gathering methods
Prior to discussing the data collection methods implemented during Phase Two of the
research, the researcher will firstly outline the reasoning behind the decision to collect
both qualitative and quantitative data.
The critical realist positon considers that measurement is fallible and is that reality
cannot be known with complete certainty. Therefore, an emphasis is placed on the
importance of multiple measures and observation (Madill, Jordan & Shirley, 2000;
Trochim, 2006). In line with the critical realist stance of the researcher, this study
adopts an AR design which collected both quantitative and qualitative data (Banister,
Burman, Parker, Taylor & Tindall, 1994). A range of data gathering methods was
adopted across the different phases of the research and these were evaluated and used to
inform the next phase of the model. Triangulation was used as a way of bringing a
richer picture to the research process as collecting information by studying it from more
than one standpoint; and by making use of both quantitative and qualitative data is a
powerful way of demonstrating validity and enhancing the quality and credibility of the
research (Silverman, 2001; Yin, 2009). An over reliance on one method may bias or
distort the research (Cohen et al., 2000). Within this piece of research the use of focus
groups are triangulated with the more traditional form of questionnaires.
77
4.15 Data collection
Within the research, data was collected according to the methods presented within Table
4.7 below. This outlines the data gathering methods and links them to the RQ:
What are the EPs’ views on the most effective models of evaluating consultation in relation to: 1a Assessing consultation as a process? 1b Providing outcomes which demonstrate accountability to employers and service users?
Table 4.7: Data Gathering Methods
RADIO Phase Data Gathering Method Research Question
Phase Two Phase Three
• Systematic analysis of evaluation frameworks
• Initial focus groups • Pre-pilot questionnaires –
audit of previous knowledge and preference questionnaires with all EPs
• Post-pilot questionnaires-feedback questionnaires on preferred models (COMOIRA and AI)
• Research diary
• Final focus groups Initial focus gr
Data gathered to inform Phase 3
1a 1b
Data collected during the later stages of phase 2 (Stages 7-8) were used to inform Phase
Three of the research which addressed the research questions. An overview of the data
gathering methods used within Phase Two is shown in Table 4.8 below. A research
diary was used across all stages of the research to collect pertinent data (see Section
4.18).
78
Table 4.8: Phase Two Data Gathering
Data Collection Data Gathering Audit of previous knowledge questionnaire
All EPs completed an audit questionnaire prior to the initial focus group
Initial focus groups
Two initial focus groups were conducted with the separate EPSs
Preference questionnaire
Following the initial focus group all EPs completed a questionnaire indicating their preferred model/models
Feedback questionnaire (COMOIRA and AI)
Following the piloting of COMOIRA and AI all EPs completed a feedback questionnaire
Each data collection method will now be discussed. Rather than presenting these in the
chronological order as outlined by Table 4.8 they are discussed as individual data
gathering methods.
4.16 Self –report questionnaire
Questionnaires are considered to be an efficient way of getting large amounts of data
over a short time period Robson (2002). A questionnaire was selected as a data
gathering tool alongside the focus groups. Four questionnaires were designed for the
study. Two were incorporated into the pre-pilot phase of the research and two into the
post-pilot phase of the research.
4.16.1 Pre-pilot questionnaires
Two questionnaires were used to collect data prior to the piloting stage of the research
(see Table 4.8). Both questionnaires were designed alongside the PEP and one SEP and
pre-tested informally with a small group of EPs. The questionnaires were then modified
in light of the given feedback. The questionnaires were designed with the purpose of
driving forward the next phase of the AR and looked to link the research questions with
those of the questionnaire (Robson, 2002).
79
The audit of previous knowledge questionnaire (see Appendix D) collected data from
the EPs around how familiar they were with five professional practice models and
whether they implemented any of these within their professional practice (see Table
5.4). This questionnaire was completed by the EPs prior to the initial two focus groups.
The preference questionnaire was completed by the EPs following the first two initial
focus groups (see Appendix E). The EPs ranked their preferred professional practice
models and indicated how likely they would be to use a particular framework within
their practice (see Table 5.5).
Both questionnaires used closed questions to generate quantifiable data which could
then be analysed and considered alongside the information from the focus groups. The
questionnaires incorporated a semantic differential rating scale as a way of building a
range of responses whilst generating numbers. These are useful as they combine the
opportunity for a flexible response whilst generating quantitative measures that allows
analysis to be conducted and correlations to be made (Cohen et al., 2000, Yin, 2011).
The preference questionnaire also included a rank order question which allowed for
degrees of response and reflected the preference and priority of the participants (Cohen
et al., 2000). This assessed the level of consensus around the different models as each
focus group member were asked to indicate their views as to whether a particular model
should be selected. This information, alongside the data collected from the initial focus
groups led to the selection of COMOIRA and AI as the preferred models to be piloted
in the next stage of the research process.
4.16.2 Post-pilot questionnaires
All the EPs across both services piloted two frameworks over the course of a full school
term. Within the first half-term period (approximately seven weeks) COMOIRA was
used to evaluate three individual teacher consultations. Within the second half-term
period (approximately seven weeks) AI was used to evaluate three individual teacher
consultations (see Table 4.8). Following the piloting period each EP completed a
feedback questionnaire about the framework (Appendix F). The post-pilot feedback
questionnaires were co-designed with the PEP and one SEP and piloted with several
EPs. This allowed the researcher to check the clarity of the questions, eliminate any
ambiguities, gather feedback on the response categories and rating scales and gain
feedback on the validity of the questionnaire items (Cohen et al., 2000).
80
The same feedback questionnaire design was used to collect data following the post-
piloting period for both professional practice frameworks. The purpose of the
questionnaire was to gather data on such factors as the ease of use of the model and
whether the model was able to evaluate consultation effectively. The questionnaire
incorporated a set of scaled questions but the researcher noted the limits of scaled
questions. In an attempt to mitigate against this, open ended questions were also
included. Cohen et al. (2000) outline how rating scales can be limited in their usefulness
by the need for a participant to select from a given choice. The use of open-ended
questions enables respondents to reply using their own comments and opinions.
However, open-ended questions were kept to a minimum for analyses purposes
(Robson, 2002).
4.17 Initial focus groups
Due to the EPSs being separately located, a decision was made to run two groups rather
than one. The time allowed for the groups was 60 minutes. The first group consisted of
one PEP, one SEP and four EPs who constituted one of the EPSs and the second group
consisted of the same PEP and six EPs from the neighbouring authority. The initial
focus groups looked to gather the EPs’ views on the perceived strengths, limitations and
applicability to EP practice of the professional practice frameworks discussed (see
Section 5.6.4.1- 5.6.4.5).
Focus groups were selected as this method allows for the interviewing of several
participants at once (Hendricks, 2008). It was logistically much more feasible to arrange
for participants to meet in two groups in terms of time and expense. Focus groups are
advantageous in that the responses of one person can help others recall important
information they wish to share. The members, therefore, respond not only to the
facilitator but to the others in the group which can generate much richer data
(Hendricks, 2008). It is important for the researcher to consider the group context,
particularly within established groups where it is important to recognise group
dynamics. The researcher must aim to facilitate the group so that all members are giving
the opportunity to participate and the most powerful voices are managed (Cohen et al.,
2000; Coolican, 2009). Therefore, it was important for the researcher to retain
awareness of the potential hierarchical structure with the attendance of the PEP and
81
SEP. This is a particularly challenging role for the facilitator but without this
generalisation becomes problematic (Cohen et al., 2000).
Prior to the initial two focus groups being undertaken, an information briefing sheet was
sent to all participants, this included brief summary of each model/framework that had
been selected following the systematic analysis. This provided each EP with the
opportunity to read around the professional practice models prior to the focus group
activity. The sheet also explained the aims of the focus group and how it would provide
an opportunity to discuss the professional practice models in turn so a decision could
then be made about the model/models to be taken forward into the piloting phase of the
research (see Appendix G).
Before the initial focus groups began the questions were written on a piece of flip chart
paper (see Appendix H) .At the request of the participants, each group was given an
additional 30 minutes reading time before the focus group started. During this time they
were encouraged to reflect, re-read the briefing paper and make notes if they so wished.
The focus groups began with an introduction and an explanation of the aims and
objectives of the focus group. General ground rules and levels of confidentiality for the
group were established. Within my role as the facilitator I had to ensure that I guided
the discussion in order to keep it focused and not allow it to deviate from the topic (Yin,
2011). The focus group was audio recorded and minimal notes were taken of the key
themes and written onto flip chart paper. This was as a way of ensuring that the
researcher was able to actively listen to all the participant views, facilitate the
interactions between the participants, contribute appropriate questions and deal with any
issues that arose. A time keeper was nominated to ensure that each professional model
was allocated the same discussion time.
The focus groups were conducted based around a set of open ended questions (Robson,
2002) which considered the strengths and limitations of each model and general
thoughts around the applicability of the model to EP practice. Each professional practice
framework was discussed in turn, the key points of the discussion were recorded on the
flip chart and clarification sought to ensure that it reflected what the participants had
82
said accurately. Following each framework discussion the initial key themes that had
emerged were verbally summarised to the group.
4.18 Research diary
Cutting across all phases of the research the researcher kept a self-completion diary.
This provides a way of recording impromptu data and participant views that may be
shared outside the focus groups and questionnaires (Corti, 1993). It allows the
researcher to reflect upon the research process (Nadin & Cassell, 2006). A diary was
kept which noted salient points throughout the study. This was used to record the
discussions with stakeholders at each phase of the RADIO approach and during the
regular joint ‘research update’ meetings held across the two EPSs (see Appendix I). It
was used to capture views of the EPs when discussions were held around reflections on
methodologies and future planning e.g. whether it was necessary to include the semi-
structured interview process (Altrichter & Holly, 2005).
When using the diary the researcher was mindful of the fact that diaries can be prone to
errors which may arise from incomplete recording of information, underreporting and
insufficient recollection of the discussion (Corti, 1993).
4.19 Phase Two data analysis
The data generated from this stage of the research were analysed using a variety of
methods. The data were aggregated at each stage of the AR process in order to inform
the subsequent phase of the AR cycle. Within this section the data analysis methods for
Phase Two will be explored. The research data for Phase Two were analysed according
to the methods presented within Table 4.9 below.
83
Table 4.9: Phase Two Data Analysis
Data Collection Data Gathered Data Analysis
Audit of previous knowledge questionnaire
Quantitative data SNAP
Initial focus groups
Qualitative data Content analysis
Preference questionnaire Quantitative data
SNAP
Post-pilot feedback questionnaire (COMOIRA)
Quantitative Qualitative data
SNAP Content analysis
Post-pilot feedback questionnaire (AI)
Quantitative Qualitative data
SNAP Content analysis
4.20 Questionnaires
Quantitative data were gathered from both the audit of previous knowledge and
preference questionnaires which were completed before the piloting of COMOIRA and
AI. Quantitative and qualitative data were generated from the post - pilot feedback
questionnaires which followed the piloting of COMOIRA and AI.
For all four questionnaires quantitative data were collected from the rating scale
questions. For coding purposes, each response was assigned a number which were then
transferred into SNAP Questionnaire and Analysis summary software for analysis
purposes. SNAP is a questionnaire and analysis software tool. The responses are
collated using a code method and an analysis is carried out on the responses received to
suit what information is required from the reporting. SNAP was used to organise the
collection of data into categories. This generated statistics highlighting the number of
responses in each category and presented this as an overall percentage. The data
gathered from the questionnaires were considered alongside the information from the
focus groups and then used to inform the subsequent phase of the research, in line with
the AR model.
The post-pilot feedback questionnaires also generated data from a set of open-ended
questions. The coding of responses involved combining the information contained in the
responses into a limited number of categories which facilitated a simple description of
the data for analysis purposes. All responses to a particular question were placed onto
84
one sheet of paper and then content analysis was used to organise the data into a small
set of categories.
4.21 Content Analysis
Qualitative data can be analysed in different ways (Braun & Clarke, 2006). Content
Analysis (CA) was selected as the method of data analysis for the initial focus groups
and the reasons for this will now be considered.
A summative manifest CA was selected as the researcher wanted specifically to identify
certain words and describe the obvious components (Graneheim & Lundman, 2004;
Hsieh & Shannon, 2005). Keywords were identified before and during the data analysis
process and were derived from the researcher’s areas of focus during the focus group
discussions. The CA focused on the strengths of the professional practice models, the
limitations of the models and their application to EP practice. The focus group
discussions were considered as a whole and then the content of the text was separated
into the three areas outlined (themes) and then the data for each area were then
organised according to content categories. Categories are defined as patterns within the
text. A coding process was designed by the researcher to organise the data into content
categories. The content categories were then analysed and divided into condensed
meaning units as observed in the research of Graneheim and Lundman (2004).
Summative content analysis is different from other types of content analysis in that the
text is approached in relation to a particular content and further analysis leads to an
interpretation of the contextual meaning (Hsieh & Shannon, 2005). For the purposes of
this research the analysis was both quantitative and qualitative, first focusing on
counting the number of strengths/ limitations for each model and the frequency of
specific words. Additionally, it also included a qualitative element which considered the
content of the words and some basic interpretation.
4.22 Phase Three data gathering methods and data analysis
The data for this phase of the research were subject to more substantive analysis due to
the fact that the data generated were used to directly answer the research question.
85
Phase Three of the research collected and analysed data according to the methods
presented within Table 4.10 below, in order to address part 1a of the RQ: “What are the
EPs’ views on the most effective models of evaluating consultation in relation to
assessing consultation as a process”? and part 1b of the RQ: “What are the EPs’ views
on the most effective models of evaluating consultation in relation to providing
outcomes which demonstrate accountability to employers and service users”?
Table 4.10: Phase Three Data Collection and Analysis
Data Collection Data Gathered Data Analysis
Semi-structured interviews
Not conducted Section 4.22.1
Final focus groups
Qualitative data Thematic analysis
Each data collection method and data analysis method will now be discussed separately.
4.22.1 Semi –structured interviews
A collaborative decision was made not to conduct the semi-structured interviews that
had been originally planned. The reasons for this are explained within 4.13.1.
4.22.2 Final focus groups
A final two focus groups were held during this phase of the research. The time allowed
for the groups was 60 minutes. The first group consisted of one PEP, one SEP and four
EPs who constituted one of the EPSs and the second group consisted of the same PEP
and six EPs from the neighbouring authority. Before the final focus groups began the
questions were written on a piece of flip chart paper (see Appendix J). These focus
groups were run according to the same set of principles as outlined in section 4.17.
Data collated from this process were to inform a decision being made about the future
framework being adopted across the EPSs. As this was the final step in the research
process, these data were subject to more rigorous analysis. Thematic Analysis (TA) was
used to analyse the data gathered from the final two focus groups according to the
process outlined below within section 4.22.3.
86
4.22.3 Thematic Analysis
TA was selected as the method of data analysis for the final focus groups and the
reasons for this will now be considered.
TA was selected due to its flexible and dynamic nature (Crabtree & Miller, 1999). TA is
not affiliated with any particular epistemological positon and is also able to positon
itself between the essentialist and constructionist paradigms within psychology which
means it fits with the critical realist stance of the researcher. It is a method which works
both to reflect reality and to unpick reality (Braun & Clarke, 2006).
As TA is independent of theory it can be used as a specific research tool rather than a
research method and can be used to analyse most types of qualitative data (Braun &
Clarke, 2006). TA was chosen as the most appropriate way to investigate the data from
the focus groups. The literature was reviewed to provide a detailed protocol to carry out
the TA. Up until recently TA has been perceived as not sufficiently rigorous to permit
necessary peer scrutiny. However, more recently published protocols for carrying out
TA have been produced (Floersch, Longhofer, Kranke & Townsend, 2010). Braun and
Clarke (2006) assert that used appropriately and applied rigorously, TA is a robust
technique. The researcher adopted the version by Braun and Clarke (2006) because of
experience of using this approach in previous research conducted for the degree of
Doctor of Educational and Child Psychology. It was also a familiar model to the
members of the professional reference group who scrutinised the coding.
The initial focus group TA informed the selection of the models: COMOIRA and AI
which were then piloted as evaluative frameworks by all EPs. The process for TA is
outlined in Section 4.22.4 below.
4.22.4 Thematic Analysis process
TA is a method for identifying, analysing and reporting themes within data. Attride-
Stirling (2001) notes that in order for qualitative research to yield meaningful and useful
results it is crucial that the data is analysed in a methodical manner. Braun and Clarke’s
(2006) six phase step by step guide was used to guide the analysis of the focus group
data (refer to Table 4.11 below). Although this is presented as a linear, staged approach,
the analysis is an iterative and reflexive process.
87
Table 4.11: Phases of thematic analysis (Braun and Clarke, 2006, p.87)
Phase Description of the process
1. Familiarising yourself with the data:
Full transcription of data which involved becoming fully ‘immersed’ in the data. Data read and re-read and noted down initial ideas. Discussed with EPs to check accuracy.
2. Generating initial codes:
Initial codes were generated. Data revisited in a systematic fashion across the entire data set and more succinct codes were generated.
3. Searching for themes:
Relevant extracts were collated into potential themes. These were checked against one another and back to original data set. Preliminary themes identified, coded data placed into relevant themes.
4. Reviewing themes: Checked and finalised the themes. Generated a thematic ‘map’ of the analysis.
5. Defining and naming themes:
Themes and sub-themes were defined and named. The ‘essence’ of the theme was identified.
6. Producing the report:
The final opportunity for analysis. Selection of vivid, compelling extract examples, final analysis of selected extracts, relating back to research questions and literature, producing a report.
In order to fully immerse herself in the data the researcher carried out a full
transcription of all focus groups as recommended by Braun and Clarke (2006). All
recordings were listened to by the researcher and typed onto a Microsoft Word
document and then transferred onto an encrypted memory stick. Transcripts were
regularly checked against the original recordings during this process to ensure accuracy.
The two final focus groups were analysed separately and the themes were then collated
into one data set.
When considering what constitutes a theme Braun and Clarke (2006) suggest it should
capture something salient about the data. The authors do not give any indication as to
the amount of data required to constitute a theme but consider that a theme represents
“some level of patterned response or meaning within the data set” (p. 86). It is down to
the judgment of the researcher to determine the ‘keyness’ of a theme and this decision is
not necessarily dependent on any form of quantifiable measures but rather it is in
88
relation to whether it captures something important in relation to the overall research.
When conducting thematic analysis, the researcher can identify themes within the data
using either use an inductive “bottom-up” approach or a deductive/theoretical “top-
down” way (Braun and Clarke, 2006). A deductive approach can be conceptualized as
being driven by the researcher and involves analysing and coding the data according to
the researcher’s interest in a particular area.
The researcher identified themes at an inductive level. Braun and Clarke (2006) suggest
that this provides a richer description of the data. At an inductive level the themes are
data driven and do not fit into a pre-existing coding system. When conducting TA it is
also important for the researcher to decide upon the level at which the themes are to be
analysed. Themes can be identified at a latent or interpretative level or at a semantic or
explicit level (Boyatis, 1998). When analysing at a semantic level, themes are identified
at a surface level and the emphasis is on the explicit meaning of the data. Conversely,
when analysing data at a latent level, there is an attempt to identify underlying ideas
where the researcher looks to interpret the data. Within this research the themes were
analysed at a semantic level in line with the critical realist position adopted. Braun and
Clarke (2006) suggest that within a realist approach, you can theorise meaning in a
simpler or more straight-forward way.
4.23 Critique of methods
The selected research design and methods will now be considered and critiqued in turn.
4.23.1 Action Research framework
The close and collaborative relationship between the researcher and participants was an
important aspect of this piece of research and fit well within the flexible, qualitative
design. However, AR can be criticised for this relationship and the effect it can have
upon the research process. It could be argued that it is not possible to engage with
people and systems without this having a significant impact upon the researcher’s
decisions, about aspects of the design and data collection. When notions of
collaboration and participation are taken seriously, then some decision making powers
about aspects of the design and data collection are lost (Robson, 2002).
89
Due to the researcher’s immersion within the process it can be difficult to remain
reflective and retain critical ways of knowing (Kidd & Kral, 2005). It could be argued
that the researcher is at risk of allowing their own perspectives to influence the other
participants which could undermine the study. It was important for the researcher to
maintain reflexivity in relation to the impact of their contribution. Throughout the
research I reflected upon my values, beliefs and attitudes and considered the way in
which my involvement shaped and informed the research (Nightingale & Cromby,
1999).
Fortunately, during the study, no challenges arose within the group of EPs in relation to
the research. Such difficulties can arise within AR, researchers may be faced with
members who are in opposition to the process or there may be a loss of motivation or
commitment (Kidd & Kral, 2005). The researcher did find that during the piloting phase
of the frameworks some EPs used the frameworks for fewer consultations than was
originally requested. This introduced a challenge to the process. However, it can be
argued that by incorporating a multi-methods approach the validity of this data is
increased. The data generated from the focus groups were triangulated against the
questionnaire data which increased the likelihood of the reliability of the findings.
4.23.2 Questionnaire
Within this piece of research four questionnaires were used. It has been suggested that
questionnaires are problematic in that their design is often based around the researcher’s
assumptions as to what is important information to collect and may, therefore, not
necessarily reflect what the participant considers to be important (Robson, 2002). The
researcher attempted to resolve this by designing the questionnaires alongside the PEP
and SEP (see RADIO Stage 6) and testing them informally with a small group of
colleagues.
The feedback questionnaires which followed on from the piloting of COMOIRA and AI
incorporated open-ended questions. The process of coding open-ended questions can be
criticised for being very subjective. As a way of overcoming this, the researcher asked
fellow doctoral student colleagues to study the coding system applied to the open-ended
questionnaire data. This was essential in both challenging and verifying the findings of
the research.
90
4.23.3 Focus groups
Focus groups have been criticised for not capturing a range of views. By virtue of the
number of participants the researcher’s ability to control the interactions is reduced,
offering the potential for the participants to follow their own agenda (Silverman, 2004).
Due to the interactive forum the process has to be facilitated carefully so a small
minority do not dominate and influence the responses (Kruger & Casey, 2000). This is a
potential barrier and can place bias upon the data. The researcher attempted to remedy
this by ensuring there were a variety of speakers during the focus groups. In addition, a
questionnaire was completed at the end of each focus group so participants’ views were
captured.
The researcher also felt it was important to recognise the impact that the presence of the
PEP and SEP may have upon the group. However, the composition of the group was
discussed openly and the participants are used to frequently sharing their views with one
another around a range of professional topics. It was also important to consider the
researchers’ role and the potential for impact. Due to the knowledge the researcher had
gathered around the professional frameworks and the potential for bias towards specific
models it was important to remain neutral during the discussions so as not skew the
content and compromise the credibility of the data collected within the focus group.
The researcher tried to manage any possible misinterpretation of participant’s views by
seeking clarification of statements and through the use of flip chart recording. The
comments on the flip chart were reflected upon following the discussion of each
framework to provide the participants with an opportunity to comment and correct.
4.23.4 Thematic Analysis
TA has been criticised as there are a number of things which can cause a poor analysis
and deliver potential pitfalls. Braun and Clarke (2006) suggest that, due to the relatively
straight-forward method of TA, the researcher should avoid a weak analysis which can
result when a collection of extracts are placed together at a superficial level. It is the
researcher’s role to ensure that their analysis is consistent with the data in order to
define a plausible theme (Braun & Clarke, 2006). The flexibility afforded by TA can
cause the data to become anecdotal or leave it open to bias which can compromise the
validity of the research process (Willig, 2001). It is the job of the researcher to
91
determine the relevance of the data and consider how and when themes are identified.
The ‘keyness’ of a code does not simply rely upon its frequency (Braun & Clarke,
2006). During analysis, it was not always easy to make a decision on what constituted a
theme. In order to minimise the bias and bring reliability to this process the researcher
asked a fellow colleague to study and code randomly selected excerpts of the
transcribed focus groups. At subsequent meetings both sets of analyses were compared
and collated into potential themes (refer to Appendix M). These were then checked
against the original data set and preliminary themes were identified and finalised. A
thematic map was generated and subthemes were defined and named (Braun and
Clarke, 2006).
Another concern is that, as TA does not require the same detailed theoretical knowledge
as other qualitative analysis approaches such as grounded theory or IPA, it could be
considered as having limited analytical and interpretative properties. As such, it
provides just a description of the data. The researcher remained mindful of this and
aimed to ensure that the analysis process was rigorous and driven by the research
questions (Braun & Clarke, 2006). Additionally, TA was only one of the methods used
to analyse data within this piece of AR.
4.23.5 Content Analysis
As with other areas of qualitative research CA has been criticised in relation to the
concepts of generalisability, credibility and dependability (Graneheim & Lundman,
2004; Lincoln & Guba, 1985). In an attempt to achieve credibility within the CA the
researcher took particular care over the selection of the condensed meaning units to
avoid the pitfalls highlighted by Granheim and Lundman (2004) where meaning units
can be too broad to manage the data or too narrow that meaning can become lost. As
with TA, it was not always easy to constitute what made up a category so the researcher
sought the views of colleagues and fellow doctoral students and compared their
analyses of selected excerpts with her own in order to minimise bias.
4.23.6 Multi-methods design validity/reliability
This piece of research attempted to ensure rigour and trustworthiness through the
triangulation of methods. A multi-method approach brings about more confidence in the
data (Cohen, Manion & Morrison, 2003). The researcher’s intention was that one
92
method would enhance the next and once the information is collated this provides
contextually richer data than if it were seen from only one vantage point (Biggerstaff,
2012). The over reliance on one method can create an amount of bias which may lead to
a distortion of the overall picture. If the outcomes of a multi-method approach
correspond the researcher can be more convinced by the findings (Robson, 2002).
4.24 Ethical considerations
The researcher conducted the research in line with the University of Manchester
Research Ethics Committee guidelines (2014) and the British Psychological Society’s
(2009) Professional Code of Practice. The researcher adhered to all recommendations
provided by the University Ethics Committee and permission was granted for the minor
amendments made to the study.
The aims and objectives of this study were given to all EPs in writing prior to the
research commencing and all potential participants were presented with an ‘informed
consent’ form (refer to Appendix K). This asked for written consent for each part of the
research process. The participant information sheets, consent forms and all other
documentation relating to the research were emailed to the researcher’s supervisor and
approved by the University of Manchester ethics panel. All participants had the right to
withdraw from the research at any point, without reason. They participants were given
the option to ask for their information to be deleted at any time up until the data has
been anonymised.
At the beginning of each focus group the researcher reiterated the purpose of the
research, her role within the research process and provided an explanation of how the
data would be used. No individuals were named during the analysis and the researcher
was the only person with access to the dictaphone. Paper and electronic copies
transcripts were used for analysis and a diary was kept for reflection. Electronic copies
and records were kept on an encrypted laptop in a locked office and paper copies and
notebooks were stored in a locked filing cupboard in the same office. The
questionnaires were anonymised to protect the participant’s identity and kept in a
locked filing cabinet. All participants were informed after the data had been collated.
This information sharing occurred at the regular joint team meeting events which were
built into the AR methodology.
93
All forms of the data analysis process were carried out by the researcher. However, it is
crucial that, when analysing and coding data, alternative explanations are explored and
considered (Yin, 2009). For this purpose, the researcher asked colleagues and fellow
doctoral students to study randomly selected excerpts of transcribed focus groups and
then compared their coding and comments with their own. This provided the researcher
with different perspectives which created a depth of understanding around the data
(Patton, 2002). The coding systems applied to the questionnaire data were also studied
by colleagues and fellow doctoral students and the similarities and differences in the
coding decisions were considered. This was essential in both challenging and verifying
the findings of the research.
94
Chapter Five: Results
5.1 Chapter outline
This chapter outlines how the research emerged. The three phases of the RADIO model
are detailed and due to the cyclical nature of this piece of AR each phase is discussed in
turn. Each phase of the research informed the next so for each stage the data gathering
methods will be considered along with the data analysis methods and the data analysis
outcomes. The results aimed to answer the following research question:
RQ1: What are the EPs’ views on the most effective models of evaluating
consultation in relation to:
1a) Assessing consultation as a process?
1b) Providing outcomes which demonstrate accountability to employers and
service users?
Multiple sources of data were generated and the links between data sets will be
discussed at relevant points. Data sets are outlined in Table 5.1 below.
Table 5.1: Phases of the Research
Phases of research Data sets Sections Pre –research Phase and Phase One
Researcher’s diary 5.2
Phase Two
Systematic analysis, audit of previous knowledge questionnaire, preference questionnaire, initial focus groups, post-pilot questionnaires
5.4
Phase Three
Final focus groups 5.11
The stages of RADIO covered across all phases of the research are summarised within
Table 4.1. The main emphasis of this chapter lies on the data gathered from the final
focus groups (see Section 5.12.1). A detailed thematic analysis was conducted at this
stage of the research to inform future decisions around the selection of a framework for
the evaluation of consultation. The earlier stages of the data analysis are covered in less
depth but were essential to the overall research process.
95
5.2 Pre -research Phase and research Phase One
The Pre- research Phase and Phase One of the study incorporated Stages 1-4 of the
RADIO process. Within these phases of the research process a clear mandate for the
research was established which is briefly summarised in Section 4.11.
5.3 RADIO Stages 1-4
The researcher started by creating a ‘rich picture’ prior to the commencement of the
piece of research. This information contributed to Stages 1-4 of the RADIO model and
was recorded within the researcher diary. The following stakeholders were consulted
during several informal meetings in order to gain information to inform the research:
• PEP
• SEP from one of the LAs
Discussions with the PEP and SEP, documented within the researcher diary, highlighted
the need for the two EPSs to find a tool that would collect relevant information on the
impact of our service delivery. As a doctoral student the researcher was requested to
explore potential professional frameworks that would facilitate the EPs within the two
services, to successfully and appropriately evaluate consultation and gather relevant
data about the effectiveness of the EPSs for the LA. A thorough review of the literature
confirmed the need for further research in this area. The processes for collaboration with
the stakeholders were identified and regular meetings were arranged.
The data for Stages 1-4 was recorded within the researcher’s diary (see Appendix I).
This information was used to inform the next stages of the research process. This data
were not subjected to any form of robust methodological analysis as it served to
highlight the rational for the study rather than serve as the main data set.
5.4 Research Phase Two
Phase Two of the research incorporated Stages 5-8 of the RADIO process. Sections 5.5-
5.10 outline the different methods used to gather data at each stage of phase two,
followed by the data analysis methods and outcomes. Sections 5.5.1-5.5.16 provides a
description of the systematic analysis conducted to select the professional practice
frameworks to present for discussion at the two initial focus groups. Following on from
this, the data gathered from the initial questionnaires and focus groups and the outcomes
96
of the analysis are discussed. The outcome of the post-pilot questionnaire data is also
covered.
5.5 Stage 5 and Stage 6- agreeing the focus of concern and negotiating framework
for information gathering
The PEP and researcher agreed the focus of the research and established that the main
aims of the research were to find a way to evaluate consultation that would enable EPs
to assess consultation as a process and provide outcomes which would demonstrate
accountability to employers and service users.
Further discussions with the PEP and SEP revealed the need to search the literature to
source suitable professional frameworks which could then be carried forward into the
next stage of the research. A thorough review of the literature was conducted. Through
this process nine possible frameworks were identified (see Section 3.3):
• AI
• AT
• COMOIRA
• DECP
• Friedman’s Results Based Accountability Model (RBA)
• IFF
• Measuring the impact of Casework (MtiC)
• RADIO
• TME
5.5.1 Systematic Analysis
Stage 6 involved the devising of a framework which could be used to guide the
selection process for choosing the professional models/frameworks that would be
presented to the EPs and discussed at the initial focus groups. This involved a five step
process as outlined below.
5.5.2 Step One: Framework for selection process
In order to select the most suitable models a framework was constructed and used to act
as a guide to inform the selection of the most relevant consultation evaluation models.
97
The work of Wagner (1995; 2000) was consulted to facilitate the development of this
framework. The general theories and principles underpinning consultation were
incorporated into the framework which is presented within Table 5.2 below
Table 5.2: Frameworks for the Selection Process
A) Does the model utilise the main theories of consultation?
1) Interactionist 2) Systemic 3) social constructionist
B) Does the model utilise the main processes of consultation? 1) Facilitate a collaborative approach 2) Allow joint problem analysis 3) Facilitate dialogue 4) Allow solution-oriented thinking 5) Facilitate the formulation of an action plan 6) Facilitate a plan, do, review process which allows the participants to
reflect upon the process and evaluate the changes
C) Can it be applied across the contexts in which the EP works? 1) Individual 2) Group 3) Organisational
D) Can it be easily implemented into the consultation process? E) Is it sufficiently flexible to accommodate the diverse nature of EP work, including the changing needs of :
1) different people? 2) different situations? 3) different groups?
F) Can it supply data to demonstrate the ‘added value’ and accountability? G) Does the model employ a range of both quantitative and qualitative techniques to triangulate the data? H) Does it allow the collection of data from a range of sources and contexts? I) Has the model been used in other services and, if so, which types of Services? J) Is there any evaluation data on the impact of the model?
5.5.3 Step Two: Systematic Analysis
The nine identified frameworks (see Section 5.5) were considered in detail alongside the
pro forma. A systematic analysis took place of each model using the framework in
98
Table 5.2 and recorded within Table 5.3 below. This process guided the decisions
around which models would be included and excluded from the research. The models
that were included met all of the criteria. The reasons by which models were excluded
are clarified in Step Three and Step Four below.
Table 5.3: Framework pro forma
DE
CP
CO
MO
IRA
IFF
Activity
Theory
TM
E
RA
DIO
MtiC
AI
RB
A
A1 � � � � � � � � x
A2 � � � � � � � � �
A3 � � � � � � � � X
B1 � � � � � � � � �
B2 � � � � � � � � �
B3 � � � � � � � � �
B4 � � � � � � � � �
B5 � � � � � � � � �
B6 � � � � � � � � �
C1 � � � � � x � � �
C2 � � � � � � x � �
C3 � � � � � � x � �
D � � � � � � � � �
E1 � � � � � � � � �
E2 � � � � � � x � �
E3 � � � � � � x � �
F � � � � � � � � �
G � � � � x � � � �
H � � � � � � x � �
I � � � � � � � � �
J � � � � � � � � �
99
5.5.4 Step Three: Exclusion of frameworks based on paradigms
All the models were initially considered in terms of their paradigms. As Wagner’s
model of consultation draws upon social constructionism (Gutkin & Curtis, 1999;
Wagner, 1995) it was considered essential that the framework adopted for evaluation
would also need to fit the constructionist paradigm. As argued by Gameson et al. (2003)
the paradigm used to evaluate should fit the paradigm of the approach.
The RBA model was considered not to fit comfortably within the paradigm of
consultation. The reasons for this are summarised below.
5.5.5 Results Based Accountability Model
One of the EPSs already uses a model based on Friedman’s RBA Framework
(Friedman, 2005). This approach distinguishes between ‘performance’ and ‘population’
results. Population results cover outcomes for the population and individual
organisations then focus on managing and improving their contribution to this
population result. The EPS uses the RBA model to measure the impact of their service
on service users (Welsh Audit Office, 2010).
RBA is based on a four-quadrant framework which aims to answer questions about
service delivery (see Section 3.3.9).This framework appears to rely on quantifiable
results which would place it in the reductionist, positivist paradigm. When considering
this framework in relation to the systematic analysis the following questions were asked
as to: “Would this framework encapsulate the complexity of the EP role due to its focus
on what is measurable”? “Could it be applied across a range of contexts in which EPs
work”? “Is it sufficiently flexible to accommodate the diverse nature of EP work”? Due
to its focus on quantifiable outcomes and its position within a positivist paradigm it was
felt that it did not sufficiently and fully provide a convincing answer to these questions
and a decision was made to exclude it from the study.
As demonstrated by the research of Keevers, Treleavan, Backhouse and Darcy (2010),
when using RBA as an evaluation tool, activities could potentially become
decontextualised for the purposes of quantification and this would hamper the inclusion
of the local practice experience of the EPs.
100
5.5.6 Step Four: Further exclusion of frameworks
Although the following frameworks were considered to sit comfortably within the
paradigms of consultation during the systematic analysis process they were excluded
from the research because they did not fully meet the criteria. Reasons for this are
explained in the sections below.
5.5.7 Target Monitoring and Evaluation
TME (Timmins et al., 2003) was designed to be used within a consultative model of
service delivery. It derived from GAS (Kiresuk & Sherman, 1968). A summary of TME
is provided in Section 3.3.7. Lengthy discussions with the PEP commissioner and other
stakeholders raised questions around whether TME would encapsulate the complexity
of the role of the EP and what they do during consultations. The focus on measurement
raised the following questions: What does it actually measure? Does it measure the
influence of the EP or the teacher’s ability to implement an intervention effectively? Is
it just a glorified rating scale? Does it provide useful data about the impact of the EP?
A decision was made to exclude TME as a framework because it was concluded that it
appeared positivist, reductionist in nature, focusing only on outcomes that are
measurable. Despite being an important aspect, pupil outcomes alone do not illuminate
EP effectiveness. In the complex context of EP work there is often a crucial difference
between an identifiable outcome (concrete, measurable) and what is achieved
(subjective, judgement).
5.5.8 Research and Development in Organisations
RADIO (Timmins et al., 2006) is collaborative in nature and steers away from a
positivist design (see Section 3.3.5).There are different research phases which include
clarifying concerns, researching the concern and creating change. Within this method
there is space for all types of approaches including interviews, questionnaires, surveys,
action research, case study as long as the chosen approaches meet address the research
questions.
The RADIO model does fit the constructionist, relativist paradigm and did, therefore, fit
with the theoretical perspective of consultation. Questions were raised around whether
RADIO could be applied to an individual consultation (see section 5.5.3, C. 1). A
101
decision was made to exclude RADIO as it was considered that it lends itself more
readily to systemic work at an organisational level and this research was focused on
individual consultation meetings. Timmins et al. (2003) discuss how the processing of
information with stakeholders can lead to organisational change. Despite being
excluded as a framework for the individual consultation meetings RADIO was adopted
as the framework used to guide the action research process of the study (see Section
4.9).
5.5.9 MtiC
Turner et al. (2010) developed a method of evaluating casework that could be
administered by EPs with a view to capturing the impact that an EP had on outcomes
for pupils. The casework evaluation was carried out individually by the EPs. This
approach was embedded in a constructionist relativist paradigm. A decision was made
to exclude this model from the research due to the following factors. This framework
was devised to evaluate individual casework and does not, therefore, gather data from a
range of contexts or accommodate the diverse nature of EP work across different groups
and situations (see section 5.5.3, C.2 & C.3). It can only be used to look at the outcomes
of individual casework and does not evaluate every consultation across different
contexts. Despite this piece of research focusing on individual consultations the PEP
questioned whether this model could encapsulate the complexity of what EPs do
through consultation.
5.5.10 Step 5 – Inclusion of frameworks
The frameworks that met the selection process criteria were carried forward into the
next phase of the research. These were selected on the basis that they were well
developed, comprehensive and flexible frameworks that fit with the model of
consultation and met the criteria, as defined in Table 5.2. It was hypothesised that the
consultation model of working could be evaluated and rationalised through an
application of these frameworks into the everyday practice of the services. It was
anticipated that a robust exploration of these frameworks would provide a relevant,
transparent and ethical process for the evaluation of EP consultations within this piece
of research.
The five frameworks that met the selection process criteria were as follows:
102
• AI
• AT
• COMOIRA
• DECP
• IFF
These five frameworks are summarised within Section 3.3.
5.6 RADIO Stage 7: gathering information
Within this stage of the research process information was gathered using the agreed data
collection methods and the data sets were analysed. The outcomes of the data analysis
process are discussed within this section.
5.6.1 Audit of previous knowledge questionnaire
Two weeks prior to the two initial focus groups with the two EPSs, all EPs were asked
to complete an audit of previous knowledge questionnaire (see Appendix D). This
collected data around how familiar the EPs were with the five models (Q1) and whether
they implemented any of these within their current professional practice (Q2).
5.6.2 Data analysis outcome
The audit of previous knowledge questionnaire was analysed using SNAP software (see
Section 4.20). This generated statistics highlighting the number of responses in each
category and this was also presented as an overall percentage. The results are illustrated
in Table 5.4 and Table 5.5.
103
Table 5.4: Audit of previous EP knowledge Q1
1. How familiar are you with
the following professional
practise models /
frameworks?
Very
familiar %
Fairly
familiar %
Know
a little %
Not at
all
familiar
% Total %
COMOIRA 3 25.0% 3 25.0% 4 33.3% 2 16.7% 12 100.0%
Appreciative Inquiry 3 25.0% 6 50.0% 2 16.7% 1 8.3% 12 100.0%
Activity Theory 0
0.0% 2 16.7% 2 16.7% 8 66.6% 12 100.0%
DECP 2
16.0% 1 8.3% 3 25.0% 6 50.0% 12 100.0%
Interactive Factors Framework 0 0.0% 2 16.7% 3 25.0% 7 58.3% 12 100.0%
104
From the data presented within Table 5.4 it was evident that the majority of EPs were
most familiar with AI and COMOIRA. Only two EPs were not at all familiar with
COMOIRA and only one EP was not at all familiar with AI. Interestingly, both of the
EPSs have regular trainees from Cardiff and, therefore, those supervising the trainees
have regular discussions around the COMOIRA model. Both EPSs have, in recent
years, had a professional development session based on AI.
None of the EPs were very familiar with either AT or IFF and the majority of EPs
reported that they knew only a little or were not at all familiar with AT, DECP or IFF.
105
Table 5.5: Audit of previous EP knowledge Q2
2. Are you currently using any of the models / frameworks in your professional practice? Yes % No %
No
response % Total %
COMOIRA 5 41.7% 6 50.0% 1 8.3% 12 100.0%
Appreciative Inquiry 6 50.0% 5 41.7% 1 8.3% 12 100.0%
Activity Theory 0 0.0% 10 83.3% 2 16.7% 12 100.0%
DECP 1 8.4% 10 83.3% 1 8.3% 12 100.0%
Interactive Factors
Framework 0 0.0% 10 83.3% 2 16.7% 12 100.0%
106
Results generated when asking whether the EPs currently use any of the models
within their practice (see Table 5.5) revealed that approximately half of the EPs used
COMOIRA and AI. The majority of EPs had not used AT, DECP or IFF within their
practice.
Those EPs who were using professional practice models within their practice were
asked to briefly describe how they were using the models within their practice and
these written responses were collated (see Table 5.6).
107
Table 5.6: EP responses from the Audit of Previous Knowledge Questionnaire
Question
Comments
If you answered 'yes' to any of the models/frameworks in question 2 please describe very briefly how you are using them.
Appreciative Inquiry is used as a framework for working with a school to develop positive change as well as using it to structure certain types of consultation. - COMOIRA is used particularly with supervision of TEPS as this is the model they use. Using informally in a small number of school settings. Looking to extend this next year in a more formal/planned way (AI) All the time (COMOIRA) Casework, everyday use (COMOIRA) As an awareness and in conversations with parents and teachers (AI) I use the forward thinking solution oriented aspect of AI in my approach to consultation. COMOIRA used with EPiTs (Cardiff) Reflecting through supervision with Cardiff students.
108
5.6.3 Initial focus groups
Two weeks prior to the initial focus groups, all EPs were provided with information
briefing sheets (see Appendix G). This provided the EPs with information about the
five frameworks selected for the research and references for further reading where
required.
All EPs attended one of the two initial service focus groups. Each focus group lasted
approximately 60 minutes. The EPs gave consent for the groups to be audio recorded
for the researcher’s reference. During each of the focus groups the selected
frameworks were discussed in turn. The EPs were initially asked to comment on the
perceived strengths of the framework, then upon the perceived limitations and finally
asked for their views on the applicability of the framework to EP practice. The
researcher wrote the key points that emerged from the discussion on a flip chart.
Clarification was sought by the researcher on the meaning and interpretation of the
data with the participants. Participants were given the opportunity to elaborate on
key points and discuss with colleagues.
5.6.4 Data analysis outcome
The initial two focus groups were analysed using a qualitative summative approach
to content analysis, often referred to as manifest content analysis (Graneheim &
Lundman, 2004) in order to inform the ongoing process of identifying and evaluating
preferred models. Keywords had been identified before and during the data analysis
process and were derived from the researcher’s areas of focus during the focus group
discussions. These targeted the strengths, limitations and applicability to EP practice
of the five models. The content of the text was separated into these three areas
(themes) and then the data for each area was then organised according to content
categories. Categories are defined as patterns or themes within the text. A coding
process was designed by the researcher to organise the data into content categories.
Summative content analysis is different from other types of content analysis in that
the text is approached in relation to a particular content and further analysis leads to
an interpretation of the contextual meaning (Hsieh & Shannon, 2005). For the
purposes of this research the analysis was both quantitative and qualitative, first
focusing on counting the number of strengths/ limitations for each model and the
109
frequency of specific words. Additionally, it also included a qualitative element
which considered the content of the words and some basic interpretation. The EPs’
views regarding each model were collated and put into tables. Significant amounts of
information were gathered for each model and these will be discussed in more depth
(see Sections 5.6.4.1 - 5.6.4.5).
5.6.4.1 COMOIRA
Themes emerging within the three categories which related to the COMOIRA model
are shown in Table 5.7 below. The focus group text was read through several times
and then sorted into content categories. The content categories were then analysed
and divided into condensed meaning units as observed in the research of Graneheim
and Lundman (2004).
110
Table 5.7: Content analysis of data for the COMOIRA model
COMOIRA
Theme Content categories Condensed meaning unit
Strengths of the model
Constructivist People interacting together Correct rules to judge the game
Reflective Emphasis on reflection is an important element It fits with what EPs do generally
Reflexive It applies as much to EPs as everyone else in the process
Promotes change At different levels: organisational, group, individual Fits with consultation
Developed for EPs Draws upon and focuses upon psychology as the core Works with our values and beliefs Based on EP practice
Flexible
Its perpetual, it allows you to regroup and review Can come at it from any angle Not linear, can complete any box
Promotes equal partnership Emphasis on supporting each other Collaborative
Limitations of the model
Time demands Designed for trainees rather than EPs EPs don’t have the same time as trainees to complete it
Too flexible Data produced will be too varied as it’s so flexible Too much data to evaluate
111
COMOIRA
Theme Content categories Condensed meaning unit
Application to the evaluation of EP practice
Training Yes we could use it but would need training Would need time to look at how to use it
112
Analysis of the data surrounding the strengths of COMOIRA revealed the EPs
perceived this framework to be reflective as well as reflexive. Discussion revealed
that the EPs thought these were important elements. Words and phrases such as
“emphasis on reflection is an important element” and “it applies as much to EPs as
everyone else” were generated. It was seen to promote change at a number of levels
which was considered an important element as this fit within the consultation model
of service delivery. The fact that the model was developed for EPs was considered to
be important as this meant it was “based on EP practice”, “works with our values and
beliefs” and “focuses upon psychology as the core”. It was seen to be a flexible
model and one which promotes equal partnership through its “collaborative” nature.
Within the data collated surrounding the limitations of the model only two categories
were identified: “time demands” and “too flexible”. As COMOIRA was designed for
TEPs, some of the team felt that they would not have the time required of them to
complete the framework and whilst some of the team felt that the flexibility of the
model was strength some considered it to be a limitation. They thought it would
‘produce too much data to evaluate’ and that the data would be too “varied”.
When considering the application of the model to evaluate EP practice, the general
views were very positive. Some EPs identified that EPs “would need training” and
“would need time to look at how to use it”.
113
5.6.4.2 IFF
Table 5.8: Content Analysis of data for the IFF model
Interactive Factors Framework
Theme Content categories Condensed meaning unit
Strengths of the model
Constructivist
It comes from a constructivist perspective It is holistic It sees problems in context which EPs do Strikes a chord with what is unique to EPs: seeing situation in context
Designed for EPs It’s been designed to be used by TEPs Its thorough- it’s been developed over years and adapted for EPs It looks for evidence to support practice
Reflective It encourages you to be reflective It allows for changing hypotheses
Systematic The structure/phases are useful It has a sense of order which is helpful
Constructionist It works at different levels
Transparent It promotes transparency of EP practice
114
Interactive Factors Framework
Theme Content categories Condensed meaning unit
Limitations of model
Rigid
Not a natural process Doesn’t fit with the way we work Too mechanistic to apply to evaluation Applies to consultation but not evaluation Too linear Lack of flexibility in terms of application
Time consuming Too wordy How would we fit it in
Too scientific
You set a hypotheses and test it It sees things as an absolute Less empathy than COMOIRA Lacks understanding of others’ perspectives
Sense of hierarchy EP leads and collates all information EP seen as expert EP seen as leading process-not equal partnership
Too complicated Needs translating to be user friendly Language needs to be more friendly Needs translating for stakeholders
115
Interactive Factors Framework
Theme Content categories Condensed meaning unit
Application to the evaluation of EP practice
Translating it to stakeholders
It would need translating to others/stakeholders
Reflective
Promotes reflective practice but wouldn’t use it to collect data for evaluation
116
Analysis of the data surrounding the strengths of IFF, (refer to Table 5.8), revealed
that many of the EPs considered one of its main strengths is the fact that it is based
on a “constructivist” perspective. This fits with the EP perspective of “seeing
problems in context” which is something “unique” to EPs. As with COMOIRA
(Gameson et al., 2003) the EPs saw the fact it is based on EP practice in a positive
light. It was seen as a model which “is thorough” because it’s been developed “over
years and adapted for EPs”. One EP liked the fact that it “looks for evidence to
support practice”. As with COMOIRA the idea that IFF is “reflective” was seen to
be a strength and the fact that it “allows for changing hypotheses”. The systematic
structure was seen by some EPs to be useful as was its “transparent” nature which
“promotes transparency of EP practice”. IFF was considered to be a model that
would “work at different levels” of EP practice which would, therefore, fit with the
consultation model of delivery.
Significant amounts of information were gleaned when discussing the limitations of
IFF and upon analysis five content categories were identified. Many of the EPs
considered it to be too “rigid” with phrases used like: “not a natural process,” “too
linear,” “lack of flexibility” and “doesn’t fit with the way we work (consultation)”.
Some of the EPs thought it could apply to consultation but was “too mechanistic to
apply to evaluation”. It was considered to be a “time consuming” model and some
EPs felt that they would not be able to “fit it in” to their practice. It was viewed a
“scientific” framework with phrases such as: “you set a hypotheses and test it”; “it
sees things as absolute”. EPs felt that it lacked an understanding of the perspective of
others and had “less empathy than COMOIRA”. Another limitation was seen to be
the “sense of hierarchy” with the view that the EP is “seen as the expert” and as
“leading the process” rather than it being an “equal process”. IFF was seen as “too
complicated” by many of the EPs who commented particularly upon the language
which they believe needs to be “more user friendly” and needs “translating” as it was
particularly complicated.
When analysing the data generated from the discussion around the application of the
model to EP practice two content categories were identified. Most of the EPs
believed that a lot of work needed to be done to “translate” the language of IFF in
order for it to be implemented and understood by others. The EPs generally struggled
117
to see how it could be used to evaluate what EPs do through consultation with
phrases such as “wouldn’t use it to collect data for evaluation”.
118
5.6.4.3 AI
Table 5.9: Content Analysis of data for the AI model
Appreciative Inquiry
Theme Content categories Condensed meaning unit
Strengths of the model
Positive Psychology
Looks at what works Strengths based Like the ‘dream’ Solution focused-questions focus on strengths rather than problem Fits with consultation and what we do as EPs Problem-solving- allows you to reframe problems- only a problem because of the view we are taking
Simple
Quick Simple and easy to use Easy to understand
Evaluative
Cyclical Useful for generating valuable data-data you can work towards Can use it to facilitate evaluative discussion-generates feedback from consultee Generates rich and complex data Lends itself to evaluation at a number of levels
Promotes autonomy
Allows EP to be autonomous Doesn’t interfere with EP practice Flexible-doesn’t restrict EP work
119
Appreciative Inquiry
Theme Content categories Condensed meaning unit
Allows creativity Fits beautifully with Person-centred planning
Flexible It fits with working at a number of levels: individual, group Fits with most ways of working
Limitations of the model
Looseness Structure is too loose
Language The language can be off putting
Too positive Doesn’t facilitate looking at what’s not going well Cannot look at problems Doesn’t allow you to explore weaknesses within EP practice
Application to the evaluation of EP practice
Problem solving element
How to introduce problem solving element May only be able to use ‘discover’ element for evaluation purposes
Clear and simple Simple and easy to use May be too simple. May need to create a framework to apply to evaluation Handbook is user friendly
Fits well with EP practice
Fits well with EP practice Works well with person centred planning tools Promotes positive psychology and solution focused work which we do as EPs
120
When analysing the data from AI around the perceived strengths of the model six
content categories were identified. Words such as “quick”, “simple” and “easy to
use” were generated and many of the EPs though that as well as being a simple
model it is “easy to understand” . The fact that it uses “positive psychology” was
seen by many EPs as a strength. With it using “solution focused-questions” and due
to its “focus on strengths rather than problems”, it was seen to fit “with consultation
and the work of EPs”. Discussion revealed that often problems are generated by the
“view we are taking” and that AI “allows you to reframe problems” or the view we
take of problems. Many EPs thought it would be a useful “evaluative” tool due to its
“cyclical” nature. Discussion revealed that AI may be “useful for generating valuable
data, data you can work towards” and that the data generated would be “rich”. It was
considered to be a tool that could “facilitate evaluative discussion”, generate
“feedback from (the) consultee” and that it would lend “itself to evaluation at a
number of levels” which would fit well with what EPs do. EPs liked the fact that it
“promotes autonomy” and “doesn’t interfere with EP practice” and “its flexibility
doesn’t restrict EP work”. Across the two services the EPs are involved in promoting
Person Centred Planning (PCP) to schools and AI was seen as allowing the creativity
to “fit beautifully with PCP”. The flexibility of the model was considered an
important element as it fits with “most ways of working” and “at a number of
levels”.
Three content categories were identified when looking at the data surrounding
perceived limitations of AI. The lack of “problem” focus was a factor for some EPs,
in terms of it not facilitating what is not going well or enabling EPs to “explore
weaknesses” within their own practice. The language was considered by one EP as
“off putting” and its structure was considered too loose by another EP.
When considering the application to EP practice, one of the categories that emerged
indicated that it might be useful to introduce a “problem solving element” to the
model. One EP felt that it would only be possible to use the “discover” element of AI
for evaluation purposes. Many of the EPs thought that AI would be “simple” and
“easy to use” although one EP thought it might be helpful to create a framework in
order to apply it for evaluative purposes. One fundamental factor was in terms of it
fitting well with EP practice. Many EPs were in agreement that it “promotes positive
121
psychology”, and “solution focused” work which is central to the consultation
process. It was also considered to be a useful tool which would work well with PCP,
a topic which EPs are currently presenting and offering training to schools.
122
5.6.4.4 DECP
Table 5.10: Content Analysis of data for the DECP framework
DECP
Theme Content categories Condensed meaning unit
Strengths of the model Assessment It’s useful for guiding individual assessments The framework is good for guiding assessment
Limitations of the model
Applicability to EP practice
It doesn’t fit with consultation Not applicable to consultation
‘Within child’ focused
Its within child focused Referral based ‘within child’ language Outdated Very scientist practitioner-assessment based
Application to EP practice
Not applicable Doesn’t fit with consultation Doesn’t fit with evaluation
123
Only one content category was identified when analysing the data for the strengths
of the DECP framework. From an “assessment” perspective it was considered useful
for “guiding individual assessments” and the framework itself was viewed as being
“good for guiding assessment”.
When analysing the data around perceived limitations of the DECP framework the
applicability to EP practice within the two services was identified. It was evident that
many of the EPs did not believe it would fit with consultation as a model of service
delivery and was not “applicable” to consultation. The DECP framework was also
seen as a “within-child” focused model and this was identified as a fundamental
weakness with many of the EPs agreeing. Phrases such as “very scientist-
practitioner based” and “referral based within child language” were used and many
of the EPs expressed the opinion that the framework is a very ‘outdated’ version of
EP practice.
The main category when analysing the data around the applicability to EP practice
was that the DECP framework was “not applicable”. It was not perceived as a model
which would fit with consultation or the evaluation of consultation.
124
5.6.4.5: Activity Theory
Table 5.11: Content Analysis of data for the AT model
Activity Theory
Theme Content categories Condensed meaning unit
Strengths of the model
Fits with EP practice
Social constructionist-allows for different perspectives Fits with psychology of consultation Makes sense of situations and systems Incorporates lots of different voices Based on dialectic materialism-examines contradictions and opposing views to make sense of a situation Captures the complexity of EP practice
Well developed at different levels
Three generations Based on EP practice Can be used at different levels: organisation, multi-agency working ,in schools
Language based
The language we use can alter things/situations The language we use can often create the problem
Develops a rich picture Incorporates lots of people’s views Gives opportunities to analyse things
125
Activity Theory
Theme Content categories Condensed meaning unit
Limitations of the model
Complexity
Very hard to understand Too complicated Needs translating Don’t understand it The framework is confusing The diagram is off putting-makes it more difficult than it needs to be How would you translate it to others
Too descriptive It’s a descriptive tool Wouldn’t allow for evaluation
Application to EP practice
Training Would need additional training Need to understand it Need to adapt it to use for evaluation purposes
126
When analysing the data from Activity Theory around the perceived strengths of the
model, four content categories were identified. One of the fundamental factors was
that it fits with EP practice and “captures the complexity of EP practice”. The “social
constructionist” paradigm of AT “fits with the psychology of consultation”. It was
viewed as a model that could make “sense of situations and systems” which is an
important aspect of consultation. A further strength was that it was seen as a model
that “incorporates lots of different voices” and “allows for different perspectives”.
Discussion revealed that the model facilitates the development of a rich picture
because it “incorporates lots of people’s views” and “gives opportunities to analyse”.
The importance of language was raised and it was recognised that the language used
in situations is fundamentally important as this can “alter situations” and “can often
create the problem”. AT was viewed as a model that was well developed at different
levels. It was seen as applicable across a range of contexts: organisation, multi-
agency working, in schools. The fact that it was based on EP practice and had been
developed across three generations was seen as strength by some of the EPs.
Two content categories appeared from the data around the limitations of AT as a
model. One of the fundamental factors that arose was related to the complexity of the
model with many of the EPs in agreement. Comments such as “too complicated”,
“don’t understand it” and, “confusing” were generated. The diagram was seen as “off
putting” and as making the model more “difficult than it needs to be”. How the
model would be described to others was also seen as potentially problematic, due to
its complexity. The model was also seen as very descriptive which was seen as a
limitation when applying it to the evaluation of consultation.
When discussing the application of AT to EP practice the majority of EPs considered
that in order take AT forward they would require additional training in order to
“understand it” and to “adapt it for evaluation purposes”.
5.7 Preference questionnaire
At the end of the two initial focus groups all the EPs completed a preference
questionnaire. The questionnaires were anonymous and collated by the researcher.
The EPs firstly ranked their preferred professional practice models and then indicated
127
how likely they would be to use each particular model for the evaluation of their
practice.
The preference questionnaire was analysed using SNAP software (Section 4.20).
This generated statistics highlighting the number of responses in each category. The
results are summarised in Table 5.12 below.
Table 5.12: Preference Questionnaire rank responses
Please rank from 1-5 (1 = most preferred, 5 = least preferred) the models/frameworks
It was evident from the Rank 1 responses that the majority of EPs most preferred the
models of COMOIRA and AI. IFF and AT both had one Rank 1 response. The
DECP model was the only framework that did not receive any responses within the
most preferred rank. COMOIRA, IFF and AI received the highest number of Rank 2
responses with eight EPs selecting these models. Again the DECP model was not
selected and AT had one Rank 2 response. COMOIRA and AT both received one
response within the Rank 3 category. AI and IFF had the most Rank 3 responses.
The majority of the EPs gave AT and DECP a number 4 ranking. Only one other
model received a Rank 4 response, with one EP selecting COMOIRA. Half of the
EPs chose the DECP model as their least preferred framework. AT and IFF had 2
responses within the Rank 5 category. COMOIRA and AI did not receive any
responses within this category. Therefore, results from the preference questionnaire
revealed that the majority of EPs most preferred the COMOIRA and AI models.
Following the ranking of responses the EPs were then asked to indicate how likely
they would be to use the professional practice models to evaluate EP practice. The
results are presented in Table 5.13 below.
Model Number of rank responses Rank 1 Rank 2 Rank 3 Rank 4 Rank 5
COMOIRA 4 3 1 1 0 AI 3 2 5 0 0 IFF 1 3 3 0 2 AT 1 1 1 5 2 DECP 0 0 0 4 6 No response 3 3 2 2 2
128
Table 5.13: Preference Questionnaire responses
Very likely to
use
% Fairly likely to
use
% Would use a
little
% Not likely to
use
% No
response
% Total %
COMOIRA 5 41.6% 3 25.0% 2 16.7% 0 0.0% 2 16.7% 12 100.0%
Appreciative Inquiry 7 58.3% 2 16.7% 1 8.3% 0 0.0% 2 16.7% 12 100.0%
Activity Theory 1 8.3% 2 16.7% 3 25.0% 4 33.3% 2 16.7% 12 100.0%
DECP 1 8.3% 1 8.3% 4 33.3% 4 33.3% 2 16.7% 12 100.0%
Interactive Factors
Framework
0
0.0%
5
41.6%
3
25.0%
2
16.7%
2
16.7%
12
100.0%
129
It is evident from these results that the EPs were ‘very likely to use’ COMOIRA and
AI. The EP responses for COMOIRA indicated that the majority of EPs would be
either ‘very likely’ or ‘fairly likely’ to use this framework to evaluate EP practice.
Only two out of the ten responses for this category indicated that they ‘would use a
little’ and none of the EPs said that they would ‘not (be) likely to use the model’.
Results for AI revealed that nine of the EPs would be ‘very likely’ or ‘fairly likely’
to use this as a model to evaluate EP practice. Only one EP showed that they ‘would
use a little’ and there were no responses within the ‘not likely to use’ category.
Conversely, the responses for AT revealed that the majority of EPs would either not
be likely to use this framework or would only use it a little. Three of the ten
responses fell within the ‘fairly’ to ‘very likely’ to use category.
AT and DECP had the highest number of responses within the ‘not likely to use’
category. Both had four responses.
The DECP framework had the least number of responses within the ‘very likely’ and
‘fairly likely’ to use categories. Only two EPs indicated that they would use this
model to evaluate EP practice. Four of the EPs reported that they ‘would use a little’
and four indicated that they were ‘not likely to use’ this framework.
IFF was the only framework that didn’t have a response within the ‘very likely to
use’ category but five of the EPs indicated that they would be ‘fairly likely to use’
this model to evaluate EP practice. The results revealed that five EPs would either
‘not be likely’ to use IFF or ‘would use a little’.
5.8 Data analysis feedback meeting
The information from the focus groups, alongside the data from the preference
questionnaires acted as a guide to select the models to take forward to the piloting
stage of the research.
A meeting was held with the PEP and SEP to discuss the outcome of the data
analysis and make a decision on the models that were to be taken through to the
130
piloting stage of the research. The researcher diary was used to record the meeting
discussions.
Based on the data collated from the initial two focus groups and the preference
questionnaire a decision was made to take the models COMOIRA and AI into the
piloting stage of the research process.
5.9 Piloting of professional practice frameworks
This stage of the research involved the piloting of the two selected frameworks:
COMOIRA and AI. Each model was trialled for approximately a seven week period
and was used to evaluate three individual teacher consultations. EPs were provided
with an Information Sheet (see Appendix L) which provided examples of both the
COMOIRA and AI models. Following each piloting period each EP completed a
feedback questionnaire (see Appendix F). Evaluations of each of the models are now
be examined in turn.
5.9.1 COMOIRA Quantitative data analysis outcome
The post-pilot feedback questionnaires were analysed using SNAP software. This
generated statistics highlighting the number of responses in each category and this
was also presented as an overall percentage. The results are illustrated in Table 5.14.
131
Table 5.14: Post-pilot feedback questionnaire: COMOIRA
Please indicate to what extent you agree with the following statements.
Strongly Agree
% Agree %
Neither agree nor
disagree
% Disagree % Strongly disagree
% No
Response % Total %
I have found the model easy to use
2 16.7% 2 16.7% 3 25.0% 1 8.3% 0 0.0% 4 33.3% 12 100.0%
COMOIRA provides a useful structure / guide for the evaluation of consultation.
1 8.3% 4 33.3% 2 16.8% 1 8.3% 0 0.0% 4 33.3% 12 100.0%
COMOIRA hinders / restricts the EP’s working practices.
1 8.3% 1 8.3% 4 33.3% 2 16.8% 0 0.0% 4 33.3% 12 100.0%
132
Please indicate to what extent you agree with the following statements.
Strongly Agree
% Agree %
Neither agree nor
disagree
% Disagree % Strongly disagree
% No
Response % Total %
COMOIRA is effective at demonstrating that the consultation has made a difference.
1 8.3% 2 16.7% 5 41.7% 0 0.0% 0 0.0% 4 33.3% 12 100.0%
COMOIRA can evaluate individual consultations.
2 16.7% 3 25.0% 2 16.7% 1 8.3% 0 0.0% 4 33.3% 12 100.0%
133
Four of the post-pilot feedback questionnaires were not completed, the reasons for
which were unknown. Results revealed that half of the respondents either “agreed”
or “strongly agreed” that COMOIRA was an easy model to use. As the responses
were anonymous it was not possible to tell whether these responses had been from
the EPs who were supervisors of the Cardiff TEPs who would have been familiar
with the model. Five of the EPs who responded either “strongly agreed” or “agreed”
that COMOIRA provides a useful structure for the evaluation of consultation. Only
one of the respondents answered “disagree” to this question and none of the EPs
“strongly disagreed”. Interestingly, two of the respondents said that they “strongly
agreed” or “agreed” that ‘COMOIRA hinders/restricts the EPs’ working practices
although four of the EPs “neither agreed nor disagreed” with this question and two
EPs “disagreed”. When considering whether COMOIRA is effective at
demonstrating whether the consultation has made a difference, three of the
respondents either “strongly agreed” or “agreed” with five EPs “neither agreeing nor
disagreeing”. Responses to the question about whether COMOIRA can evaluate
individual consultations revealed that five EPs either “strongly agreed” or “agreed”
with this statement. Two of the respondents “neither agreed nor disagreed” and one
EP “strongly disagreed”.
5.9.2 COMOIRA Qualitative data analysis outcome
The qualitative data from the open-ended responses was summarised and used to
triangulate the data from the scaling questions. A summary of qualitative responses
for the open-ended questions is summarised in Table 5.15. All the data are presented
but in order to make them more accessible they were organised into themes
following a content analysis of the responses.
134
Table 5.15: Open-ended responses from the post-pilot COMOIRA Questionnaire
Question Themes Comments
What are the strengths of COMOIRA when using the model to evaluate consultation?
Evaluative tool
Really useful evaluation tool for the EP Knowing you are completing an evaluation means you think that way in the consultation Has helped me be clearer about me constructing hypotheses though these happen after events and through reflection It enables me to be clearer for next consultation Helped me know what I need to develop in my own skills
Change
It focuses on different ways of looking at things and encourages the idea that change will happen It is quite specific about what the changes area and who will make them It goes even further to review if the changes have happened and helped
Structure
Structure, Self-reflection - individual and joint Process useful for evaluation Structures: The questions I understand make sense to me Provides some prompts for during consultation. Provides same structure. Good for reflection Thorough Being able to choose the starting point and recognising the relationships between areas
135
Question Themes Comments
What are the limitations of COMOIRA when using the model to evaluate consultation?
Structure
Too structured I didn't understand all the terms I found it hard for both models - to post hoc apply a model Would be fine if I worked with COMOIRA probably
Evaluation
It doesn't really let you know if the consultation was effective for the other person as you don't ask them Sometimes the boxes don't make sense Too many boxes that you feel you need to fill in The model did not help my evaluation, instead felt I needed to fill in boxes
Information laden Could be perceived as too much information to process
Problem focussed
Does it focus on problems and things that need to be changed quite heavily? Perhaps a bit more of "what successful changes have already been made and recognition of positive ideas etc. that have made a difference/progress, may be helpful at some stage?
Process driven Can feel a bit process driven rather than person led
136
Question Themes Comments
What are your general thoughts about the application of COMOIRA when evaluating consultation?
Reflection
Would like to use it with the person I am consulting with at the end of the consultation because what I think went well and what is good for others might not be the same thing It is helpful to aid more structured reflection Would like to use it more within practice and think I would then be more skilled at using it to evaluate Very useful structured test allowing self-reflection and thinking following Consultation Links consultation to theory i.e. theory and practice Feels like it brings more psychology to what we do The core of COMOIRA is helpful on reflection of consultations Useful to have time to reflect and look/consider a range of factors that might be playing a role in a situation
Evaluation
“Have key people made the change and have these helped?" could be useful for evaluation and "what has happened so far and how has this gone” as both these may give some idea of outcomes The model may be more evaluative if completed all together
Training
As I am not COMOIRA trained, I found it hard use COMOIRA post hoc to review consultants for which I hadn't used COMOIRA. Does that make sense? I would need training in COMOIRA I am not sure I fully understand the model to use it effectively I felt it did not help me to evaluate and also felt I was repeating myself in different boxes Some questions were easier to answer than others
137
5.9.3 AI Quantitative data analysis outcome
The post-pilot feedback questionnaires were analysed using SNAP software. This
generated statistics highlighting the number of responses in each category and this
was also presented as an overall percentage. The results are illustrated in Table 5.16.
138
Table 5.16: Post-pilot feedback questionnaire: AI
Please indicate to what extent you agree with the following statements.
Strongly Agree
% Agree %
Neither agree nor
disagree
% Disagree % Strongly disagree
% No
Response % Total %
I have found the model easy to use
1 8.3% 3 25.0% 1 8.3% 2 16.7% 1 8.3% 4 33.3% 12 100.0%
Appreciate Inquiry provides a useful structure / guide for the evaluation of consultation.
0 0.0% 3 25.0% 1 8.3% 3 25.0% 1 8.3% 4 33.3% 12 100.0%
Appreciate Inquiry hinders / restricts the EP’s working practices.
0 0.0% 0 0.0% 2 16.7% 3 25.0% 3 25.0% 4 33.3% 12 100.0%
139
Please indicate to what extent you agree with the following statements.
Strongly Agree
% Agree %
Neither agree nor
disagree
% Disagree % Strongly disagree
% No
Response % Total %
Appreciate Inquiry is effective at demonstrating that the consultation has made a difference.
0 0.0% 4 33.3% 3 25.0% 1 8.3% 0 0.0% 4 33.3% 12 100.0%
Appreciate Inquiry can evaluate individual consultations.
0 0.0% 6 50.0% 2 16.7% 0 0.0% 0 0.0% 4 33.3% 12 100.0%
140
Four of the post-pilot feedback questionnaires were not completed for reasons which
are unknown to the researcher. Results revealed that half of the respondents either
agreed or strongly agreed that AI was an easy model to use with three of the EPs
“disagreeing” or “strongly disagreeing” with this statement. Whilst three of the
respondents “agreed” that AI provides a useful structure for the evaluation of
consultation, half of the EPs who responded “disagreed” or “strongly disagreed”
with this. Interestingly, six of the respondents said that they “strongly disagreed” or
“disagreed” that AI hinders/restricts EPs’ working practices. None of the EPs’
responses indicated that they believed AI was restrictive to EP practice.
When considering whether AI is effective at demonstrating whether the consultation
has made a difference, four of the respondents “agreed” with this statement and only
one EP “disagreed”. Responses to the question about whether AI can evaluate
individual consultations revealed that the majority of the EPs “agreed” that this was
the case. No EPs “disagreed” or “strongly disagreed” with this statement.
5.9.4 AI Qualitative data analysis outcome
The qualitative data from the open-ended responses was summarised and used to
triangulate the data from the scaling questions. A summary of qualitative responses
for the open-ended questions is summarised in Table 5.17 below. As with
COMOIRA, all the data are presented but in order to make them more accessible
they were organised into themes following a content analysis of the responses.
141
Table 5.17: Open-ended responses from the post-pilot AI Questionnaire Question Themes Comments
What are the strengths of AI when using the model to evaluate consultation?
Structure Opens up thinking Less structure Structure and simple
Develops practice Looks at developing practice - allows you to think about what you will do in the future - like the "what could be" bit
Applies psychology Useful application of positive psychology It is based on a social constructionist perspective
Positive focus
Makes you think positively about the consultation, look for ways it could be improved Makes you look for strengths rather than negatives Brings psychology Allows free thinking Positive focus
Simple Simple I am familiar with model and process
142
Question Themes Comments
What are the limitations of AI when using the model to evaluate consultation?
Interpretation of AI
Interpreting what thinking was required of me Some aspects don't seem to apply to some situations but this could be me getting used to the model
No weaknesses
There are no weaknesses if you accept the premise and creating the reality in the moment
Structure
Too loose to provide any dependability? Evaluation Too vague, open maybe? More focus/challenge? Lack of structure Doesn't really allow you to think about what went on as much as COMOIRA - needed more structure so harder to think it through Restrictive
143
Question Themes Comments
What are your general thoughts about the application of Appreciative Inquiry when evaluating consultation?
Training and practice
With practice/more use I think this will get better in that it would be an embedded part of thinking processes I don't feel I know enough about AI to be able to use effectively
Flexible
Yes generally a flexible positive model.
Evaluation
Quite difficult to evaluate with this tool Not sure how it could give us structured feedbacks about what we do. Very loose. It is sound in principle and practice. Care needs to be taken in separating the 'evaluation element' too distinctly.
Next steps
Would be good to see how teachers might respond to thinking about consultation like this Need to engage with others, not just post hoc, by me?
5.10 RADIO Stage 8: Processing information with stakeholders
As the research progressed the PEP and SEP had regular meetings with the
researcher in order to share the research findings following the data analysis process
and make decisions about the next steps. Members of the EPS were consulted
regularly during joint team meetings.
5.11 Phase Three
Phase Three of the research incorporated Stages 9 and10 of the RADIO process.
Phase Three consisted of two final focus groups, one for each EPS. The data
gathering methods, data analysis and outcomes are discussed in turn. The data for
this phase of the research were subject to more substantive data analysis due to the
fact that the data generated were used to address the research questions and to inform
Chapter Six of the thesis.
5.12 RADIO Stage 9: Agreeing areas for future action
Within this stage of the research process the data from the two final focus groups
were analysed using thematic analysis and feedback was given to the stakeholders.
5.12.1 Final focus groups
Both EPSs were asked to participate in a final focus group within their own EPS.
Each lasted approximately 60 minutes. The data generated from the discussions were
audio recorded with the participants’ consent. The questions were displayed on flip
chart paper and the researcher read these out in turn to the group (see Appendix J).
After allowing time for the EPs to discuss some working examples of how the
frameworks had been used for evaluation purposes, each question was then discussed
in turn. A record was made of the pertinent points and the researcher checked out
meaning and interpretation of the data with the participants. Clarification was sought
and the participants were given the opportunity to elaborate on key points and to
discuss the questions with colleagues which allowed for both individual and group
views to be obtained.
145
5.12.2 Data analysis
TA was carried out on the fully transcribed audio-recorded discussions from the two
focus groups (see Appendix M). This was conducted according to the six step phase
Braun and Clarke (2006) method outlined in Table 4.11.The themes from
COMOIRA and AI were considered separately. Emerging themes and concepts were
coded and revisited systematically to allow the assessment/reassessment of relevant
data.
For COMOIRA four themes were identified and AI yielded three major themes. The
researcher then created a thematic map of the emergent themes for both COMOIRA
and AI and subsequently thematic maps of each of the individual themes were
identified (see Appendix N). Given the researcher’s proximity to the research and for
validity and reliability purposes, the emergent themes and codes were checked by a
small group of EPs for inter-coder agreement. At this point, it was necessary for the
researcher to justify and explain their interpretation of the data. This process was
invaluable as it generated changes in the overall structure and labelling of the final
thematic map.
The models will now be considered separately and each theme identified and
discussed in turn.
5.12.3 Data analysis outcome: COMOIRA
Themes identified from the transcripts of COMOIRA will now be detailed. Within
the text the main codes resulting from the emergent themes are highlighted in bold
italics and the sub codes resulting from the subthemes are highlighted in normal text
italics hereafter. The first thematic map below (see Figure 5.1) details the themes
that emerged in relation to the strengths and limitations of COMOIRA, its ability to
provide outcomes which demonstrate accountability and preferred ways forward.
The subsequent thematic maps explore each individual theme and sub theme in
detail.
146
Figure 5.1: Thematic Map for COMOIRA
5.12.3.1 Change
The first subordinate theme change held a number of subthemes as highlighted in
Figure 5.2 below.
Figure 5.2: Change
COMOIRA
Change
Accountability
Applying
psychology
Future
implications
Change
Capturing
change
Who controls
change?
Barriers to change
Reframing
Using reflection
147
One theme of particular interest was that of change as it generated a great deal of
discussion around how EPs capture the changes made through consultation with one
respondent asking “ how do you capture the seeds that we place that then go away
and change systems?”
Another EP commented that often EPs are not aware of the changes that are made
following their involvement stating “the change could be that behaviour has become
normalised or the situation has got better so we don’t hear again, so those changes
we don’t really monitor do we”. To some extent, this notion of change being
difficult to identify or ‘capture’ adds weight to the rationale for a service evaluation
model of consultation.
The issue of who controls change was an interesting subtheme which generated
questions around whether some teachers had the power or control necessary to make
the changes that were identified during the consultation. EP comments included “is
the ability to change it even within their control?” and “how far are the key people
able to make change within the constraints of the system they are in”.
The theme of who controls change is important in relation to the consultation
process and the contribution the EP can make to this. Some of the EPs reflected on
who the consultations were conducted with and the influence they had within wider
systems. One EP noted that “you could have an amazing consultation with someone
but they’re in a system where they have no power to create change whatsoever and
then you are really consulting with the wrong person because what can they do?”
This highlighted how the systems around the consultee are especially important to
the subsequent implementation of the change process and could potentially be a
barrier to change. The barriers to change were considered to be a critical element
with the ‘willingness of the consultee to make the changes’ an important factor that
led to any subsequent changes that then occurred. One EP questioned why some
consultees do not want to implement the proposed changes, asking “what are their
anxieties about change; is there something they don’t want to give up.”
148
Interestingly, the barriers to change could be seen to have a direct impact on the
consultation process and whether the EP was able to contribute positively. Some
respondents saw the willingness of the consultee to implement suggested changes as
being fundamental to this “how far are people willing to make the changes and how
able are they? That sort of affects my practice and how the conversations might go.”
Therefore the interaction of the EP with the consultee can be seen to be of particular
importance and critical to the consultation process. This relationship and the
conversations that subsequently developed were identified as having a direct
influence on the outcome of the consultation and as to whether the consultee would
then make the necessary changes. One EP’s comments included: “I’m thinking about
how I interact, if it’s difficult for them (consultation), then you are not going to get
that change are you out of them?”
The questions used within COMOIRA were viewed by some EPs as being helpful to
the change process. The question: ‘Have key people made the changes and have
these helped’ was seen as a useful reflective tool that would encourage others to
focus on changes that have been made (see Appendix O). COMOIRA was also
considered to be a useful tool for reframing the problem which may then bring about
change in itself “a different perspective, a different view might make a difference in
their own behaviour regarding the situation.”
This angle of reflecting and reframing appears to be beneficial to the change process
as it encourages the positive notion that changes can be made and enables problems
to be viewed differently in terms of finding solutions based on what has been
successful previously.
149
5.12.3.2 Accountability
Figure 5.3: Accountability
Another important theme was related to accountability and the issue of whether
using COMOIRA to evaluate EP practice would demonstrate to the LA that the EPS
had made a difference through their consultations with others. COMOIRA was
considered a useful tool in that it would allow EPs to demonstrate changes that had
been made through their involvements. One respondent stated “COMOIRA shows
this is the change that has happened…to demonstrate we have had a positive effect.”
Gathering evidence around this positive effect was seen to be crucial to the issue of
accountability with the EPs discussing how to demonstrate the impact they make
through consultation and how to identify what a reasonable outcome would look
like. The issue of who EPs are accountable to and what they are accountable for
indicated that this needed further clarification. Questions arose such as “accountable
to the LA or to who we are working with?”
The explicit psychology within the model was seen as enhancing the accountability
of the service as it would help to answer questions around “what psychologists do”
and the differences EPs make through applying psychology. Respondents
Accountability
Explicit
psychology
Demonstrate
changes
Evidence
Reservation
150
commented “when we have got senior managers coming and saying “what is it that
your service offers? We have a paper trial” and “I think that it might be a good way
of capturing that we use psychology to make a difference.”
As well as providing a tool that captured the psychology used within consultation
COMOIRA was also recognised as a model that could be applied across other areas
of working to demonstrate accountability and provide evidence about what EPs do
within other parts of their practice, for example, group consultation and training were
mentioned. The difference in the approach of the EP, when compared to other
professionals, was considered to be related to the “psychology behind the question”
and COMOIRA was seen as a tool that could provide evidence for this explicitly.
One EP’s comments reflected this “it might enhance our accountability because of
the explicitly of the psychology. We don’t do that anywhere else.”
It is important to note that there were also some reservations about using this model
to demonstrate accountability due to its subjectivity and how it would be interpreted
by managers who were not psychologists. Consideration was given to the fact that
managers may be looking for hard measures to demonstrate the difference EPs make
e.g. “children’s learning measures”. If this was the case then some of the EPs did not
see COMOIRA as the “right tool to do that”. One EPs comments captured this
“personally I don’t think that it’s the tool to do this on its own.”
151
5.12.3.3 Applying Psychology
Figure 5.4: Applying Psychology
A clear and important theme that emerged around the COMOIRA model was related
to applying psychology. The transparency of the model and its clear link to the core
of psychology provided a tool whereby the application of psychology was “clearly
demonstrated for other people.” It appeared that the transparency the model
provided was a particularly valuable function of COMOIRA as it made the
psychology used within the consultation process explicit to the service users. One EP
noted that “it’s a strong tool because it makes the psychology very clear.”
The reflexive and reflective nature of the tool was viewed very positively as it
naturally lends itself to the evaluation process. Some of the respondents’ comments
included “I liked it as a reflective, evaluative tool” and “as a reflective tool it’s got a
lot about it.”
The reflexive element was seen as strength as it applied to the EP as well as the
consultee. This encouraged the EP not only to reflect upon the practice of others but
Applying
psychology
Written for EPs
Reflective/ reflexive
Transparency of
psychology
Social
constructionism
152
also to consider their contribution to the consultation process. Therefore, the
psychology was as applicable to the EPs’ practice as it was to the consultation
process. One respondent stated “another strength is its reflexive in the sense that the
psychology applied to us as much as it does to everyone else.”
The fact that the COMOIRA model was written for EPs by EPs was viewed as
strength and appeared to give the model credibility. It was seen to fit with EP
practice due to the fact that it “sits easily with social constructionism.” It was also
considered to facilitate psychology through the conversations that evolved when
implementing the model during the evaluation of the consultation. In this sense
COMOIRA could be seen to promote the application of psychology to the evaluation
process.
5.12.3.4 Future Implications
Figure 5.5: Future Implications
Under the theme of future implications it became apparent that further knowledge of
COMOIRA was considered necessary for some EPs if the model was going to be
taken forward and used for future evaluations. With this in mind, some EPs thought
it would be beneficial to use COMOIRA during their consultations as “it must be a
Future
implications
Time and systems
Adapt form
Trial with teachers
Knowledge
Evaluation
153
lot easier if you are doing COMOIRA, to evaluate using COMOIRA.” Further
suggestions included having a trial with teachers in order to gain and further the
EPs’ knowledge around COMOIRA. Using the model in future consultations with
teachers, rather than as a self-evaluation tool, was considered as one way forward.
EP comments included “that could be a possible way forward to try it at the end of a
consultation” and “I think you need to share it with the person you are working
with.”
The aspect of needing time and appropriate systems raised a fundamental issue in
terms of time constraints. Therefore, it would be important for managers to consider
the time required for further piloting and reviewing of the model as this was
considered to be crucial in order to take the model forward: “we need to build in
some time and some systems to follow it through properly otherwise it’s not worth
doing it.”
For one EP, the issue of time was related to the idea that COMOIRA was a
“laborious” tool and that having to record what was already “internalised” was
“another job to have to do”. This raises an issue with regard to how evaluation is
implemented so it is not viewed as additional work that is unnecessary.
Some of the EPs believed that COMOIRA could only be used as an evaluative tool if
this process was facilitated by an EP; otherwise the psychology behind the process of
consultation would be lost. One EP’s comments captured this:
we know there is a huge psychological theory that underpins that
(consultation) and that’s what we are tapping into by asking these questions
so if you are evaluating us, if someone without psychology is evaluating
using those questions you are going to get a very different evaluation.
Therefore, the EPs’ involvement in the evaluation process would need further
consideration as it was viewed as a necessity in terms of the capturing of the
psychology used within the consultation process.
154
In relation to the future use of COMOIRA as an evaluative tool, some of the EPs felt
there was scope for adapting the form to use with clients and this would be a useful
next step. Suggestions included “part of the next step could be to adapt the form for
use with clients” and “work needs to be done on it [the form] in order to make it fit
for purpose.”
This raises some further implications for future consideration. It may be necessary to
allow some time to ascertain whether the COMOIRA form encompasses all the
aspects required for the evaluation of consultation or whether further adaption is
necessary before it can be implemented with service users.
5.13 Data analysis outcome: AI
Themes identified from the transcripts of AI will now be detailed. Within the text the
emergent themes are highlighted in bold italics and the subthemes are highlighted in
normal text italics. The first thematic map (see Figure 5.6) details the themes that
emerged in relation to the strengths and limitations of AI, its ability to provide
outcomes which demonstrate accountability and preferred ways forward. The
subsequent thematic maps explore each individual theme and sub theme in detail.
Figure 5.6: Thematic Map for AI
Limitations
Evaluation
Strengths
AI
155
5.13.1 Evaluation
Figure 5.7: Evaluation
The first theme related to evaluation provided a lot of rich data and yielded a
number of sub-themes. The interpretation of the model for some EPs caused an
element of confusion. This mainly related to the way it was being used for evaluative
purposes. One EP was unsure whether they were reflecting upon “my dream or their
dream?” The language of the 4Ds (see Section 3.3.8) was cited by another EP as
“strange” and they were unclear about the dream question. Interestingly a lack of
knowledge of the model appeared to be a factor in this confusion as the same EP
commented that they knew very little about the model of AI. However, this
highlights the need for further training if the AI model is to be used in future EPS
evaluation.
Evaluation
Interpretation
Change
Future
implications
Potential
Part of a package
Further use
Confusion
Lack of Knowledge
Demonstrating accountability
156
One sub-theme of particular interest was related to the evaluation of change and how
this can be captured by an evaluation tool, with one respondent asking “what does
change look like and how do we capture it?”
This sub-theme raised the issue of how to assess the difference EPs make and this
was recognised as a particular area of difficulty. It was acknowledged that from an
interactionist perspective EPs are working with others to create change so there has
to be a willingness from the other person to “want to change.” This highlights the
ongoing challenge that faces the EP profession who are often working through others
to bring about change and are, therefore, dependent upon others to implement the
necessary changes. This led onto the issue of demonstrating accountability and the
question: How does AI help EPs to demonstrate that what they do makes a
difference? Interestingly, AI was seen as a tool that would enable EPs to “improve
their practice” in the future and “create positive change” in their role.
A strong sub-theme to emerge under evaluation was related to future implications.
AI was viewed by many of the EPs as having the potential to develop further and, as
a model, to cater for the differing styles of the EPs across the two teams. EP
comments included, “As a team of differing EPs who bring different theoretical
frameworks to the game I feel AI allows everyone to fit in” and “if we are looking at
which to move forward as an evaluative tool I would want to invest time in AI as it
has more potential as a tool.”
In order to explore how AI could be developed in the future for evaluative purposes
the EPs felt that it would require further use and “experimentation”. One EP
suggested a professional development session with worked examples so the
framework could be developed. Another EP felt it would be helpful to “populate it
with extra questions” which would help to “generate outcomes”. Some of the EPs
believed that the model would be best placed as part of an evaluative package. With
one EP suggesting that “it would need to be part of a bigger picture.” Another EP
thought that “it would work best as part of an evaluative package.”
This raises an important implication for the EPSs in relation to the allocation of time
to develop this model either as part of a package or for further piloting and reviewing
157
of the model for future evaluation purposes. Interestingly, certain EPs did not think
that the AI model required any further work and thought that it already fitted well
with the process of consultation. The same EPs felt that it had the potential on its
own to capture what EPs do through consultation. The framework was seen as
complementing “the process we often go through.” This was captured by one EPs
comments:
(the framework looks at) what is happening now that needs to stick around
or change, what it might look like, how you are going to get to that, what
you are going to put in place for it to be successful.
These contrasting opinions suggest that it may be beneficial to have further
discussions between the two EPSs to explore these views in more detail as these
views could potentially impact on the effectiveness of the future implementation of
AI if it was to be selected as an evaluative tool.
5.13.2 Strengths
Figure 5.8: Strengths
Strengths
Captures complexity
Structure
Positive psychology
Solution focused Reframing
Empowering
Creative
Change
158
The subordinate theme strengths also produced a number of sub-themes. AI was
seen as a tool that captures the complexity of consultation despite its simplicity.
Due to its “fluid”, “broad” nature it was seen as “open to wider interpretation”
which was thought to create a richer picture. The structure of AI was seen as
encouraging creativity as that there are no “right or wrong” answers. It appeared
that this open structure appealed to the EPs due to its inclusive nature which
allowed the integration of individual ways of working and, therefore, did not
restrict EP practice or force the EPs into a specific way of working. Interestingly
AI is a familiar model to many of the EPs due to recent training and is used by
some within their practice. It may be the familiarity of the model that instilled such
confidence in the model as many of the EPs were confident in the application of AI
approaches within their working practices.
The positive psychology used within the model and the solution focused element
was considered to be strength of the approach and matched with the belief systems
of the EPs. This was reflected in one of the EPs’ comments “there is a lot of
positive psychology, which I use a lot of anyway.”
Many of the EPs liked the emphasis on “thinking what went well”. This was seen
to be an important factor for encouraging change. Solution-focused theory is very
much part of consultation so AI can be seen to complement this process (Wagner,
1995). The language of the 4 Ds was considered to facilitate the evaluation of
consultation and facilitate a positive rather than “problem saturated” discovery
pattern. The emphasis on reframing enabled the EPs to have conversations that led
to a shift in the thinking of others rather than focusing on the problems. This could
be seen to focus the thinking on the positive differences that could be made rather
than ruminating over the negative issues. One respondent stated that “it makes
them think that a difference (positive) can be made.”
Interestingly, this led to AI being viewed as an empowering model, something that
could be used to “empower others within their own practice”. The positive, solution
focused psychology used within the model was considered to have the ability to
encourage others to look at “something in a way that enables them to make a
positive change” and this was considered to be very empowering. This is an
159
important finding when considering this issues that relate to EPs working through
others to try and bring about positive change.
5.13.3 Limitations
Figure 5.9: Limitations
It is important to note that some of the perceived strengths of AI were seen as
limitations by two of the EPs. In these instances, the framework was considered too
simple to capture the complexity of the problems that are presented to EPs. The
looseness of the model was seen as contributing to its simplicity and one of the EPs
said that they would benefit from a more structured model. Two EPs reflected on the
visionary nature of the model and their comments reflected that it was too visionary
and “so far removed from reality”. One of the EPs stated “it’s all very rose tinted
glasses.”
Having a vision that was too far removed from reality was thought to be “dispiriting
as it would never be achieved”. Another limitation was related to the fact that AI
does not “acknowledge people’s difficulties” or recognise concerns and one EP
considered this to be a fundamental missing element of this framework: “you have to
acknowledge people’s concerns somewhere.”
Limitations
Doesn’t recognise concerns
Too visionary
Too simple
Looseness
160
It may be important to explore these concerns further in order to move forward with
the AI model. It could be that adapting the form to incorporate an element that
recognises the concerns of the service users would help to allay some of these
perceived problems with AI. A further suggestion could be that additional training or
peer discussion may alleviate some of the EPs’ concerns.
5.14 RADIO Stage 10: action planning
The researcher originally intended to complete Stage 10 as part of the research
process. Following the data analysis of the final focus groups, the researcher
provided feedback to the PEP and SEP. It was originally proposed that, at this
meeting, a decision would be made around the professional practice frameworks and
the potential of the models as a future service evaluation tool. However, due to the
competing demands placed particularly upon the PEP and organisational factors
affecting the EPSs within the LAs during the final stages of this research, Stage 10
was not completed. It is crucial that Stage 10 is concluded in terms of moving the
research forward as this stage is needed to inform Stage 11: Implementing Action
and Stage 12: Evaluating Action (see section 4.13.1). In the researcher’s opinion an
important next step would be to consider whether the services are going to
implement one of the models. The findings suggest that the data are not conclusive
and indicated that some of the EPs had a preference towards COMOIRA and some
towards AI. Therefore, the researcher would recommend further piloting and data
gathering around the individual models. Additionally, further consideration could be
given as to whether the EPSs look at using more than one model or design a bespoke
evaluation tool based on these future research findings.
5.15 RADIO Stage 11 and Stage 12: implementing action and evaluating action
The researcher was aware at the start of the research that, due to time constraints,
Stages 11 and 12 of the RADIO framework would not be completed within the time
scale of this piece of research. These stages are due to be completed outside the
parameters of this study.
The next section of the research (Chapter Six) will consider how the results relate to
the research questions and literature.
161
Chapter Six: Discussion
6.1 Introduction
This study aimed to explore the most effective professional practice frameworks that
would assist a group of EPs’, across two EPSs, to successfully and appropriately
evaluate consultation and gather relevant data about the effectiveness of the EPSs for
the LA. Chapter Six considers the research findings and how these relate to the
research questions, with reference to the existing literature. RQ1a is addressed first
and then RQ1b is considered. The future implications arising from the research are
then considered. Following this the personal reflections of the researcher will be
discussed, followed by the theoretical contribution of this piece of research and the
limitations of the study. Finally, the wider implications of the study are explored.
The future research directions are outlined within Chapter Seven: Conclusion.
The study aimed to address the following research question:
RQ1: What are the EPs’ views on the most effective models of evaluating
consultation in relation to:
1a) Assessing consultation as a process?
1b) Providing outcomes which demonstrate accountability to employers
and service users?
An AR design was used to explore the research questions and this was conducted
using the RADIO framework (Timmins et al., 2003). Ten EPs from two EPSs
participated within the research alongside the PEP for both services and an SEP from
one of the services. The research was conducted as outlined within Table 4.1. It is
important to note that the data considered in this section are gathered from the final
phases of the project. Due to the AR design of the study the data gathered during the
earlier phases of the research was used to inform the next steps during Stages 1- 7 of
research process (see Sections 5.2-5.6). The information collected during the final
phase (Stage 9) of the research is the focus for this section of the study (see Sections
5.11-5.14).
162
6.2 RQ 1a and 1b
As outlined, the results for RQ1a and RQ1b derived mainly from the data generated
from the final focus groups. Alongside this, the data from the post-pilot
questionnaires was also consulted to support the validity of the focus group data (see
Sections 5.9-5.10). The final focus group questions generated a number of themes
that fit with the research questions and some themes that did not fit with the research
questions but had pertinent issues in relation to future implications. The themes
generated are presented in Table 6.1 and linked to RQ1a, 1b and also to the future
implications (see Section 6.5).
163
Table 6.1: Final Focus Group Themes
Professional Practice Model
Main Themes Sub-theme Research Question
Section
COMOIRA Applying Psychology
Reflective and reflexive
1a 6.3.1
Transparency of psychology
1a 6.3.1
Written for EPs 1a
6.3.1
COMOIRA Change
Capturing Change
1a 6.3.4
Barriers to Change
1a 6.3.4
Reframing Change 1a 6.3.4
Using Reflection
1a 6.3.4
Who Controls Change
1a 6.3.4
COMOIRA Accountability
Explicit psychology
1b 6.4.1
Evidence
1b 6.4.1
Reservation
1b 6.4.1
Demonstrate change
1b 6.4.1
COMOIRA Future Implications
Time and systems
Future Implications
6.5
Adapt form 6.5
Use as an evaluative tool 6.5
Knowledge 6.5
AI
Evaluation
Future implications
Future implications
6.5
Change
1a 6.3.4
Interpretation Future Implications
6.5
AI Strengths
Structure 1a & Future Implications
6.3.2; 6.5
Captures complexity
1a 6.3.2
Positive psychology
1a
6.3.2
164
Professional Practice Model
Main Themes Sub-theme Research Question
Section
AI Limitations
Too visionary
1a 6.3.4
Too simple
1a 6.3.2
Doesn’t recognise concerns
1a 6.3.4
6.3 RQ1a: : What are the EPs’ views on the most effective models of evaluating
consultation in relation to assessing consultation as a process
As previously outlined, the results for RQ1a predominantly derived from the final
focus groups. The data from the post-pilot questionnaires were also consulted to
support the validity of the focus group data. The data analysis feedback meeting
outlined in Section 5.8 selected COMOIRA and AI as the models that would be
looked at specifically within this study. Following the piloting of each model, the
EPs considered what they thought about the models of COMOIRA and AI in relation
to the evaluation of individual teacher consultations. It was not within the scope of
the thesis to discuss each theme that was generated in detail; rather this section will
focus specifically on the themes that appeared to have the most significance for the
service. In terms of COMOIRA, the themes: applying psychology and change were
the most significant. In terms of AI the theme: strengths was the most significant. In
addition, a number of AI sub themes were linked to RQ1a and the main themes
resulting from COMOIRA. Therefore, the themes applying psychology, strengths
and change were identified as being most pertinent following a process where the
researcher and fellow doctoral colleagues considered the dataset and selected the
information most relevant to the service. These themes were shared with
stakeholders. Table 6.3 below outlines the themes and links them with the dataset.
165
Table 6.2: RQ1a - Links to the dataset
RQ1a: What are the EPs’ views on the most effective models of evaluating consultation in relation to assessing consultation as a process
Theme COMOIRA
Theme COMOIRA Sub-theme
Section AI Theme AI Sub- theme
Section
Applying psychology
Applying psychology
Reflective/reflexive Transparency of Psychology Written for EPs
5.12.3.3
Change Change
Capturing change Reframing Change Barriers to change Who controls change
5.12.3.1
Strengths Evaluation
Positive psychology Change
5.13.2 5.13.1
Strengths
Strengths Limitations
Creative Captures complexity Positive psychology Structure Too simple Too visionary
5.13.2 5.13.3
6.3.1 COMOIRA: Applying psychology
When considering whether the COMOIRA model was able to assess consultation as
a process, one theme of particular interest was that of applying psychology (see
Section 5.12.3.3).
The EPs considered that one of the ways that COMOIRA lends itself to the
assessment of consultation as a process is in relation to the transparency of
psychology provided through its psychological core (Gameson et al., 2003). The
transparency of the COMOIRA model was seen by many of the EPs as making the
psychology of EP practice more explicit and, therefore, demonstrable to other
166
people. Furthermore, it was seen as facilitating psychology through the conversations
that evolved whilst implementing the model. This was considered to promote the
application of psychology to the evaluation process. Therefore, it could be suggested
that this emphasis focused the evaluation of the consultation explicitly on the
psychology that had been used within the consultation. This research suggests that,
COMOIRA could potentially provide a tool whereby the psychological knowledge
of the EP could be utilised to shape the evaluation of the consultation; and enable the
EP to effectively communicate the psychological principles behind the consultation
process.
The fact that it was written by EPs for EPs was considered to be strength of
COMOIRA and this was seen to give the model credibility. There was a common
belief amongst the EPs that the reflexive element of COMOIRA was a particularly
useful tool (see Section 5.12.3.3). The reflexive questions were considered to
provide opportunities for EPs to reflect upon their own practice and to instigate them
to look at ways to develop their practice in the future. This is in line with Kelly et
al.’s research (2008) that discusses how COMOIRA encourages EPs to use
psychology explicitly in order to guide their practice. Another one of the key aspects
of the COMOIRA model is its emphasis on reflection and the application of
reflective psychology (Gameson & Rhydderch, 2010). In this study, these elements
encouraged the EPs to reflect upon their own practice within the consultation, so
from an evaluation viewpoint it was seen to be as relevant for the EP as the
consultee. Some of the EPs considered how using COMOIRA as an evaluative tool
had helped them to be clearer around constructing hypotheses, due to the focus on
reflection, and this then guided thinking around future consultations.
6.3.2 Appreciative Inquiry: Strengths
When considering whether the AI model was able to assess consultation as a process
one theme of particular interest was related to the strengths of the model (see Section
5.13.2). Within AI, the positive psychology which is central to the model was
considered to be a particular strength, cited as being complimentary to EP practice
and as fitting with the belief systems of many of the EPs. Additionally, the solution
focused element that it offered was seen to fit with the psychological approaches
used during consultation sessions. Despite its simple structure, AI was seen as
167
encouraging creativity which appeared to allow the EPs the flexibility to integrate
their own ways of working as psychologists when assessing the consultations and it
was not seen as restricting individual practice.
The research of Wimbush and Watson (2000) considered one possible way to
formalise high quality evaluation was to integrate it through the implementation of
transparent practice frameworks. However, it was considered to be essential that
such a framework would encapsulate the complexity of the EP role. The systematic
literature review that was conducted as part of the research (see Section 5.5.1)
ensured that the selected models would complement the process of consultation but it
was uncertain whether they could capture and evaluate the complexity of the
consultation process. The findings from this research demonstrate that, interestingly,
for some EPs, AI was seen as a tool that was able to capture the complexity of
consultation due to its ‘broad’ nature (see Section 5.13.2). Some of the EPs reported
that, it’s open simple structure was one of the strengths of AI. The fluid, broad nature
of the model was seen as being open to interpretation and thus creating a richer
picture. This is a valuable insight when considering AI in relation to the assessment
of consultation as a process, as it would suggest that, as an evaluative tool, AI has
the potential to capture the complexity of the consultation process and the
psychology applied within the consultation. However, it is important to note that the
simplicity and looseness of AI was considered by two EPs to be a limitation and AI
would benefit from a more structured approach when applying it to the assessment of
consultation.
6.3.3 Application of psychology within the assessment of consultation
The findings from the themes COMOIRA: applying psychology and AI: strengths
suggest that using such frameworks within the assessment of consultation as a
process may contribute towards the recommendations for EPs to communicate their
psychological knowledge (BPS, 2007). The application of psychology is seen to be a
fundamental part of the EP role as highlighted in the reviews on the role of the EP
(DfEE, 2000; Farrell et al., 2006; Scottish Executive, 2002). Indeed, such reviews
called for a more sound use of psychological knowledge across the variety of
contexts in which EPs work (Farrell et al., 2006) and for the ‘conceptualisation’ of
the role as that of ‘scientist-practitioner’(Fallon et al., 2010; Lane & Corrie, 2006). It
168
could be suggested from these findings that professional practice frameworks may
potentially provide a resource that links the application of psychology to the
evaluation of consultation (Frederickson & Miller, 2008; Gameson et al., 2003). As
such, implementing such models for evaluative purposes may offer one way for EPs
to demonstrate that their professional practice is informed by psychological theories
(Gameson et al., 2003).
6.3.4. Change
Another interesting theme that emerged from the COMOIRA data with regard to
assessing consultation as a process was in relation to change. The issue of change
also occurred as a subtheme across two of the main themes within the AI dataset:
evaluation and strengths. Therefore, the data across both the COMOIRA and AI
models relating to change and the assessment of consultation as a process will be
presented in the following section.
The majority of the EPs identified the issue of change as one that produces a
significant challenge for EPs, particularly in terms of how to capture and measure the
changes that occur during and following consultation. This supports the research of
Pilgrim (2007) which found that working consultatively leaves EPs with the
challenge of defining outcomes that are measurable and can show a positive impact.
The issue of how to measure the impact of EPs has been greatly debated within the
literature (Cherry, 1998; Dowling & Leibowitz, 1994; Dunsmuir et al., 2009; Sharp
et al., 2010; Turner, 2010) particularly in relation to the complexity around what is
measurable and whether what is measured correlates with the actual outcomes and
whether these can be linked to the EPs’ contribution in the process (Cherry, 1998;
HEPS, 2010; Sharp et al., 2000; Turner et al., 2010).
The questions used within COMOIRA were seen by some EPs as being helpful to
the change process because they encouraged the focus to be specifically on the
changes that had occurred from the consultation. Therefore, COMOIRA could
potentially enable EPs to demonstrate the changes that they had made as a result of
their consultations via the process of reframing the problem (see Section 5.12.3.1).
Gameson and Rhydderch (2010) encourage the adaptation of the formal language of
the model and provide examples of how the language of the model could be adapted
169
(Appendix O). These suggested adaptations were used by some of the EPs in the
study and the focus of the questions within this framework, for example: ‘Have key
people made the changes and have these helped?’ were considered to be helpful and
seen to encourage others to focus on the changes that had been made. This appeared
to be related to the fact that COMOIRA facilitates the reframing of the problem
which can then lead to constructive changes occurring. Furthermore, the specific
emphasis of the questions around the change process meant that during the
evaluation of the consultation there was a focus on what changes had happened and
what had facilitated this course of change. This is in line with the research of
Gameson et al. (2003) and Gameson and Rhyderrch (2010) which highlights how
COMIORA focuses on promoting, monitoring and evaluating change. Therefore,
this angle of using reflection could be interpreted as being beneficial to the change
process as it encourages the notion that changes can be made and it enables problems
to be viewed differently.
With AI, the positive emphasis on thinking about what went well during the
consultation was seen as strength and an important factor for facilitating the change
process. The positive psychology used within the model through the language of the
4 D’s was considered to be a strength and encouraged a positive rather than problem
saturated discovery pattern. This solution focused emphasis is very much part of
consultation so AI can be seen to complement this process (Wagner, 1995). Again,
as with COMOIRA, the emphasis within the AI model on reframing was considered
to be a strength and something that would enable the EPs to have conversations that
led to a shift in the thinking of others. Consequently, it was seen that this would
allow the consultee to believe that positive changes could be made. This element of
AI was considered to be empowering and something that could be used to “empower
others within their own practice” (see Section 5.13.2). Interestingly, for some of the
EPs the positive emphasis of AI was viewed as a limitation and seen as too visionary
(Section 5.13.3). The fact that it was considered to be so far removed from reality
was thought to be dispiriting for the consultee as it was offering something that was
unachievable. The exclusion of the recognition of the consultee’s concerns was
considered by some EPs to be a fundamental missing element of this framework.
170
Within this research the issue of barriers to change, raised within the COMOIRA
focus groups, offered an insight into the factors that can be instrumental where no
subsequent changes occur following a consultation. This further contributes to the
difficulties associated with assessing the impact of the EP during consultation. The
EPs considered how one specific complication when evaluating consultation is that it
is often the consultee rather than the EP who implements the recommendations made
which leaves EPSs with the challenge of demonstrating what difference they have
made. The issue of how to measure the impact for a client group with whom EPs do
not work directly with has been greatly debated within previous literature (Cherry,
1998; Dowling and Leibowitz, 1994; Dunsmuir et al., 2009; Sharp et al., 2000;
Turner et al., 2010). The issue of barriers to change was considered to have a direct
impact on the consultation process and as to whether the EP could contribute
positively to this.
Within the AI dataset for evaluation (see Section 5.13.1) it was acknowledged that
from an interactionist perspective EPs are working with others to create change so
there has to be a willingness from the other person to want to change. The readiness
of the consultee to make the necessary changes was perceived as having a direct
influence upon the success of consultations themselves. This finding is particularly
pertinent when considering how to assess consultation as a process and highlights
the relevance of the literature around individual reactions to the change process.
Bovey and Hede (2001) conducted a piece of organisational research in Australia in
nine organisations undergoing major change. Despite a number of limitations in
relation to the methodology used the findings are interesting. They highlight how
certain individuals react to the change process and the defence mechanisms that can
be triggered when individuals are faced with change, for example, resistance. When
considering this piece of research in relation to EP practice, it demonstrates how
important it is for EPs to understand the human factors linked with reluctance to
change and how these can impact on the success of the consultation process.
In addition, when considering the willingness of the consultee to engage in the
consultation process, the interaction of the EP with the consultee was seen to have a
direct influence on the outcome of the consultation and as to whether the consultee
would go on to make the necessary changes. This finding within the COMOIRA
171
dataset (see Section 5.12.3.1), was of particular interest as it appeared that a
particularly important function of any future evaluative framework is that it should
emphasise reflexive thinking and develop this within EP evaluation. This encourages
the EPs to consider their role in the change process and supports the findings of
Gameson et al. (2003) that is not acceptable to attribute the lack of change to the
service user, without considering the EP’s part in the process. Furthermore, it may be
beneficial for the service user and practitioner to monitor and evaluate changes in
relation to their co-constructed questions and change issues. Gameson and
Rhydderch (2008) consider how one of the eight key decision points within the
COMOIRA model: Construct and Explore Relevant Hypotheses, could potentially
enable EPs and service users to explore together the implications of their potentially
competing hypotheses which may be an inhibiter of the change process and present
difficulties within the collaborative working process. This is particularly relevant
when considering these findings and suggests that COMOIRA could potentially
provide a tool for the EPs in the two LAs to begin to address this particular aspect of
the barriers to change.
A further element associated with change, within the COMOIRA dataset, was in
relation to who is in control of the change (see Section 5.12.3.1) and how far is the
consultee able to make changes within the constraints of the systems they are in.
Consideration was given as to whether some of the consultees were part of a system
where they did not have the power to create change and, therefore, whether the EP
was consulting with the right person. The findings of this research suggest that the
use of professional practice models as an evaluation tool within EP practice could
potentially help with these change issues specifically if they were implemented as an
evaluative tool in collaboration with the consultee. The frameworks could be
implemented as a tool to examine whether changes had been made following the
consultation and, if not, explore the reasons for this. In addition, these findings
highlight how important it is for the EP to try and develop an understanding of the
organisations in which they work, as the organisational culture often determines how
receptive the organisation (school) will be to the suggested intervention (Bryan,
Klein & Elias, 2007). If EPs can begin to understand the contextual factors of the
organisation, for example, the common beliefs and attitudes, they can begin to
explore and evaluate how certain schools may react to the recommendations
172
generated from consultations that are related to organisational change factors (Bryan
et al., 2007; Schein, 1998) and how this then impacts upon the staff members within
the school. Therefore, it seems important for EPs to try and develop an
understanding of the impact that organisational culture can place upon the members
of staff with whom they work (Bryan et al., 2007). This insight would be an
important element in the evaluation process.
6.4 RQ1b: What are the EPs’ views on the most effective models of evaluating
consultation in relation to providing outcomes which demonstrate
accountability to employers and service users
The results for RQ1b again predominantly derived from the final focus groups. The
data from the post-pilot questionnaires were also consulted to support the validity of
the focus group data. Following the piloting of COMOIRA and AI, the EPs
considered what they thought about the models in relation to the evaluation of
individual consultations and their ability to provide outcomes which demonstrate
accountability to employers and service users. As with RQ1a, it was not within the
scope of the thesis to discuss each theme that was generated in detail; rather this
section will focus specifically on the themes that appeared to have the most
significance for the service. The themes: accountability and evaluation were
observed as the most noteworthy due to their overarching quality. These themes
were identified as being most pertinent for the service following a process where the
researcher and fellow doctoral colleagues considered the dataset and selected the
most relevant information. Table 6.4 below outlines the themes and links them with
the dataset.
173
Table 6.3: RQ1b - Links to the dataset
RQ1b: What are the EPs’ views on the most effective models of evaluating consultation in relation to providing outcomes which demonstrate accountability to employers and service
users
Theme COMOIRA Theme
COMOIRA Sub-theme
Section AI Theme AI Sub- theme
Section
Accountability Accountability
Explicit psychology Evidence Demonstrate changes Reservation
5.12.3.2 Evaluation
Demonstrating accountability
5.13.1-
Evaluation Future Implications
Use as an evaluative tool
5.12.3.4 Evaluation Interpretation
5.13.1
6.4.1 COMOIRA: Accountability
When considering whether the COMOIRA model was able to provide outcomes
which demonstrate accountability to employers and service users, one theme of
particular interest was that of accountability (Section 5.12.3.2). Up until relatively
recently, EPSs within the UK had been protected from market pressures due to their
statutory role (Pugh, 2010). However, in the past few years there has been a clearer
focus on offering value for money within public services and the outcomes of EPs
work has come under scrutiny with increased accountability to stakeholders (Fallon
et al., 2010). In recent years contextual pressures have arisen within the EPSs where
this study was conducted and questions asked around whether other agencies can do
the job of an EP (Cameron, 2006; Farrell et al., 2006). This created the impetus from
the PEP for this piece of research to look for ways to provide outcomes that
demonstrate accountability to employers and service users.
Interestingly, within this study COMOIRA was viewed by some EPs as a model that
could potentially provide an evaluative tool that would enhance the accountability of
the service, through the explicit psychology present within the model (see Section
5.12.3.2).The explicit psychology was seen as enhancing the accountability of the
service to employers and service users as it would help to answer questions around
‘what psychologists do’ and ‘the differences EPs make’. COMOIRA was seen to be
174
a good way of capturing the psychology that EPs use and as providing a tool that
would enable EPs to demonstrate the changes and positive effects that had been
made through their involvements. These findings indicate that, as an evaluative tool,
COMOIRA could potentially provide clear and explicit evidence of the psychology
used within consultation, through the application of the core psychological principles
of: social constructionism, systemic thinking, enabling dialogue and informed
reasoned action. It was also suggested that it could possibly offer a paper trail for
managers to present as evidence to senior managers about the distinctive
contribution of the EPSs. Furthermore, the conversations that evolved when
implementing COMOIRA as an evaluative tool demonstrated that many of the EPs
believed that it could be applied at a number of different levels, not just during
consultations with teachers. Therefore, these findings suggest that COMOIRA may
have the potential, as an evaluative tool, to demonstrate accountability and provide
evidence about what EPs do within other parts of their practice, for example, group
consultation (see Section 5.12.3.2).
However, it is important to note that not all of the EPs were in agreement with the
ability of COMOIRA to demonstrate accountability and one of the EPs felt that
COMOIRA, as a tool on its own, was unable to do this (see Section 5.12.3.2). A
further reservation levied at COMOIRA in relation to its ability to demonstrate
accountability was with regard to its subjectivity and how it would be interpreted by
managers who were not psychologists. It was recognised that that managers may be
looking for hard measures to demonstrate the difference EPs make and, in view of
this, some of the EPs felt that COMOIRA was not the right tool to do this. Taking
this into account, a possible suggested way forward may be to continue to pilot the
models for a period of time to collect further data within this area. Equally,
discussions could be had with stakeholders to emphasise the importance of looking
beyond easily measurable outcomes which are not necessarily robust enough to
capture the complexity of the consultation process. This is in line with the findings
of Turner et al. (2010) that emphasise how such information does not shed enough
light on the input and effectiveness of an EP.
175
6.4.2 Appreciative Inquiry: Evaluation
In relation to AI the issue of demonstrating accountability was a sub-theme within
evaluation (see Section 5.13.1). AI was seen by the EPs as a tool that would enable
EPs to “improve their practice” and “create positive change”. This point was not
elaborated upon, but it appeared that the positive psychology in the AI model
encouraged the reframing of problems and focused thinking around what could be
done differently to improve practice to bring about positive change. Such evidence
could then be provided as evidence to demonstrate accountability to services users
and employers.
6.4.3 Demonstrating accountability
As Norwich (2000) purports, the challenge that EPs have faced over the years of
adapting theory to practice and the gap between theory and practice is one of the
reasons put forward within the literature for the lack of clarity around the role of the
EP. Indeed, the two EPSs involved in this research are currently trying to provide
evidence around their unique role and their distinctive contribution for employers.
These findings indicate that the models of COMOIRA and AI could be seen to
potentially offer a way for the EPs to bridge the link between theory and practice and
may conceivably be a way of to provide clarity around the EP role to employers and
service users. This is in line with the work of Kelly et al. (2008), which argues that
practice frameworks are an essential resource for the EP profession as they provide a
means of addressing the challenge that EPs face with regard to clarifying the
complex relationship between theory and practice and enable the integration and
cohesion of complex theoretical perspectives with equally complex practice
methodology. Therefore, the models of COMOIRA and AI could provide a potential
way of helping to provide answers and evidence to employers and service users
around what psychologists do in comparison to other professionals (Kelly et al.,
2006; Wicks, 2013) and, as identified within previous research, secure the centrality
of psychology and the distinctiveness of the EPs’ contribution (Fallon et al., 2010;
Frederickson & Miller, 2008; Gameson et al., 2005; Leadbetter, 2005; Kelly et al.,
2008). Furthermore, the implementation of professional practice models to evaluate
what EPs do would offer a further way to meet one of the central functions of
educational psychology which is to “communicate psychological knowledge” (BPS,
2007, p.5).
176
6.4.4 Evaluation
Evaluation was a theme that emerged from the AI data and it generated a great deal
of discussion around the ability of the model to evaluate change which is explored in
detail within Section 6.3.4; and the future implications of the model which is
discussed in Section 6.5. However, in relation to RQ1b and the ability of AI to
provide outcomes which demonstrate accountability to employers and service users
it is important to highlight the concerns of some of the EPs around the AI model in
relation to evaluation (Section 5.13.1). For some EPs, the interpretation of the AI
framework was considered to be confusing in relation to the way it was being used
for evaluative purposes. Interestingly, a lack of knowledge of the model appeared to
be a contributing factor as the same EPs commented that they knew very little about
the model. This issues arising from this are explored within the future implications
section (Section 6.5).
Within the COMOIRA dataset evaluation was a sub-theme within the future
implications theme (Section 5.12.3.4). In relation to evaluation process itself, the
EPs’ involvement in the process was seen to be pivotal in order for the psychology
not to be lost. Indeed, both the AI and COMOIRA models are theoretically complex
and, therefore, it would seem advisable for an EP to be part of the evaluation to make
it accessible to the consultee. The EP would need to facilitate the evaluation
procedure in relation to: the translating of the psychology behind the model, guiding
the process of self-reflection and the decision making around future direction.
Without this, the ability of the models to demonstrate accountability could be lost.
Some of the EPs suggested that the evaluation of the consultation should be a
collaborative process undertaken alongside the consultee following the meeting.
This is in line with the research of Kelly et al. (2008) that the review and evaluation
of consultation is key aspect of individual consultations and should take place at
each distinct consultation meeting with the aim of reviewing and evaluating the
meeting. As highlighted by the work of Gameson and Rhydderch (2010), one of the
distinctive features of COMOIRA is a commitment to embedding on-going
evaluation throughout the EPs’ work with service users. Implementing this finding
would enable the EPSs to generate outcomes at every consultation which would then
demonstrate accountability for service users. This information could then be collated
177
across the two EPSs and presented to stakeholders. Therefore, these findings are
potentially significant for the EPSs within this study who are under increased
scrutiny due to the current political and economic climate with stakeholders making
decisions about the future of services based on reduced budgets (Fallon et al., 2010;
Farrell et al., 2006). Furthermore, using the models as collaborative evaluative tools,
alongside service users, would enable EPs to find out whether what they were
delivering was meeting the needs of the service users. This is a useful consideration
as previous literature has highlighted the disparities between what EPs want to
deliver and what schools want (Ashton & Roberts, 2006; Mackay, 2002). Therefore,
these models can be seen to potentially provide a way for EPs to: constantly reflect
upon their own practice, consider how this can be improved and as a tool that
facilitates them to meet the needs of their client group.
The theme of evaluation generated a number of issues which need further
consideration by service managers. Conducting further trails in collaboration with
the consultee was offered as a way to develop the COMOIRA model in the future
and seen as an important aspect to furthering EP knowledge of the model and its role
in the evaluation of consultation. Therefore, it may be necessary for further research
to be conducted using the models collaboratively with service users, in order to
consider fully their potential as frameworks which can demonstrate accountability to
employers and service users. In addition, further training across the EPSs may be
required. These issues are explored further within Section 6.5.
However, these initial findings are interesting and it may be that, following further
research, these professional practice frameworks may have the potential to formalise
the evaluation of individual consultations and they could potentially provide a way
that EPs can demonstrate evidence about their performance and the results they
produce (Farrell et al., 2006).
6.5 Future Implications
The final focus group data generated a number of themes that had pertinent issues in
relation to future implications (see Table 6.4 below). The issues arising from the
future implications dataset will now be considered.
178
Table 6.4: Future implications
Future implications for COMOIRA and AI
COMOIRA Theme
COMOIRA Sub-theme
Section AI Theme AI Sub- theme Section
Future implications
Time and systems Evaluation Knowledge
5.12.3.4 Evaluation
Future implications Interpretation
5.13.1
Accountability Evidence 5.12.3.2 Strengths Structure
5.13.2
When considering future implications it appeared that a deeper understanding of the
models was required by certain EPs in order for them to be adopted as an evaluative
tool. Within the AI theme of evaluation, the interpretation of the language of AI
caused an element of confusion for some EPs which mainly related to the way it was
being used for evaluative purposes (Section 5.13.1). Interestingly, a lack of
knowledge of the model appeared to be one of the factors behind this confusion as
the same EPs commented that they knew very little about the model of AI. In
relation to the COMOIRA model some of the EPs considered that further knowledge
of the model was required in order for them to use it in the future for evaluative
purposes (see Section 5.12.3.4). For some of the EPs, the idea of using COMOIRA
as a framework during consultation was considered as something that would be
beneficial for professional development purposes and to enhance their knowledge of
the model (see Section 5.12.3.4). Therefore, it is apparent that further training
implications exist if either model is to be implemented in the future. This training
may benefit from strategic planning and consideration would need to be given to
follow up sessions and supervision.
With both frameworks some adaptations to the models were considered necessary by
some EPs in order to move them forward and to make them fit for purpose. For
COMOIRA, some of the EPs thought that there was scope in adapting the form to
use with clients and this would be a useful next step within the evaluation process
(see Section 5.12.3.4). This reflects the suggestions made by Gameson and
179
Rhydderch (2010) for applied psychologists to adapt and personalise the COMOIRA
model for use in their local context. For AI one recommendation was to “populate it
with extra questions” (Section 5.13.1) which could possibly help to generate more
measurable quantitative outcomes. It was suggested that one possible way forward
would be to set up professional development sessions and look through worked
examples so the frameworks could be developed. A further suggestion for AI was
that it would be best placed as part of an evaluative package which would help to
build up a richer evaluative tool (Section 5.13.1). Additionally, with both
frameworks, some of the EPs felt that they would require further piloting and
experimentation in order to consider thoroughly their future scope for evaluation.
The demands of time were illuminated throughout the discussions. The potential
investment of time and appropriate systems (see Section 5.12.3.4) for further training
and piloting of the models could be viewed as a potential barrier for the development
of any future evaluation tool. The dissonance between what training is required and
how this could be managed within current time and financial restraints is a concern
in terms of moving both of the models forward. Furthermore, the recent competing
demands and priorities of the EPSs are a concern. These competing demands have
heightened due to financial pressures placed upon the EPSs with a view to transition
into traded services; however, this also adds weight to the rationale for the
development of a service evaluation model of consultation.
When considering the future potential of COMOIRA and AI as evaluative tools, the
findings from this research suggested that both models could be applied across a
number of different levels of EP practice. For COMOIRA, group consultation was
highlighted which indicates that COMOIRA could possibly be adopted not just for
individual teacher consultations but to evaluate consultation on a wider level which
is aimed at the level of the individual, group or organisation as discussed in Section
2.6.2 (Gutkin & Curtis, 1999; Wagner, 1995). Training was another area identified
as a specific area of work that COMOIRA could be used to evaluate. The fact that AI
was open to such wide interpretation and had a natural structure which encouraged
creativity (see Section 5.13.2), could be perceived as providing it with the potential
to be implemented across a range of EP practice. Interestingly, this finding indicates
that the models could possibly be used as an evaluative tool within other areas EPs
180
work to define and showcase the range of skills beyond consultation to employers
and service users.
6.6 Personal reflections
6.6.1 Action Research model
The close and collaborative relationship between the researcher and participants was
an important aspect of this piece of research, however, AR can be criticised for this
relationship and the effect it can have upon the research process. The researcher tried
to ensure that she remained reflective and considered the impact she potentially
brought to the research process. Fortunately, during the study, no challenges arose
within the group of EPs in relation to the research which can be an issue within AR
(Kidd & Kral, 2005). During the piloting phase of the frameworks, some EPs used
the frameworks for fewer consultations than was originally requested. This
introduced a challenge to the research procedure, however, it was hoped that by
incorporating a multi-methods approach offered validity to the data. The data
generated from the focus groups were triangulated against the questionnaire data
which looked to increase the likelihood of the reliability of the findings.
When reflecting upon the choice of the AR design for this piece of research, the
researcher considered that it proved to be a valuable methodology. The collaborative
approach created opportunities for joint decision making and contributed towards a
shared vision for the development of a future evaluation framework within the EPSs.
Furthermore, the RADIO structure provided a framework whereby stakeholders were
involved in the research process. Its emphasis on collaboration promoted a less
manager-directed approach to service evaluation and enabled the EPs to play a part
in service development issues. This is in line with the findings of (Timmins et al.,
2003). RADIO proved useful in facilitating communication between all those
involved in the process through the provision of its clear research structure. It proved
valuable in identifying the needs of the EPS, through Stages 1-4 of the process, and
it provided a framework whereby the researcher and stakeholders regularly came
together to reflect upon the outcomes of the various stages of the research and use
these to inform the next steps. As with the findings of Timmins et al. (2003), RADIO
provided a valuable framework for negotiation and addressed the need for a dynamic
181
research process that was able to respond to the range of perspectives within the
EPSs.
In terms of personal reflection in relation to the researchers’ own practice, this piece
of AR has offered the researcher an insight into how organisational theory could
prove to be a valuable area for future professional development. It has demonstrated
to the researcher how important it is to try and develop an understanding of the
school organisations in which EPs work and the impact of that organisational culture
upon the members of staff with whom EPs work (Bryan et al., 2007). It seems that
this understanding is a key element when evaluating the changes that occur through
consultation (Bryan et al., 2007; Schein, 1996).
6.6.2 Future implementation
In terms of the RADIO model, Stages 10-12 were not addressed. This was a result of
the time scale of the research project and due to the competing demands placed upon
the EPSs. This is discussed further in section 6.8.3. As this was a piece of AR, the
researcher has not made any decisions around the potential future model that should
be taken forward as a service evaluation tool but, instead, will offer some personal
reflections. In the researcher’s opinion the findings from this study have identified
some interesting implications (see Section 6.9) and highlighted some important
issues for the EPSs to consider. In addition, it has also illustrated that further
research needs to be done within this area.
One particularly interesting observation when the EPs were asked to consider the
future implementation of both models as evaluative tools was that there was a
divided opinion between the EPs. AI was viewed by some of the EPs as having the
most potential to develop further as it was seen to cater for the differing styles of the
EPs across the two teams. When thinking of ways forward, these EPs wanted to
invest time in AI as an evaluative tool. However, two EPs believed AI to be too
simple to capture the complexity of consultation and commented on its visionary
nature being too far removed from reality (see Section 5.13.3). COMOIRA was
viewed by some EPs as a useful tool for evaluation due to the reflective element and
its ability to capture the psychology used within the consultation. These EPs thought
182
that COMOIRA would provide a way to capture the psychology that EPs use to
make a difference.
When considering the future implementation of these models; for the COMOIRA
model, several suggestions were generated by the EPs in relation to steps that would
need to be put in place prior to future implementation. Further knowledge of the
model was a requirement for some EPs and others thought a trialling of the model
would be beneficial. Another suggestion included sharing the model with the
consultee and trialling it at the end of individual consultations. The issue of the
investment of time came up for both models but was emphasised more clearly for the
COMOIRA framework. The idea of building in time and systems to prioritise this
area was considered necessary.
When considering the future implementation of the models the researcher would
recommend to stakeholders that both EPSs have further training around the two
models. This could include peer training as the data from the audit of previous
knowledge questionnaire (see Table 5.4) indicated that around half of the EPs were
either very familiar or fairly familiar with COMOIRA and AI. Furthermore, an
extended piloting period would seem preferable in order to trail the models alongside
the service users and gather feedback following this process.
In addition, it would be important for stakeholders to consider the potential of these
models alongside the new initiatives being offered by the EPSs. All school staff
across both LAs are currently being trained by some of the EPs in PCP. PCP is being
driven forward by the LA and all schools are being encouraged to adopt this
approach within the next couple of years, with support from the EPSs. Therefore, it
would be beneficial for a future evaluative framework to have the capacity to capture
the effectiveness of this approach and its place within consultation and to evaluate
the role of the EP within this process.
6.7 Theoretical contribution to knowledge
A review of the literature revealed that whilst professional practice models had been
studied on an individual basis, for example the research by Gameson et al. (2003;
2005) and Gameson and Rhydderch (2010) on the COMOIRA model, research that
183
compares and contrasts professional practice models in a systematic and data driven
way had not been widely conducted. This piece of research was conducted in a
rigorous way, included multiple data sets and incorporated both quantitative and
qualitative information. As well as conducting this research in a systematic and data
driven way a further contribution to knowledge was that it considered the models in
terms of their potential as evaluative tools for EP consultation, rather than
specifically as professional practice frameworks. In the researcher’s knowledge, this
is the first research of its kind in relation to COMOIRA and AI. In addition to this, a
critical assessment was carried out, which considered the application of the
COMOIRA and AI models to the evaluation of individual teacher consultations. This
research is unique in this field and is the first of its kind to consider the application
of a range of professional frameworks in the evaluation of EP practice. Furthermore;
this study is unique in that it elicits EPs’ views on a range of professional practice
models.
This study adds to the research around self-evaluation and how EPs evaluate
themselves. This is very relevant when considering the literature around evaluation
within field. The HEPS (2010) research highlighted that EPSs were struggling to
find an adequate evaluation instrument to measure the impact of what they do
(HEPS, 2010). Furthermore, it found that EPSs recognised that evaluating their
impact was the ‘big question’ they needed to address and many were grappling with
how best to achieve this due to the inherent difficulties associated with this task. This
study has begun to explore this research area and has, therefore, hopefully
contributed to knowledge in this field.
Previous literature has contended that there needs to be a formalised transparent
framework of evaluation which can be embedded in the day-to-day world of an EP
and suggested that applied psychologists they should use their knowledge and skills
to refine the tools and processes required for evaluating their work. (Currie, 2002;
Turner et al., 2010). In addition, Wicks (2013) called for a more extensive use of
practice frameworks and further research to consider their effectiveness as evaluation
tools. It is hoped that this study has contributed to these suggested areas of research.
184
6.8 Limitations of the research
6.8.1 Model selection
A main limitation of this study is the narrow focus on two professional practice
models. It is noted that had further models been included within the research then
this would have generated richer data. Future research could include a wider range of
models as it would be interesting to gather information around the effectiveness of
other models in relation to the evaluation of EP consultation. Future research could
incorporate the models that met the selection process criteria during the systematic
review (see Section 5.5.1) but were not then taking forward to be piloted.
It should be noted that the most well-known and well used models were selected by
the EPs for the piloting process. This is evident when considering the data from the
audit of previous knowledge questionnaire (see Table 5.4) which indicated that
around half of the EPs were either very familiar or fairly familiar with COMOIRA
and AI in comparison to, for example, AT where only two of the twelve participants
were in these categories. Familiarity and previous experience with models may have
skewed the selection process, which could indicate an element of bias. Had more
training been available on the other models this may have affected the choices of the
EPs.
6.8.2 Transferability of findings
The specific contextual factors of the study are noted. A critique of an AR design
conducted across two small services is that it cannot be generalised to other service
contexts (Robson, 2002). It would be important, therefore, for future research to take
note of these contextual limitations and consider how research could begin to explore
using a wider sample, for example, include more EPSs. Furthermore, it is
acknowledged that the focus of the research was limited to only one area of
consultation and it was not applied to further areas of consultation, for example with
group consultations. Further research in this area could incorporate the wider aspects
of EP consultation or extend the focus to other areas of service delivery.
6.8.3 Time implications
As discussed within section 6.6.2 the researcher acknowledges that due to the time
frame of the study and the competing work priorities and pressures on the EPs, the
185
SEP and PEP, the full stages of the RADIO model were not operationalised. The
original plan included Stage 10 of the RADIO model. During Stage 10 the findings
of final focus groups were due to be shared with the stakeholders and decisions made
around the potential of the models as a service evaluation tool. This element is
crucial in terms of moving the research forward and is needed to inform Stage 11:
Implementing Action and Stage 12: Evaluating Action which were due to be
conducted outside the parameters of this study.
6.8.4 Methodological considerations
One critique in relation to the methodology of the study is that the individual
interviews were not conducted for the reasons outlined in Section 4.13.1. It is
important to note that his may have strengthened the reliability and validity of the
data through the richness of the information that can be captured within individual
interviews. However, despite this the researcher asserts that the research is
methodologically strong due to the incorporation of a multi-methods approach. The
data generated from the focus groups were triangulated against the questionnaire
data which increased the likelihood of the reliability of the findings and offered
validity to the data.
6.9 Implications for practice
Despite the limitations noted within section 6.8, there are some potentially valuable
implications resulting from this piece of research to consider for the EP profession.
The following suggestions are aimed at EPSs and offer further pointers when
considering future ways to evaluate EPSs.
6.9.1 Data driven
A key element to the government’s agenda to improve the quality of public services
is that services should be delivered on the basis of evidence – based practice. This
posits that data should be collected in a systematic manner with a clear focus upon
outcomes which can then be used to inform judgements about the effectiveness of
services (Dunsmuir et al., 2009). Consequently LAs have explored potential ways to
collect information to validate and evaluate their services. This research integrated
this guidance by using a systematic process to guide the study. This proved useful as
it provided a clear rationale for the decisions that were subsequently made and the
186
actions that were then taken. Furthermore, collecting data in such a robust way
harnessed valuable information about the potential of professional practice models in
relation to the evaluation of EP consultations. The potential benefit to collecting
evidence in this way is that EPSs are under increasing pressure from stakeholders to
demonstrate accountability. As highlighted by Fallon et al. (2010) the relatively
recent introduction of a traded model service requires EPs to have a clear focus on
the outcomes of their work and further highlights the need for EPs to evaluate what
they do and demonstrate their unique contribution
An important finding from this study is that when used evaluatively, practice
frameworks could potentially be seen to provide evidence around the distinctiveness
of the EPs’ contribution to the consultation process which could be used to enhance
the accountability of EPSs. This supports previous research showing that effective
practice frameworks can be seen to secure the centrality of psychology and the
distinctiveness of the EPs contribution in their professional interactions (Fallon et al.,
2010; Frederickson & Miller, 2008; Gameson et al., 2005; Kelly et al., 2008;
Leadbetter, 2005). An important consideration for the EP profession is the potential
of these frameworks to provide a form of effective evaluation which could help to
safeguard EPSs (Lloyd & Harrington, 2012).
6.9.2 Applying psychology
A further interesting consideration from this study is in relation to the importance of
the reflective and reflexive components within the professional practice models and
how this contributed to the evaluation process. This element of the frameworks
linked the application of psychology to the evaluation process, with the EPs
reflecting on how the consultation went and upon their own contribution to the
individual consultation. This finding can be seen to meets the requirement of the
HCPC which states that practitioner psychologists must be able to reflect critically
upon and review their practice and record the outcomes of such reflection (HCPC,
2015).
The strength of the models in their ability to utilise psychology and link this with the
evaluation of the consultation could be viewed as important for the EP profession
and potentially offer a tool to bridge the gap between theory and practice. This
187
supports the findings of Kelly et al. (2008) that practice frameworks are an essential
resource for the EP profession as they provide a means of addressing the challenge
that EPs face with regard to clarifying the complex relationship between theory and
practice. Indeed, recent reviews have called for a more sound use of psychological
knowledge across the variety of contexts in which EPs work (Farrell et al., 2006) and
for the ‘conceptualisation’ of the role as that of ‘scientist-practitioner’ (Fallon et al.,
2010; Lane & Corrie, 2006). It could be suggested from these findings that
professional practice frameworks may potentially provide a resource that help EPs
demonstrate that their professional practice is informed by psychological theories
and research evidence (Gameson et al., 2003). This finding provides useful
information for the EP profession and indicates that implementing practice models
within the evaluation of consultation may contribute towards the direction for EPs to
draw upon psychological knowledge and skills (Frederickson and Miller, 2008;
Gameson et al., 2003) and apply this psychology to their practice. This is seen to be
a fundamental part of the EP role (DfEE, 2000; Farrell et al., 2006; Scottish
Executive, 2002).
6.9.3 Stakeholder and service user feedback
Another interesting implication is that the focus on stakeholder feedback proved to
be a useful contribution to the research process. As outlined by Timmins et al (2003)
RADIO provides a process whereby stakeholders are involved in the research
process. The involvement of stakeholders throughout meant that the decision making
was collaborative and guided by the findings of the stages of the research. In
addition the key people required in order to agree to further action were involved and
able to sanction this at key decision making stages.
6.9.4 Future implications
This study has highlighted the importance of strategic planning. This is an important
finding and managers will need to consider this in relation to the future development
of these professional practice models as evaluative tools. This research has
implications for future continued professional development opportunities. It will be
important for stakeholders to consider the need for further training for the EPs, in
relation to the models themselves but there may also be some potential in the
developing the knowledge of EPs in relation to organisational psychology. The
188
findings from this research highlighted how certain individuals react to the change
process and the defence mechanisms that can be triggered when individuals are faced
with change, for example, resistance. When considering this piece of research in
relation to EP practice it demonstrates how important it is for EPs to understand the
human factors linked with reluctance to change and how these can impact on the
success of the consultation process (Bovey & Hede, 2001). Furthermore, these
findings emphasise how important it is for EPs to try and develop an understanding
of the organisations in which they work as the organisational culture often
determines how receptive the organisation (school) will be to the suggested
intervention (Bryan et al., 2007) and how this then impacts upon the staff members
within the school (Bryan et al., 2007; Schein, 1996). Further insight into this area
would be an important element in the development of a future evaluation tool.
Further implications that arise from this research include the need for stakeholders
and service managers to consider the service systems in relation to professional
development. It is recommended that ongoing supervision and the sharing of practice
across both EPSs should be considered an important factor in the future development
of the professional practice frameworks. This has time implications for both EPSs in
relation to peer supervision and, furthermore, appropriate systems would need
consideration in order for this aspect of work to be prioritised and not lost amongst
the competing pressures that the EPSs have to face. It would also be important for
stakeholders and managers to build in appropriate systems for the review of this
aspect of service development.
189
Chapter Seven: Conclusion
7.1 Chapter Introduction
The aim of this piece of research was to establish an evaluation tool that would
collect relevant information on the impact of EP service delivery in relation to
consultation. It focused on finding a method that would facilitate two EPSs to
successfully and appropriately assess individual teacher consultation and gather
relevant data about the effectiveness of the EPSs for the LA. This research adopted
an AR approach which added a positive dimension to the process. It created a greatly
needed dialogue around ways to evaluate what EPs do through consultation in an
ever increasingly accountable climate where EPSs are developing traded models of
service in order to generate income (DfE, 2011).
This research identified many positive and valuable aspects with regards to both the
COMOIRA and AI models when these were implemented to evaluate consultation. It
demonstrated that they have the potential to assess certain aspects of the consultation
process and, as such, could conceivably contribute towards providing outcomes
which can demonstrate accountability to employers and service users. The study
findings also proposed that professional practice models could possibly be used to
evaluate the wider aspects of EP practice and, therefore, be of broader benefit to
EPSs. It is hoped that this research has contributed to knowledge within the area of
self-evaluation within EPSs as this was the ‘big question’ that many EPs needed to
and were struggling to address (HEPS, 2010). This piece of research is unique in that
it was conducted in a rigorous way and included multiple data sets incorporating
both quantitative and qualitative information. It is the first of its kind to consider the
models of COMOIRA and AI in terms of their potential as evaluative tools in the
field of individual EP consultation.
The researcher hopes that this study will instigate future research in this area and will
stimulate EPSs to continue to explore how professional practice models can be used
effectively to evaluate EP practice.
190
7.2 Future Research Directions
• Future focus could be given to the adaptation and development of the AI and
COMOIRA models within the context of current EP practice. AI was not developed
specifically as a framework for EP practice and despite COMOIRA being developed
for EP practice it was designed in 2003 so the context in which EPs work has
changed dramatically. Indeed, these models predate many of the new initiatives
within education, for example, PCP and Education, Health and Care plans (DfE,
2015). In relation to this, the models may benefit from further piloting and
consideration could be given to how any further adaptations and development would
complement the current context within which the EP profession works.
• This research focused specifically on the two models of COMOIRA and AI.
Future research could incorporate the models that met the selection process criteria
during the systematic review (see Section 5.5.1) but were not chosen to be piloted
as part of this research. For example, the models: AT (Leadbetter, 2008) and the
IFF (Frederickson & Cline, 2002). In addition, it may be useful for future research
to consider the design of a bespoke evaluative framework that combines key
elements from different professional frameworks.
• The focus of the research was limited to only one area of service delivery. It
focused upon individual teacher consultations and not at consultation on a wider
level which is aimed at the level of the individual, group or organisation (Gutkin &
Curtis, 1999; Wagner, 1995). Future research could look at consultation on a
broader level, for example, parent consultations, group consultation which is a
current model of service delivery within the EPSs involved in this study.
Furthermore, future research could look beyond consultation and incorporate
different areas of service delivery within the evaluation process, for example,
training was another area identified as a specific area of work that COMOIRA
could be used to evaluate.
• This study was conducted across two small services. Future research could
look to use a wider sample group and involve a higher number of EPSs. Indeed,
191
neighbouring EPSs have expressed an interest in this piece of research and wish to
share the findings.
• Future research could consider implementing the models collaboratively,
alongside teachers, to facilitate a joint evaluation of the consultation. The models
could be used to encourage the practitioner and service user to reflect upon the
consultation and assess its impact. It would be interesting to collect data around
whether this process provides opportunities for EPs to develop their practice in the
future, helps them to provide evidence about their performance and the results they
produce. This is in line with one of the recommendations made by Farrell et al.
(2006).
• The researcher would recommend that any future research incorporates
Stages 10, 11 and 12 of RADIO which includes the following stages: action
planning, implementing action and evaluating action.
192
References Alison, P., & Pomeroy, E. (2000). ‘How Shall we “Know?” Epistemological Concerns in Research in Experiential Education. Journal of Experiential Education, 23 (2), 91-98. Altrichter, H. & Holly, M. L. (2005). Research diaries. In B. Somekh & C. Lewin (Eds), Research in Social Sciences (pp. 23-33). London: Sage Publications. American Psychological Association. (2010). Concise Rules of APA Style. Sixth Edition. Washington DC: American Psychological Association Ashton, R. (2009). ‘Using the Research and Development in Organisations model to improve transition to high school’. Educational Psychology in Practice, 25 (3), 221-232. Ashton, R., & Roberts, E. (2006). What is valuable and unique about the Educational Psychologist? Educational Psychology in Practice, 22 (2), 111–124. Association of Educational Psychologists (AEP). (2008). Educational Psychologists in Multidisciplinary Settings: Investigations into the Work of Educational Psychologists in Children’s Services Authorities. Durham: AEP. Association of Educational Psychologists. (2010). Principles for the Delivery of Educational Psychology Services. Durham: AEP. Association of Educational Psychologists. (2011). The Delivery of Educational Psychology Services. Durham: AEP. Athun, R. (2000). Parents’ views of quality in Educational Psychology Services. Educational Psychology in Practice, 16, 141-157. Atkinson, C. (2011). Action Research. Unpublished notes. University of Manchester. Attride-Stirling, J. (2001). Thematic networks: an analytic tool for qualitative research. Qualitative Research, 1 (3), 385-405. Banister, P., Burman. E., Parker, I., Taylor, M., & Tindall, C. (1994). Qualitative methods in psychology: A research guide. Buckingham: Open University Press. Bhasker, R. (1978). A realist theory of science. Brighton: Harvest Press. Biggerstaff, D. (2012). Qualitative Research Methods in Psychology, Psychology- Selected Papers, Dr Gina Rossi (Ed). Retrieved from: http://intechpen.com/books/pschology-selected-papers/qualitative-research-methods-in-psychology. Bovey, W. H., & Hede, A. (2001). Resistance to organisational change: the role of defence mechanisms. Journal of Managerial Psychology, 16(7), 534-548.
193
Boyatzis, R. (1998). Transforming qualitative information: Thematic analysis and code development. Thousand Oaks, CA: Sage. Boyle, C., & Lauchlan, F. (2009). Applied psychology and the case for individual casework: some reflections on the role of the educational psychologist. Educational Psychology in Practice, 25 (1), 71-84. Boyle, C., & MacKay, T. (1990). Pupils with learning difficulties: What do schools expect of their educational psychologists? A Scottish perspective. Learn, 12, 24-32. Boyle, C., & MacKay, T. (2007). Evidence for the efficacy of systemic models of practice from a cross-sectional survey of schools’ satisfaction with their educational psychologists. Educational Psychology in Practice, 23(1), 19-31. Boyne, G.A., Gould-Williams, J. S., Law, J., and Walker, R. M. (2004). Toward the Self-Evaluating Organization? An Empirical Test of the Wildavsky Model. Public Administration Review, 64(4), 463-473, Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in Psychology, 3, 77-101. British Psychological Society (BPS) Division of Educational and Child Psychology (DECP).(2007). Criteria for the accreditation of three-year training programmes in educational psychology in England, Northern Ireland and Wales. Leicester: BPS. British Psychological Society (2009). Code of Ethics and Conduct. Guidance Published by the Ethics Committee of the British Psychological Society. Leicester: BPS. Bryan, K. S., Klein, D. A., & Elias, M. J. (2007). Applying Organizational Theories to Action Research in Community Settings: A Case Study in Urban Schools. Journal of Community Psychology, 35 (3), 383-398 Brydon-Miller, M., Greenwood, D. & Maguire, P. (2003). Why Action Research? Action Research, 1(1), 9–28. Cabinet Office. (1999). Modernising Government. Retrieved from: http://archive.cabinet-office.gov.uk Cameron, R. J. (2006). Educational Psychology: The distinctive contribution. Educational Psychology in Practice, 22 (4), 289-304. Cameron, R. J., & Monsen, J. J. (2005). Quality psychological advice for teachers, parents/carers and LEA decision-makers with respect to children and young people with special needs. Educational Psychology in Practice, 21(4), 286-306. Caplan, G. (1970). The theory and practice of mental health consultation. New York: Basic Books.
194
Carter, S. M., & Little, M. (2007). Justifying Knowledge, Justifying Method, Taking Action: Epistemologies, Methodologies, and Methods in Qualitative Research. Qualitative Health Research, 17(10), 1316-1328. Checkland, P. & Scholes, J. (1990). Soft Systems Methodology in Practice. Chichester, GB:Wiley. Cherry, C. (1998). Evaluation of an educational psychology service in the context of LEA inspection. Educational Psychology in Practice, 14(2), 118-127. Clegg, S. (2005). Evidence- based practice in educational research: a critical realist critique of systematic review. British Journal of Sociology of Education, 26 (3), 415-428. Cohen, L., Manion, L., & Morrison, K. (2000). Research methods in education. London: Routledge / Falmer. Cohen, L., Manion, L., & Morrison, K. (2003). Research Methods in Education. 5th Ed. Taylor and Francis. Cohen, L., Manion, L., & Morrison, K. (2007). Research Methods in Education. 6th Ed. London: Routledge Falmer. Coolican, H. (2009). Research Methods and Statistics in Psychology. London: Hodder. Education. Cooperider, D. L., & Whitney, D., & Stavros, J. M. (2008). Appreciative Inquiry: A positive revolution in change. San Fransisco, Californai: Berrett- Koehler Publishers, Inc. Coneley, J.C., & Coneley, C.W. (1982). School consultation: a guide to practice and theory. London: Pergamon. Coneley, C. W., Coneley, J. C., Ivey, D. C., & Scheel, M. J. (1991), Enhancing Consultation by Matching the Consultee's Perspectives. Journal of Counseling & Development, 69: 546–549. Corban, I. (2011). Educational Psychologists’ views of factors that influence job approbation, job satisfaction and dissatisfaction when working within multi-agency local authority contexts. Unpublished doctoral thesis: University of Manchester. Cording, J. (2011). A study of Educational Psychologists use of consultation and users’ views on what a service should deliver. Unpublished doctoral thesis: University of Exeter. Corson, D. (1991). Educational Research and Bhasker’s Conception of Discovery. Educational Theory, 41 (2), 189-198. Corti, L. (1993). Using diaries in social research. Social Research Update, 2.
195
Crabtree, B., & Miller, W. (1999). Doing qualitative research. Newbury Park, CA: Sage. Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five traditions. London: Sage. Creswell, J. W. ., & Clark, V. L. (2006). Designing and Conducting Mixed methods Research London: SAGE. Currie, E. (2002). Review of the provision of educational psychology services in Scotland. Edinburgh: Scottish Executive. Dennis, R. (2004). Delivering educational psychology services to non-maintained EY settings: Inclusion ABC. Educational and Child Psychology, 21 (2), 27-38. DES (1968). Psychologists in Education Services (The Summerfield report). London: HMSO. Department for Children, Schools and Families (DCSF). (2009). Lamb inquiry: special educational needs and parental confidence. London: DCSF. Department for Education (DfE). (2011). SEN Green Paper. London: DfE. Department for Education (DfE). (2011). School Funding Reform: next steps towards a fairer system. London: DfE. Department for Education. (DfE). (2014). Educational Psychology Workforce Survey 2013 Research report. London: DfE. Department for Education. (DfE). (2015). Statutory guidance publications for schools and local authorities. www.gov.uk. Department for Education and Department of Health. (DfE). (2014). Special educational needs and disability code of practice: 0-25 years. London: DfE. Department for Education and Employment (DfEE). (1997). Excellence for all children- meeting special educational needs. The Green Paper. London: DfEE. Department for Education and Employment (DfEE). (1998). Meeting Special Educational Needs A Programme of Action. London: HMSO. Department for Employment and Education (DfEE). (2000). Educational psychology services (England): Current role, good practice and future directions. The Report of the Working Group. London. DfEE Publications. Department for Education and Skills (DfES). (2001) The special educational needs code of practice London, UK: Department for Education and Skills. Department for Education and Skills (DfES). (2003). Every Child Matters (Green Paper). London: HMSO.
196
Department for Education and Skills (DfES). (2004). Common Assessment Framework. London: HMSO. Department for Education and Skills (DfES). (2004). Every Child Matters: Change for Children. London. HMSO. Dickinson, D. (2000). ‘Consultation: Assuring the Quality and Outcomes’. Educational Psychology in Practice, 16(1), 19-23.
Division of Educational and Child Psychology (DECP). (1998). A Framework for Psychological Assessment and Intervention: Draft 1. Leicester: British Psychological Society.
Division of Educational and Child Psychology (DECP). (1999). ‘A framework for psychological assessment and intervention.’ DECP Newsletter (89). 6-9.
Division of Educational and Child Psychology (DECP). (2002). Professional Practice Guidelines. Leicester: British Psychological Society.
Dowling, J., & Leibowitz, D. (1994). Evaluation of Educational Psychology Services: Past and Present. Educational Psychology in Practice, 9 (4), 241- 250.
Dunsmuir, S., Brown, E., Iyadurai, S., & Monsen, J. (2009). Evidence-based practice and evaluation: from insight to impact. Educational Psychology in Practice, 25, 53-70
Easton, G. (2010). Critical Realism in Case Study Research. Industrial Marketing Management, (39), 118-128.
Education Act. (1944). Retrieved from: www.parliament.uk. Education Act. (1981). Retrieved from: www.parliament.uk. Education Act. (1994). Retrieved from: www.legislation.gov.uk. Education Act. (1996). Retrieved from: www.legislation.gov.uk. Engestrom, Y. (1999). ‘Activity Theory and Individual and Social Transformation’. In Y. Engestrom, Y., Miettinen, R., & Punamaki, R.L. (eds). Perspectives on Activity Theory. Cambridge: Cambridge University Press.
Fallon, K., Woods, K., & Rooney, S. (2010). A discussion of the developing role of educational psychologists within Children’s Services. Educational Psychology in Practice, 26 (1), 1469-5839. Farrell, P., Woods, K., Lewis, S., Rooney, S., Squires, S. & O’Connor, M. (2006). A Review of the Functions Contribution of Educational Psychologists in England and Wales in Light of “Every Child Matters”. London: DfES Research Publication.
197
Faupel, A.W., & Norgate, R. (1993). Where to, Educational Psychology? Roles, Responsibilities in the World of the 1993 Education Act. Educational Psychology in Practice, 9 (3), 131-138. Floersch, J., Longhofer, J., Kranke, D. and Townsend, L. (2010). Integrating thematic, grounded theory and narrative analysis: A case study of adolescent psychotropic treatment. Qualitative Social Work Journal, 9, 407-425 Frederickson, N., & Cline, T. (2002). Special Educational Needs, Inclusion and Diversity: A Textbook. Buckingham: Open University Press. Frederickson, N., & Miller, A. (2008). What Do Educational Psychologists Do? In N. Frederickson, A. Miller and T. Cline, Educational Psychology. London: Hodder Education. Frederickson, N., & Reason, R. (1995). Discrepancy definitions of specific learning difficulties. Educational Psychology in Practice, 10 (4), 195-205. Friedman., M. (1997). A Guide to Developing and Using Performance Measures in Results-Based Budgeting. Washington, DC: Finance Project. Friedman, M. (2005). Trying hard is not good enough. Trafford on Demand Pub. Gameson, J., Rhyderrch, G., Ellis, D., & Carroll, T. (2003). ‘Constructing a flexible model to integrate professional practice- Part 1: Conceptual and theoretical issues.’ Educational and Child Psychology, 22 (4), 96-108. Gameson, J., Rhyderrch, G., Ellis, D., & Carroll, T. (2005). ‘Constructing a flexible model to integrate professional practice- Part 2: Process and practice issues.’ Educational and Child Psychology, 22 (4), 41-55. Gameson, J., & Rhydderch, G. (2008). The Constructionist Model of Informed and Reasoned Action (COMOIRA). In Woolfson, K., & Boyle, J. Frameworks for Practice in Educational Psychology. London: Jessica Kingsley. Gameson, J., & Rhyderrch, G. (2010). Constructing a flexible model of integrated professional practice- Part 3: the model in practice. Educational Psychology in Practice, 26(2), 123-149. Gaskell, S., & Leadbetter, J. (2009). Educational psychologists and multi-agency working: Exploring professional identity. Educational Psychology in Practice, 25 (2), 97-112. Gephart, R. (1999). Paradigms and Resrach Methods. Research Methods Forum. 4. Gerrig, J. (2004). Qualitative Methods. Boston University, Vol. 2. Gersch, I. (2009). A positive future for educational psychology- if the profession gets it right. Educational Psychology in Practice, 25(1), 9-19.
198
Gillham, B. (1978). Reconstructing Educational Psychology. London: Croom Helm. Glaser, B. G. (1992). Basics of grounded theory: Emergence vs. forcing. Mill Valley: CA. Graneheim, U. H., & Lundman, B. (2004). Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Education Today, 24, 105-112. Gray, P. (2001). Developing Support for more Inclusive Schooling: A review of the role of SEN Support Services in English LEAs. London: Department for Education and Employment. Greenhouse, P. M. (2013). Activity theory: a framework for understanding multi-agency working and engaging service users change. Educational Psychology in Practice, 29(4), 404-415. Guba, E. G., & Lincoln, Y. S. (1994). Competing paradigms in qualitative research. In : Denzin, N., K. & Lincoln, Y., S. eds. Handbook of qualitative research. London: Sage. Gutkin, T. B., & Curtis, M. J. (1990). School-based Consultation: Theory and techniques. In C. R. Reynolds & T. B. Gutkin (eds.). The handbook of school psychology. New York: Wiley. Gutkin, T. B., & Curtis, M. J. (1999). School – Based Consultation Theory and Practice: TheArt and Science of Indirect Dervice Delivery, in C. R. Reynolds and T.B. Gutkins (eds). The Handbook of School Psychology, 3rd edn. New York: Wiley. Hammond, S.A. (1998). The Thin Book of Appreciative Inquiry. Thin Book Publishing Co. Hampshire Educational Psychology Service. (2010). How do Educational Psychology Services Currently Evaluate Themselves? NAPEP Survey. Hampshire: Hampshire County Council. Health and Care Professions Council. (2015). Standards of proficiency- Practitioner Psychologist. London: HCPC. Heath, H., & Cowley, S. (2004). Developing a grounded approach: a comparison of Glaser and Strauss. International Journal of Nursing Studies, 41(2), 141-150. Heish, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15 (9), 1277-1288. Hendricks, C. (2008). Improving Schools through Action Research. A Comprehensive Guide for Educators. University of Georgia: Pearson.
199
Her Majesty's Stationery Office. (1968). The Summerfield Report: Psychologists in Education Services. London: HMSO.
Idol, L. (1990). The scientific art of classroom consultation. Journal of Educational and Psychological Consultation, 1(1), 3-22.
Kaslow, N. J. (2004). Competencies in Professional Psychology. American Psychologist, 59(8), 774-781. Keevers, L., Treleavan, L., Backhouse, H., & Darcy, M. (2010). Locally-based community organisations and social inclusion. Sydney: The University of Sydney. Kelly, B., Woolfson, L., & Boyle, J. (Eds). (2008). Frameworks for practice in educational psychology. London: Jessica Kingsley Publishers. Kelly, D., & Gray, C. (2000). Educational psychology services (England): Current Role , good practice and future directions- the research report. Nottingham: DfEE Publications.
Kennedy, E., Cameron, R.J., & Monsen, J. (2009). Effective Consultation in Educational and Child Psychology Practice: Professional Training for Both Competence and Capability. School Psychology International, 30(6), 603-625.
Kidd, S. A., & Kral, M. J. (2005). Practicing Participatory Action Research. Journal of Counselling Psychology, 52(2), 187-195.
Kiersuk, T. J., & Sherman, R. E. (1968). Goal attainment scaling: A general method for evaluating comprehensive community mental health programs. Community Mental Health Journal, 4(6), 443-453.
Kinderman, P. (2005). A Psychological Model of mental Disorder. Harvard Review of Psychiatry, 13(4), 206-217.
Knight, G., & Timmins, P. (1995). Research and Development in Organisations (RADIO). In Timmins,P., Sheperd, D., & Kelly, T. (2003). The research and development in organisations approach and the evaluation of a mainstream support initiative. Educational Psychology in Practice, 19 (3), 229-242. Kreuger, R. A., & Casey, M. A. (2000). Focus groups: A practical guide for applied researchers (3rd ed.). Thousand Oaks, CA. Sage. Lane, D.A., & Corrie, S. (2006). The Modern Scientist- Practitioner. A Guide to Pratice in psychology. London: Routledge. Larney, R. (2003). School- Based Consultation in the United Kingdom: Principles, Practice and Effectiveness. School Psychology International, 24(1), 5-19. Leadbetter, J. (2005). Activity theory as a conceptual framework and analytical tool within the practice of educational psychology. Educational and Child Psychology, 22(1), 18-28.
200
Leadbetter, J. (2006). Investigating and Conceptualising the Notion of Consultation to Facilitate Multi-agency Work. Educational Psychology in Practice, 22(1) 19-31. Leadbetter, J. (2008). Activity theory and the professional practice of educational psychology. Frameworks for practice in educational psychology: A textbook for trainees and practitioners. London: Jessica Kingsley Publishers. Levin, B., & Rock, T. (2003). The Effects of Collaborative Action Research on Preservice and Experienced Teachers Partners in professional Development Schools. Journal of Teacher Education, 54, 135-149. Leyden, G. (1999). Time for change: The reformulation of applied psychology for LEAs and schools. Educational Psychology in Practice, 14(4), 222-228. Lightfoot, L.M. (2013). The Role of Practitioner Educational Psychologists in Supporting a Residential Setting to Meet the Needs of Looked After Children. Unpublished doctoral thesis: University of Manchester. Lincoln, Y.S., & Guba, E.G. (1985). Naturalistic Inquiry. London: SAGE Publications. Lindsay, G. (1985). Educational Psychology in England and Wales. Journal of School Psychology, 23(4), 305-317. Lloyd, N., & Harrington, L. (2012). The challenge of effective outcome evaluation of a national, multi-agency initiative: The experience of Sure Start. Evaluation, 18(1), 93-109. Love, P. (2009). Educational psychologists: The early search for an identity. Educational Psychology in Practice, 25(1), 3-8. Mackay, T. (2002). The future of educational psychology. Educational Psychology in Practice, 18(3), 245-253. MacKay, T., & Boyle, J. (1994). Meeting the needs of pupils with learning difficulties. What do primary and secondary schools expect of their educational psychologist. Educational Research, 36 (2), 187-196. Madill, A., Jordan, A., & Shirley, C. (2000). Objectivity and reliability in qualitative analysis: Realist, contextualist and radical constructionist epistemologies. British Journal of Psychology, 91, 1-20. Marrow, S. L. (2007). Qualitative Research in Counselling Psychology: Conceptual Foundations. The Counselling Psychologist, 35: 209. McNiff, J., & Whitehead, J. (2003). You and your research project. London: Routledge Falmer.
201
McNiff, J., & Whitehead, J. (2006). All You Need to Know About Action Research. London: SAGE Publications. McNiff, J., & Whitehead, J. (2011). All You Need to Know About Action Research. (2nd Edition ). London: SAGE Publications. Miller, G. (1969). Psychology as a means of promoting human welfare. American Psychologist, 24, 1063–1075. Miller, A. (1996). Pupil Behaviour and Teacher Culture. London: Cassell. Miller, A., & Frederickson, N. (2006). Generalizable findings and idiographi problems: Struggles and successes for educational psychologists as scientist-practitioners. In D.A. Lane & S. Corrie (Eds). The modern scientist practitioner. A guide to practice in psychology. Hove: Routledge. Monsen, J., G, B., Frederickson, N., & Cameron, R.J. (1998). Problem analysis and professional training in educational psychology. Educational Psychology in Practice, 13 (4), 234-249. Moran, P., Jacobs, C., Bunn, A., & Bifulco. (2007). Multi-agency working: implications for an early- intervention social work team. Child and Family Social Work, 12(2), 143-151 Nadin, S. & Cassell, C. (2006). The use of a research diary as a tool for reflexive practice: Some reflections from management research. Qualitative Research in Accounting & Management, 3(3), 208-217. National Audit Office. (2001). Purchasing Professional Services: Report by the Controller and Auditor General. London: national Audit Office. Nightingale. D.J., & Cromby, J. (1999). Social Constructionist Psychology: a critical analysis of theory and practice. Buckingham: Open University Press. Ng, K., & Hase, S. (2008). Grounded Suggestions for doing Grounded Theory Business Research. The Electronic Journal of Business Research Methods, 6(2), 155-170. Norwich, B. (2000). Educational psychology and special educational needs: How they relate and where is the relationship going? Educational and Child Psychology, 17 (2), 5-15. Norwich, B. (2005). Future directions for Professional Educational Psychology. School Psychology International, 26, 387-397. Patomaki, H., & Wight, C. (2000). After postpositivism? The promises of Critical realism. International Studies Quarterly, 44, 213-237. Patton, M. (2002). Qualitative research and evaluation methods. (3rd edition). Thousand Oaks, CA. Sage.
202
Parlett, M., & Deardon, G. (1977). The illuminative evaluation in higher education. California: Pacific Sounding Press. Parlett, M., & Hamilton, D. (1972). “Evaluation as illumination: A New Approach to the Study of Innovatory Programs”. Edinburgh: Edinburgh University. Centre for Research in the Educational sciences. Pennick, J., & Lagunowitsch, H. (2010). Executive Summary: The Perceived facilitators and barriers of Educational Psychologist’s working in a systemic way: a school perspective. University of Bristol: Research Commission. Pilgrim, D. (2007). Evidence based mental health. Clinical Psychology, 9, 5-6. Potter, J., & Wetherell, M. (1987). Discourse and Social Psychology: Beyond Attitudes and Behaviour. London: SAGE. Pugh, J. (2010). Cognitive behaviour therapy in schools: the role of educational psychology in the dissemination of empirically supported interventions. Educational Psychology in Practice, 26 (4), 391-150. Reason, P. & Bradbury, H. (2001). Handbook of Action Research: Participative Inquiry and Practice. London: Sage. Robson, C. (2002). Real World Research (2nd Edition). Oxford: Basil Blackwell. Sarantakos, S. (1998). Social Research (2nd Edition). London: Macmillan. Schein, E. (1998). Organisational Culture and Leadership. San Francisco,CA: Jossey-Bass.. School Funding Regulations. (2010). Retrieved from: www. assembly.wales.
Sharp, S., Frederickson, N., & Laws, K. (2000). Changing the profile of an educational psychology service. Educational and Child Psychology, 17(1), 98-111. Sheridan, S. M.., & Cowan, R. J. (2004). Handbook of Paediatric Psychology in School Settings. London: Lawrence Erlbaum Associates, Inc. Silverman, D. (2001). Interpreting qualitative data: methods for analysing talk, text and interaction, 2nd edition. London: Sage. Silverman, D. (2004). Qualitative Research: Theory, Method and Practice. London: SAGE Publications. Simm, J., & Ingram, R. (2008). Collaborative action research to develop the use of solution-focused approaches. Educational Psychology in Practice, 24(1), 43-53. Stobie, I. (2002). Processes of change and continuity in educational psychology, Part 1. Educational Psychology in Practice, 18 (3), 214-237.
203
Summerfield, A. (1968). Psychologists in Education Services. London: HMSO. Timmins,P., Sheperd, D., & Kelly, T. (2003). The research and development in organisations approach and the evaluation of a mainstream support initiative. Educational Psychology in Practice, 19 (3), 229-242. Timmins, P., Mohammed,B., McFadyen,J., & Ward, J. (2006). ‘Teachers and consultation: Applying research and development in organisations (RADIO).’ Educational psychology in Practice, 22 (4), 305-319. Torrance, H. (2004). Using action research to generate knowledge about educational practice. In G. Thomas, & R. Pring (Eds). Evidence-based practice in education. Maidenhead: Open University Press. Trochim, Dr .W. (2006). The Research Methods Knowledge Base (3rd edition). Atomic Dog Publishing. Turner, S., Randall, L., & Mohammed, A. (2010). Doing an effective job? Measuring the impact of casework. Educational Psychology in Practice, 26(4), 313-329. United Nations (1989) Convention on the Rights of the Child. Retrieved 2nd May 2008 from http://www2.ohchr. Wagner, P. (1995). School Consultation: Frameworks for the Practising Educational Psychologist: A Handbook. UK: Kensington and Chelsea Education Psychology Service. Wagner, P. (2000). Consultation: Developing a comprehensive approach to service delivery. Educational Psychology in Practice, 16 (1), 9-18. Wagner, P. (2008). Consultation as a framework for practice. In B. Kelly, L. Woolfson, & J. Boyle, (Eds), Frameworks for Practice in Educational Psychology: A Textbook for Trainees and Practitioners. London: Jessica Kingsley Publishers. Warnock Report. (1978). Special Educational Needs Report of the Committee of Enquiry into the Education of Handicapped Children and Young People. London: HMSO. Watkins, C. (2000). Introduction to the articles on consultation. Educational Psychology in Practice, 16(1), 5-8. Welsh Assembly Government. (2008). School Effectiveness Framework. Retrieved from: www.sefwales.co.uk/framework. Welsh Assembly Government. (2011). Children and Young People: Rights to Action. Welsh Assembly Government.
204
Wales Audit Office. (2010). Councils’ response to the financial challenges: Key messages from the Wales Audit Office preliminary corporate assessments. Cardiff: Auditor General for Wales. Whitehead, J. (2009). Generating living theory and understanding in action research studies. London: Sage Wicks, A. (2013). Do frameworks enable educational psychologists to work effectively and efficiently in practice? A critical discussion of the development of executive frameworks. Educational Psychology in Practice, 29(2), 152-162. Willdridge, K. (2013). An exploration of educational psychologists’ views of their role and job satisfaction. DEdPsy Thesis: Cardiff University. Wilkinson, L. A. (2006). Monitoring Treatment Integrity: An Alternative to the ‘Consult and Hope’ Strategy in School-Based Behavioural Consultation. School Psychology International, 27(4), 426-438. Willig, C. (2001). Introducing Qualitative Research in Psychology. Buckingham: Open University Press. Willig, C. (2003). Discourse analysis. In Smith, J.A. Qualitative psychology; a practical guide to research methods. London: Sage. Wimbush, E., & Watson, J. (2000). An Evaluation Framework for Health Promotion: Theory, Quality and Effectiveness. Evaluation, 6(3), 301-321.
Woods, K. (1994). Towards National Criteria for Special Educational Needs: Some Conceptual and Practical Considerations for Educational Psychologists. Educational Psychology in Practice, 10(2), 85- 92.
Woolfson, L. (2008). The Woolfson et al. Integrated Framework: An Executive Framework for Service- Wide Delivery. Frameworks for Practice in Educational Psychology: A textbook for trainees and practitioners. London: Jessica Kingsley Publishers. Woolfson, L., Whaling, R., Stewart, A., & Monsen, J.J. (2003). ‘ An integrated framework to guide educational psychologist practice.’ Educational Psychology in Practice, 19 (4), 283-302. Yin, R.K. (2009). Case Study research: Design and Methods, 4th Ed. Applied Social Research Methods Series (Vol 5). London: Sage. Zachariadis, M., Scott, S., & Barrett, M. (2010). Exploring critical realism as the theoretical foundation of mixed-method research: evidence from the economics of IS innovations. Cambridge: Cambridge Judge Business School, University of Cambridge.
205
Appendix A: Search Strategy example for Literature review Stage One Ovid: Search form Page 1 of 3 Search History
Searches
Results
Advanced Search Type
1
exp evaluation
15499
Advanced
Display
2
exp public sector
76913
Advanced
Display
3
exp self-evaluation
9934
Advanced
Display
4
combine 1 and 2
115
Advanced
Display
206
Appendix B: COMOIRA model
207
Appendix C: Appreciative Inquiry Model
208
Appendix D: Audit of Previous Knowledge Questionnaire 1. How familiar are you with the following professional practice models/frameworks? (Please tick as appropriate) Model
Very familiar
Fairly Familiar
Know a Little
Not at all familiar
COMOIRA
APPRECIATIVE INQUIRY (AI)
ACTIVITY THEORY (AT)
DECP
INTERACTIVE FACTORS FRAMEWORK (IFF)
2. Are you currently using any of the models/frameworks in your professional practice? YES NO COMOIRA AI AT DECP IFF 3. If you answered ‘yes’ to any of the models/frameworks in question 2 please describe very briefly how you are using them
Thank you for taking the time to complete this questionnaire Adrienne Eddleston
209
Appendix E: Preference Questionnaire Appendix E: Professional Practice Model Preference Questionnaire 1. Please rank from 1-5 (1=most preferred, 5=least preferred) the models/frameworks 1. 2. 3. 4. 5. 2. How likely would you be to use the following professional practice models/frameworks? Model
Very likely to use
Fairly likely to use
Would use a little
Not likely to use
COMOIRA
APPRECIATIVE INQUIRY (AI)
ACTIVITY THEORY (AT)
DECP
INTERACTIVE FACTORS FRAMEWORK (IFF)
Thank you for taking the time to complete this questionnaire
Adrienne Eddleston
210
Appendix F: Feedback Questionnaires for COMOIRA and AI
Evaluation Model Questionnaire: Appreciative Inquiry Please indicate the number of consultations you have used the model for up to this point …………………………….. Please indicate on the scales below to what extent you agree with the following statements. 1) I have found the model easy to use? Strongly agree
5 Agree
4 Neither agree nor disagree
3
Disagree 2
Strongly disagree
1
2) Appreciative Inquiry provides a useful structure/guide for the evaluation of consultation? Strongly agree
5 Agree
4 Neither agree nor disagree
3
Disagree 2
Strongly disagree
1
3) Appreciative Inquiry hinders/restricts the EPs working practices? Strongly agree
5 Agree
4 Neither agree nor disagree
3
Disagree 2
Strongly disagree
1
4) Appreciative Inquiry is effective at demonstrating that the consultation has made a difference? Strongly agree
5 Agree
4 Neither agree nor disagree
3
Disagree 2
Strongly disagree
1
211
5) Appreciative Inquiry can evaluate individual consultations? Strongly agree
5 Agree
4 Neither agree nor disagree
3
Disagree 2
Strongly disagree
1
Please answer the following questions about Appreciative Inquiry: 6) What are the strengths of Appreciative Inquiry? 7) What are the limitations of Appreciative Inquiry? 8) What are your general thoughts about the application of Appreciative Inquiry when evaluating consultation? Evaluation Model Questionnaire: COMOIRA Please indicate the number of consultations you have used the model for up to this point …………………………….. Please indicate on the scales below to what extent you agree with the following statements. 1) I have found the model easy to use? Strongly agree
5 Agree
4 Neither agree nor disagree
3
Disagree 2
Strongly disagree
1
2) COMOIRA provides a useful structure/guide for the evaluation of consultation? Strongly agree
5 Agree
4 Neither agree nor disagree
3
Disagree 2
Strongly disagree
1
212
3) COMOIRA hinders/restricts the EPs working practices? Strongly agree
5 Agree
4 Neither agree nor disagree
3
Disagree 2
Strongly disagree
1
4) COMOIRA is effective at demonstrating that the consultation has made a difference? Strongly agree
5 Agree
4 Neither agree nor disagree
3
Disagree 2
Strongly disagree
1
5) COMOIRA can evaluate individual consultations? Strongly agree
5 Agree
4 Neither agree nor disagree
3
Disagree 2
Strongly disagree
1
Please answer the following questions about Appreciative Inquiry: 6) What are the strengths of COMOIRA? 7) What are the limitations of COMOIRA? 8) What are your general thoughts about the application of COMOIRA when evaluating consultation?
213
Appendix G: Briefing Sheet PROFESSIONAL PRACTICE MODEL/FRAMEWORK BRIEFING SHEET
As you are aware I am undertaking a piece of research to find an evaluation tool which will collect relevant information across the X and X Educational Psychology Services. I have carried out a systematic review of the literature and following further consultation with our Principal Educational Psychologist and Flintshire Senior Educational Psychologist the following professional practice models were selected: The Constructionist Model of Informed Reasoned Action (COMOIRA), The Interactive Factors Framework, Activity Theory, Appreciative Inquiry, The framework for psychological assessment and intervention (DECP). I have provided a brief summary of each model/framework and also included some references if you wish to find out any further information. We will have the opportunity to discuss each professional practice model in the focus groups that are due to be held in July 2014. The focus group will provide us with the opportunity to discuss each model so a decision can then be made about which model/models will be taken forward into the piloting phase of the research.
Many thanks for your time
Adrienne Eddleston The Constructionist Model of Informed Reasoned Action (COMOIRA) The first COMOIRA paper proposed a new and flexible model for professional practice which was designed to integrate theory and practice (Gameson et al., 2003). COMOIRA (Gameson & Rhydderch, 2008) has been used recently to guide the practice of trainee EPs at Cardiff University during their fieldwork placements. COMOIRA encourages and summarises the idea of rebuilding. It focuses on processes which consider and promote change at the organisational, systems, group or individual level. It can be used at these different levels in a variety of ways.
One of the key features of COMOIRA is the degree of emphasis on the reflective and reflexive practice that EPs use to facilitate change (Gameson & Rhydderch, 2010). COMOIRA emphasises the importance for practitioners to ‘adapt’ and ‘personalise’ the model. This will ensure that it is suited to the service users and fits the EPs personal style and paradigm base. However, it is crucial to maintain the key features and central concepts of social constructionism and systematic thinking.
The structure and basic processes underpinning COMOIRA were first published in Gameson et al (2003) in a visual form (see Appendix One). COMOIRA comprises a set or core principles, concepts and theories with eight key decision points (Kelly,Woolfson & Boyle, 2008). The core and key decision points are supported by a series of reflective and reflexive questions (Gameson et al, 2003) designed to aid all parties to think clearly about the complex processes associated with change and to use psychology explicitly (see Appendix Two : COMOIRA short form).
214
The flexible framework enables the model to be used in relation to a range of diverse issues and needs within everyday practice. It ensures that the model is suitable across different contexts and at a variety of different levels i.e. individual, group and organisational. The movement between key decision points is always through the core reinforcing the idea that core elements underpin all aspects of the process. However, the order in which key decision points are used is flexible and the sequence can start anywhere, follow any path, include any number of and repeat key decision points.
Core
At the core of COMOIRA sit Social Constructionism, Systemic Thinking, Enabling Dialogues and Informed and Reasoned Action. These underpin and influence all aspects of the process. Each of the decision points will be briefly outlined below.
Construct and Explore Relevant Hypotheses
This allows those involved to collaboratively explore the belief systems, assumptions and expectations that are likely to impact on the process (Kelly ,Woolfson & Boyle, 2008). How the relevant people construct and explore relevant hypotheses in relation to:
• factors that are causing and/or contributing to the issue or concern
• factors that are maintaining the issue or concern
• what needs to be done to improve or find a solution to the issue or concern (Rhydderch & Gameson, 2010)
Explore Constructions of Intention to Change
This enables all relevant people to explore together how far each:
• maintain ownership of relevant issues and processes
• intend to do something different in order to promote the chosen changes
• is ready and willing to change
• is committed to investing time and energy in making the desired or intended change (Kelly ,Woolfson & Boyle, 2008)
Explore Constructions of Ability to Change
This helps all relevant people explore together how far they and others:
• consider they have the skills required to make the desired or intended changes
• feel they have the right to make those changes
215
• feel confident in their ability to make and maintain those changes ((Kelly,Woolfson & Boyle, 2008)
Reflect, Reframe and Reconstruct
This prompts the relevant people to:
• reflect together on issues emerging from the four main aspects of the core as well as those emerging from the request and the other key decision points
• consider and explore together some possible alternative ways to re-frame/reconstruct these issues in order to promote the process of change
• consider together what needs to be done next and what part of the model may help promote and maintain relevant and appropriate change (Kelly ,Woolfson & Boyle, 2008)
Facilitate change/s
This prompts all relevant people to decide who will make the desired or intended changes. The main function is to facilitate changes in ways that empower people to maintain and manage those changes independently (Kelly,Woolfson & Boyle, 2008)
Evaluate the change/s
This helps all relevant people to monitor and evaluate the outcomes together in relation to the desired/ intended change (Kelly ,Woolfson & Boyle, 2008).
Review the process
This allows the relevant people to:
• consider what has happened and how the process has gone so far
• consider people’s roles in the process so far
• take account of how COMOIRA has been used so far
• check what other options might be relevant at this point
• decide what needs to be done next (Kelly ,Woolfson & Boyle, 2008)
Construct and Clarify Key Change Issues
This allows relevant people to construct:
• what they would like to be different
• who they think should make those changes
216
• when, where and how they should make the change/s
References:
Gameson,J.,Rhydderch,G.,Ellis,D.,& Carroll,T. (2003). Constructing a flexible model of integrated professional practice. Part 1- Conceptual and theoretical issues. Educational and Child Psychology, 20 (4),96-106. Gameson,J.,Rhydderch,G.,Ellis,D.,& Carroll,T. (2005). Constructing a flexible model of integrated professional practice. Part 2- Process and practice issues. Educational and Child Psychology, 22 (4),41-52. Kelly,B., Woolfson,L. & Boyle,J. (2008). Frameworks for Practice in Educational Psychology:A textbook for trainees and practitioners.London:Jessica Kingsley Publishers. The Interactive Factors Framework The Interactive Factors Framework (IFF) was developed and integrated into the problem-analysis process by Frederickson and Cline (2002). As defined by Woolfson, Whaling., Stewart & Monsen (2003) the Problem-Analysis Framework (PAF) has an emphasis on the generation of and testing of hypothesis; problem solving and the evaluation of evidence and psychological knowledge. It was developed to inform and structure the EPiT (Educational Psychologists in Training) training at the University College of London. Following its implementation a number of areas for development were identified (Monsen , Graham,Frederickson&Cameron, 1998) and The University College of London spent the next 6 years developing a more systematic approach which became the IFF. The IFF requires EPs to actively consider different levels of analysis and their interactions when formulating initial guiding hypotheses and subsequent implementation (Woolfson et al, 2003). It aims to retain a systematic structure to guide EPs thinking, actions and reporting. Key principles within IFF are the need for transparency of EP practice, collaborative working between the EPs and the various problem stakeholders, recognising that EPs/parents and carers, teachers, children all bring their own perspective. Knowledge of the framework should be shared. The IFF offers a five phase approach, with action points for the EP. Finally, a critical reflection is carried out by the team on what went well and areas that require development. The analysis can be addressed at the individual pupil, group and the school level. The client has a significant part in the decision making process. The phases will be briefly described in the following section. Phase 1: Establishing roles and expectations This phase involves laying the foundations for the problem solving relationship by clarifying and negotiating the role of the ‘problem owner’, the expectations and desired outcomes. Other stakeholders are identified e.g. family, child. The role of the EP is clarified. The framework is explicitly shared with all stakeholders to ensure
217
transparency. By the end of this phase there should be a clear understanding of how everyone will proceed and the individual roles within this process. EP Phase 1 action points:
• Clarify what school seeks to achieve with EP involvement • Identify who the stakeholders are and arrange a joint meeting • Share the framework • Negotiate all stakeholders roles in the process • Negotiate who will be responsible for recording minutes of the meetings-
preferably school as the problem owner • Move to phase 2 or agree date of phase 2 meeting • Agree distribution list for minutes
Phase 2: Guiding hypotheses and information gathering The EP facilitates the gathering of relevant background information. The stakeholders discuss their views of the problem and evaluate the source and validity of their views. Views can be reframed as hypotheses for which more data will be gathered. Hypotheses from all stakeholders should be shared and documented. This includes hypotheses based on psychological knowledge that the EP wishes to explore. EP Phase 2 action points
• Facilitate expression of hypotheses about problem area, encouraging stakeholders to be specific and provide evidence
• Ensure children are also viewed as stakeholders and consulted where appropriate
• Introduce EPs hypotheses based on psychological knowledge, theory and research
• Encourage reframing of view of problem if viewed as intrinsic to child • Conceptualise and record hypotheses at different ecological levels,
identifying the source and who is responsible for data collection • Agree time plan for information gathering and date of Phase 3 feedback
meeting to arrange joint intervention plan • EP arranges to collate stakeholders findings to devise preliminary problem
analysis for Phase 3 feedback meeting • Minutes recorded and sent • Carry out information gathering
Phase 3: Joint problem analysis The EP collates the information gathered in Phase 2 and reflects upon it the aim being to identify problem dimensions that have emerged. Reflection is an important aspect of this phase. New hypotheses may arise that require a temporary return to Phase 2. Joint agreement can then be reached on what the identified problems are and how these relate with each other to result in the problem situation. The
218
ecological systems structure from Phase 2 is used again to focus attention on ways in which the solutions may be targeted. Priorities for intervention are agreed. EP Phase 3 action points
• Examine hypotheses that have been confirmed and integrate these into an ecological analysis of the dimensions of the problem at the level of child, family, school, community
• Draw the different problem dimensions as a diagram (Woolfson et al. 2003, p. 293) interlinked with arrows to provide a visual representation ( see Appendix 3)
• Share problem analysis with other stakeholders at feedback meeting • Identify and discuss with stakeholders hypotheses that were not confirmed to
help reframe their view of the problem • Agree with stakeholders including child where appropriate the new shared
view of the problem and priorities for intervention • Ensure recording and distribution of minutes
Phase 4: Joint action plan and implementation An intervention plan is drawn up and implemented for the priorities agreed at Phase 3. Specifics should be agreed and recorded e.g. what action will be taken, who will be responsible, within what time scale EP Phase 4 action points
• EP shares professional knowledge of possible interventions with stakeholders taking account of their views and experiences of similar intervention
• Clear action plan to be agreed and recorded with details of stakeholder’s roles regarding implementation, who is to do what and when
• Evaluation procedures are agreed and recorded with details of who, what and when
• Phase 5 meeting arranged • Recorded minutes are distributed
Phase 5: Evaluate, reflect and monitor All stakeholders critically review outcomes and agree on next steps. Plans for maintenance of positive changes that have occurred and for future monitoring should be put in place. Roles and remits should be specific. How to address areas where the ‘hoped for’ progress was not made are discussed which may require a return to Phase 2. The group can work through Phase 2, 3 and 4 again if required, taking account of new/updated information and consider new hypotheses and gather additional data if needed. A critical reflection is carried out by the team on what went well and areas that require development. EP Phase 5 action points
• Analysis of what actually happened following Phase 4
219
• Evaluation data gathered and shared • Reflect jointly with stakeholders on evaluation outcomes and develop a
maintenance plan • Where outcomes not successful identify underlying issues and return to
earlier phase if required • EP supports stakeholders in reflecting on the experience and how it could
inform future working • EP reflects on his/her involvement and identifies strengths, weaknesses and
personal development needs References: Frederickson,N.&Cline,T. (2002). Special Educational Needs, Inclusion and Diversity:a textbook. Buckingham&Philadelphia,PA: Open University Press. Monsen,J.,Graham,B.,Frederickson,N&Cameron,(. (1998).bProblem analysis and professional training in educational psychology. Educational Psychology In Practice. 13,234-249. Woolfson,L., Whaling,R., Stewart,A. & Monsen,J. (2003). An integrated framework to guide educational psychologist practice. Educational Psychology In Practice.19, 283-302. Appreciative Inquiry I will only provide a brief summary of this framework as we have had previous training in this area at one of our joint team meetings (Cooperrider,Whitner & Stavros, 2008). Appreciative Inquiry was developed in the mid 1980s by David Cooperrider and colleagues for their work in organisations. It is based on the belief that groups within organisations often share a set of assumptions which cause them to act and think in certain ways. Often these assumptions operate at an unconscious level. A key aspect of Appreciative Inquiry is the belief that it is important to verify these assumptions every so often to check they are still current (Hammond, 1998). A four process cycle is used to evaluate/investigate these assumptions:-
5) Discover – what works well, searching for the best experiences, successes 6) Dream- what would work well in future/ develop a vision 7) Design- working together to plan/prioritise processes around how to achieve
the vision 8) Destiny/deliver – making the vision real/, implementation of proposed
design This model could be applied to evaluate the work of the EPs. For example some Discover and Dream questions could include:
• What do you value/have you valued most whilst working alongside the EP Service?
220
• Please provide an example/examples of when you felt that working with
the EP Service produced the most effective outcomes
• What do you think are the most useful/valuable ways to work with the EP Service in the future?
References: Cooperrider,D.L.,Whitner,D.,Stavros,J.M. (2008). Appreciative Inquiry Handbook: for Leaders of Change. 2nd Edition.Crown Custom Publishing.Inc Hammond,S.A. (1998). Thin Book of Appreciaitve Inquiry. Sue Annis Hammond. A framework for psychological assessment and intervention This framework was drafted by the Division of Educational and Child Psychology (DECP) in 2002 (see Appendix Four). It is intended to guide assessment by EPs. It emphasises the direct link between assessment and intervention and the written recording and reporting. The framework recognises that, although psychological assessment is a highly individualised, complex and creative process, there are some fundamental under-pinning principles relevant to all EPs (DECP, 2002). It acknowledges how the assessment of children and young people has moved away from the positivist frameworks which dominated the EP profession for many years. The DECP (2002) argue that current models of assessment need to reflect the body of psychological knowledge, which emphasises social constructionism. The framework recognises how psychological assessment involves the use of a variety of tools, techniques and approaches which draw upon relevant theory and evidence based research. The assessment generates an understanding of what is happening, who is concerned, why there is a problem and what can be done to make a difference to the situation. it seeks to provide information on the processes of learning, the young person’s cognition, social and emotional development and the impact of the context on those areas (DECP, 2002). References: Division of Educational and Child Psychology (2002). Professional Practice Guidelines. Leicester:DECP. Activity Theory Leadbetter (2008) in Kelly,B.,Woolfson,L.&Boyle,J (2008) outlines how Activity Theory (AT) has developed over the past 70 years from the original ideas of Vygotsky and Leontiev. Researchers such as Cole (1996), Kozulin (1998) and Engestrom (1999) have expanded this work.
221
Engestrom extended the first-generation Activity System triangle to produce the ‘second’ and ‘third’ Activity Theory models (see Appendix Five). Although Engestrom is not the only researcher working with Activity Theory he has written extensively in this area. Five principles of AT can help us understand better the complexities of this approach as summarised in Kelly ,Woolfson & Boyle (2008):
1. The prime unit of analysis of AT is ‘a collective, artefact-mediated and object –oriented activity system, seen in its network relations to other activity systems’ (Daniels 2001, p.93).
2. Activity Systems are usually multi-voiced reflecting a multiple of viewpoints with different interests
3. Activity Systems develop over long periods of time and are transformed and transforming
4. Contradictions are central to an understanding of AT as they are sources of tension and eventually change and development. By examining contradictions, new objects can be created and new ways of working developed
5. Through examination of contradictions participants may question established patterns of working and new motives and objects may be formed. These transformations may occur over lengthy periods of time and result in a much wider range of possibilities for action.
Kelly ,Woolfson & Boyle (2008) discuss how AT can be applied to the practice of EPs and the tasks undertaken by them across a number of important domains. It is a framework that can be applied to any situation where human action is taking place. A Descriptive Framework Leadbetter (2008) details how second-generation AT can be used when considering an individual child’s learning and development. EPs can use the framework to conceptualise their thinking and the planning of a piece of work and this can be then shared with others e.g. teachers. This model has been used by trainee psychologists for planning purposes and as a tool for considering possible ways of working with staff in schools (Leadbetter, 2008). AT can be used as a descriptive tool and to examine and make sense of systems and situations. An analytic device AT can be applied to more complex areas of work. Analysis and comparison across different systems can provide helpful data. AT can be used to track changes over time by constructing models from data collection, people’s views, document analysis and other methods and then repeating the data collection when changes occur (Kelly,Woolfson & Boyle, 2008). Contradictions AT can be used to consider contradictions emerging from different subject positions (Kelly ,Woolfson & Boyle,2008).
222
Organisational Development Approach AT has been developed as a way of engaging with organisations to examine and expand efficient working systems. Multi-agency teamwork Engestrom’s third-generation model can be used to understand forms of multi-agency working. AT provides a framework for understanding human activity within the sociocultural context it occurs and it is capable of mapping the complex systems and practices that present within multi-agency teams. (Greenhouse, 2013). Service Development and Service Delivery AT can be used to facilitate both small and large scale changes within EP Services (Kelly, Woolfson & Boyle, 2008). References: Cole,M. (1996). Cultural psychology. A once and future discipline. Cambridge:Harvard University Press. Engestrom,Y., Miettinen,R. & Punamaki,R.L. (1999). Perspectives on Activity Theory.Cambridge: Cambridge University Press Greenhouse, P.M. (2013). Activity Theory: a framework for understanding multi-agency working and engaging service users in change. Educational Psychology in Practice. 29 (4), 404-415. Kelly,B., Woolfson,L. & Boyle,J. (2008). Frameworks for Practice in Educational Psychology:A textbook for trainees and practitioners.London:Jessica Kingsley Publishers. Kozulin,A. (1998). Psychological Tools: A Sociocultural Approach to Education. Cambridge:Harvard Press.
223
Appendix H: Initial Focus Group Questions
224
Appendix I: Researcher Diary
225
Appendix J: Final Focus group questions Q1: What are your views on the use of COMOIRA as an evaluative tool in relation to:-
A) Its strengths when using it to assess consultation as a process? B) Its limitations when using it to assess consultation as a process? C) Its ability to provide outcomes which demonstrate accountability?
Q2: What are your views on the use of AI as an evaluative tool in relation to:-
A) Its strengths when using it to assess consultation as a process? B) Its limitations when using it to assess consultation as a process? C) Its ability to provide outcomes which demonstrate accountability?
Q3: What are the preferred ways forward from here?
226
Appendix K: Participant Information Sheet and Consent Form
Selecting an Effective Model to evaluate consultation
Participant Information Sheet
You are being invited to take part in a research study aiming to find an evaluation tool which will collect relevant information on the impact of our service delivery.
Before you decide it is important for you to understand why the research is being done and what it will involve. Please take time to read the following information carefully and discuss it with others if you wish. Please ask if there is anything that is not clear or if you would like more information. Take time to decide whether or not you wish to take part. Thank you for reading this.
Who will conduct the research?
The research is being conducted by Adrienne Eddleston, Educational Psychologist, Educational Psychology Service.
Title of the Research:
Selecting an Effective Model to evaluate consultation
What is the aim of the research?
This study will aim to find a professional practice model which will collect relevant information on the impact of our service delivery .It will focus on finding a method that facilitates us, as Educational Psychologists, to successfully and appropriately evaluate consultation. This professional practice model should enable us to gather data about the effectiveness of the Service for the LA. The research will focus on piloting different professional practice models as part of our consultations.
Why have I been chosen?
All Educational Psychologists are invited to take part in the research.
What would I be asked to do if I took part?
1. In order to determine the ‘best fit’ models to pilot across the two services a rigorous review of relevant evidence based literature has been carried out and a summary of the selected models has been written and presented in a briefing sheet . This will be sent out to you for you to read.
227
2. You will then be asked to attend an initial focus group (60-90 minutes) which will aim to gather data to guide the selection of the professional practice models to be piloted across the two services.
3. You will take part in the piloting process (12 weeks). 4. During the piloting of the various professional practice models every EP will
be asked to fill in a questionnaire to rate different aspects of the model. This questionnaire will be designed in conjunction with the Principal Educational Psychologist and Flintshire’s Senior Educational Psychologist.
5. Individual interviews (60 minutes) will be held following the piloting of the different models.
6. Once all the data has been analysed it will be presented to both the EP Services.
7. You will then be asked to attend a final focus group (60-90 minutes) which will be held in the spring term 2015 (date to be agreed) with the purpose of making a decision about the evaluation models/ methods that will subsequently be adopted.
What happens to the data collected?
Anonymous data will be stored securely and analysed by the researcher, as follows:
The focus group discussion will be recorded on a dictaphone and this will be analysed during the data analysis. No individual will be named for the recording or during the transcript and I will be the only person who will have access to the dictaphone. The recording will be deleted after the data has been transcribed. You will be asked to complete a questionnaire during the piloting of the evaluation models. This will take place after a minimum of six consultations. The questionnaires will be piloted by a small group of EPs. The questionnaires will be anonymous but will have a code for the specific model being tested. The data from the questionnaires will be analysed. During the study I will be using a diary to record any discussions that take place outside the focus group discussions and interview sessions. I will be the only person with access to the diary and I will only record comments and not names. You will be asked to participate in a semi-structured interview following the testing of the various models. Questions will be based around your views on the model you trialled. The discussion will be recorded on a dictaphone. No individual will be named for the recording or during the transcript and I will be the only person who will have access to the dictaphone. The recording will be deleted after the data has been transcribed. How is confidentiality maintained?
All data will be anonymised and individual participant’s’ responses will not be identifiable. Any focus group references which make participants potentially identifiable will be removed.
228
What happens if I do not want to take part or if I change my mind?
You are under no obligation to participate in the research and it is up to you whether or not to take part in the research. You will have the right to withdraw from the research at any point, without reason, before or at any point during the research. You can ask for your information to be deleted at any time up until the data has been anonymised.
Will I be paid for participating in the research?
No
What is the duration of the research?
September 2014 - first focus group (60-90 minutes)
September to December 2014, 12 week piloting of model. The piloting of the models will be incorporated into our normal school consultation visits so there should not be any additional time commitment. Mid –way through the piloting you will be asked to complete a questionnaire.
January 2015 - interview (60 minutes)
Date to be agreed for final focus group (60-90 minutes)
Where will the research be conducted?
Within the EP Services
Will the outcomes of the research be published?
Findings will be collated to form the basis of my Professional Doctorate in Educational Psychology (University of Manchester ) and published in peer review journals
Contact for further information
Adrienne.Eddleston@ xxxxxxxxxxxxxxxxxxxxx Or Supervisor: Dr Cathy Atkinson, Room A6.5, Ellen Wilkinson Building, Oxford Rd, University of Manchester. Email: [email protected] What if something goes wrong?
If there are any issues regarding this research that you would prefer not to discuss with members of the research team, please contact the Research Practice and Governance Co-ordinator by either writing to 'The Research Practice and
229
Governance Co-ordinator, Research Office, Christie Building, The University of Manchester, Oxford Road, Manchester M13 9PL', by emailing: [email protected], or by telephoning 0161 275 7583 or 275 8093
CONSENT FORM
If you are happy to participate please complete and sign the consent form below
1. I confirm that I have read the attached information sheet on the above study and have had the opportunity to consider the information and ask questions and had these answered satisfactorily.
2. I understand that my participation in the study is voluntary and that I am free to withdraw at any time without giving a reason.
3. I understand that the interviews and focus groups will be audio recorded
4. I agree to the use of anonymous quotes taken from the focus groups, diary, questionnaires and interviews.
5. I agree that any data collected may be published in anonymous form in academic books or journals.
I agree to take part in the above project
Name of participant
Date Signature
Name of person taking consent
Date Signature
230
Appendix L: Information Sheet PROFESSIONAL PRACTICE MODEL/FRAMEWORK INFORMATION SHEET
As you are aware I am undertaking a piece of research to find an evaluation tool which will collect relevant information across the X and X Educational Psychology Services. Following on from our focus group discussion I met with X and X to discuss the data collated from the focus group and the questionnaires. We have selected COMOIRA and Appreciative Inquiry (see attached) as the professional practice models we will now take forward into the piloting phase of the research. A decision was made to pilot the COMOIRA framework first and then follow this with the Appreciative Inquiry model. We are hoping that the piloting process for both evaluation tools will be completed by December 2014. As previously discussed the framework is being used as an evaluative tool so you are not being asked to change what you do during your consultation sessions. What next:-
1. Use the COMOIRA framework to evaluate three individual teacher consultations.
2. You will be asked to complete a questionnaire about the framework. 3. Use the Appreciative Inquiry framework to evaluate three individual teacher
consultations. 4. You will be asked to complete a questionnaire about the framework. 5. You will be asked to take part in an interview to discuss both frameworks.
Many thanks for your time
Adrienne Eddleston
231
Appendix M: Description of the final focus group analysis process for COMOIRA
Step 1: full transcription of data carried out Step 2: initial codes were generated and then the data was revisited in order to gather more succinct codes. Step 3: Relevant extracts were collated into potential themes and preliminary themes were identified as in the example below: Preliminary theme: Applying psychology EP9: and I suppose if you were sharing it with your audience, the hidden psychology is very clear isn’t it that there is a core of psychology and we are going to have this conversation around that but we are using, the way we are working together, is actually using the psychology and in that it’s a strong tool cos it makes that very clear and it’s not about having a cosy chat and a coffee and look there’s an outcome
EP9: but I suppose like you were saying, that affects the whole hidden agenda, in a positive way though, that we are applying psychology, because you’re applying psychology at lots of different levels aren’t you? Step 4: Final themes checked and finalised and thematic maps were generated defining themes and subthemes
232
Appendix N: Example of a thematic map
233
Appendix O: COMOIRA Questions
234
